ProgramBench: Can Language Models Rebuild Programs from Scratch?

Imagine giving a neural network a binary file or a description of how a program works, and it returns the source code. Sounds like science fiction? Researchers from several universities have introduced ProgramBench, a benchmark that evaluates exactly this ability in language models.

What is ProgramBench? It's a set of tasks where the model must "rebuild" a program given only:

• a natural language description of functionality;
• input-output examples;
• or even binary code.

The benchmark includes 150 tasks of varying complexity — from simple scripts to small applications. Evaluation is based on three metrics: reconstruction accuracy, syntactic correctness, and execution time.

Results: Not perfect yet The best models (GPT-4, Claude 3) handle simple tasks in 60-70% of cases, but drop to 20% on complex examples. Interestingly, models often generate syntactically correct but semantically incorrect code — the program compiles but doesn't do what it should.

Why this matters for developers? If such models are refined, they could help restore lost code, reverse-engineer legacy systems, or even automatically generate documentation from program behavior. For now, it's mostly a research playground, but the potential is huge.

METABYTE studio comment: ProgramBench reminds us that even the smartest neural networks can't yet replace humans in understanding business logic. But as a tool for refactoring or analyzing legacy code — why not?