ProgramBench

Citation frequency 2/3
What does it measure?
Made by the ProgramBench team: whether an AI can rebuild a command-line program from scratch, given only the executable and a behavioral spec.
Who evaluates it, and how?
200 public tasks: the AI writes new source code by running the program, without internet access or decompilers. A task counts as resolved only if every hidden test passes. Vals AI runs every model on the same harness, with six hours and 1,000 steps per task.
Reading the score
Scores run from 0 to 100%. Higher is better. This is the strictest measure, so even top models stay under 20%. The share of tasks 'almost resolved' (at least 95% of tests passed) is much higher.
Top score19.0%
Models tested57
Scoring basisHigher is better
Last updated2026.11.19
Model release date
  1. 1
    Anthropic2026.11.09
    19.0%
  2. 2
    Anthropic2026.09.22
    18.5%
  3. 3
    Anthropic2026.09.01
    7.0%
  4. 4
    Anthropic2026.09.28
    6.5%
  5. 5
    OpenAI2026.09.04 · max
    5.5%
  6. 6
    OpenAI2026.09.29 · max
    3.0%
  7. 6
    Anthropic2026.07.24
    3.0%
  8. 8
    Anthropic2026.10.07
    2.5%
  9. 8
    Google2026.09.30 · high
    2.5%
  10. 8
    Meta2026.09.02 · max
    2.5%

The "harness" is the agent program the AI used to carry out the task. The same model can score very differently depending on its harness and reasoning effort.