ProgramBench
Citation frequency 2/3
- What does it measure?
- Made by the ProgramBench team: whether an AI can rebuild a command-line program from scratch, given only the executable and a behavioral spec.
- Who evaluates it, and how?
- 200 public tasks: the AI writes new source code by running the program, without internet access or decompilers. A task counts as resolved only if every hidden test passes. Vals AI runs every model on the same harness, with six hours and 1,000 steps per task.
- Reading the score
- Scores run from 0 to 100%. Higher is better. This is the strictest measure, so even top models stay under 20%. The share of tasks 'almost resolved' (at least 95% of tests passed) is much higher.
Top score19.0%
Models tested57
Scoring basisHigher is better
Last updated2026.11.19
Model release date
RankModelSettingReleasedPrice/1MScore
- 12026.11.09$3219.0%
- 22026.09.22$1618.5%
- 32026.09.01$407.0%
- 42026.09.28$86.5%
- 5max2026.09.04$405.5%
- 6max2026.09.29$83.0%
- 62026.07.24$203.0%
- 82026.10.07$0.402.5%
- 8high2026.09.30$82.5%

- 8max2026.09.02$3.502.5%
The "harness" is the agent program the AI used to carry out the task. The same model can score very differently depending on its harness and reasoning effort.