Coding AI Benchmark Leader Scores Just 45.5%

Cognition announced FrontierCode 2.0 on its official blog on the 27th of last month, introducing a coding evaluation tool designed to reflect real software development environments. It measures whether developers can review an AI’s changes and incorporate them into an existing project, unlike benchmarks that mainly test whether code works as intended. Cognition said existing scores were rising quickly but did not adequately show the limits AI faces in real development work.

The evaluation includes 132 tasks based on real open-source projects. Tasks cover complex work such as fixing security vulnerabilities and optimizing performance, and both the projects and code changes are larger than in the previous version. A model receives no credit for a task if it fails to meet mandatory requirements.

Claude Opus 5.5 topped the official leaderboard at 45.5%. GPT-6 Astra scored 42.8%, Claude Sonnet 5.5 41.0%, Gemini 4 Carbon 40.0%, and GPT-6.1 Sol 39.0%. Every major AI coding model scored below 50%. On FrontierCode 1.1, Opus 5.5 scored 54.6% and GPT-6 Astra 53.3%; Cognition said the lower scores reflected more complex tasks and higher code-quality requirements, rather than a decline in model capability.

The weakest area was completeness: whether a model considers existing code and development context, going beyond the explicitly requested feature to handle unstated requirements. Across four major models, the failure rate on mandatory completeness criteria was 19.2%, higher than the 14.9% failure rate for implementing core functionality. Even at the highest reasoning setting, the completeness failure rate was 15.3%.

In one task, models had to fix an error in a mock service worker for a JavaScript library. Opus 5.5 corrected an error that initialized a specific feature but failed to shut down existing work properly, potentially leaving unnecessary resources behind or making the program unstable. Opus 5.5 was run five times at each of five reasoning settings, for 25 runs total, and none passed the evaluation criteria. Cognition said the issue concerned understanding the context of the whole task, rather than a simple coding mistake.

Adding more reasoning resources did not necessarily improve code quality. Cognition observed cases where models changed too many files or substantially altered existing designs. GPT-6 Astra fixed a data-transfer ordering problem in mobile analytics software PostHog in all five runs at a low reasoning setting, but succeeded only once in five runs at the maximum setting. During its fix, it removed a public program interface, breaking compatibility with other software.

Cognition said AI must interpret incomplete requirements and understand existing project constraints to function as an independent software developer. It also said real development requires judging the scope of changes needed to solve a problem, preserving existing features, and preventing side effects.

Cognition announced FrontierCode 2.0 on its official blog on the 27th of last month, introducing a coding evaluation tool designed to reflect real software development environments. It measures whether developers can review an AI’s changes and incorporate them into an existing project, unlike benchmarks that mainly test whether code works as intended. Cognition said existing scores were rising quickly but did not adequately show the limits AI faces in real development work.

The evaluation includes 132 tasks based on real open-source projects. Tasks cover complex work such as fixing security vulnerabilities and optimizing performance, and both the projects and code changes are larger than in the previous version. A model receives no credit for a task if it fails to meet mandatory requirements.

Claude Opus 5.5 topped the official leaderboard at 45.5%. GPT-6 Astra scored 42.8%, Claude Sonnet 5.5 41.0%, Gemini 4 Carbon 40.0%, and GPT-6.1 Sol 39.0%. Every major AI coding model scored below 50%. On FrontierCode 1.1, Opus 5.5 scored 54.6% and GPT-6 Astra 53.3%; Cognition said the lower scores reflected more complex tasks and higher code-quality requirements, rather than a decline in model capability.

The weakest area was completeness: whether a model considers existing code and development context, going beyond the explicitly requested feature to handle unstated requirements. Across four major models, the failure rate on mandatory completeness criteria was 19.2%, higher than the 14.9% failure rate for implementing core functionality. Even at the highest reasoning setting, the completeness failure rate was 15.3%.

In one task, models had to fix an error in a mock service worker for a JavaScript library. Opus 5.5 corrected an error that initialized a specific feature but failed to shut down existing work properly, potentially leaving unnecessary resources behind or making the program unstable. Opus 5.5 was run five times at each of five reasoning settings, for 25 runs total, and none passed the evaluation criteria. Cognition said the issue concerned understanding the context of the whole task, rather than a simple coding mistake.

Adding more reasoning resources did not necessarily improve code quality. Cognition observed cases where models changed too many files or substantially altered existing designs. GPT-6 Astra fixed a data-transfer ordering problem in mobile analytics software PostHog in all five runs at a low reasoning setting, but succeeded only once in five runs at the maximum setting. During its fix, it removed a public program interface, breaking compatibility with other software.

Cognition said AI must interpret incomplete requirements and understand existing project constraints to function as an independent software developer. It also said real development requires judging the scope of changes needed to solve a problem, preserving existing features, and preventing side effects.