IdeaTaste Finds LLMs Trail Researchers on Ideas

A research team introduced IdeaTaste, a benchmark that compares language-model research proposals with papers published by the same research communities in 2026. The benchmark uses literature snapshots frozen at 2025 to prompt proposals, then measures how closely those proposals match later work. The study covers six language models, four research areas ranging from established to emerging fields, and more than 90,000 proposals.

Across the experiments, AI agents reached about 26% of the human level in matching citations and 18% in matching ideas. Their populations cited literature as diversely as human communities and took up similar tasks and methods, but rarely produced the specific approaches and research plans that communities later pursued. Adding richer field context or larger agent populations did not substantially narrow the gap.

In blinded expert reviews, AI proposals came closest to 2026 papers on presentation and fell furthest behind on expected success. Proposals whose ideas appeared in no human paper received the weakest assessments. The authors say the central gap is turning broad research directions into concrete approaches. They released IdeaTaste as an open leaderboard that will be updated continuously to compare AI proposals with future publication cycles.

A research team introduced IdeaTaste, a benchmark that compares language-model research proposals with papers published by the same research communities in 2026. The benchmark uses literature snapshots frozen at 2025 to prompt proposals, then measures how closely those proposals match later work. The study covers six language models, four research areas ranging from established to emerging fields, and more than 90,000 proposals.

Across the experiments, AI agents reached about 26% of the human level in matching citations and 18% in matching ideas. Their populations cited literature as diversely as human communities and took up similar tasks and methods, but rarely produced the specific approaches and research plans that communities later pursued. Adding richer field context or larger agent populations did not substantially narrow the gap.

In blinded expert reviews, AI proposals came closest to 2026 papers on presentation and fell furthest behind on expected success. Proposals whose ideas appeared in no human paper received the weakest assessments. The authors say the central gap is turning broad research directions into concrete approaches. They released IdeaTaste as an open leaderboard that will be updated continuously to compare AI proposals with future publication cycles.