Notion Upgrades Model Router Auto, Claims 90% Performance at 90% Lower Cost

Notion published a blog post detailing how it evaluated and upgraded Auto, its model router, which routes customer AI requests to different language models. Notion said it built its evaluation around real knowledge-work tasks rather than relying on a single LLM judge, because a model grading another's output does not reliably capture real-world complexity.

Notion first grouped common Notion AI use cases into task clusters, such as maintaining CRM contacts and follow-ups, tracking inventory and shipment records, and resolving customer support cases. It selected the 20 largest clusters and randomly sampled hundreds of real-world tasks from each, forming its evaluation dataset.

To run the tasks, Notion built a simulated environment called Gymmy, which runs Notion's application code, databases and search services in isolated sandboxes. Each sandbox was loaded with a saved copy of a workspace's data, so different models started from the same state and used Notion's actual tools. The tasks were replayed using GPT-6 Astra, Opus 5.5, GPT-6.1 Sol and Auto.

Because asking one LLM to score outputs proved noisy, with judges favoring their own provider or giving inconsistent scores, Notion first ran pairwise comparisons. An ensemble of frontier models chose the preferred output of two models under a custom rubric, the majority winner was recorded, and Elo scores were computed from the results, a ranking method similar to rating chess players. Notion then calibrated a single LLM judge to reproduce those Elo rankings so thousands of outputs could be scored independently. Pairwise and pointwise evaluations agreed more than 80% of the time. Judge scores also tracked customer thumbs-up and thumbs-down feedback on Notion AI outputs, with thumbs-up responses generally ranked higher.

Using this method, Notion reported that Auto delivers roughly 90% of the performance of frontier models such as GPT-6 Astra or Claude Opus 5.5 at about 90% lower cost. Notion said this can make users' credits go up to 10 times further for everyday work. Its credit comparison showed simple tasks such as polishing an email or translating a document at 5 credits versus 50 for frontier models, medium tasks such as writing a work report or transcribing a one-hour meeting at 10 versus 85, and complex tasks such as creating databases or maintaining CRM contacts at 25 versus 150.

Notion said it will keep testing new models on real Notion tasks and upgrading its model router. It also noted that AI meeting notes use Auto and will begin consuming monthly Standard credit allowance starting December 1, with a one-hour meeting estimated at about 10 Standard credits.

Notion published a blog post detailing how it evaluated and upgraded Auto, its model router, which routes customer AI requests to different language models. Notion said it built its evaluation around real knowledge-work tasks rather than relying on a single LLM judge, because a model grading another's output does not reliably capture real-world complexity.

Notion first grouped common Notion AI use cases into task clusters, such as maintaining CRM contacts and follow-ups, tracking inventory and shipment records, and resolving customer support cases. It selected the 20 largest clusters and randomly sampled hundreds of real-world tasks from each, forming its evaluation dataset.

To run the tasks, Notion built a simulated environment called Gymmy, which runs Notion's application code, databases and search services in isolated sandboxes. Each sandbox was loaded with a saved copy of a workspace's data, so different models started from the same state and used Notion's actual tools. The tasks were replayed using GPT-6 Astra, Opus 5.5, GPT-6.1 Sol and Auto.

Because asking one LLM to score outputs proved noisy, with judges favoring their own provider or giving inconsistent scores, Notion first ran pairwise comparisons. An ensemble of frontier models chose the preferred output of two models under a custom rubric, the majority winner was recorded, and Elo scores were computed from the results, a ranking method similar to rating chess players. Notion then calibrated a single LLM judge to reproduce those Elo rankings so thousands of outputs could be scored independently. Pairwise and pointwise evaluations agreed more than 80% of the time. Judge scores also tracked customer thumbs-up and thumbs-down feedback on Notion AI outputs, with thumbs-up responses generally ranked higher.

Using this method, Notion reported that Auto delivers roughly 90% of the performance of frontier models such as GPT-6 Astra or Claude Opus 5.5 at about 90% lower cost. Notion said this can make users' credits go up to 10 times further for everyday work. Its credit comparison showed simple tasks such as polishing an email or translating a document at 5 credits versus 50 for frontier models, medium tasks such as writing a work report or transcribing a one-hour meeting at 10 versus 85, and complex tasks such as creating databases or maintaining CRM contacts at 25 versus 150.

Notion said it will keep testing new models on real Notion tasks and upgrading its model router. It also noted that AI meeting notes use Auto and will begin consuming monthly Standard credit allowance starting December 1, with a one-hour meeting estimated at about 10 Standard credits.