Models That Choose Instead of Generate: Inside 100+ Decision Models

Which team should handle a customer inquiry? Does a document meet the review criteria? Should a task continue automatically or be handed to a person? These workflows often need one answer from a defined set of options, and a measure of how much to trust that decision, before they need a long explanation.

Decision models are designed for this gap. Jev, AWS, Microsoft, Perplexity, Cloudflare, and Liquid AI have introduced models that take different approaches. Examining 105 implementations whose base models can be identified from public materials reveals that many differently named models share the same starting point. A large model's high decision score does not necessarily translate into inexpensive operation, either.

In our earlier article, Decision AI in Korean and Japanese: Near-Frontier Results at Lower Cost, we looked at cases where Jev approached general-purpose AI on certain tasks, including answer review, while reducing processing time and recorded cost. Decision models offer two practical strengths: answers stay within predefined options, making them easy to connect to conditional branches in code, and producing less lengthy text can create room to reduce latency and cost. Once the gap between option probabilities and actual accuracy has been tested, those probabilities can help set thresholds for handing cases to people. Which models and tasks realize these benefits still needs to be tested directly.

This article explains how decision models are built, how to read their performance tables, and how to test candidates on your team's tasks. AIB's Japanese and Korean evaluations, and its proposed training experiments using Korean data, provide concrete examples of the evaluation process. You do not need to start by training a model. Choose one recurring task, then write down its real inputs, options, and decision criteria: that is the easiest starting point.

Inputs and three outputs of decision models

A decision model typically receives a state, a question, and a restricted range of answers. For example, the state might be a customer's message, “The integration keeps failing”; the question, “Which team should handle this?”; and the options, “Billing, technical support, or sales.” The output is a result of a defined type, together with probabilities.

The three representative formats documented by TypeSafe make their roles clear.

• Choice — Question: Which candidate should we choose? Returns: Probabilities for each option and the selected option. Work example: Assigning a customer inquiry to a team.
• Noul — Question: Is this condition true? Returns: The probability of yes, from 0 to 1. Work example: Checking whether a document contains required items.
• Score — Question: Which level applies? Returns: Probabilities over ordered rating levels and an expected score. Work example: Evaluating how well an answer meets a set of criteria.

Even when formats have the same name, check each provider's input specification and returned fields. In particular, a Score of 4, a Choice probability of 0.8, and a Noul probability of yes are different quantities.

A general-purpose LLM can also turn the logits of answer symbols such as A, B, and C—the scores before normalization—into option probabilities. Decision models put this procedure at the center of the product and arrange for selection or evaluation results to feed directly into a program's next action. Reducing the process of generating a long answer and then interpreting that text can help with repeated classification or routing.

However, not generating new sentences does not guarantee a correct decision. With flawed criteria or training data unlike the real task, a model may quickly return a well-formed wrong answer.

---

Base models, readout heads, and training

When evaluating a new decision model, distinguish its starting point, how its results are read, and what has been trained.

A base model is the pretrained model used as the starting point. A checkpoint is a file of model weights at a particular point in time. Even within the Qwen family, changes in size, version, training, or execution settings can produce different results.

The readout head converts internal representations into scores for options. Approaches include a pointer mechanism that compares representations at option positions, reading scores for symbols such as A, B, and C, and a separate head that handles multiple questions and fields together. LoRA is a training method that adds small trainable matrices to the original weights; it is not the name of a readout head.

AWS Strands Decider 2B provides a concrete example. Its developers removed the language generation head from the Qwen3.5-2B backbone and trained a pointer head with roughly 1 million parameters, along with rank 16 LoRA. The launch article reports median latency of about 115 ms on an RTX 3090. That is the article's v18 measurement; it does not automatically describe the public v19 checkpoint, other devices, or different input lengths.

“Small LLM + new readout head + LoRA” is a promising route, but it is not a universal recipe for the field. Some implementations leave the base unchanged and only read logits. Others use full fine-tuning or merge the results of multiple training runs.

Cloudflare Clef uses a small Transformer readout to process several questions in a schema together. Fastino GLiNER2.5-Decide uses an encoder to compute scores and select structured results subject to constraints. Liquid's d1 uses fine-tuning and merging procedures that differ by model. Reducing all products from one provider to “LoRA models” hides important distinctions.

---

Base models behind major providers' products

The following starting points were confirmed in official launch announcements and model cards. Whether a base is publicly disclosed is separate from how the finished product is offered.

Microsoft's official launch announcement says the current model starts from Qwen3.5-9B. Plans to consider other bases, such as MAI or OpenAI families, should not be confused with its present architecture.

Perplexity v1.1 uses a separate readout and removes the causal mask from some attention operations. Running it with ordinary causal inference will not reproduce the evaluation configuration described in its model card. The public card's usage instructions also specify conditions such as authenticated access and about 49 GiB of weight memory. The model's name alone is no reason to assume it will be lightweight to run immediately.

Fastino's GLiDE is another exception. This API product uses additional inference when uncertain, demonstrating why not every decision model can be described as always finishing in “one forward pass with 0 output tokens.” Nor can we assume GLiDE's undisclosed base matches GLiNER2.5-Decide's simply because both products come from the same company.

The key comparison is less about a provider's size than the conditions and costs under which it takes our inputs and returns the results we need.

---

105 implementations: lineage and top rankings

The 105 entries in this survey are decoder-based and diffusion-based implementations classified by base model using public materials. They combine 102 entries with measurements on the current board and 3 added from announcements and public documentation. AWS Strands Decider 2B, Microsoft-Decision-1, and Liquid d1-3B have no comparable Capability measurement in this list.

Jev and GLiDE, whose bases are undisclosed, as well as encoders, rerankers, and base-model controls with no additional training, are excluded from this lineage count. The total of 105 therefore does not mean that every model took the same test, or that there are 105 independently developed bases.

• Qwen — Survey count: 74. Share: About 70%. Entries with scores: 72. Mean Capability: 58.9. Highest: 82.8.
• Gemma — Survey count: 23. Share: About 22%. Entries with scores: 23. Mean Capability: 67.9. Highest: 81.7.
• Other — Survey count: 8. Share: About 8%. Entries with scores: 7. Mean Capability: 48.2. Highest: 58.4.
• Total — Survey count: 105. Share: 100%. Entries with scores: 102. Mean Capability: —. Highest: —.

Means include only implementations with measurements. The Other value of 48.2 is the mean across its 7 scored implementations. Means for the individual base families within Other range from 43.3 to 58.4, a different quantity from the mean across all implementations in that group. The highest values in this list are Capability scores that include models outside the cost and latency limits; distinguish them from official top rankings within those limits.

29 implementations start from Qwen3.5-4B. That indicates active experimentation with output methods, data, and inference code on a base of the same size. Even when an implementation builds on the Plumb or deck families, we trace beyond its final name to the upstream base and classify it in the Qwen lineage.

By contrast, 19 of the official Composite top 20 in Benchmark Heaven v1.6.1, board v1.7.30, are in the Gemma lineage. The remaining entry is Qwen3.5-4B-based H2O-Lightning-4B v1.1. This count includes Gemma-lineage Diffusion Jev and excludes the unranked Jev reference row.

These findings answer different questions. Qwen is the most frequently chosen starting point in this survey. Gemma makes up a large share of the top overall scores in this version. Counts of submissions or implementations, decision capability, and operational efficiency do not produce the same ranking.

Family averages also mix versions, sizes, training methods, readout heads, and execution devices. They cannot establish that “the Gemma base itself is the cause.” Use them as a signal for narrowing practical candidates, then compare individual models under similar conditions.

---

27B costs: 5.1–22.2 times the Jev reference

This list contains 15 Qwen3.8-27B-family implementations. Where the direct base is a derivative, we checked the upstream lineage; for some historical torchcast base versions, we also considered the public board's estimated labels. Flash-Next and models distilled to 2B are not counted as 27B.

Comparing the same board's cost per 1,000 decisions with the fixed Jev v1.5 reference cost of about $0.0323, these 15 entries range from about 5.1 to 22.2 times that cost.

Saying “27B-family models cost 6–7 times as much as Jev” therefore fits some cases, such as Perplexity, but does not describe the whole family.

This column includes estimated costs calculated from token usage and reference prices for self-hosted models. Actual API billing prices, hourly costs for rented GPUs, and savings from batching must be calculated separately. Do not use these cost ratios as if they were a contract quote.

Perplexity v1.1 has Capability of 82.8, the mean of Intelligence 75.7 and Calibration 89.9. v1 and v1.1 are different checkpoints, and the model card's Decision Index score is from a different evaluation than this JevBench value.

Even with high Capability, exceeding the Jev-class cost limit excludes a model from that headline ranking. The Composite ranking, which also reflects cost, can differ too. What matters more than which leaderboard a model tops is which candidates remain within our task's budget and response-time limits.

---

JevBench capability and overall efficiency

Benchmark Heaven's JevBench evaluates four axes: Intelligence, Calibration, Speed, and Cost. Capability and Composite may look similar on screen, but they are calculated differently and answer different questions.

The current Jev-class limits are about $0.0646 per 1,000 decisions and about 1.23 seconds adjusted median latency. Exceeding either the cost or latency limit makes a model ineligible for the headline ranking, even if its Capability is high.

“Equally weighted” Composite does not mean simply adding the four scores and dividing by 4. A harmonic mean is strongly affected by low axes, and additional penalties apply when Intelligence, Speed, or Cost falls below specified thresholds. A large model can therefore make good decisions yet receive a low overall score because of high cost.

Historical rankings also need to be kept distinct. In v1.5.4, retained on BenchLM, Cygnet at 73.70 and Winnow at 73.23 were statistically similar leading entries. In the current v1.6.1 examined here, H2O-Lightning-4B v1.1 ranks first in Composite. Test composition, methods, and candidates can change, so differences between versions should not all be interpreted as improvements in the models.

A different JevBench, “JevBench: coherence tests for typed decision models,” is a separate project that tests the coherence of typed decisions. Despite the shared name, it must not be cited interchangeably with Benchmark Heaven's leaderboard combining accuracy, cost, and speed. Decision Index is another separate evaluation; its scores should not be mixed with Capability as if they used the same scale.

---

A probability of 0.8 is not confidence of 0.8

For practical automation, “What did it choose?” is followed by “When should we hand it to a person?” First check how the service defines probability and confidence.

A top Choice option probability of 0.8 is the value the model assigns to that option. TypeSafe's Choice confidence removes the uniform-distribution baseline determined by the number of options.

confidence = (highest probability - 1/number of options) ÷ (1 - 1/number of options)

For example, with 4 options and a highest probability of 0.8, confidence is about 0.733. Noul returns the probability of yes without a separate confidence value. Check this distinction before applying the same operational threshold to a value of 0.8 from different services.

Calibration describes how well actual outcomes match predicted probabilities across cases assigned similar probabilities. If accuracy in a group of cases assigned 0.8 is much lower, the model is overconfident. ECE measures the gap between confidence and accuracy within probability bins; the Brier score measures the difference between predicted probabilities and actual outcomes.

The public-item accuracy of 167/231 in the AWS v19 card, and the calibration metrics on that card, are the developer's measurements under those evaluation conditions. That ratio cannot be called overall JevBench Capability or transferred to the success rate on real Korean-language customer inquiries.

When setting thresholds using work data, consider accuracy among automatically handled cases alongside the share of cases handled automatically and the share handed to people. High accuracy achieved by handing almost everything to people has different operational value from accuracy achieved while handling most cases. Also test option order and repeated runs on the same input. Having no output tokens does not make every model equally reproducible.

---

Korean-built bases and Korean-language performance

There is one EXAONE-family entry in the list of 105. This is a count of included bases developed in Korea, not a claim that only one Korean-language decision model exists. Qwen- and Gemma-family models can also be evaluated on Korean. A base's country of origin does not determine its Korean-language performance.

EXAONE-4.0-1.2B-JEV is an individual developer's derivative, not LG's official decision model. The v0.3 card describes full fine-tuning on English data and a method that reads options with their order swapped twice, then averages the results. It explicitly states an English-only scope and the limitation that it may be overconfident in unfamiliar domains.

The fact that the base EXAONE-4.0-1.2B supports Korean must not be extended into a claim of verified Korean decision performance for this derivative. Both the base and derivative model cards also have non-commercial license conditions, so check the terms before considering them for business use.

AIB DecisionBench-KJ: Japanese and Korean results

AIB's Japanese and Korean evaluation provides a concrete example. AIB DecisionBench-KJ, operated by AIB, compares decision models using 8 public datasets: 1,250 Korean items and 750 Japanese items, for 2,000 in total. Version 0.2 on the public page on October 10, 2026 includes 12 decision models and general-purpose controls GPT-6.1 Sol and Claude Sonnet 5.5. The current results cover more models than the October 4 results in our earlier Jev article.

Tasks are chosen for decisions that can be expressed as yes/no, option selection, or an ordered rating, rather than lengthy answer writing. Human labels from the source datasets serve as ground truth, and the option assigned the highest model probability is scored as the answer.

• KoBEST — 250 Korean items: Passage-based yes/no, 100; cause and effect, 90; sentiment classification, 60.
• KLUE — 250 Korean items: Evidence–statement agreement, 150; sentence similarity, 100.
• JGLUE — 250 Japanese items: Common sense, 75; evidence agreement, 100; sentence similarity, 75.
• KMMLU / JMMLU — 125 items each in Korean and Japanese: Knowledge at the level of Korean qualification exams and Japanese school curricula.
• M-RewardBench — 125 items each in Korean and Japanese: Selecting the better of two AI answers.
• PolyGuardPrompts — 125 items each in Korean and Japanese: Identifying harmful requests.
• K-MHaS — 250 Korean items: Identifying hate speech.
• RTP-LX — 125 items each in Korean and Japanese: Rating the degree of toxicity on an ordered scale.

Overall accuracy gives equal weight to the accuracies of the 8 datasets. Korean and Japanese scores also equally weight the datasets available in each language, so the overall score is not simply the mean of the two language scores. For ordered scores such as sentence similarity from 0 to 5 or toxicity from 1 to 5, an answer counts as correct only if it exactly matches the labeled level.

There are two more measures to read alongside accuracy. Probability calibration (ECE) asks whether answers given “90% confidence” are actually correct that often. Option-order instability measures how often the answer changes when the same question's options are reversed. Lower is better for both. Order comparisons add up to 1,340 pairs, giving 3,340 requests per decision model including the basic 2,000 items.

For four currently measured models, read the following as overall accuracy / calibration error / option-order instability:

• Perplexity Decider v1.1 27B: 81.7 points / 3.4 percentage points / 6.9%.
• Jev 1.13: 78.6 points / 3.7 percentage points / 6.5%.
• Microsoft-Decision-1: 77.5 points / 8.6 percentage points / 5.6%.
• Perplexity Decider v1 27B: 77.2 points / 2.3 percentage points / 14.9%.

In these results, the accuracy leader is not the calibration leader. Perplexity v1.1 has the highest overall accuracy, while v1 has smaller calibration error. Clef Flash also has low option-order instability of 4.0%, but overall accuracy of 69.4 points. This is why correctness, probabilities, and order stability need to be read separately.

The current table also lets you compare time and cost. Perplexity v1.1 records 6 minutes 52 seconds and $0.0046 per 1,000 cases, while Jev records 3 minutes 45 seconds and $0.022. The GPT-6.1 Sol control scores 83.9 points in accuracy, with 34 minutes 25 seconds and $0.826. In this measurement, Perplexity v1.1 is 2.2 points behind in accuracy, with normalized processing time about 5 times shorter and recorded cost about one 180th as large. These ratios describe this particular version, item set, and calling configuration.

Measurements span October 3–10, and response versions and calling routes are disclosed. For example, Jev was measured through OpenRouter as typesafe/jev-1.13-20260917, and Perplexity v1.1 through Perplexity as pplx-decider-v1.1-27b. Response times include intermediary routing time. Costs are based on actual billing records, and temporarily free versions are excluded from paid-cost comparisons. Provider refusals to make a decision count as wrong answers.

Items are stratified samples drawn with seed 20261003 and held fixed within the version. Item fingerprints (SHA-256) for each set and the recorded source revisions are public. Adding models does not change the item set. For AIB, a possible next research step is to build on this evaluation with experiments extending Korean-language work, preference, and routing tasks.

After public evaluations narrow the candidates, test them again using your team's expressions, work criteria, document lengths, and exceptional cases. A customer's “cancel” might mean cancelling a payment or a reservation. “There is no problem” might confirm that a check is complete or be an indirect expression of dissatisfaction. One overall average score cannot reveal these distinctions.

---

Four research proposals for AIB

This lineage survey suggests four research and evaluation proposals for AIB. The following are proposals for use and evaluation, not statements that these features are already deployed or that training is complete.

1. Learning Korean answer preferences. Choices users make between two answers to the same question could support an experiment in learning preference judgments. Preferences, however, are different from objective correctness or safety judgments. Before considering training, check repeated voting, question and model biases, and noise in data whose consent and usage conditions have been confirmed.

2. A model router. Supply the type, length, and required quality of a task, together with its budget and waiting-time constraints, to choose which model receives it. Compare policies that first send tasks to an inexpensive model and escalate difficult cases to another model or a person. Include the router's own cost and the cost of reprocessing incorrectly routed tasks in the total.

3. Reviewing vote quality. Use Noul or Choice experimentally to flag question–answer relevance, insufficient comparison material, and patterns that need review. First evaluate these as supporting signals that send exceptions to a reviewer. Do not expand them into a feature that declares users trustworthy or fraudulent based on a single model score.

4. Extending Korean decision benchmarks. Add real work classification, option-order changes, preference and quality judgments, and model-routing tasks to the existing DecisionBench-KJ. Separating tasks with objective answers from those on which people have different preferences makes it clearer which decisions a model handles well.

Start with one small, evaluable problem. For example, collect classifications finalized by staff from past inquiries that have been anonymized and whose permitted uses have been checked, then see whether a model assigns teams using the same criteria. These proposals do not authorize extracting private votes or using them for training.

---

AIB's proposed Korean-language training

For AIB's proposed training experiments using Korean data, candidate bases are Qwen3.5-4B, Gemma-4-12B, and EXAONE-4.0-1.2B. The aim is to determine whether Korean data contains decision signals that a model can learn.

These three models differ in size. Interpret the experiment as a comparison of quality and resource requirements across different deployment candidates, rather than a pure ranking of bases of the same size. Record factors that constrain actual adoption, such as EXAONE's usage conditions, alongside the results.

Split related data together so that the same source text or variants of the same question do not appear in both training and test sets. Record evaluations with reordered options, errors, and abstentions. If model architectures differ, matching LoRA rank alone can still result in different numbers of trainable parameters.

Compare results from adding a head and LoRA with results from reading the original model's logits unchanged. That is how you can determine whether a more complicated training procedure actually helps. A practical sequence is to verify a useful signal on representative work before increasing the scale of the data and models.

---

Practical model selection and automation policies

If considering adoption now, you do not need to test all 105 implementations. First filter by input language and length, the need for images or audio, usage conditions, and operating model. Then compare two or three candidates under similar conditions.

1. Write down the decision criteria. Define input data, options, and the correct answer or conditions for human review. Check whether the options need “Cannot determine” or a path to human review.
2. Collect representative cases and difficult exceptions. Attach the criteria used by the people responsible, and examine high-loss errors separately where the work requires it.
3. Record outputs and costs in the same units. Compare not only accuracy but option probabilities, failures and abstentions, input length, cost per 1,000 cases, and response time.
4. Set thresholds together with a human-review policy. At the same share of automatically handled cases, check which candidate is more accurate and which has lower error and reprocessing costs.
5. Check new case types during operation. When inquiry content, document formats, or options change, reassess whether previous calibration and thresholds still hold.

Does our work need the highest public ranking, or a trustworthy next action within a defined budget and time limit?

Decision models let you isolate and evaluate the selection step in a workflow. Base-model lineage helps narrow candidates, and understanding how probabilities and costs are defined makes comparisons easier. The final choice still needs to be verified with real work data and a processing policy.

---

Sources and further reading

Decision APIs and probability definitions: Introduction - TypeSafe AI · Confidence - TypeSafe AI · Introducing System One Models & Jev - TypeSafe AI Blog

Provider launches and implementations: Microsoft-Decision-1: Our model for fast decision-making · Introducing Strands Decider 2B: a small, open source, decision model | Strands Agents · Open d1: Edge decision models for text, vision, and audio | Blog | Liquid AI

Public model cards: StrandsAgents/strands-decider-2B-hobson-v19 · Hugging Face · perplexity-ai/pplx-decider-v1.1-27b · Hugging Face · Cloudflare/clef · Hugging Face · Cloudflare/clef-flash · Hugging Face

Encoders and inference methods: GLiNER2.5-Decide: An Open-Weight Model for Structured Decision Making · fastino/GLiNER2.5-Decide · Hugging Face · Introducing GLiDE: The First Thinking Decision Model

EXAONE and comparison candidates: LGAI-EXAONE/EXAONE-4.0-1.2B · Hugging Face · carrtesy/EXAONE-4.0-1.2B-JEV · Hugging Face · Qwen/Qwen3.5-4B-Base · Hugging Face · google/gemma-4-12B-it · Hugging Face

Evaluations and data: JevBench — AI decision model benchmark & leaderboard | Benchmark Heaven · JevBench dataset card & decision model results | Benchmark Heaven · Best AI Decision Models: Rankings, Cost & Use Cases | BenchLM.ai · JevBench: coherence tests for typed decision models

AIB's Korean and Japanese evaluation and earlier article: AIB DecisionBench-KJ: a Korean & Japanese benchmark for decision models | AIB · Decision AI in Korean and Japanese: Near-Frontier Results at Lower Cost

---

Choosing AI for your work and budget

AIB is a young startup operating an independent AI comparison platform that helps people compare models without favoring a particular AI company. By considering both the quality a task needs and its cost, we aim to prevent differences in information and budget from becoming gaps in access to useful AI.

If you are deciding how much recurring work to automate, which AI to connect it to, or how to compare cost and waiting time, talk with us.

Contact AIB about AI adoption, costs, and practical uses