VocabX Cuts Essay Grader Inference Cost 77%

Dharma-AI researchers report that VocabX, a method for tailoring a language model’s vocabulary to a specific task, cut measured inference cost by 77.0% in a production Portuguese essay grader. The result comes from the current version of a paper that is still under review and may change before publication. VocabX kept the grader’s Qwen3-4B base model, data, training recipe and seed unchanged; the tokenizer was the only difference among seven versions tested on 500 requests using an RTX 5090 GPU and the vLLM inference server.

The production grader reads Portuguese essays, prompts and supporting texts, then returns a JSON assessment with a grade and a comment for each competency. Its original vocabulary had 151,669 entries, many rarely used for this task. The average response fell from 350 tokens to 101, while the bench score—which combines grading accuracy with screening for off-topic or rule-breaking essays—rose from 0.703 to 0.810.

The team says general-purpose tokenizers can split frequently repeated domain text, such as the grader’s JSON structure and competency names, into multiple tokens. They also make a model score vocabulary entries it almost never uses. This matters especially during response generation: in the grader’s measurements, generating one token cost 7.4 times as much as reading one input token. That ratio varies with hardware, task, batch size and context length.

VocabX selects vocabulary changes according to estimated cost and time saved, rather than token frequency alone. It works at both the sub-word level, changing how words are split, and the multi-word level, where repeated spans such as full JSON lines can become a single “macro token.” Five automatic searches identify repeated long words, lines, expressions, sentences and punctuation-based structures. Across eight corpora, the punctuation-structure search contributed the most on average; on code, it accounted for 17.5 points. Without all five searches, token reduction on the grader would have fallen by 55.6 points. The grader’s output layer also shrank from about 152,000 rows to about 65,000.

To estimate the value of candidates missing from a sample, VocabX uses a statistical estimate instead of assigning them zero savings. The authors give the example of an expression appearing 37 times in 50,000 documents: it would appear in a 400-document sample only about 26% of the time. In a contracts corpus, just 16.6% of candidates appeared in a 400-document sample. The estimate raised token reduction on that corpus from 44.7% to 52.8%, using the same sample and budget.

In five public domains—contracts, medical papers, scientific papers, news and Python code—and three model families, Qwen3, Llama-3.1 and Gemma-3, VocabX compressed more than the strongest reference method in all 15 combinations, by 1.2 to 9.3 points. Its pruned version also had the lowest estimated inference cost in all 15; average margins were 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. On contracts, it reduced tokens by 62.6% with 113,426 vocabulary entries, compared with 61.8% for the best in-place BPE configuration using 207,803 entries.

An early VocabX version combined each competency’s grade and surrounding JSON into one macro token. Its bench score was 0.689, below the production grader’s 0.703. Giving the grade its own token added 2 tokens per response, raised the score to 0.810 and produced a 77.0% cost reduction, versus 73.3% for the fused version. Dharma-AI says the full method and experiment tables will appear in a VocabX paper it plans to publish soon.

Dharma-AI researchers report that VocabX, a method for tailoring a language model’s vocabulary to a specific task, cut measured inference cost by 77.0% in a production Portuguese essay grader. The result comes from the current version of a paper that is still under review and may change before publication. VocabX kept the grader’s Qwen3-4B base model, data, training recipe and seed unchanged; the tokenizer was the only difference among seven versions tested on 500 requests using an RTX 5090 GPU and the vLLM inference server.

The production grader reads Portuguese essays, prompts and supporting texts, then returns a JSON assessment with a grade and a comment for each competency. Its original vocabulary had 151,669 entries, many rarely used for this task. The average response fell from 350 tokens to 101, while the bench score—which combines grading accuracy with screening for off-topic or rule-breaking essays—rose from 0.703 to 0.810.

The team says general-purpose tokenizers can split frequently repeated domain text, such as the grader’s JSON structure and competency names, into multiple tokens. They also make a model score vocabulary entries it almost never uses. This matters especially during response generation: in the grader’s measurements, generating one token cost 7.4 times as much as reading one input token. That ratio varies with hardware, task, batch size and context length.

VocabX selects vocabulary changes according to estimated cost and time saved, rather than token frequency alone. It works at both the sub-word level, changing how words are split, and the multi-word level, where repeated spans such as full JSON lines can become a single “macro token.” Five automatic searches identify repeated long words, lines, expressions, sentences and punctuation-based structures. Across eight corpora, the punctuation-structure search contributed the most on average; on code, it accounted for 17.5 points. Without all five searches, token reduction on the grader would have fallen by 55.6 points. The grader’s output layer also shrank from about 152,000 rows to about 65,000.

To estimate the value of candidates missing from a sample, VocabX uses a statistical estimate instead of assigning them zero savings. The authors give the example of an expression appearing 37 times in 50,000 documents: it would appear in a 400-document sample only about 26% of the time. In a contracts corpus, just 16.6% of candidates appeared in a 400-document sample. The estimate raised token reduction on that corpus from 44.7% to 52.8%, using the same sample and budget.

In five public domains—contracts, medical papers, scientific papers, news and Python code—and three model families, Qwen3, Llama-3.1 and Gemma-3, VocabX compressed more than the strongest reference method in all 15 combinations, by 1.2 to 9.3 points. Its pruned version also had the lowest estimated inference cost in all 15; average margins were 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. On contracts, it reduced tokens by 62.6% with 113,426 vocabulary entries, compared with 61.8% for the best in-place BPE configuration using 207,803 entries.

An early VocabX version combined each competency’s grade and surrounding JSON into one macro token. Its bench score was 0.689, below the production grader’s 0.703. Giving the grade its own token added 2 tokens per response, raised the score to 0.810 and produced a 77.0% cost reduction, versus 73.3% for the fused version. Dharma-AI says the full method and experiment tables will appear in a VocabX paper it plans to publish soon.