A new verification pipeline for large language models (LLMs) organizes verification into a tiered architecture to reduce computational costs and improve accuracy. Based on insights from development community feedback, the strategy moves from deterministic code checks to LLM-based semantic analysis, and finally to human review for ambiguous cases.

The layered architecture consists of four distinct stages. Layer 0 and Layer 1 perform deterministic checks—such as verifying minimum length, keyword blacklists, and punctuation ratios—to catch garbage output before incurring model costs. Layer 2 applies LLM semantic judgment only to the remaining residual samples. Finally, Layer 3 escalates cases with divergent judgment (where models disagree) to human reviewers. This design ensures that LLM resources are reserved for genuinely ambiguous scenarios rather than simple format errors.

Experiment F, which tested this pipeline across 38 scenarios, demonstrated a 33% reduction in single-judge LLM calls. On a test set of 30 samples, this layered approach achieved a zero-cost catch rate for garbage data, while legitimate semantic edge cases correctly proceeded to Layer 2. The pipeline does not eliminate the inherent limitations of LLM judgment—the difficulty in evaluating subjective 'is this enough?' questions—but it effectively shrinks the volume of work and prevents common false-negative errors.

Furthermore, cost-weight analysis reveals that symmetric metrics like F1-score often hide the impact of false negatives (FN). A false negative triggers a full repair loop, which consumes three times the tokens and latency of a standard call. By using cost-weighted analysis, the pipeline prioritizes strictness to avoid FN-heavy repair loops, even if it occasionally allows a single false positive. Divergence in multi-perspective voting, rather than being treated as noise, serves as a critical signal to route ambiguous work to humans.

A new verification pipeline for large language models (LLMs) organizes verification into a tiered architecture to reduce computational costs and improve accuracy. Based on insights from development community feedback, the strategy moves from deterministic code checks to LLM-based semantic analysis, and finally to human review for ambiguous cases.

The layered architecture consists of four distinct stages. Layer 0 and Layer 1 perform deterministic checks—such as verifying minimum length, keyword blacklists, and punctuation ratios—to catch garbage output before incurring model costs. Layer 2 applies LLM semantic judgment only to the remaining residual samples. Finally, Layer 3 escalates cases with divergent judgment (where models disagree) to human reviewers. This design ensures that LLM resources are reserved for genuinely ambiguous scenarios rather than simple format errors.

Experiment F, which tested this pipeline across 38 scenarios, demonstrated a 33% reduction in single-judge LLM calls. On a test set of 30 samples, this layered approach achieved a zero-cost catch rate for garbage data, while legitimate semantic edge cases correctly proceeded to Layer 2. The pipeline does not eliminate the inherent limitations of LLM judgment—the difficulty in evaluating subjective 'is this enough?' questions—but it effectively shrinks the volume of work and prevents common false-negative errors.

Furthermore, cost-weight analysis reveals that symmetric metrics like F1-score often hide the impact of false negatives (FN). A false negative triggers a full repair loop, which consumes three times the tokens and latency of a standard call. By using cost-weighted analysis, the pipeline prioritizes strictness to avoid FN-heavy repair loops, even if it occasionally allows a single false positive. Divergence in multi-perspective voting, rather than being treated as noise, serves as a critical signal to route ambiguous work to humans.