Don Johnson, a developer building tools for AI agents, found that 389 out of 390 tests in his Rust library harness passed despite a fundamental error in a calculator's regression calculation. He discovered the mistake by introducing a mutation—replacing a multiplication sign with addition—which the existing test suite failed to flag. The single test that did identify the failure relied on the NIST (National Institute of Standards and Technology) StRD (Statistical Reference Datasets) project, specifically the Longley regression dataset. This highlighted that while deterministic programs are repeatable, they are not inherently trustworthy without independent, evidence-based verification.

The author argues that AI agent workflows should adopt a clear boundary between model inference and tool execution. By using a 'five-layer challenge stack' for agent tools, developers can establish provisional authority for instruments within a defined contract. This stack requires an explicit, executable contract; an independent reference for expected values; property-based tests for checking mathematical laws; mutation testing to verify test sensitivity; and bounded, typed failure reports. The goal is to move beyond simple example-based testing toward an approach where tools provide 'sources under challenge' that can be inspected, replayed, and cross-examined by agents when conflicts arise.

The experiment demonstrated that even with strict input validation, tools may fail to enforce resource boundaries or maintain schema consistency. Johnson repaired his calculator to reject unknown fields in JSON requests and added hard limits—100,000 iterations and 1,000,000 grid points—to the optimization interface. By utilizing cargo-mutants, he ensured that subsequent guards could catch eleven different plausible defects, completing a loop of claim, attack, failure, repair, and re-mutation. This process aims to replace blind trust in code with evidence-based calibration, allowing agents to respect a tool's output only within the specific, tested contract it has demonstrably earned.

Don Johnson, a developer building tools for AI agents, found that 389 out of 390 tests in his Rust library harness passed despite a fundamental error in a calculator's regression calculation. He discovered the mistake by introducing a mutation—replacing a multiplication sign with addition—which the existing test suite failed to flag. The single test that did identify the failure relied on the NIST (National Institute of Standards and Technology) StRD (Statistical Reference Datasets) project, specifically the Longley regression dataset. This highlighted that while deterministic programs are repeatable, they are not inherently trustworthy without independent, evidence-based verification.

The author argues that AI agent workflows should adopt a clear boundary between model inference and tool execution. By using a 'five-layer challenge stack' for agent tools, developers can establish provisional authority for instruments within a defined contract. This stack requires an explicit, executable contract; an independent reference for expected values; property-based tests for checking mathematical laws; mutation testing to verify test sensitivity; and bounded, typed failure reports. The goal is to move beyond simple example-based testing toward an approach where tools provide 'sources under challenge' that can be inspected, replayed, and cross-examined by agents when conflicts arise.

The experiment demonstrated that even with strict input validation, tools may fail to enforce resource boundaries or maintain schema consistency. Johnson repaired his calculator to reject unknown fields in JSON requests and added hard limits—100,000 iterations and 1,000,000 grid points—to the optimization interface. By utilizing cargo-mutants, he ensured that subsequent guards could catch eleven different plausible defects, completing a loop of claim, attack, failure, repair, and re-mutation. This process aims to replace blind trust in code with evidence-based calibration, allowing agents to respect a tool's output only within the specific, tested contract it has demonstrably earned.