Debashish Ghosal released AgentEval Forge as a public agent evaluation harness on GitHub and PyPI after a two-week follow-up to his earlier argument that agent evaluation must judge a full run, not only a final answer. The v0.1.0 project validates agents through launch scenarios, regression workflows, adversarial case generation, and CI gating, with a design that treats safety failures as run-ending failures rather than errors that a correct final answer can offset.

AgentEval Forge includes 20 Critical User Journeys, five core components, 17 deterministic checks, 11 LLM-as-judge metrics, a regression engine, an adversarial generator, and a WBS of 118 tasks across 12 milestones from M0 scaffold to M12 launch. The tool supports five agent surfaces: subprocess, Python import, HTTP, LangGraph, and PydanticAI. Each adapter sends a restricted invocation payload with scenario input, allowed tools, disallowed tools, and a budget, while hiding expected answers and scoring thresholds to prevent ground-truth leakage.

Ghosal also built a security model with sandbox mode, trust policies, audit trails, and API key sanitization, plus CI integration for GitHub Actions, GitLab CI, and Docker sandbox. The documented capability list includes 20 launch scenarios across 10 families, 8 security scenarios, 17 deterministic scorers, 11 LLM-as-judge metrics, five framework adapters, and a product hierarchy of Safety > Correctness > Efficiency.

Field testing changed the project because unit tests and mock agents were hiding integration problems. Ghosal searched 150+ GitHub repositories and found 19 that looked testable: 11 LangGraph agents and 8 PydanticAI agents. Each agent received its own configuration file, scenario pack, and virtual environment, and the tests ran across three tiers: a local MLX setup serving Qwen3.5-9B-MLX-4bit, a cheap cloud tier using gpt-4o-mini, and a better cloud tier using gpt-4o.

The field test produced a 9% pass rate across 19 agents and 95 scenarios, with only 9 passes. Ghosal said that result measured adapter realism more than agent quality, because wrappers that kept the harness alive but returned blank completions received zero scores. The cheap and better cloud tiers both produced identical outcomes, 9/95 passes, and the better model did not reveal any regression or improvement missed by the cheap model.

Configuration failures became the main blocker. One agent hardcoded ChatOpenAI() at module scope with an unsupported model name, another wrote to /root at import time, three brought ormsgpack with a C extension ABI mismatch, one repository put pyproject.toml in a subdirectory where uv sync at the root did nothing, and one agent was removed for using an outdated PydanticAI API version. Ghosal wrote eight compatibility wrapper modules, created three pyproject.toml files for broken packaging, and quarantined three otherwise viable agents.

One LangGraph multi-agent repository, lg-mcp-agents, achieved 5/5 passes on both cloud tiers after targeted adapter work. Ghosal said that result showed the adapter was exercising real agent behavior because the repository had previously been non-runnable outside its original Streamlit app. The field harness was added late because mock agents followed the adapter contract too neatly, while real agents imported ffmpeg, created incompatible virtual environments, wrote to absolute paths, hardcoded API keys at module scope, and nested project files in subdirectories.

AgentEval Forge v0.1.0 can import, configure, invoke, and score real third-party agents from GitHub at scale, according to Ghosal, but it does not yet prove that the scorer can rank a large mixed roster meaningfully. Blank completions in some PydanticAI wrappers, the ormsgpack mismatch, roster expansion, adapter quality, and default CI integration for SWE-bench and WebArena remain unfinished work. The package installs with pip install agent-eval-forge, and the repo includes a README, user guide, field test report dated 08.02.2026, and hard-won lessons document.

Debashish Ghosal released AgentEval Forge as a public agent evaluation harness on GitHub and PyPI after a two-week follow-up to his earlier argument that agent evaluation must judge a full run, not only a final answer. The v0.1.0 project validates agents through launch scenarios, regression workflows, adversarial case generation, and CI gating, with a design that treats safety failures as run-ending failures rather than errors that a correct final answer can offset.

AgentEval Forge includes 20 Critical User Journeys, five core components, 17 deterministic checks, 11 LLM-as-judge metrics, a regression engine, an adversarial generator, and a WBS of 118 tasks across 12 milestones from M0 scaffold to M12 launch. The tool supports five agent surfaces: subprocess, Python import, HTTP, LangGraph, and PydanticAI. Each adapter sends a restricted invocation payload with scenario input, allowed tools, disallowed tools, and a budget, while hiding expected answers and scoring thresholds to prevent ground-truth leakage.

Ghosal also built a security model with sandbox mode, trust policies, audit trails, and API key sanitization, plus CI integration for GitHub Actions, GitLab CI, and Docker sandbox. The documented capability list includes 20 launch scenarios across 10 families, 8 security scenarios, 17 deterministic scorers, 11 LLM-as-judge metrics, five framework adapters, and a product hierarchy of Safety > Correctness > Efficiency.

Field testing changed the project because unit tests and mock agents were hiding integration problems. Ghosal searched 150+ GitHub repositories and found 19 that looked testable: 11 LangGraph agents and 8 PydanticAI agents. Each agent received its own configuration file, scenario pack, and virtual environment, and the tests ran across three tiers: a local MLX setup serving Qwen3.5-9B-MLX-4bit, a cheap cloud tier using gpt-4o-mini, and a better cloud tier using gpt-4o.

The field test produced a 9% pass rate across 19 agents and 95 scenarios, with only 9 passes. Ghosal said that result measured adapter realism more than agent quality, because wrappers that kept the harness alive but returned blank completions received zero scores. The cheap and better cloud tiers both produced identical outcomes, 9/95 passes, and the better model did not reveal any regression or improvement missed by the cheap model.

Configuration failures became the main blocker. One agent hardcoded ChatOpenAI() at module scope with an unsupported model name, another wrote to /root at import time, three brought ormsgpack with a C extension ABI mismatch, one repository put pyproject.toml in a subdirectory where uv sync at the root did nothing, and one agent was removed for using an outdated PydanticAI API version. Ghosal wrote eight compatibility wrapper modules, created three pyproject.toml files for broken packaging, and quarantined three otherwise viable agents.

One LangGraph multi-agent repository, lg-mcp-agents, achieved 5/5 passes on both cloud tiers after targeted adapter work. Ghosal said that result showed the adapter was exercising real agent behavior because the repository had previously been non-runnable outside its original Streamlit app. The field harness was added late because mock agents followed the adapter contract too neatly, while real agents imported ffmpeg, created incompatible virtual environments, wrote to absolute paths, hardcoded API keys at module scope, and nested project files in subdirectories.

AgentEval Forge v0.1.0 can import, configure, invoke, and score real third-party agents from GitHub at scale, according to Ghosal, but it does not yet prove that the scorer can rank a large mixed roster meaningfully. Blank completions in some PydanticAI wrappers, the ormsgpack mismatch, roster expansion, adapter quality, and default CI integration for SWE-bench and WebArena remain unfinished work. The package installs with pip install agent-eval-forge, and the repo includes a README, user guide, field test report dated 08.02.2026, and hard-won lessons document.