Google Shows Agents CLI Evaluation Workflow

Google Cloud product manager Pier Paolo Ippolito and solutions specialist Elia Secchi published a walkthrough on October 28, 2026, showing how the Agents CLI evaluation commands can test agent behavior, compare changes, track regressions and support scheduled checks. The workflow starts with a LangChain template in Agents CLI v1.8.0 and a four-case dataset covering a greeting, weather questions for San Francisco and Rome, and an off-topic question about the 2024 World Series. A custom metric uses gemini-3.8-flash as a judge to score accuracy, relevance and clarity on a 1-to-5 scale.

The baseline scored a mean of 4.0000 across 4 valid cases, with a standard deviation of 1.4142. To demonstrate a regression, the walkthrough replaces the weather agent’s helpful system prompt with a chef persona. The mean then falls to 2.7500, with a standard deviation of 0.9574, a 1.25-point drop on the 1-to-5 scale. Three cases decline and one stays level: the greeting falls from 5.0 to 4.0, San Francisco weather from 4.0 to 2.0, Rome weather stays at 2.0, and the off-topic case falls from 5.0 to 3.0. In the San Francisco case, the agent stops using its registered weather tool and invents conditions.

The walkthrough says eval compare shows per-case results alongside summary scores, helping identify whether a change affects tone, tool use or the scope of answers. In a separate model-backend experiment, replacing gemini-3.8-flash with gemini-2.5-flash lowers the mean from 4.00 to 2.75. The greeting remains at 5.0 and Rome weather at 2.0; San Francisco weather drops from 4.0 to 3.0, while the off-topic case drops from 5.0 to 1.0. The article attributes the latter result to a refusal based on the event supposedly not having happened yet, which the judge scored harshly. It cautions that the mean alone makes this model swap look as bad as the chef-prompt regression, though the case-by-case results differ. It also says model comparisons should account for prompts tuned to each model, and keep the judge model constant; changing the judge is a separate experiment.

Agents CLI does not include a scheduler, but its evaluation commands can run through cron or Cloud Scheduler and save results to persistent storage. The walkthrough recommends nightly checks against a stable regression dataset to catch behavior changes from model updates, tool data and shifting dependencies. Such scheduled datasets typically contain dozens to hundreds of cases, compared with a few cases in the faster development loop. A wrapper can compare each run’s mean with a baseline and trigger an alert when the drop exceeds a threshold; the example uses 0.05 as its default and demonstrates a 0.1 threshold, which flags the 4.000-to-2.750 regression.

For ADK projects, eval optimize uses the GEPA framework to generate and score prompt variants, retain those with higher scores and rank candidates for human review. It is not supported for the LangChain extension. Optimization changes the prompt while keeping tools, agent structure and model fixed. The authors caution that a prompt can score better by overfitting to a metric’s wording, so a person must review candidate prompts before promoting one.

Google Cloud product manager Pier Paolo Ippolito and solutions specialist Elia Secchi published a walkthrough on October 28, 2026, showing how the Agents CLI evaluation commands can test agent behavior, compare changes, track regressions and support scheduled checks. The workflow starts with a LangChain template in Agents CLI v1.8.0 and a four-case dataset covering a greeting, weather questions for San Francisco and Rome, and an off-topic question about the 2024 World Series. A custom metric uses gemini-3.8-flash as a judge to score accuracy, relevance and clarity on a 1-to-5 scale.

The baseline scored a mean of 4.0000 across 4 valid cases, with a standard deviation of 1.4142. To demonstrate a regression, the walkthrough replaces the weather agent’s helpful system prompt with a chef persona. The mean then falls to 2.7500, with a standard deviation of 0.9574, a 1.25-point drop on the 1-to-5 scale. Three cases decline and one stays level: the greeting falls from 5.0 to 4.0, San Francisco weather from 4.0 to 2.0, Rome weather stays at 2.0, and the off-topic case falls from 5.0 to 3.0. In the San Francisco case, the agent stops using its registered weather tool and invents conditions.

The walkthrough says eval compare shows per-case results alongside summary scores, helping identify whether a change affects tone, tool use or the scope of answers. In a separate model-backend experiment, replacing gemini-3.8-flash with gemini-2.5-flash lowers the mean from 4.00 to 2.75. The greeting remains at 5.0 and Rome weather at 2.0; San Francisco weather drops from 4.0 to 3.0, while the off-topic case drops from 5.0 to 1.0. The article attributes the latter result to a refusal based on the event supposedly not having happened yet, which the judge scored harshly. It cautions that the mean alone makes this model swap look as bad as the chef-prompt regression, though the case-by-case results differ. It also says model comparisons should account for prompts tuned to each model, and keep the judge model constant; changing the judge is a separate experiment.

Agents CLI does not include a scheduler, but its evaluation commands can run through cron or Cloud Scheduler and save results to persistent storage. The walkthrough recommends nightly checks against a stable regression dataset to catch behavior changes from model updates, tool data and shifting dependencies. Such scheduled datasets typically contain dozens to hundreds of cases, compared with a few cases in the faster development loop. A wrapper can compare each run’s mean with a baseline and trigger an alert when the drop exceeds a threshold; the example uses 0.05 as its default and demonstrates a 0.1 threshold, which flags the 4.000-to-2.750 regression.

For ADK projects, eval optimize uses the GEPA framework to generate and score prompt variants, retain those with higher scores and rank candidates for human review. It is not supported for the LangChain extension. Optimization changes the prompt while keeping tools, agent structure and model fixed. The authors caution that a prompt can score better by overfitting to a metric’s wording, so a person must review candidate prompts before promoting one.