AWS’s Sales, Marketing, and Global Services Operations team built a continuous evaluation framework with Amazon Bedrock AgentCore Evaluations for a production multi-agent system serving AWS Field Sales worldwide. Sellers use Amazon Quick to ask business questions in plain English, while leaders use the same system for Weekly Business Reviews (WBRs). Specialized agents provide territory-scoped revenue and performance insights, customer intelligence, and text-to-SQL access to analytics data; answers follow each user’s role-level security.

Before launch, evaluation of the Customer Insights Agent caught four errors even though system monitoring stayed green: it described $410.08 in a customer’s three-month service revenue as “$410.08 Million,” mislabeled Sales Revenue as excluding Adjustments, estimated Annual Recurring Revenue absent from the data, and presented its own calculations as sourced facts. A pre-production test also found that a mismatch between coordinated data, tool, and prompt changes lowered the pass rate from 78 percent to 38 percent. The team says weekly data refreshes and changes to agents, prompts, code, or models can create further quality risks, so it needs to check answers before releases and continuously in production.

The framework uses AgentCore Evaluations alongside Amazon Bedrock Guardrails for safety, Amazon CloudWatch for metrics, alarms, and tracing, Amazon S3 and Amazon Redshift to persist evaluations, Amazon Quick for dashboards, and AWS Lambda for custom evaluators. It assesses five dimensions: safety, quality, performance, cost, and operational control. The team prioritizes safety as a requirement for launching and continuing to run an agent; quality receives the most attention in the post because it is described as the hardest dimension to get right.

The nine evaluation stages run from defining quality at the first commit through improving a live system: Foundation, Experiment, Merge (CI), Gate (CD), Canary, Adversary, Drift, Pulse, and Feedback Loop. They ask teams to define success, compare versions, catch regressions, assess production readiness, detect silent failures, test attacks, monitor quality changes, check live operation, and automate improvement. The framework applies at three levels: individual tools, end-to-end sub-agents, and multi-agent orchestration. Each level has its own golden set, evaluators, and promotion criteria.

At the foundation stage, the team starts with business outcomes for sellers and leaders, then maps them to measurable evaluation metrics with thresholds and alarm conditions. For sellers, examples include entity resolution accuracy and grounding rate; for leadership, accuracy pass rate and response coverage. It also calls for safety policies, p50, p90, and p99 latency targets per tool and agent, cost-per-query budgets, and service level agreements (SLAs) linked to alarms and runbooks. The team defines six accuracy failure types from production errors: metric swaps, wrong periods, wrong entities, pure hallucinations, wrong currency or unit symbols, and unauthorized calculations. It tracks overall accuracy pass rate as the headline measure and monitors dimensions such as relevance, faithfulness, helpfulness, instruction following, and tool selection accuracy. The supplied article text ends while describing the evaluator stack, after introducing built-in and customized evaluators.

AWS’s Sales, Marketing, and Global Services Operations team built a continuous evaluation framework with Amazon Bedrock AgentCore Evaluations for a production multi-agent system serving AWS Field Sales worldwide. Sellers use Amazon Quick to ask business questions in plain English, while leaders use the same system for Weekly Business Reviews (WBRs). Specialized agents provide territory-scoped revenue and performance insights, customer intelligence, and text-to-SQL access to analytics data; answers follow each user’s role-level security.

Before launch, evaluation of the Customer Insights Agent caught four errors even though system monitoring stayed green: it described $410.08 in a customer’s three-month service revenue as “$410.08 Million,” mislabeled Sales Revenue as excluding Adjustments, estimated Annual Recurring Revenue absent from the data, and presented its own calculations as sourced facts. A pre-production test also found that a mismatch between coordinated data, tool, and prompt changes lowered the pass rate from 78 percent to 38 percent. The team says weekly data refreshes and changes to agents, prompts, code, or models can create further quality risks, so it needs to check answers before releases and continuously in production.

The framework uses AgentCore Evaluations alongside Amazon Bedrock Guardrails for safety, Amazon CloudWatch for metrics, alarms, and tracing, Amazon S3 and Amazon Redshift to persist evaluations, Amazon Quick for dashboards, and AWS Lambda for custom evaluators. It assesses five dimensions: safety, quality, performance, cost, and operational control. The team prioritizes safety as a requirement for launching and continuing to run an agent; quality receives the most attention in the post because it is described as the hardest dimension to get right.

The nine evaluation stages run from defining quality at the first commit through improving a live system: Foundation, Experiment, Merge (CI), Gate (CD), Canary, Adversary, Drift, Pulse, and Feedback Loop. They ask teams to define success, compare versions, catch regressions, assess production readiness, detect silent failures, test attacks, monitor quality changes, check live operation, and automate improvement. The framework applies at three levels: individual tools, end-to-end sub-agents, and multi-agent orchestration. Each level has its own golden set, evaluators, and promotion criteria.

At the foundation stage, the team starts with business outcomes for sellers and leaders, then maps them to measurable evaluation metrics with thresholds and alarm conditions. For sellers, examples include entity resolution accuracy and grounding rate; for leadership, accuracy pass rate and response coverage. It also calls for safety policies, p50, p90, and p99 latency targets per tool and agent, cost-per-query budgets, and service level agreements (SLAs) linked to alarms and runbooks. The team defines six accuracy failure types from production errors: metric swaps, wrong periods, wrong entities, pure hallucinations, wrong currency or unit symbols, and unauthorized calculations. It tracks overall accuracy pass rate as the headline measure and monitors dimensions such as relevance, faithfulness, helpfulness, instruction following, and tool selection accuracy. The supplied article text ends while describing the evaluator stack, after introducing built-in and customized evaluators.