Rescale Deploys AI Test-Triage Agent

Rescale built an AI agent on AWS to classify and route nightly end-to-end test failures, reaching staging in five days and production within weeks. Since launch, agreement with human triage has risen from 44% to 68%, while time blocking production deployment during failure analysis has fallen 20%.

The agent runs as a step in Rescale’s Argo Workflows CI/CD pipeline on Amazon EKS. When hundreds of Playwright tests produce failures, it uses Claude Opus 4.8 on Amazon Bedrock to classify each as an infrastructure issue, code bug, flaky test, or environment problem. It retrieves historical patterns from a knowledge base, queries pod logs through Model Context Protocol (MCP, a standard for connecting tools to AI systems), and produces a report with file references, function names, and recommended fixes for engineers to review. It completes the workflow in seconds; engineers validate the report before action.

AWS architects and Rescale engineers built the system in a five-day, in-person AI-Driven Development Lifecycle (AI-DLC) workshop. During the first two days, the team answered more than 20 AI-generated clarification questions, scoped Phase 1, deferred Phase 2 items, and consolidated five user stories into four work units: a historical-pattern knowledge base, the triage agent, Argo integration, and an evaluation harness. The team explicitly deferred features including automatic pull request generation.

During construction on days two through four, the team reviewed architecture and interface contracts. It shifted hosting from Amazon Bedrock AgentCore to Amazon EKS, where Rescale already runs Argo, and changed log retrieval from a third-party service to Amazon CloudWatch while retaining the MCP pattern. Built with the Strands SDK, the containerized agent supports concurrent failure analysis and structured JSON and HTML output. By day four, the pipeline was running in staging. On day five, the team set up a PyTest evaluation framework with live and stubbed modes and an initial classification-agreement baseline. In a live demonstration, the agent correctly classified an infrastructure failure and a code bug, with file-level fix recommendations.

The team matched design effort to each work unit: the knowledge base was deployed with Terraform in hours; the triage agent received an architecture review, interface contracts, 12 design questions, and iterative code review; Argo integration focused on infrastructure contracts and IAM; and the evaluation harness had lightweight design and substantial implementation. The agent has run in production since the workshop, processing nightly failures and giving engineers an automated first pass to validate. Its agreement with human triage rose from 44% at initial deployment to 68%, as the team refined its golden dataset and prompts. The measured 20% drop in time blocking production deployment applies to the period when failures are analyzed; Rescale’s releases are weekly.

Rescale is evaluating an internal AI-DLC Center of Excellence to codify practices and support adoption by other teams. It is incorporating AI-DLC into development of new engineering simulation products and applying the approach to simulation application catalog management, container vulnerability tracking and remediation for government compliance, and workflow-based cloud coding agents. Rescale says the method structures design discussions into reviewed specifications, allowing coding agents to work on defined tasks while engineers remain accountable for correctness.

Rescale built an AI agent on AWS to classify and route nightly end-to-end test failures, reaching staging in five days and production within weeks. Since launch, agreement with human triage has risen from 44% to 68%, while time blocking production deployment during failure analysis has fallen 20%.

The agent runs as a step in Rescale’s Argo Workflows CI/CD pipeline on Amazon EKS. When hundreds of Playwright tests produce failures, it uses Claude Opus 4.8 on Amazon Bedrock to classify each as an infrastructure issue, code bug, flaky test, or environment problem. It retrieves historical patterns from a knowledge base, queries pod logs through Model Context Protocol (MCP, a standard for connecting tools to AI systems), and produces a report with file references, function names, and recommended fixes for engineers to review. It completes the workflow in seconds; engineers validate the report before action.

AWS architects and Rescale engineers built the system in a five-day, in-person AI-Driven Development Lifecycle (AI-DLC) workshop. During the first two days, the team answered more than 20 AI-generated clarification questions, scoped Phase 1, deferred Phase 2 items, and consolidated five user stories into four work units: a historical-pattern knowledge base, the triage agent, Argo integration, and an evaluation harness. The team explicitly deferred features including automatic pull request generation.

During construction on days two through four, the team reviewed architecture and interface contracts. It shifted hosting from Amazon Bedrock AgentCore to Amazon EKS, where Rescale already runs Argo, and changed log retrieval from a third-party service to Amazon CloudWatch while retaining the MCP pattern. Built with the Strands SDK, the containerized agent supports concurrent failure analysis and structured JSON and HTML output. By day four, the pipeline was running in staging. On day five, the team set up a PyTest evaluation framework with live and stubbed modes and an initial classification-agreement baseline. In a live demonstration, the agent correctly classified an infrastructure failure and a code bug, with file-level fix recommendations.

The team matched design effort to each work unit: the knowledge base was deployed with Terraform in hours; the triage agent received an architecture review, interface contracts, 12 design questions, and iterative code review; Argo integration focused on infrastructure contracts and IAM; and the evaluation harness had lightweight design and substantial implementation. The agent has run in production since the workshop, processing nightly failures and giving engineers an automated first pass to validate. Its agreement with human triage rose from 44% at initial deployment to 68%, as the team refined its golden dataset and prompts. The measured 20% drop in time blocking production deployment applies to the period when failures are analyzed; Rescale’s releases are weekly.

Rescale is evaluating an internal AI-DLC Center of Excellence to codify practices and support adoption by other teams. It is incorporating AI-DLC into development of new engineering simulation products and applying the approach to simulation application catalog management, container vulnerability tracking and remediation for government compliance, and workflow-based cloud coding agents. Rescale says the method structures design discussions into reviewed specifications, allowing coding agents to work on defined tasks while engineers remain accountable for correctness.