Circle K Cuts Incident Triage to Under 15 Minutes

Circle K deployed an AI incident-triage system on Amazon Bedrock AgentCore in April 2026, reducing mean triage time from 45–60 minutes to under 15 minutes, a 67–75 percent reduction according to the company’s internal incident-response metrics. The system investigates alerts across Circle K’s AWS infrastructure and sends engineers a structured summary in Slack. Circle K operates more than 17,200 convenience and fuel retail stores across 27 countries, with AWS supporting point-of-sale, loyalty, and electric vehicle charging applications.

Before automation, engineers responding to Grafana alerts checked CloudWatch logs, OpenSearch traces, ECS task status, and GitHub runbooks and code. Circle K built the system to automate this repeated context gathering while leaving operational judgment with engineers. It chose Amazon Bedrock over self-managed open-source agent frameworks to avoid managing model hosting, prompt versions, tool coordination, and security controls. The company says Bedrock fit its existing AWS Identity and Access Management policies, Virtual Private Cloud setup, and logging pipelines; it also says customer data is not shared with model providers or used to train foundation models.

The initial system used Amazon Bedrock Flows to coordinate three specialized agents. A Grafana alert went through Amazon SNS and an SQS queue to a Lambda function, which invoked the Flow. In parallel, one agent checked AWS resources, including ECS task health, load balancer targets, CloudWatch metrics, and logs; another queried OpenSearch logs and traces and retrieved service information from a managed knowledge base. A third agent then searched the affected service’s GitHub repositories, infrastructure definitions, and recent commits. A final model call combined these findings into a structured summary of the probable cause, evidence, affected service and owner, and recommended remediation. The process took 5–15 minutes, depending on incident complexity, and engineers could scan the consistent summary in under 2 minutes.

In an example, an alert reported an HTTP 5xx error rate above 5% for 3 consecutive minutes. The agents found an exhausted database connection pool, “connection refused” errors between the service and PostgreSQL, and code opening a new database connection for every request. The final analysis linked the timing to a marketing campaign that had started 10 minutes before the alert and recommended a code-level fix.

After using Flows for months, Circle K found some investigations needed more time as data volumes and context grew. Filtering logs, shortening context windows, and splitting services across Flows helped, but did not resolve the longer-running analysis needs. The team moved the system to AgentCore Runtime, which supports sessions of up to 8 hours; the current investigations did not need that full duration. It consolidated the three agents into one containerized agent session, while Lambda starts the session and posts its result to Slack within 15 minutes. AgentCore Gateway provides access to OpenSearch, Bedrock Knowledge Bases, and GitHub as tools, with centralized access control.

Circle K says straightforward incidents now receive summaries in 5–8 minutes, while complex multi-service incidents that previously took more than an hour to triage manually finish within 15 minutes. The system also gives on-call engineers consistent summaries, relevant context for escalation to another team, and a quicker way to acknowledge alerts when services are healthy and no errors correlate with the alert.

The team reports that persistent session state lets the agent make a targeted second log review without losing earlier findings. AgentCore captures session traces using OpenTelemetry standards and surfaces them through CloudWatch, helping the team inspect step duration, knowledge-base queries, and bottlenecks for prompt tuning. Runtime also supports containerized agents built with different frameworks. Circle K’s lessons include passing relevant log lines directly to the agent instead of only log-group names and time ranges, testing retrieval settings (the team found the knowledge base’s default configuration worked best for its data), tuning the system prompt and temperature, and running GitHub searches after log and resource analysis has narrowed the likely cause. The provided source ends while describing these lessons.

Circle K deployed an AI incident-triage system on Amazon Bedrock AgentCore in April 2026, reducing mean triage time from 45–60 minutes to under 15 minutes, a 67–75 percent reduction according to the company’s internal incident-response metrics. The system investigates alerts across Circle K’s AWS infrastructure and sends engineers a structured summary in Slack. Circle K operates more than 17,200 convenience and fuel retail stores across 27 countries, with AWS supporting point-of-sale, loyalty, and electric vehicle charging applications.

Before automation, engineers responding to Grafana alerts checked CloudWatch logs, OpenSearch traces, ECS task status, and GitHub runbooks and code. Circle K built the system to automate this repeated context gathering while leaving operational judgment with engineers. It chose Amazon Bedrock over self-managed open-source agent frameworks to avoid managing model hosting, prompt versions, tool coordination, and security controls. The company says Bedrock fit its existing AWS Identity and Access Management policies, Virtual Private Cloud setup, and logging pipelines; it also says customer data is not shared with model providers or used to train foundation models.

The initial system used Amazon Bedrock Flows to coordinate three specialized agents. A Grafana alert went through Amazon SNS and an SQS queue to a Lambda function, which invoked the Flow. In parallel, one agent checked AWS resources, including ECS task health, load balancer targets, CloudWatch metrics, and logs; another queried OpenSearch logs and traces and retrieved service information from a managed knowledge base. A third agent then searched the affected service’s GitHub repositories, infrastructure definitions, and recent commits. A final model call combined these findings into a structured summary of the probable cause, evidence, affected service and owner, and recommended remediation. The process took 5–15 minutes, depending on incident complexity, and engineers could scan the consistent summary in under 2 minutes.

In an example, an alert reported an HTTP 5xx error rate above 5% for 3 consecutive minutes. The agents found an exhausted database connection pool, “connection refused” errors between the service and PostgreSQL, and code opening a new database connection for every request. The final analysis linked the timing to a marketing campaign that had started 10 minutes before the alert and recommended a code-level fix.

After using Flows for months, Circle K found some investigations needed more time as data volumes and context grew. Filtering logs, shortening context windows, and splitting services across Flows helped, but did not resolve the longer-running analysis needs. The team moved the system to AgentCore Runtime, which supports sessions of up to 8 hours; the current investigations did not need that full duration. It consolidated the three agents into one containerized agent session, while Lambda starts the session and posts its result to Slack within 15 minutes. AgentCore Gateway provides access to OpenSearch, Bedrock Knowledge Bases, and GitHub as tools, with centralized access control.

Circle K says straightforward incidents now receive summaries in 5–8 minutes, while complex multi-service incidents that previously took more than an hour to triage manually finish within 15 minutes. The system also gives on-call engineers consistent summaries, relevant context for escalation to another team, and a quicker way to acknowledge alerts when services are healthy and no errors correlate with the alert.

The team reports that persistent session state lets the agent make a targeted second log review without losing earlier findings. AgentCore captures session traces using OpenTelemetry standards and surfaces them through CloudWatch, helping the team inspect step duration, knowledge-base queries, and bottlenecks for prompt tuning. Runtime also supports containerized agents built with different frameworks. Circle K’s lessons include passing relevant log lines directly to the agent instead of only log-group names and time ranges, testing retrieval settings (the team found the knowledge base’s default configuration worked best for its data), tuning the system prompt and temperature, and running GitHub searches after log and resource analysis has narrowed the likely cause. The provided source ends while describing these lessons.