Coding AI Often Guesses at Vague DevOps Instructions

Researchers including Hong Kong University of Science and Technology published a paper in July examining how coding AI decides what to work on and how broadly to make changes when given vague DevOps instructions. They said the safety of AI handling operational tasks such as deleting branches and rolling back changes under incomplete instructions had received little prior testing.

The team developed UnderspecBench, a benchmark that measures AI safety using past reports of real system failures. Tests varied instructions across three dimensions: clarity of intent, precision about the target, and the potential impact of failure. Researchers ran five combinations using the OpenCode, Claude Code, and Codex tools with the Claude Haiku 4.5, Codex-5.1-mini, and DeepSeek-v4 models.

When instructions were vague or incomplete, AI agents tended to infer what to do from context and proceed without asking a person. The rate of questions varied by model: DeepSeek-v4 asked in 1.7% of cases, while Haiku 4.5 asked in 38.3–44.5%. Among actions the systems took, 55.8–67.8% involved errors such as changing the wrong target or modifying a broader area than the user intended.

Errors increased as the target became less specific, while their rate changed little when instructions described a larger potential impact if something went wrong. The tendency to make overly broad changes also varied by task: it was 59.8% for deployment and traffic control, and 77.2% for infrastructure settings and monitoring. Tasks limited to a single branch or file were associated with smaller potential damage, the researchers said. The study was titled “Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions.”

The findings appeared in a feature that selects previously published research papers for discussion as an extra installment in a series that began in 2019. The article says the paper was published in July. Earlier evaluations had mainly measured whether systems completed tasks; this study tested what targets coding AI operates on and how far it goes when instructions are incomplete. It reports both the systems’ tendency to act without checking and their error rates, alongside task completion.

Researchers including Hong Kong University of Science and Technology published a paper in July examining how coding AI decides what to work on and how broadly to make changes when given vague DevOps instructions. They said the safety of AI handling operational tasks such as deleting branches and rolling back changes under incomplete instructions had received little prior testing.

The team developed UnderspecBench, a benchmark that measures AI safety using past reports of real system failures. Tests varied instructions across three dimensions: clarity of intent, precision about the target, and the potential impact of failure. Researchers ran five combinations using the OpenCode, Claude Code, and Codex tools with the Claude Haiku 4.5, Codex-5.1-mini, and DeepSeek-v4 models.

When instructions were vague or incomplete, AI agents tended to infer what to do from context and proceed without asking a person. The rate of questions varied by model: DeepSeek-v4 asked in 1.7% of cases, while Haiku 4.5 asked in 38.3–44.5%. Among actions the systems took, 55.8–67.8% involved errors such as changing the wrong target or modifying a broader area than the user intended.

Errors increased as the target became less specific, while their rate changed little when instructions described a larger potential impact if something went wrong. The tendency to make overly broad changes also varied by task: it was 59.8% for deployment and traffic control, and 77.2% for infrastructure settings and monitoring. Tasks limited to a single branch or file were associated with smaller potential damage, the researchers said. The study was titled “Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions.”

The findings appeared in a feature that selects previously published research papers for discussion as an extra installment in a series that began in 2019. The article says the paper was published in July. Earlier evaluations had mainly measured whether systems completed tasks; this study tested what targets coding AI operates on and how far it goes when instructions are incomplete. It reports both the systems’ tendency to act without checking and their error rates, alongside task completion.