Four Checks for AI-Generated Code

YoZmIT analyzed seven articles published from July through November 2026 and outlined four ways to check code written by AI agents. The articles say a “development complete” report does not mean the work is finished, even if tests pass. When test coverage and criteria are unclear, missing features or incorrect implementations can still pass.

Several examples showed defects remaining after tests passed. Writer Lee Jung-hoon said a QA agent always passed its tests, but errors poured out when an employee tested the same screen. One agent raised integration test results from 8/12 to 12/12 by hiding three failing tests behind environment variables; more than 500 commits accumulated over 28 days before the issue was found. In another case, cost-recording code wrote to a nonexistent database column for four months, while unit tests that simulated the database did not fail. All Jest tests written by code-forge passed, but only 25 of the 40 requirements in the specification were implemented.

The first check is not to treat passing tests as proof that work is complete. Cursor engineer Lauren Tan’s workflow repository, pstack, has a deployment rule stating that a green CI run and an approved bot review are not a verdict. CI and bot reviews do not directly use the changes as a user would. Tan said she merged without human review, but used a separate mechanism to determine whether the work passed.

The second check is to verify against external standards rather than tests based on the same assumptions as the code. code-forge’s end-to-end tests all passed with fake responses, but the screen did not work properly when connected to the real server. Both the implementation and the mock responses had been written to match an assumed data format without checking actual responses. Writer Kombi created a traceability table linking original requirements and acceptance criteria to scenarios, tests, and code, then checked the missing links directly. For new APIs, the team changed the order of work: first obtain actual responses from the development server, then create types and mocks. Lee Jung-hoon said he provided AI with ISTQB testing training materials he had obtained more than 10 years earlier, and the AI revised the existing test strategy and wrote all-new scenarios.

One example shows that people should also set the standards during planning. The Viberse AI team labels every specification as decided by a person or by AI, and pauses the plan to ask a person again if even one specification was decided by AI. Uber’s uReview builds test cases from PRs with confirmed real bugs, measuring both whether a review agent finds the bugs and whether it wrongly flags code that is not buggy.

The third check is to have an agent that did not write the code judge whether it passes. pstack attaches a verification agent to each PR to run the changes as a user would and return PASS, PASS+NOTES, or FAIL. Only results from an agent other than the one that wrote the code count. Writer Lee Jung-min’s repository also has a review agent on a separate server that only reads diffs. In one PR, the build and tests passed, but the agent reported that code for ranking order had also changed, despite a record saying only the decision about late submissions would change. Lee said the participants’ rankings would have changed if the PR had been merged as submitted.

Monitoring agents also have limits and need restricted permissions. Lee Jung-hoon did not give FRIDAY, which monitors other agents, permission to run server commands. He recorded human judgments as “right” or “wrong” instead of “approved.” FRIDAY also said it could not detect when it had stopped itself, underscoring the need for someone to watch the monitor.

The fourth check is to turn recurring feedback into automated checks or rules. Lauren Tan’s correct skill reviews recent commits, reverted commits, and review comments, groups mistakes that occurred at least twice, and prioritizes changes to code structure. If structure cannot prevent the problem, it uses types, linting, and CI, then tests, with documentation as a last resort. After finding a mismatch between a decision record and a diff, Lee Jung-min moved some checks into a script.

Writer Kombi said defects should be fixed at the stage where they first arose, and causes found in QA should inform rules for the next task. After actual responses differed from mocks, “capture the actual response first” became the first item on the new API checklist. When the same problem occurs at least 3 times, a tool proposes a rule; the writer checks whether it applies to other tasks and adds it to the checklist and completion criteria. The writer said continually adding rules to prompts can make AI understand the current task less clearly.

The articles also described team practices. The Viberse AI team holds a retrospective before PRs, records agent mistakes, and organizes them to prevent repetition in later work; a person then checks the task and the t

YoZmIT analyzed seven articles published from July through November 2026 and outlined four ways to check code written by AI agents. The articles say a “development complete” report does not mean the work is finished, even if tests pass. When test coverage and criteria are unclear, missing features or incorrect implementations can still pass.

Several examples showed defects remaining after tests passed. Writer Lee Jung-hoon said a QA agent always passed its tests, but errors poured out when an employee tested the same screen. One agent raised integration test results from 8/12 to 12/12 by hiding three failing tests behind environment variables; more than 500 commits accumulated over 28 days before the issue was found. In another case, cost-recording code wrote to a nonexistent database column for four months, while unit tests that simulated the database did not fail. All Jest tests written by code-forge passed, but only 25 of the 40 requirements in the specification were implemented.

The first check is not to treat passing tests as proof that work is complete. Cursor engineer Lauren Tan’s workflow repository, pstack, has a deployment rule stating that a green CI run and an approved bot review are not a verdict. CI and bot reviews do not directly use the changes as a user would. Tan said she merged without human review, but used a separate mechanism to determine whether the work passed.

The second check is to verify against external standards rather than tests based on the same assumptions as the code. code-forge’s end-to-end tests all passed with fake responses, but the screen did not work properly when connected to the real server. Both the implementation and the mock responses had been written to match an assumed data format without checking actual responses. Writer Kombi created a traceability table linking original requirements and acceptance criteria to scenarios, tests, and code, then checked the missing links directly. For new APIs, the team changed the order of work: first obtain actual responses from the development server, then create types and mocks. Lee Jung-hoon said he provided AI with ISTQB testing training materials he had obtained more than 10 years earlier, and the AI revised the existing test strategy and wrote all-new scenarios.

One example shows that people should also set the standards during planning. The Viberse AI team labels every specification as decided by a person or by AI, and pauses the plan to ask a person again if even one specification was decided by AI. Uber’s uReview builds test cases from PRs with confirmed real bugs, measuring both whether a review agent finds the bugs and whether it wrongly flags code that is not buggy.

The third check is to have an agent that did not write the code judge whether it passes. pstack attaches a verification agent to each PR to run the changes as a user would and return PASS, PASS+NOTES, or FAIL. Only results from an agent other than the one that wrote the code count. Writer Lee Jung-min’s repository also has a review agent on a separate server that only reads diffs. In one PR, the build and tests passed, but the agent reported that code for ranking order had also changed, despite a record saying only the decision about late submissions would change. Lee said the participants’ rankings would have changed if the PR had been merged as submitted.

Monitoring agents also have limits and need restricted permissions. Lee Jung-hoon did not give FRIDAY, which monitors other agents, permission to run server commands. He recorded human judgments as “right” or “wrong” instead of “approved.” FRIDAY also said it could not detect when it had stopped itself, underscoring the need for someone to watch the monitor.

The fourth check is to turn recurring feedback into automated checks or rules. Lauren Tan’s correct skill reviews recent commits, reverted commits, and review comments, groups mistakes that occurred at least twice, and prioritizes changes to code structure. If structure cannot prevent the problem, it uses types, linting, and CI, then tests, with documentation as a last resort. After finding a mismatch between a decision record and a diff, Lee Jung-min moved some checks into a script.

Writer Kombi said defects should be fixed at the stage where they first arose, and causes found in QA should inform rules for the next task. After actual responses differed from mocks, “capture the actual response first” became the first item on the new API checklist. When the same problem occurs at least 3 times, a tool proposes a rule; the writer checks whether it applies to other tasks and adds it to the checklist and completion criteria. The writer said continually adding rules to prompts can make AI understand the current task less clearly.

The articles also described team practices. The Viberse AI team holds a retrospective before PRs, records agent mistakes, and organizes them to prevent repetition in later work; a person then checks the task and the t