Brian Jin reported on August 7, 2026, that Study 011 of the open-source Judgment Pack evaluator experiments asked one AI model, gpt-5.6-sol, to independently author structured test records for the same synthetic vendor-screening policy 50 times. The policy included sanctions rules, country restrictions, personal-data conditions, a high-risk threshold at 70, and a personal-data threshold at 40. Before the batch, six possible defect classes had been preregistered, and the model did not see the Judgment Pack or the later defects being tested.

The experiment used one fixed prompt, one fixed synthetic policy, one pinned CLI and binary, fresh isolated environments for each call, 50 sequential authoring calls, and six preregistered coverage classes. Of the 50 runs, 49 passed pipeline checks and 1 was rejected before scoring. All 49 valid runs covered all 6 of 6 boundary classes, each valid run produced exactly 16 accepted records, 784 of 784 accepted records agreed with the reference policy semantics, and all 49 valid completions were different.

The six classes were exact risk = 70; 70 <= risk < 71; personal data with 40 <= risk < 41; 40 <= risk < 70; registered country = SY; and personal data with 39 <= risk < 40. Each class was covered in 49 / 49 valid runs, a 100% observed rate with the same exact 95% Clopper-Pearson interval of [0.9275, 1.0000]. Jin said that interval matters because 49 successes out of 49 valid runs cannot establish a true rate of 100%, only a lower bound of about 92.75% under that experimental cell.

The final class, personal data with 39 <= risk < 40, received special caution. The synthetic policy explicitly mentioned the threshold at 40, but did not ask the model to test a hidden family boundary at 39. All 49 valid runs still produced a record inside that interval: 28 runs used 39.99 and 21 runs used 39.999. Jin said the narrower explanation is that the prompt requested borderline cases around 40 and the model generated values immediately below 40.

The outputs were not byte-identical repeats. All 49 valid completions were distinct, and the largest group of byte-identical outputs was 1. The model varied names, examples, and record composition while still landing on the same semantic regions. Jin noted that different outputs do not prove statistically independent draws, because provider-side cross-session behavior was not observable from the retained artifacts.

Across the valid runs, each completion produced 16 records, giving 49 x 16 = 784 accepted records. The study reported 784 policy-concordant records, 0 mislabelled records, and 0 dropped records, for pooled label agreement of 784 / 784 = 1.000. Jin did not attach a binomial confidence interval to that figure because the 16 records inside one completion were not treated as independent trials; the run remained the repeated unit.

Run 026 was excluded after the pre-prompt developer context failed to match the locked golden context, even though the model process exited successfully. The pipeline returned transcript-refused, producing an observed pipeline-invalid rate of 1 / 50 = 2% with a 95% interval of [0.05%, 10.65%]. The likely explanation was service-side boilerplate variation, but the study did not inspect the refused transcript and override the allowlist rule.

The preregistered review-depth mapping used the lower bound of the confidence interval: lower >= 0.80 meant LIGHT review, lower >= 0.40 meant STANDARD review, and lower < 0.40 meant FULL review. All six classes ended with lower bound = 0.9275, mislabel share = 0, and tier = LIGHT. Jin emphasized that the mapping is experimental and has not been operationally validated.

The study’s claim was limited to one prompt, one model, and one small synthetic policy. It did not show that AI can reliably author arbitrary organizational judgment, did not test other models or prompts, and did not evaluate real policies with interacting exceptions, multiple documents, ambiguous language, precedence, temporal conditions, missing evidence, conflicting authorities, or human approval requirements. It also measured coverage of regions that could expose defects, not defect detection itself; Study 010 handled a single defect-detection draw separately.

Brian Jin reported on August 7, 2026, that Study 011 of the open-source Judgment Pack evaluator experiments asked one AI model, gpt-5.6-sol, to independently author structured test records for the same synthetic vendor-screening policy 50 times. The policy included sanctions rules, country restrictions, personal-data conditions, a high-risk threshold at 70, and a personal-data threshold at 40. Before the batch, six possible defect classes had been preregistered, and the model did not see the Judgment Pack or the later defects being tested.

The experiment used one fixed prompt, one fixed synthetic policy, one pinned CLI and binary, fresh isolated environments for each call, 50 sequential authoring calls, and six preregistered coverage classes. Of the 50 runs, 49 passed pipeline checks and 1 was rejected before scoring. All 49 valid runs covered all 6 of 6 boundary classes, each valid run produced exactly 16 accepted records, 784 of 784 accepted records agreed with the reference policy semantics, and all 49 valid completions were different.

The six classes were exact risk = 70; 70 <= risk < 71; personal data with 40 <= risk < 41; 40 <= risk < 70; registered country = SY; and personal data with 39 <= risk < 40. Each class was covered in 49 / 49 valid runs, a 100% observed rate with the same exact 95% Clopper-Pearson interval of [0.9275, 1.0000]. Jin said that interval matters because 49 successes out of 49 valid runs cannot establish a true rate of 100%, only a lower bound of about 92.75% under that experimental cell.

The final class, personal data with 39 <= risk < 40, received special caution. The synthetic policy explicitly mentioned the threshold at 40, but did not ask the model to test a hidden family boundary at 39. All 49 valid runs still produced a record inside that interval: 28 runs used 39.99 and 21 runs used 39.999. Jin said the narrower explanation is that the prompt requested borderline cases around 40 and the model generated values immediately below 40.

The outputs were not byte-identical repeats. All 49 valid completions were distinct, and the largest group of byte-identical outputs was 1. The model varied names, examples, and record composition while still landing on the same semantic regions. Jin noted that different outputs do not prove statistically independent draws, because provider-side cross-session behavior was not observable from the retained artifacts.

Across the valid runs, each completion produced 16 records, giving 49 x 16 = 784 accepted records. The study reported 784 policy-concordant records, 0 mislabelled records, and 0 dropped records, for pooled label agreement of 784 / 784 = 1.000. Jin did not attach a binomial confidence interval to that figure because the 16 records inside one completion were not treated as independent trials; the run remained the repeated unit.

Run 026 was excluded after the pre-prompt developer context failed to match the locked golden context, even though the model process exited successfully. The pipeline returned transcript-refused, producing an observed pipeline-invalid rate of 1 / 50 = 2% with a 95% interval of [0.05%, 10.65%]. The likely explanation was service-side boilerplate variation, but the study did not inspect the refused transcript and override the allowlist rule.

The preregistered review-depth mapping used the lower bound of the confidence interval: lower >= 0.80 meant LIGHT review, lower >= 0.40 meant STANDARD review, and lower < 0.40 meant FULL review. All six classes ended with lower bound = 0.9275, mislabel share = 0, and tier = LIGHT. Jin emphasized that the mapping is experimental and has not been operationally validated.

The study’s claim was limited to one prompt, one model, and one small synthetic policy. It did not show that AI can reliably author arbitrary organizational judgment, did not test other models or prompts, and did not evaluate real policies with interacting exceptions, multiple documents, ambiguous language, precedence, temporal conditions, missing evidence, conflicting authorities, or human approval requirements. It also measured coverage of regions that could expose defects, not defect detection itself; Study 010 handled a single defect-detection draw separately.