John, the author of hexisteme notes, reported on August 2, 2026 that metrics from coding-agent sub-agents cannot be directly compared with metrics from main interactive threads in his single-machine fleet. His logging pipeline had collected a few thousand threads across seven anonymized model versions and five behavioral metrics, but a role × model cross-tab showed that role changed some numbers by up to 135x and that each model had a different mix of main-thread and sub-agent rows.

The fleet used two roles for the same model weights: a long interactive main thread driven by the operator, and a short one-shot sub-agent spawned and discarded by a main thread. Median output tokens per thread differed sharply by role: model A had 200 main rows with a 478,238 median versus 2,761 sub-agent rows with a 6,212 median, or 77x; model C was 427,838 versus 8,784, or 49x; model D was 185,534 versus 1,371, or 135x; model G was 128,415 versus 11,358, or 11x. Model B was 16,432 versus 14,157, or 1.2x, while model F flipped direction at 4,194 versus 51,562, or 0.08x.

Behavioral metrics also changed meaning across roles. Same-file re-edit rate had main-thread medians of 0.40 / 0.44 / 0.50 / 0.53 for four models, but exactly 0 for six of seven sub-agent models. Error-recovery sequences had main-thread medians of 1 / 2 / 2 / 2 for four models, but exactly 0 for six of seven sub-agent models. Validation runs had a median of 0 for six of seven main-thread models and 0 for all seven sub-agent models. The author said sub-agent rows carried no between-model signal for re-edit rate and recovery count, while main-thread rows held the possible signal despite making up only 7% of rows.

The role mix also varied by model. Across the attributed corpus, 4,533 of 4,890 rows, or 92.7%, were sub-agent rows, but main share ranged from 1.5% for models E and B to 71.4% for model F. The cross-tab listed E at 2 main rows and 132 sub-agent rows, B at 20 and 1,338, A at 200 and 2,761, D at 11 and 122, C at 88 and 156, G at 21 and 18, and F at 15 and 6. The author attributed the imbalance to delegation policy, including mechanical fan-out work sent to cheaper tiers; model A spawned 1,788 sub-agents that were also A and 730 that were B, while model C spawned 393 that were B.

Pooling models A and C across roles produced a misleading comparison. A pooled calculation gave A as (200 × 684,639 + 2,761 × 13,691) / 2,961 = 59,010 and C as (88 × 565,678 + 156 × 13,913) / 244 = 212,910, making C appear to burn 3.6x the output tokens of A. Within role strata, C was lower than A on main threads at 565,678 versus 684,639, or 0.83x, and nearly tied on sub-agent runs at 13,913 versus 13,691, or 1.02x. The author called the gap an amalgamation effect rather than textbook Simpson's paradox (pooled trend differs from grouped trends), because the pooled 3.6x came from different weights, not a meaningful within-role effect.

The author said stratifying by role is only the first step because a metric may measure a different construct in each stratum. A completion proxy failed on sub-agent threads when one model had 92 of 99 threads ending on a tool-result line because of a harness logging convention. He also noted that units can change: model A had 200 main attribution rows but 154 usable main threads for the completion proxy. His checklist now includes printing the stratum × treatment cross-tab first, setting thresholds in advance at n ≥ 5 to appear and n ≥ 20 on both sides for an interval, forbidding cross-stratum comparison, testing construct validity per stratum, rejecting unexplained zero-width intervals, and publishing both pooled and stratified results when they disagree.

The article also warned that a [0, 0] confidence interval can reflect tied data rather than precision. For validation-run counts on main threads, every model's median was 0, while means were A 4.11, C 2.38, G 1.67 and B 0.60. Percentile bootstrap on all six model pairs returned a median difference of 0 and a 95% interval of [0, 0], even with a ~7x spread in means. The author said the evidence remained non-causal because routing was not randomized and was entangled with task difficulty, project and week; model A's main-thread re-edit median also ranged 0.34 / 0.45 / 0.53 / 0.70 across four projects.

John, the author of hexisteme notes, reported on August 2, 2026 that metrics from coding-agent sub-agents cannot be directly compared with metrics from main interactive threads in his single-machine fleet. His logging pipeline had collected a few thousand threads across seven anonymized model versions and five behavioral metrics, but a role × model cross-tab showed that role changed some numbers by up to 135x and that each model had a different mix of main-thread and sub-agent rows.

The fleet used two roles for the same model weights: a long interactive main thread driven by the operator, and a short one-shot sub-agent spawned and discarded by a main thread. Median output tokens per thread differed sharply by role: model A had 200 main rows with a 478,238 median versus 2,761 sub-agent rows with a 6,212 median, or 77x; model C was 427,838 versus 8,784, or 49x; model D was 185,534 versus 1,371, or 135x; model G was 128,415 versus 11,358, or 11x. Model B was 16,432 versus 14,157, or 1.2x, while model F flipped direction at 4,194 versus 51,562, or 0.08x.

Behavioral metrics also changed meaning across roles. Same-file re-edit rate had main-thread medians of 0.40 / 0.44 / 0.50 / 0.53 for four models, but exactly 0 for six of seven sub-agent models. Error-recovery sequences had main-thread medians of 1 / 2 / 2 / 2 for four models, but exactly 0 for six of seven sub-agent models. Validation runs had a median of 0 for six of seven main-thread models and 0 for all seven sub-agent models. The author said sub-agent rows carried no between-model signal for re-edit rate and recovery count, while main-thread rows held the possible signal despite making up only 7% of rows.

The role mix also varied by model. Across the attributed corpus, 4,533 of 4,890 rows, or 92.7%, were sub-agent rows, but main share ranged from 1.5% for models E and B to 71.4% for model F. The cross-tab listed E at 2 main rows and 132 sub-agent rows, B at 20 and 1,338, A at 200 and 2,761, D at 11 and 122, C at 88 and 156, G at 21 and 18, and F at 15 and 6. The author attributed the imbalance to delegation policy, including mechanical fan-out work sent to cheaper tiers; model A spawned 1,788 sub-agents that were also A and 730 that were B, while model C spawned 393 that were B.

Pooling models A and C across roles produced a misleading comparison. A pooled calculation gave A as (200 × 684,639 + 2,761 × 13,691) / 2,961 = 59,010 and C as (88 × 565,678 + 156 × 13,913) / 244 = 212,910, making C appear to burn 3.6x the output tokens of A. Within role strata, C was lower than A on main threads at 565,678 versus 684,639, or 0.83x, and nearly tied on sub-agent runs at 13,913 versus 13,691, or 1.02x. The author called the gap an amalgamation effect rather than textbook Simpson's paradox (pooled trend differs from grouped trends), because the pooled 3.6x came from different weights, not a meaningful within-role effect.

The author said stratifying by role is only the first step because a metric may measure a different construct in each stratum. A completion proxy failed on sub-agent threads when one model had 92 of 99 threads ending on a tool-result line because of a harness logging convention. He also noted that units can change: model A had 200 main attribution rows but 154 usable main threads for the completion proxy. His checklist now includes printing the stratum × treatment cross-tab first, setting thresholds in advance at n ≥ 5 to appear and n ≥ 20 on both sides for an interval, forbidding cross-stratum comparison, testing construct validity per stratum, rejecting unexplained zero-width intervals, and publishing both pooled and stratified results when they disagree.

The article also warned that a [0, 0] confidence interval can reflect tied data rather than precision. For validation-run counts on main threads, every model's median was 0, while means were A 4.11, C 2.38, G 1.67 and B 0.60. Percentile bootstrap on all six model pairs returned a median difference of 0 and a 95% interval of [0, 0], even with a ~7x spread in means. The author said the evidence remained non-causal because routing was not randomized and was entangled with task difficulty, project and week; model A's main-thread re-edit median also ranged 0.34 / 0.45 / 0.53 / 0.70 across four projects.