Debashish Ghosal said on August 7, 2026, that building `agent-exec-trace`, an open-source observability layer for AI agents, shifted from a detector-design task into a data-shape problem after tests on 100,000 real agent traces from Hugging Face. The tool instruments agent runs with OpenTelemetry-style traces (standardized telemetry records), then analyzes loops, retry storms, cost spikes, hallucination patterns, and other runtime failures so developers can understand where an agent run went wrong.

Ghosal first expected the difficult work to be defining anomalies, choosing thresholds, wiring traces, running analytics, and shipping the repository. On the first large pass over the Hugging Face corpus, the `empty_response` detector fired on 100% of traces, while 28 of 35 rule-based detectors never fired. He concluded the detectors were not the main problem because the traces used different response keys, tool-name conventions, operation names, timestamp formats, and parent-child relationships. After four normalization passes, the original compatibility estimate fell to 42.4%.

`agent-exec-trace` is designed to emit spans for planning, tool calls, retrieval, memory, approvals, and cost, then surface bad runs in a usable UI. The project ships as three PyPI packages, has 794 Python tests and 34 Playwright end-to-end tests, and uses a small SDK intended to wrap real agents without redesigning them. Ghosal said the disciplined project setup, including ruff clean, mypy strict clean, green tests, and coverage above 90%, did not prevent mock-agent assumptions from being validated against unrealistic data.

After the real-trace corpus exposed the limits of the system, Ghosal built a synthetic trace generator with one million traces, ten fake agents, fourteen tools, and deliberate behavior modes including loops, retries, timeouts, inactivity gaps, intervention waits, token explosions, and memory bursts. Structural compatibility rose to 99.2%, and 20 of the 35 rule-based detectors fired. The synthetic data still distorted validation: the hallucination detector fired on 98% of synthetic traces because synthetic outputs and tool evidence had an artificially easy relationship, while the `cost_spike` detector rarely fired because generated costs were mostly at cents-level values.

Ghosal said validation now needs layers: unit tests show the logic works, synthetic traces show a detector can see required fields, and real traces show whether the result matters outside the developer’s sandbox. He also found that a green milestone for OTLP export was misleading because two bugs canceled each other out: the OTel collector’s gRPC port was not exposed in Docker Compose, and the SDK path configured local tracing instead of OTLP export. He said a real gate must prove that a real agent emits data, Jaeger receives it, analytics ingests it, and the API serves the result.

The project also forced a privacy tradeoff. Metadata-only capture avoided raw tool arguments, full tool responses, and memory values by default, but it made the hallucination detector weaker because the detector could not compare an agent claim with returned tool evidence. When truncated content was allowed, the hallucination false-positive rate dropped sharply. Ghosal said he still does not fully trust 28 detectors that have not fired meaningfully on real corpora, LLM detectors on production workloads, span-tree materialization in the API, threshold sets across workloads, or green gates that lack end-to-end proof.

Debashish Ghosal said on August 7, 2026, that building `agent-exec-trace`, an open-source observability layer for AI agents, shifted from a detector-design task into a data-shape problem after tests on 100,000 real agent traces from Hugging Face. The tool instruments agent runs with OpenTelemetry-style traces (standardized telemetry records), then analyzes loops, retry storms, cost spikes, hallucination patterns, and other runtime failures so developers can understand where an agent run went wrong.

Ghosal first expected the difficult work to be defining anomalies, choosing thresholds, wiring traces, running analytics, and shipping the repository. On the first large pass over the Hugging Face corpus, the `empty_response` detector fired on 100% of traces, while 28 of 35 rule-based detectors never fired. He concluded the detectors were not the main problem because the traces used different response keys, tool-name conventions, operation names, timestamp formats, and parent-child relationships. After four normalization passes, the original compatibility estimate fell to 42.4%.

`agent-exec-trace` is designed to emit spans for planning, tool calls, retrieval, memory, approvals, and cost, then surface bad runs in a usable UI. The project ships as three PyPI packages, has 794 Python tests and 34 Playwright end-to-end tests, and uses a small SDK intended to wrap real agents without redesigning them. Ghosal said the disciplined project setup, including ruff clean, mypy strict clean, green tests, and coverage above 90%, did not prevent mock-agent assumptions from being validated against unrealistic data.

After the real-trace corpus exposed the limits of the system, Ghosal built a synthetic trace generator with one million traces, ten fake agents, fourteen tools, and deliberate behavior modes including loops, retries, timeouts, inactivity gaps, intervention waits, token explosions, and memory bursts. Structural compatibility rose to 99.2%, and 20 of the 35 rule-based detectors fired. The synthetic data still distorted validation: the hallucination detector fired on 98% of synthetic traces because synthetic outputs and tool evidence had an artificially easy relationship, while the `cost_spike` detector rarely fired because generated costs were mostly at cents-level values.

Ghosal said validation now needs layers: unit tests show the logic works, synthetic traces show a detector can see required fields, and real traces show whether the result matters outside the developer’s sandbox. He also found that a green milestone for OTLP export was misleading because two bugs canceled each other out: the OTel collector’s gRPC port was not exposed in Docker Compose, and the SDK path configured local tracing instead of OTLP export. He said a real gate must prove that a real agent emits data, Jaeger receives it, analytics ingests it, and the API serves the result.

The project also forced a privacy tradeoff. Metadata-only capture avoided raw tool arguments, full tool responses, and memory values by default, but it made the hallucination detector weaker because the detector could not compare an agent claim with returned tool evidence. When truncated content was allowed, the hallucination false-positive rate dropped sharply. Ghosal said he still does not fully trust 28 detectors that have not fired meaningfully on real corpora, LLM detectors on production workloads, span-tree materialization in the API, threshold sets across workloads, or green gates that lack end-to-end proof.