PostHog Uses Decision Models to Filter LLM Calls

PostHog Uses Decision Models to Filter LLM Calls
Wednesday, November 18, 2026
  • •PostHog uses decision models to screen tasks before making potentially expensive LLM calls.
  • •Replay Vision lets 98.7% of summaries pass without an LLM call; a 0.4 score triggers privacy cleanup.
  • •Embedding-based reranking improved results about 25% across 20 test queries; complex reasoning remains an LLM task.

PostHog says its production systems use decision models to handle narrow classification and routing tasks before calling a large language model (LLM). These models return a typed answer with a probability score rather than free-form prose; a calibrated score of 0.8 means answers are correct about 80% of the time. The company says decision models can be up to 444.6x cheaper and 193.6x faster than frontier models, making them useful for screening whether an LLM call is needed.

In Replay Vision, PostHog checks each session summary with two yes-or-no questions: whether it names a real person and whether it shows contact details or an ID. If either answer scores 0.4 or higher, Gemini removes the personal information and the summary is checked again; otherwise it passes through. In production, 98.7% of summaries skip the LLM call. PostHog says using two narrow questions cuts false flags to about a tenth of the number produced by one question. A score of 0.85 can still mean about 150 wrong answers per 1,000 checks, so the company sets a low 0.4 threshold in this case because missing personal data is more costly than an extra LLM call. For its Slack bot, where an unwanted reply is the greater concern, only messages scoring at least 0.85 on both whether they ask for information and whether PostHog can answer are passed along.

PostHog also uses a decision model to route tagged Slack requests to a suitable model. If the user has not named one, the model chooses from options described with short notes, following the instruction to pick the cheapest model likely to complete the task correctly on the first try. The company cautions that thresholds vary by task, model and provider. When it tested auto-merging similar Error Tracking logs, thresholds changed between versions 1 and 2 of the same model; PostHog made thresholds configurable by project and kept the judge in shadow mode while recalibrating it with hand-labeled data. The company now uses an internal AI gateway to swap and test models, including self-hosted ones.

For searches such as “onboarding friction” in Replay Vision, embeddings first produce a list of session recordings; a decision model then scores how closely each summary matches the query, and the results are reranked. Across 20 test queries, PostHog says this improved ranking quality by about 25%. A decision model cannot rank all sessions from scratch when the task requires more context or several reasoning steps. An earlier attempt to select session highlights chained two questions and sent hourly requests of up to 32 sessions and 12-38k tokens each, flooding the model server and blocking other products' calls for hours.

PostHog also replaced deterministic chart-selection rules with a decision model that considers the SQL query, if an organization opts in, and the first three result rows. For a pageviews-by-browser query, its Jev model selected a bar chart split by browser where the old rules chose an aggregate line chart. In a test ranking findings from an internal PR review agent, five questions yielded inconclusive results with an AUC of 0.52. PostHog says typical decision models are not suited to this kind of reasoning because they cannot handle large requests; their long-context mode is slower and less accurate. Its in-house research team developed Jeeves, a hybrid decision model that reasons before deciding when uncertain. Jeeves is PostHog's default for decision calls without a named model; the company also names Kev as a similar variation. Its guidance is to use decision models for inputs too simple for an LLM but too fuzzy for a regular expression, and leave more complex, multi-step reasoning to LLMs.

Subscribe to the AIB Newsletter

The fastest, most reliable AI news, straight to your inbox

Companies related to this article