CrucibleBench introduces a proof-of-concept for evaluating large language models (LLMs) using a Multi-User Dungeon (MUD), a text-based, persistent virtual environment. By utilizing a constrained environment with 7 command types, 12 rooms, and 14 items, researchers aimed to measure model behavior rather than static knowledge. The framework tracks four NPCs (non-player characters) that maintain individual trust and suspicion levels ranging from 0–100, providing explicit social feedback to the models as they interact. Each test run involves within-run persistence, ensuring that actions such as taking items or earning trust remain effective throughout the interaction.

The evaluation revealed that using LLM judges in scoring stacks can significantly distort performance rankings. When removing two classifier-dependent dimensions from the scoring protocol, the rankings for frontier models shifted by as many as six positions. The researchers reported per-subject agreement between models and independent judges ranging from 21.7% to 84.8%, highlighting instability that aggregate reliability metrics often fail to capture. The study suggests that benchmarks relying on LLM judges should prioritize reporting per-subject agreement and ranking stability under judge ablation (systematic removal of components) to ensure accuracy.

Behavioral analysis using state-machine telemetry identified three primary failure modes across tested models: dialogue looping, wrong-room interaction, and exploration paralysis. Dialogue looping occurred in 14–66% of runs, where agents repeatedly used failed conversational tactics without adapting. Exploration paralysis was characterized by agents failing to translate information gathering into goal-directed actions, often observed as agents performing five consecutive 'look' commands or visiting two or fewer rooms over twenty turns. Grok 4 was noted as the only frontier model with a significant incidence of wrong-room interaction, affecting 12% of its runs.

The project, which cost approximately $99 to produce, emphasizes that its findings center on measurement methodology rather than definitive rankings. The researchers released a full artifact package including 650 transcripts, source code, and billing logs. CrucibleBench is now transitioning to a second phase focused on calibration, with a provisional $3,500 funding goal to expand the benchmark's objectives and pilot cohorts.

CrucibleBench introduces a proof-of-concept for evaluating large language models (LLMs) using a Multi-User Dungeon (MUD), a text-based, persistent virtual environment. By utilizing a constrained environment with 7 command types, 12 rooms, and 14 items, researchers aimed to measure model behavior rather than static knowledge. The framework tracks four NPCs (non-player characters) that maintain individual trust and suspicion levels ranging from 0–100, providing explicit social feedback to the models as they interact. Each test run involves within-run persistence, ensuring that actions such as taking items or earning trust remain effective throughout the interaction.

The evaluation revealed that using LLM judges in scoring stacks can significantly distort performance rankings. When removing two classifier-dependent dimensions from the scoring protocol, the rankings for frontier models shifted by as many as six positions. The researchers reported per-subject agreement between models and independent judges ranging from 21.7% to 84.8%, highlighting instability that aggregate reliability metrics often fail to capture. The study suggests that benchmarks relying on LLM judges should prioritize reporting per-subject agreement and ranking stability under judge ablation (systematic removal of components) to ensure accuracy.

Behavioral analysis using state-machine telemetry identified three primary failure modes across tested models: dialogue looping, wrong-room interaction, and exploration paralysis. Dialogue looping occurred in 14–66% of runs, where agents repeatedly used failed conversational tactics without adapting. Exploration paralysis was characterized by agents failing to translate information gathering into goal-directed actions, often observed as agents performing five consecutive 'look' commands or visiting two or fewer rooms over twenty turns. Grok 4 was noted as the only frontier model with a significant incidence of wrong-room interaction, affecting 12% of its runs.

The project, which cost approximately $99 to produce, emphasizes that its findings center on measurement methodology rather than definitive rankings. The researchers released a full artifact package including 650 transcripts, source code, and billing logs. CrucibleBench is now transitioning to a second phase focused on calibration, with a provisional $3,500 funding goal to expand the benchmark's objectives and pilot cohorts.