RoboColosseum Tests Robot Policies Across Embodiments

Researchers introduced RoboColosseum, a distributed framework for evaluating robot policies on locally operated robots across varied environments. The paper, published on Nov 2, reports tests of five vision-language-action (VLA) policies and three frontier AI agents on four robot embodiments: SO-101, Franka, Bimanual-YAM and G1 Humanoid. The framework uses unified policy serving and standardized protocols to make trials across locations comparable.

In its Open Track, the researchers ran 5,699 blinded, pairwise comparisons at 124 sites under matched local conditions, without task-specific adaptation. Frontier agents performed comparably to specialized VLA policies, but all systems struggled to complete open-world tasks reliably. The Fine-tune Track tested VLA policies on 20 tasks, giving each policy the same 100 demonstrations per task. After fine-tuning, GR00T N1.7 led on SO-101, while pi0.5 led on Franka and Bimanual-YAM.

Rankings across embodiment-specific task suites had weak agreement (Kendall W = 0.24), and Open Track rankings poorly predicted performance after fine-tuning, with substantial rank reversals on all three embodiments tested in both tracks. The researchers also measured partial task progress. On the G1 Humanoid, none of the policies completed any of the five tasks, but average progress ranged from 13.3% to 23.2% across policies. Open Track rankings became more stable as trials accumulated; active matchmaking reached comparable ranking agreement with substantially fewer physical trials than uniform sampling.

Researchers introduced RoboColosseum, a distributed framework for evaluating robot policies on locally operated robots across varied environments. The paper, published on Nov 2, reports tests of five vision-language-action (VLA) policies and three frontier AI agents on four robot embodiments: SO-101, Franka, Bimanual-YAM and G1 Humanoid. The framework uses unified policy serving and standardized protocols to make trials across locations comparable.

In its Open Track, the researchers ran 5,699 blinded, pairwise comparisons at 124 sites under matched local conditions, without task-specific adaptation. Frontier agents performed comparably to specialized VLA policies, but all systems struggled to complete open-world tasks reliably. The Fine-tune Track tested VLA policies on 20 tasks, giving each policy the same 100 demonstrations per task. After fine-tuning, GR00T N1.7 led on SO-101, while pi0.5 led on Franka and Bimanual-YAM.

Rankings across embodiment-specific task suites had weak agreement (Kendall W = 0.24), and Open Track rankings poorly predicted performance after fine-tuning, with substantial rank reversals on all three embodiments tested in both tracks. The researchers also measured partial task progress. On the G1 Humanoid, none of the policies completed any of the five tasks, but average progress ranged from 13.3% to 23.2% across policies. Open Track rankings became more stable as trials accumulated; active matchmaking reached comparable ranking agreement with substantially fewer physical trials than uniform sampling.