Artificial Analysis launched AA-Robotics v0.1 on October 14, 2026, a research preview that tests how frontier language models understand, plan and act in the physical world. The benchmark gives each model a 6-axis robot arm with five joints and a gripper, plus a simple objective. Its first release covers eight tabletop tasks, and the company says it expects tracking robotic performance by language models to become important as these models improve, even though they are not suitable for production robotics on their own.

The eight tasks test control and dexterity, spatial reasoning and adaptability through objectives such as pressing a button, placing a toy crocodile inside a rectangle, pouring a drink without spills, packing four blocks, matching shapes, writing “hello world,” building the tallest freestanding tower and measuring a box to within 2 mm. The models run in coding-agent harnesses (software environments for writing and running code) in isolated containers, where they can use four camera views and a simple HTTP API to control a Hiwonder NexArm. They receive brief instructions and must submit each attempt for review; completed tasks score 100, partial results receive 1 to 99 points, and failures score zero. All attempts are zero-shot, meaning the models receive no task-specific examples beforehand.

GPT-6 Astra ranked first with a mean score of 69.4 out of 100, followed by GPT-6.1 Sol at 64.6, Gemini 4 Argon at 55.3, Claude Opus 5.5 at 48.8 and Claude Fable 5.1 at 46.3. Artificial Analysis also reported rounded scores of 69 and 65 for Astra and Sol when comparing cost: Astra averaged $16.36 per task, more than eight times Sol’s $1.93, for a four-point score difference. The company says 70% of model spending went to input context, reflecting the visual information processed.

Models averaged about 30 minutes per task, compared with a human average of 20 seconds; Gemini 4 Argon took 52 minutes per task. The company characterizes model task times as roughly 100 times slower than human performance. Individual runs varied: GPT-6 Astra pressed the red button in 2.2 minutes, while Claude Fable 5.1 spent about ten minutes observing and planning before acting. On crocodile placement, GPT-6.1 Sol declared the task complete after five minutes, compared with 45 minutes for Claude Opus 5.5. Gemini 4 Argon spent an average of 21 minutes observing and reasoning before its first action, nearly twice as long as any other model. The most cost-effective model cost about ten times as much as a person paid the average US hourly wage. Human costs were calculated using $37.81 per hour, the preliminary September 2026 average for US private nonfarm payroll employees.

Grasping performance varied across the six tasks that involved picking up objects. GPT-6 Astra and Claude Fable 5.1 had the highest successful-grasp shares, about 45% and 44%; Gemini 4 Argon recorded 36%, Claude Opus 5.5 40%, and GPT-6.1 Sol 28%. The benchmark counted a grasp as successful when the intended object remained held during lifting or carrying. Its examples of errors include models continuing to adjust objects after they were already correctly placed, misjudging whether a task was complete, dropping or pushing objects, and making empty or misplaced grasps. Artificial Analysis says models often refine attempts after reaching a passing state; GPT-6.1 Sol built a passing tower after about 17 minutes, then later failed the task. The company says it remains to be seen how models will improve at judging task completion in physical space.

Artificial Analysis launched AA-Robotics v0.1 on October 14, 2026, a research preview that tests how frontier language models understand, plan and act in the physical world. The benchmark gives each model a 6-axis robot arm with five joints and a gripper, plus a simple objective. Its first release covers eight tabletop tasks, and the company says it expects tracking robotic performance by language models to become important as these models improve, even though they are not suitable for production robotics on their own.

The eight tasks test control and dexterity, spatial reasoning and adaptability through objectives such as pressing a button, placing a toy crocodile inside a rectangle, pouring a drink without spills, packing four blocks, matching shapes, writing “hello world,” building the tallest freestanding tower and measuring a box to within 2 mm. The models run in coding-agent harnesses (software environments for writing and running code) in isolated containers, where they can use four camera views and a simple HTTP API to control a Hiwonder NexArm. They receive brief instructions and must submit each attempt for review; completed tasks score 100, partial results receive 1 to 99 points, and failures score zero. All attempts are zero-shot, meaning the models receive no task-specific examples beforehand.

GPT-6 Astra ranked first with a mean score of 69.4 out of 100, followed by GPT-6.1 Sol at 64.6, Gemini 4 Argon at 55.3, Claude Opus 5.5 at 48.8 and Claude Fable 5.1 at 46.3. Artificial Analysis also reported rounded scores of 69 and 65 for Astra and Sol when comparing cost: Astra averaged $16.36 per task, more than eight times Sol’s $1.93, for a four-point score difference. The company says 70% of model spending went to input context, reflecting the visual information processed.

Models averaged about 30 minutes per task, compared with a human average of 20 seconds; Gemini 4 Argon took 52 minutes per task. The company characterizes model task times as roughly 100 times slower than human performance. Individual runs varied: GPT-6 Astra pressed the red button in 2.2 minutes, while Claude Fable 5.1 spent about ten minutes observing and planning before acting. On crocodile placement, GPT-6.1 Sol declared the task complete after five minutes, compared with 45 minutes for Claude Opus 5.5. Gemini 4 Argon spent an average of 21 minutes observing and reasoning before its first action, nearly twice as long as any other model. The most cost-effective model cost about ten times as much as a person paid the average US hourly wage. Human costs were calculated using $37.81 per hour, the preliminary September 2026 average for US private nonfarm payroll employees.

Grasping performance varied across the six tasks that involved picking up objects. GPT-6 Astra and Claude Fable 5.1 had the highest successful-grasp shares, about 45% and 44%; Gemini 4 Argon recorded 36%, Claude Opus 5.5 40%, and GPT-6.1 Sol 28%. The benchmark counted a grasp as successful when the intended object remained held during lifting or carrying. Its examples of errors include models continuing to adjust objects after they were already correctly placed, misjudging whether a task was complete, dropping or pushing objects, and making empty or misplaced grasps. Artificial Analysis says models often refine attempts after reaching a passing state; GPT-6.1 Sol built a passing tower after about 17 minutes, then later failed the task. The company says it remains to be seen how models will improve at judging task completion in physical space.