Stanford and Columbia researchers introduced DexAgent, an agentic framework that converts one first-person human video and a task description into robot training data. Across 11 real-world manipulation tasks, policies trained on DexAgent data achieved 63.6% average success, versus 18.2% for the strongest evaluated baseline, SPIDER, and 17.3% for V2D—about 3.5× the strongest baseline. The tasks included battery insertion, scissor cutting, rope knotting and computer installation, and covered rigid, articulated and deformable objects, tool use and longer action sequences. Each policy was tested for 10 trials per task with randomized object placements. DexAgent-trained policies reached 80% success when placing a toy in a drawer and 50% on rope knotting. The researchers used the same policy model and shared data-augmentation and visual-processing protocol for method comparisons.

DexAgent processes each demonstration in four stages: it identifies objects, their properties and task subgoals; reconstructs the objects in simulation; generates and checks robot motions; then produces varied training episodes. For a bottle-opening task, it represents the cap’s rotation and upward movement and the other hand’s role in holding the bottle steady. Its simulation checks test whether objects have the needed physical properties, such as whether turning the cap moves it upward. When a suitable reconstruction tool is unavailable, the agent can write one, refine the simulated object until checks pass, and test the new function on three additional versions with different parameters before adding it to its reusable library.

For robot motion, DexAgent uses the human pose as a guide, optimizes finger positions to reproduce relevant contacts and tests candidate grasps in simulated interactions. It links verified poses into a trajectory and can revise an earlier action if it leaves the scene in a state that blocks a later step. It then varies initial object and robot states, adapts the trajectory and checks feasibility, generating 500 verified robot episodes per task in the policy experiments. To create robot-view images, it renders the simulated robot and objects along those trajectories, removes the demonstrator’s hands and manipulated objects from the original footage, composites the renderings into the cleared scene and refines their appearance through retexturing. The resulting episodes are used to fine-tune the pretrained π0.5 policy.

The reusable library contains 103 skills and 188 verifiers. In a separate experiment processing 100 EgoDex videos, conversion time fell from 4.3 hours for the first video to 2.2 hours with skill reuse, and to 13.1 minutes when object assets could also be reused; the full run averaged 2.1 hours per video. That experiment accumulated 85 skills and 168 verifiers, a subset of the broader library. The researchers say simulation checks cannot guarantee real-world success: DexAgent may misunderstand a demonstration, reconstruct an object incorrectly, or miss an error that later carries into the library. Force-sensitive interactions are especially challenging, and the team identifies stronger verification and smaller simulation-to-reality gaps as areas for further work. It also plans to explore reinforcement learning when available skills cannot produce a valid solution.

Stanford and Columbia researchers introduced DexAgent, an agentic framework that converts one first-person human video and a task description into robot training data. Across 11 real-world manipulation tasks, policies trained on DexAgent data achieved 63.6% average success, versus 18.2% for the strongest evaluated baseline, SPIDER, and 17.3% for V2D—about 3.5× the strongest baseline. The tasks included battery insertion, scissor cutting, rope knotting and computer installation, and covered rigid, articulated and deformable objects, tool use and longer action sequences. Each policy was tested for 10 trials per task with randomized object placements. DexAgent-trained policies reached 80% success when placing a toy in a drawer and 50% on rope knotting. The researchers used the same policy model and shared data-augmentation and visual-processing protocol for method comparisons.

DexAgent processes each demonstration in four stages: it identifies objects, their properties and task subgoals; reconstructs the objects in simulation; generates and checks robot motions; then produces varied training episodes. For a bottle-opening task, it represents the cap’s rotation and upward movement and the other hand’s role in holding the bottle steady. Its simulation checks test whether objects have the needed physical properties, such as whether turning the cap moves it upward. When a suitable reconstruction tool is unavailable, the agent can write one, refine the simulated object until checks pass, and test the new function on three additional versions with different parameters before adding it to its reusable library.

For robot motion, DexAgent uses the human pose as a guide, optimizes finger positions to reproduce relevant contacts and tests candidate grasps in simulated interactions. It links verified poses into a trajectory and can revise an earlier action if it leaves the scene in a state that blocks a later step. It then varies initial object and robot states, adapts the trajectory and checks feasibility, generating 500 verified robot episodes per task in the policy experiments. To create robot-view images, it renders the simulated robot and objects along those trajectories, removes the demonstrator’s hands and manipulated objects from the original footage, composites the renderings into the cleared scene and refines their appearance through retexturing. The resulting episodes are used to fine-tune the pretrained π0.5 policy.

The reusable library contains 103 skills and 188 verifiers. In a separate experiment processing 100 EgoDex videos, conversion time fell from 4.3 hours for the first video to 2.2 hours with skill reuse, and to 13.1 minutes when object assets could also be reused; the full run averaged 2.1 hours per video. That experiment accumulated 85 skills and 168 verifiers, a subset of the broader library. The researchers say simulation checks cannot guarantee real-world success: DexAgent may misunderstand a demonstration, reconstruct an object incorrectly, or miss an error that later carries into the library. Force-sensitive interactions are especially challenging, and the team identifies stronger verification and smaller simulation-to-reality gaps as areas for further work. It also plans to explore reinforcement learning when available skills cannot produce a valid solution.