Video-Point Model Forecasts 3D Motion

Researchers introduced Video-Point Model (VPM), a system that uses past videos from multiple cameras and a language instruction to forecast future video from arbitrary viewpoints and the paths of 3D points. The paper was published on Oct 20 and submitted to Hugging Face on Oct 21. VPM is designed to make the separate camera views describe the same changing 3D scene.

The team extended a pretrained world model with camera-conditioned processing that combines observations across cameras, then added a trajectory head to predict how physical points move over time. Each trajectory follows one point across all views. Training the model on those trajectories alongside video generation ties changes in appearance across cameras to a shared account of 3D motion.

For training and evaluation, the researchers built VPM-Data, a simulation dataset pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark covering simulated and real environments. They also adapted VPM for robot control by adding robot actions as a token modality. The model predicts those actions jointly with future video, using the same point-supervised backbone.

On VP-WorldBench and existing 4D-generation benchmarks, point supervision improved video fidelity and 3D motion accuracy compared with a video-only counterpart. In simulated and real-robot experiments, it also increased manipulation success for the action-enabled model. The authors describe the shared 3D-aware representation as improving both forecasting and robot control.

Researchers introduced Video-Point Model (VPM), a system that uses past videos from multiple cameras and a language instruction to forecast future video from arbitrary viewpoints and the paths of 3D points. The paper was published on Oct 20 and submitted to Hugging Face on Oct 21. VPM is designed to make the separate camera views describe the same changing 3D scene.

The team extended a pretrained world model with camera-conditioned processing that combines observations across cameras, then added a trajectory head to predict how physical points move over time. Each trajectory follows one point across all views. Training the model on those trajectories alongside video generation ties changes in appearance across cameras to a shared account of 3D motion.

For training and evaluation, the researchers built VPM-Data, a simulation dataset pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark covering simulated and real environments. They also adapted VPM for robot control by adding robot actions as a token modality. The model predicts those actions jointly with future video, using the same point-supervised backbone.

On VP-WorldBench and existing 4D-generation benchmarks, point supervision improved video fidelity and 3D motion accuracy compared with a video-only counterpart. In simulated and real-robot experiments, it also increased manipulation success for the action-enabled model. The authors describe the shared 3D-aware representation as improving both forecasting and robot control.