ReLift Repairs Dense 3D Tracking Trajectories

Researchers from Technion Israel Institute of Technology introduced ReLift, a framework for dense 3D tracking from monocular video. The paper, published on October 31 and submitted to arXiv on November 2, treats tracking as geometric repair and completion of trajectories lifted from 2D video, without image-based training.

Existing approaches often learn point matching, depth estimation and motion together, a difficult task because 4D supervision is scarce. The researchers report that pretrained 2D trackers and 3D geometry models can already produce lifted trajectories that rival dedicated 3D trackers when paired with the same geometry. But errors in matching and depth remain, and during occlusion the estimated depth may describe the object blocking the tracked point.

ReLift uses a feed-forward trajectory corrector that alternates attention along and across trajectories to preserve reliable observations, fix corrupted positions and fill in missing motion. It trains on 3D trajectories with simulated lifting errors, using geometric inputs rather than images and remaining independent of any particular tracker or geometry model. Its correction model can improve lifted trajectories from different tracker and geometry combinations without retraining.

On WorldTrack, ReLift achieved state-of-the-art mean accuracy, exceeding the strongest baselines by 2.1 APD across all points and 3.0 APD on dynamic points. The paper says faster configurations remained competitive with leading 3D trackers while running faster than 4RC. The authors say future advances in the trackers or geometry models can benefit ReLift without retraining.

Researchers from Technion Israel Institute of Technology introduced ReLift, a framework for dense 3D tracking from monocular video. The paper, published on October 31 and submitted to arXiv on November 2, treats tracking as geometric repair and completion of trajectories lifted from 2D video, without image-based training.

Existing approaches often learn point matching, depth estimation and motion together, a difficult task because 4D supervision is scarce. The researchers report that pretrained 2D trackers and 3D geometry models can already produce lifted trajectories that rival dedicated 3D trackers when paired with the same geometry. But errors in matching and depth remain, and during occlusion the estimated depth may describe the object blocking the tracked point.

ReLift uses a feed-forward trajectory corrector that alternates attention along and across trajectories to preserve reliable observations, fix corrupted positions and fill in missing motion. It trains on 3D trajectories with simulated lifting errors, using geometric inputs rather than images and remaining independent of any particular tracker or geometry model. Its correction model can improve lifted trajectories from different tracker and geometry combinations without retraining.

On WorldTrack, ReLift achieved state-of-the-art mean accuracy, exceeding the strongest baselines by 2.1 APD across all points and 3.0 APD on dynamic points. The paper says faster configurations remained competitive with leading 3D trackers while running faster than 4RC. The authors say future advances in the trackers or geometry models can benefit ReLift without retraining.