PointMotionBench

PointMotionBench This is the official benchmark for the paper MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction. A benchmark for evaluating 3D point motion in video, covering egocentric and third-person scenes across three source datasets. Each sample pairs an RGB video clip with per-object 3D and 2D tracked surface points and a human-verified natural-language caption. Overview | Dataset | Clips | Video format | Tracks | Scene type | |---------|-------|--------------|--------|------------| | DAVIS | 90 | mp4, 24 fps | 2D + 3D | Third-person, diverse outdoor/indoor | | HOT3D | 2,475 | mp4, 30 fps | 2D + 3D | Egocentric, object manipulation (Aria) | | WorldTrack | 155 | npz (frames embedded), 30 fps | 3D (+2D) | Egocentric + studio, 4 splits | Setup Step 1 — Download PointMotionBench Benchmark data provided in this repository include annotations, captions, indices, and scripts created by Ai2 that correspond to the source datasets. Step 2 — DAVIS: Download Videos DAVIS videos should be reconstructed from the source data at the official source. Download **Trainval 2017 - Images (480p)** and **Annotations**, then convert frames to mp4 (requires `ffmpeg`): Step 3 — HOT3D: Download Videos HOT3D videos should be reconstructed from the source data at bop-benchmark/hot3d (HuggingFace). Requirements:** `huggingface_hub`, `imageio[ffmpeg]`, `imageio-ffmpeg`, `opencv-python-headless`, `numpy` This runs all three stages (download TARs → extract RGB → trim to PointMotionBench windows). For large-scale extraction, run the three scripts individually — `extract_rgbs.py` supports sharding: Step 4 — WorldTrack: Reconstruct Clips Download the WorldTrack source data (WorldTrack benchmark, introduced in St4RTrack, Feng et al., ICCV 2025 — dataset download available at HavenFeng/St4RTrack). The source data should have this layout: Then extract PointMotionBench clips using the index map from Step 1: | Split | Clips | Frames per clip | Scene type | |-------|-------|