RealCam-Vid

RealCam-Vid Dataset --> News 25/04/08: We provide torch dataset demo code for example usage of our RealCam-Vid. 25/03/26: Release our dataset RealCam-Vid v1 for metric-scale camera-controlled video generation, containing ~100K video clips with dedicated short/long captions and metric-scale camera annotations. 25/02/18: Initial commit of the project, we plan to release the full dataset and data processing code in several weeks. DiT-based models (e.g., CogVideoX) trained on our dataset will be available at RealCam-I2V. Motivation Current datasets for camera-controllable video generation face critical limitations that hinder the development of robust and versatile models. Our curated dataset and data-processing pipeline uniquely combines **diverse scene dynamics** with **metric-scale camera trajectories**, enabling generative models to learn both scene dynamics and camera motion in a unified framework. Training Data Variation Static Scene & Dynamic Camera Dynamic Scene & Static Camera Dynamic Scene & Dynamic Camera Existing datasets for camera motions and scene dynamics suffer from **domain-specific biases** and **incomplete annotations**, limiting their utility for training robust real-world models. **Static Scene & Dynamic Camera datasets** (e.g., RealEstate10K, DL3DV) - **Strengths**: High aesthetic quality, dense relative-scale camera trajectory annotations. - **Weaknesses**: Static scenes lack object dynamics, leading to models that fail to generalize to real-world dynamic environments due to overfitting to rigid structures. **Dynamic Scene & Static Camera datasets** (e.g., 360-Motion) - **Strengths**: Capture dynamic objects. - **Weaknesses**: Omit camera motion, limiting their utility for trajectory-based video generation. **Dynamic Scene & Dynamic Camera datasets** (e.g., MiraData) - **Strengths**: Exhibit rich real-world dynamics (moving objects + camera motion). - **Weaknesses**: No absolute-scale camera annotations, making them unsuitable for metric-scale tr