PhysicalAI-VANTAGE-Bench

VANTAGE-BENCH Video ANalysis Tasks Across Generalized Environments* Dataset Description VANTAGE-BENCH is the first public benchmark purpose-built for evaluating visual understanding on video captured by fixed infrastructure cameras. It spans three real-world domains — warehouse, smart city / Intelligent Transportation Systems (ITS), and smart spaces — across six spatio-temporal video understanding tasks including video question answering (VQA), temporal grounding, dense video captioning, event verification, spatial grounding, and spatio-temporal tracking. This dataset is for evaluation purposes only. Dataset Owner(s) NVIDIA Corporation Dataset Creation Date April 24, 2026 License/Terms of Use This dataset is released under the NVIDIA Evaluation Data License. Dataset Characterization Data Collection Method** Hybrid: Human, Synthetic, Automated. Video data is sourced from vendor-provided footage (GoPro captures of warehouse and smart space environments), synthetic generation (DriveSim collision and multi-camera scenarios), and publicly scraped sources (Dubuque highway/ITS footage). Labeling Method** Hybrid: Human, Synthetic, Pseudolabeled. Annotations for VQA, dense video captions, and temporal localization are primarily human-authored. Spatial grounding labels (2D/3D bounding boxes, referring expressions) use a combination of human annotation and pseudolabeling pipelines (detection + SAM for spatial pointing). Event verification labels are human-curated. Annotations are held server-side for evaluation only. Directory Structure Evaluation Tasks and Submission Formats | Category | Task | Metric | |----------|------|--------| | Semantic | VQA | Accuracy | | Semantic | Event Verification | F1 Score | | Temporal | Dense Video Captioning | SODA-c | | Temporal | Temporal Localization | mAP@tIoU | | Spatial | 2D Object Localization | F1@0.5 | | Spatial | 2D Referring Expressions | mIoU | | Spatial | 2D Spatial Pointing | Pointing Accuracy | | Spatio-Temporal | Single Object