PLM-VideoBench

Dataset Summary PLM-VideoBench is a collection of human-annotated resources for evaluating Vision Language models, focused on detailed video understanding. [\[📃 Tech Report\]](https://arxiv.org/abs/2504.13180) [\[📂 Github\]](https://github.com/facebookresearch/perception_models/) Supported Tasks PLM-VideoBench includes evaluation data for the following tasks: FGQA In this task, a model must answer a multiple-choice question (MCQ) that probes fine-grained activity understanding. Given a question and multiple options that differ in a fine-grained detail (e.g., painting vertically vs. horizontally), the model must select the correct answer. To reduce bias, we follow prior work and report multi-binary accuracy (MBAcc). Specifically, each question is split into multiple binary-choice questions, where the correct answer is compared with one distractor at a time; a prediction is considered correct only when the correct answer is consistently selected across all binary comparisons. Data fields are: `uid`: a `string` feature, unique identifier for the binary question. `qa_id`: a `string` feature, unique identifier for the Q&A sample. `video`: a `string` feature, unique identifier for the video segment. `question`: a `string` feature, the question about the video segment. `answer`: a `string` feature, the groud truth answer to the question. `options`: a `struct` feature representing the two potential answers to the binary question. `answer_index`: a `int32` feature, the index of the correct answer within the options. `metadata`: a `dict` of features, representing metadata about the video segment and Q&A pair: `source_dataset`: a `string` feature, name of the source dataset. `source_video_id`: a `string` feature, video id of untrimmed source video. `source_start_time`: a `float` feature, denoting the start time (seconds) of the video segment in the source video. `source_end_time`: a `float` feature, denoting the end time (seconds) of the video segment in the source video. `q_t