sigmacollab

Dataset Card for SigmaCollab __`SigmaCollab`__ is a dataset that enables research on human-AI physically situated collaboration. The dataset consists of a set of 85 sessions in which untrained participants were guided by a mixed-reality assistive AI agent in performing procedural tasks in the physical world. In addition 8 sessions are available in which an expert performed the same tasks. Dataset Details Dataset Description __`SigmaCollab`__, described in detail in this arxiv paper, was collected with an open-source mixed-reality AI application called Sigma (itself described in this arxiv paper and in an IEEE VR extended abstract). The _application-driven_ and _interactive_ nature of the __`SigmaCollab`__ dataset bring to the fore novel research challenges for human-AI collaboration, and provide more realistic testing grounds for various AI models operating in this space. Additionally, the Sigma system is open-source, and you can download and run it yourself to collect your own / additional data. __`SigmaCollab`__ includes a set of rich, multimodal data streams: the participant and system audio, egocentric camera views from the head-mounted device, depth maps, head, hand and gaze tracking information, as well as additional annotations performed post-hoc. The raw set of data streams included is summarized in the table below: | Stream | Representation | Avg. Frame Rate | |----------------------------------|------------------------------------------------------------------------------------------------------------------|:---------------------:| | RGB Camera View | 896 × 504 pixels @ 24bpp, with camera pose and intrinsics | 14.91 Hz | | Depth Camera View | 320 × 288 pixels @ 16bpp, with camera pose and intrinsics | 4.98 Hz | | Left Front Grayscale Camera View | 640 × 480 pixels @ 8bpp, with camera pose and intrinsics | 13.64 Hz | | Right Front Grayscale Camera View| 640 × 480 pixels @ 8bpp, with camera pose and intrinsics | 13.64 Hz | | Head Pose + Eye Gaze | tuple of h