Chinese-LiPS

Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides ⭐ Introduction The **Chinese-LiPS** dataset is a multimodal dataset designed for audio-visual speech recognition (AVSR) in Mandarin Chinese. This dataset combines speech, video, and textual transcriptions to enhance automatic speech recognition (ASR) performance, especially in educational and instructional scenarios. 🚀 Dataset Details **Total Duration:** 100.84 hours **Number of Speakers:** 207 professional speakers **Number of Clips:** 36,208 video clips **Audio Format:** Stereo WAV, 48 kHz sampling rate Video Format: **Slide Video:** 1080p resolution, 30 fps **Lip-Reading Video:** 720p resolution, 30 fps **Annotations:** JSON format with transcriptions and extracted text from slides Dataset Statistics | Split | Duration (hrs) | # Segments | # Speakers | | ---------- | -------------- | ---------- | ---------- | | Train | 85.37 | 30,341 | 175 | | Validation | 5.35 | 1,959 | 11 | | Test | 10.12 | 3,908 | 21 | | **Total** | **100.84** | **36,208** | **207** | 📂 Dataset Organization The dataset is structured into several compressed files: **image.zip**: First-frame images from slide videos (used for OCR and vision-language models). **processed_test.zip processed_val.zip processed_train.zip**: Processed data with 16 kHz audio, 96×96 25-frame lip-reading videos, and JSON annotations. train.zip, test.zip, val.zip: Data split into training, testing, and validation sets. Each contains: **meta_all.csv, meta_train.csv, meta_valid.csv, meta_test.csv**: Metadata files with ID, TOPIC, WAV, PPT, FACE, and TEXT fields. The TOPIC field is abbreviated in Chinese as follows: DZJJ = E-sports & Gaming, JKYS = Health & Wellness, KJ = Science & Technology, LY = Travel & Exploration, QC = Automobile & Industry, RWLS = Culture & History, TY = Sports & Competitions, YS = Movies & TV Series, ZX = Others. **meta_test.json**: Includes OCR and InternVL2 prompts for the test set. 📥 Download Y