nemotron-labs-audio-visual-flamingo-hf

Model Overview Nemotron-Labs-Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos Nemotron-Labs-Audio-Visual Flamingo (AVF) is a fully open audio-visual large language model for joint understanding and reasoning over audio, images, and videos. `nvidia/nemotron-labs-audio-visual-flamingo-hf` is the instruction-tuned checkpoint for multimodal understanding and text generation over long, complex, real-world videos. AVF is designed for settings where the answer depends on both what is visible and what is audible: speech, background sounds, music, visual actions, scene transitions, temporal order, and cross-modal alignment. This checkpoint generates text only; streaming TTS is not included in the Hub weights. This model is for non-commercial research purposes only.** Description AVF extends the Audio Flamingo line from audio-only understanding into joint audio-visual reasoning. The model combines: a SigLIP vision tower for images and sampled video frames Dynamic-S2 visual preprocessing for high-resolution images and long videos an AF-Whisper audio encoder for speech, sound, and music separate 2-layer projectors for visual and audio features temporal audio-visual interleaving with Constrained Rotary Time Embeddings (CRTE) a Qwen2.5-7B-family decoder-only language backbone Unlike systems that process video frames and audio as independent streams, AVF aligns synchronized visual and audio chunks, interleaves them along the time axis, and feeds the fused sequence to the language model. This helps the model attend across co-occurring visual and auditory events. Model Details **Developed by:** NVIDIA and University of Maryland **Checkpoint:** AVF-Instruct, final 2026-06-29 release checkpoint **Model type:** audio-visual large language model for conditional text generation **Backbone:** Qwen2.5-7B-family decoder-only language model **Modalities:** text, image, audio, video **Language:** English-focused release **License:** NVIDIA OneWay Noncommercial