GLM-TTS

GLM-TTS: Controllable & Emotion-Expressive Zero-shot TTS 📜 Paper   |   💻 GitHub Repository   |   🛠️ Audio.Z.AI 📖 Model Introduction GLM-TTS is a high-quality text-to-speech (TTS) synthesis system based on large language models, supporting zero-shot voice cloning and streaming inference. The system adopts a two-stage architecture combining an LLM for speech token generation and a Flow Matching model for waveform synthesis. By introducing a **Multi-Reward Reinforcement Learning** framework, GLM-TTS significantly improves the expressiveness of generated speech, achieving more natural emotional control compared to traditional TTS systems. Key Features **Zero-shot Voice Cloning:** Clone any speaker's voice with just 3-10 seconds of prompt audio. **RL-enhanced Emotion Control:** Utilizes a multi-reward reinforcement learning framework (GRPO) to optimize prosody and emotion. **High-quality Synthesis:** Generates speech comparable to commercial systems with reduced Character Error Rate (CER). **Phoneme-level Control:** Supports "Hybrid Phoneme + Text" input for precise pronunciation control (e.g., polyphones). **Streaming Inference:** Supports real-time audio generation suitable for interactive applications. **Bilingual Support:** Optimized for Chinese and English mixed text. System Architecture GLM-TTS follows a two-stage design: 1. **Stage 1 (LLM):** A Llama-based model converts input text into speech token sequences. 2. **Stage 2 (Flow Matching):** A Flow model converts token sequences into high-quality mel-spectrograms, which are then turned into waveforms by a vocoder. Reinforcement Learning Alignment To tackle flat emotional expression, GLM-TTS uses a **Group Relative Policy Optimization (GRPO)** algorithm with multiple reward functions (Similarity, CER, Emotion, Laughter) to align the LLM's generation strategy. Evaluation Results Evaluated on `seed-tts-eval`. **GLM-TTS_RL** achieves the lowest Character Error Rate (CER) while m