Qwen2.5-7B-Instruct-AWQ

Qwen2.5-7B-Instruct-AWQ Introduction Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2: Significantly **more knowledge** and has greatly improved capabilities in **coding** and **mathematics**, thanks to our specialized expert models in these domains. Significant improvements in **instruction following**, **generating long texts** (over 8K tokens), **understanding structured data** (e.g, tables), and **generating structured outputs** especially JSON. **More resilient to the diversity of system prompts**, enhancing role-play implementation and condition-setting for chatbots. **Long-context Support** up to 128K tokens and can generate up to 8K tokens. **Multilingual support** for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more. This repo contains the AWQ-quantized 4-bit instruction-tuned 72B Qwen2.5 model**, which has the following features: Type: Causal Language Models Training Stage: Pretraining & Post-training Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias Number of Parameters: 7.61B Number of Paramaters (Non-Embedding): 6.53B Number of Layers: 28 Number of Attention Heads (GQA): 28 for Q and 4 for KV Context Length: Full 131,072 tokens and generation 8192 tokens Please refer to this section for detailed instructions on how to deploy Qwen2.5 for handling long texts. Quantization: AWQ 4-bit For more details, please refer to our blog, GitHub, and Documentation. Requirements The code of Qwen2.5 has been in the latest Hugging face `transformers` and we advise you to use the latest version of `transformers`. With `transformers<4.37.0`, you will encounter the following error: Also check out our AWQ documentation for more usage guide. Quic