Hy3-GGUF

Dedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit. 📖 Documentation &nbsp&nbsp | &nbsp&nbsp🤗 Hugging Face &nbsp&nbsp | &nbsp&nbsp🤖 ModelScope &nbsp&nbsp | &nbsp&nbsp💬 WeChat Hy3 llama.cpp Quantization Low-bit quantized **Hy3 (hy_v3)** GGUFs for llama.cpp — ready to run out of the box, with MTP self-speculative decoding. This repo ships the quantized models, plus the mixed-precision recipes to build your own from a calibration set. Two parts below: **Deploy** (build & run) and **Quantization**. Quick Start Build Clone the latest llama.cpp (must include the `hy_v3` support merged in PR #25395 — i.e. any commit after `505b1ed`; the current `master` is fine) and build it following the official build guide: Binaries land in `llama.cpp-hyv3/build/bin/`. **Already downloaded an older GGUF?** Earlier GGUFs here were built against a patched llama.cpp and won't load correctly on today's upstream. To build and run those, follow the previous README (the `setup_hyv3_llama.sh` + patches flow) instead. Run Recommended setups | GPUs | build | MTP | `-c` (context) | KV cache | |---|---|---|---|---| | 1× H20 (96 GB) | IQ1_M | no | `-c 65536` | `-ctk q8_0 -ctv q8_0` | | 2× H20 (192 GB) | IQ1_M | yes | / (default) | / (f16) | | 2× H20 (192 GB) | Q4_K_M | yes | `-c 65536` | `-ctk q8_0 -ctv q8_0 -ctkd q8_0 -ctvd q8_0` | | 4× H20 (384 GB) | Q4_K_M | yes | / (default) | / (f16) | Weights: IQ1_M ~83 GiB, Q4_K_M ~166 GiB (MTP adds ~2 GiB plus a draft KV cache). 1 card fits only IQ1_M, and tightly — shrink context, quantize KV, no MTP. 2 cards run IQ1_M+MTP with room to spare (drop `-c`/KV flags), but Q4_K_M+MTP is tight, so keep `-c 65536` and q8_0 KV (main + draft). 4 cards run Q4_K_M+MTP comfortably. Quantize with Your Own Data If you have a calibration set, you can compute your own importance matrix and quantize the model yourself. All steps use stock llama.cpp tools; the mixed-precision recipes are shipped in this folder. The mixed-precision rec