SmolLM3-3B

SmolLM3 Table of Contents 1. Model Summary 2. How to use 3. Evaluation 4. Training 5. Limitations 6. License Model Summary SmolLM3 is a 3B parameter language model designed to push the boundaries of small models. It supports dual mode reasoning, 6 languages and long context. SmolLM3 is a fully open model that offers strong performance at the 3B–4B scale. The model is a decoder-only transformer using GQA and NoPE (with 3:1 ratio), it was pretrained on 11.2T tokens with a staged curriculum of web, code, math and reasoning data. Post-training included midtraining on 140B reasoning tokens followed by supervised fine-tuning and alignment via Anchored Preference Optimization (APO). Key features Instruct model optimized for **hybrid reasoning** **Fully open model**: open weights + full training details including public data mixture and training configs **Long context:** Trained on 64k context and supports up to **128k tokens** using YARN extrapolation **Multilingual**: 6 natively supported (English, French, Spanish, German, Italian, and Portuguese) For more details refer to our blog post: https://hf.co/blog/smollm3 How to use The modeling code for SmolLM3 is available in transformers `v4.53.0`, so make sure to upgrade your transformers version. You can also load the model with the latest `vllm` which uses transformers as a backend. [!TIP] We recommend setting `temperature=0.6` and `top_p=0.95` in the sampling parameters. Long context processing The current `config.json` is set for context length up to 65,536 tokens. To handle longer inputs (128k or 256k), we utilize YaRN you can change the `max_position_embeddings` and rope_scaling` to: Enabling and Disabling Extended Thinking Mode We enable extended thinking by default, so the example above generates the output with a reasoning trace. For choosing between enabling, you can provide the `/think` and `/no_think` flags through the system prompt as shown in the snippet below for extended thinking disabled. The code for generat