SmolLM3-3B-Base

SmolLM3 Table of Contents 1. Model Summary 2. How to use 3. Evaluation 4. Training 5. Limitations 6. License Model Summary SmolLM3 is a 3B parameter language model designed to push the boundaries of small models. It supports 6 languages, advanced reasoning and long context. SmolLM3 is a fully open model that offers strong performance at the 3B–4B scale. SmolLM3-3B-Base** is the base model after pretraining, you can find the instruct model at SmolLM3-3B. The model is a decoder-only transformer using GQA and NoPE, it was pretrained on 11.2T tokens with a staged curriculum of web, code, math and reasoning data. Post-training included midtraining on 140B reasoning tokens followed by supervised fine-tuning and alignment via Anchored Preference Optimization (APO). Key features Instruct model optimized for **hybrid reasoning** **Fully open model**: open weights + full training details including public data mixture and training configs **Long context:** Trained on 64k context and suppots up to **128k tokens** using YARN extrapolation **Multilingual**: 6 natively supported (English, French, Spanish, German, Italian, and Portuguese) For more details refer to our blog post: https://hf.co/blog/smollm3 How to use The modeling code for SmolLM3 is available in transformers `v4.53.0`, so make sure to upgrade your transformers version. You can also load the model with the latest `vllm` which uses transformers as a backend. For local inference, you can use `llama.cpp`, `ONNX`, `MLX` and `MLC`. You can find quantized checkpoints in this collection (https://huggingface.co/collections/HuggingFaceTB/smollm3-686d33c1fdffe8e635317e23). Long context processing The current `config.json` is set for context length up to 65,536 tokens. To handle longer inputs (128k or 256k), we utilize YaRN you can change the `max_position_embeddings` and rope_scaling` to: Evaluation In this section, we report the evaluation results of SmolLM3 model. All evaluations are zero-shot unless stated otherwise, and we