SmolLM3-3B-checkpoints

SmolLM3 Checkpoints We are releasing intermediate checkpoints of SmolLM3 to enable further research. For more details, check the SmolLM GitHub repo with the end-to-end training and evaluation code: ✓ Pretraining scripts (nanotron) ✓ Post-training code SFT + APO (TRL/alignment-handbook) ✓ Evaluation scripts to reproduce all reported metrics Pre-training We release checkpoints every 40,000 steps, which equals 94.4B tokens. The GBS (Global Batch Size) in tokens for SmolLM3-3B is 2,359,296. To calculate the number of tokens from a given step: Training Stages Stage 1:** Steps 0 to 3,450,000 (86 checkpoints) config Stage 2:** Steps 3,450,000 to 4,200,000 (19 checkpoints) config Stage 3:** Steps 4,200,000 to 4,720,000 (13 checkpoints) config Long Context Extension For the additional 2 stages that extend the context length to 64k, we sample checkpoints every 4,000 steps (9.4B tokens) for a total of 10 checkpoints: Long Context 4k to 32k** config Long Context 32k to 64k** config Post-training We release checkpoints at every step of our post-training recipe: Mid training, SFT, APO soup, and LC expert. How to Load a Checkpoint License Apache 2.0 Citation