FineMath-Llama-3B

Model Card Model summary This is a continual-pre-training of Llama-3.2-3B on a mix of 📐 FineMath (our new high quality math dataset) and FineWeb-Edu. The model demonstrates superior math performance compared to Llama 3.2 3B, while maintaining similar performance on knowledge, reasoning, and common sense benchmarks: It was trained on **160B tokens** using a mix of 40% FineWeb-Edu and 60% from FineMath (30% FineMath-4+ subset and 30% InfiWebMath-4+ subset). We use nanotron for the training, and you can find the training scripts in our SmolLM2 GitHub repo. Use Intended use This model was trained on English math data and is not instruction-tuned, making it intended for text completion in English. It is part of the FineMath ablation models we trained for FineMath (finemath-ablation-4plus-160B), and is not necessarily the best possible outcome achievable with the given dataset. Generation Training Model **Architecture**: Llama3 **Pretraining steps**: 160k **Pretraining tokens**: 160B **Precision**: bfloat16 Hardware **GPUs**: 64 H100 Software nanotron for training datatrove for tokenization lighteval for evaluation Evaluation We used the SmolLM2 setup to evaluate all our ablation models with `lighteval`. You can find the details here: https://github.com/huggingface/smollm/tree/main/evaluation#smollm2-base-models Limitations This model was predominantly trained on English math data, potentially limiting its performance in other languages. Furthermore, the model's behavior is influenced by the quality and diversity of its training data, which may include biases and harmful content. Citation