Nanbeige4.2-3B-Base

Technical Report 👁️ 1. Introduction Nanbeige4.2-3B-Base is a new base model in the Nanbeige4 series, trained from scratch with improvements to both model architecture and training data. It delivers substantial gains over the previous generation and outperforms opensource base models of comparable scale, including Qwen3.5-4B and Gemma4-E4B, across a broad range of benchmarks. Key features include: **Looped Architecture**: Adopts a Looped Transformer architecture, in which hidden states are fed back into the same Transformer layers after completing a bottom-to-top pass. The reuse schema significantly increases the model capacity without increasing the number of parameters. **Enhanced Training Data Recipe**: Expands the total amount of training data from 23T to 28T tokens, with more aggressive upsampling of math, code, STEM, and Synthetic-QA data to strengthen reasoning and coding capabilities. The accompanying `modeling_nanbeige.py` also includes our latest architectural improvements, including **LoopSplit**, **mHC with depth attention**, and **concatenated n-gram embeddings**. These features have been incorporated into Nanbeige4.5, whose training is underway for release later in 2026. 2. Model Performance We compare Nanbeige4.2-3B-Base with other base models of comparable parameter scale on a diverse benchmark suite covering mathematical reasoning, general reasoning, code generation, and knowledge-intensive tasks: Model Parameters Benchmarks Total Params Non-emb. Params GSM8K BBH MBPP MMLU-Pro SuperGPQA GPQA Gemma4-E4B-Base 8B 4B 61.8 62.5 53.5 37.6 23.3 27.5 Qwen3.5-4B-Base 5B 4B 84.4 79.1 57.1 51.8 32.1 43.1 Nanbeige4.1-3B-Base 4B 3B 85.9 70.7 60.7 47.6 24.8 36.2 Nanbeige4.2-3B-Base 4B 3B 92.7 81.6 67.6 63.8 35.2 53.3 The results demonstrate that **Nanbeige4.2-3B-Base** consistently outperforms Gemma4-E4B-Base, Qwen3.5-4B-Base, and our previous Nanbeige4.1-3B-Base across the benchmark suite. 3. Quickstart 4. Limitations While we place great emphasis on model safety