nanowhale-100m-base

nanowhale-100m-base 🐳 A small ~110M parameter language model implementing the **DeepSeek-V4 architecture** from scratch. This is the pretrained base model — see HuggingFaceTB/nanowhale-100m for the SFT/chat version. Training code**: github.com/huggingface/nanowhale Architecture This model implements key DeepSeek-V4 innovations at a miniature scale: | Component | Details | |---|---| | **Parameters** | ~110M total (41M embeddings, 69M non-embedding) | | **Hidden size** | 320 | | **Layers** | 8 | | **Attention heads** | 8 (1 KV head — MQA-style) | | **Head dim** | 96 (32 RoPE + 64 NoPE) | | **MLA** | q_lora_rank=160, o_groups=2, o_lora_rank=80 | | **MoE** | 4 routed experts + 1 shared, top-2 routing | | **Expert FFN** | SwiGLU, intermediate_size=640 | | **Routing** | sqrtsoftplus scoring, noaux_tc method | | **Hyper-Connections** | hc_mult=4, Sinkhorn routing (2 iters) | | **MTP** | 1 next-token prediction layer | | **Vocab** | 129,280 (DeepSeek-V4 tokenizer) | | **Context** | 2,048 tokens | DeepSeek-V4 Features Implemented **Multi-head Latent Attention (MLA)**: Compressed KV cache via latent projections **Mixture of Experts (MoE)**: Sparse activation — only 2 of 4 experts per token **Hyper-Connections**: Multi-copy hidden states with learned Sinkhorn routing replacing residual connections **SwiGLU FFN** with configurable limit **Grouped output projection** (o_groups) Training **Dataset**: HuggingFaceFW/fineweb-edu (streaming) **Steps**: 5,000 **Tokens seen**: ~2.6B **Batch size**: 8 × 4 gradient accumulation = 32 effective **Sequence length**: 2,048 **Learning rate**: 6e-4, cosine schedule, 3% warmup **Optimizer**: AdamW (β1=0.9, β2=0.95, weight_decay=0.1) **Precision**: bf16 mixed precision **Hardware**: 1× NVIDIA H100 80GB Training Metrics | Metric | Value | |---|---| | Final loss | ~5.3 (cross-entropy) | | Final entropy | 3.77 | | Token accuracy | 33.8% | Usage Limitations **Small model**: 110M params with 129K vocab means ~37% of parameters are in embeddings, limi