Solar-Open2-250B

Solar Open 2 Solar Open 2 is Upstage’s 250B-A15B open-weight large language model, built for agentic use cases such as office productivity, document-intensive work, and coding. Its Hybrid-Attention Mixture-of-Experts (MoE) architecture with linear attention delivers highly efficient inference even in long-context settings. Technical Report** | **Blog** | **Upstage Website** | **Try Demo (~7/31)** Highlights **Agentic Specialist:** Purpose-built for agentic workflows — tool calling, multi-step reasoning, and end-to-end task execution. Competitive with the strongest open-weight models on agent benchmarks. **Minimal Inference Cost:** A 250B-parameter MoE that activates only 15B per token, built on a hybrid attention stack that interleaves three linear-attention layers with one softmax-attention layer — large-model capacity at small-model inference cost. **1M-Token Context:** The linear-attention layers encode token order intrinsically in their recurrent state, so positional encoding is removed entirely (NoPE), lifting the RoPE extrapolation limit. Only 12 of the 48 layers keep a KV cache, holding long-context memory to roughly a quarter of an all-softmax model of the same shape. **Efficiently Trained at Low Cost:** Initialized by selective weight transfer from Solar Open 1 (102B) — only the 2.3% of weights that survive the architectural change are carried over, and everything else is randomly initialized — which raises the starting point and accelerates early convergence at 250B scale. **Multilingual:** English, Korean, and Japanese. Model Overview | Field | Value | | ------------------------------- | ----------------------------------------------------------------------- | | Model Name | Solar Open 2 (250B-A15B) | | Architecture | Hybrid-Attention Mixture-of-Experts (MoE) | | Total Parameters | 250B (250,287,794,944) | | Active Parameters | 15B (per token) | | Layers | 48 | | Hidden Size | 4096 | | Attention | Hybrid — Softmax + Linear Attention, pattern `[Softmax, Li