Motif-3-Beta

Motif-3-Beta ⚠️ **Preview / beta checkpoint — not the final release.** This repository hosts an intermediate checkpoint of **Motif-3**. The **final checkpoint will be released soon.** Motif-3** is a large-scale Mixture-of-Experts (MoE) language model built from the ground up by Motif Technologies following a fully in-house, proprietary design — not a re-parameterization of existing open-source architectures. Highlights 🧠 **~314B total parameters / ~13B active** per token (sparse MoE) 📏 **256K context length** (262,144 tokens), natively long-context ⚡ Sparse routing: 384 experts with 8 activated per token, plus 1 shared expert 🌐 **Multilingual**, general-purpose Model details | | | | ------------------ | ---------------------------------------- | | Model type | Mixture-of-Experts causal language model | | Total parameters | ~314B | | Active parameters | ~13B / token | | Hidden size | 4096 | | Layers | 53 | | Routed experts | 384 (top-8) | | Shared experts | 1 | | Context length | 262,144 (256K) | | Vocabulary | 220,160 | | Tensor type | bfloat16 | Architecture Motif-3 is a fully in-house design and introduces several custom components: **Grouped Differential Latent Attention (GDLA)** **Grouped PolyNorm activation**, applied per expert **Modified mHC** Benchmarks Artificial Analysis Intelligence Index (AAII): 44** See Artificial Analysis for details. Usage - vLLM (Beta) Docker image: `ghcr.io/motiftechnologies/vllm:v0.20.2-motif3.rc2` Tested only on B200 and H200 GPUs. If you encounter any issues, please open an HF issue. Example vLLM command: Usage - HF.generate The shipped modeling code has been fixed and HF `.generate` now runs and produces coherent output. For production or high-throughput serving, vLLM (above) is recommended. The model ships with custom modeling code, so load it with `trust_remote_code=True`: Access This model is **openly available** — anyone can download the weights, no access request required. License Permission is granted to use, modify, and r