bitnet-embedding-270m

BitNet Embeddings Model Description BitNet-Embeddings is a family of multilingual text embedding models developed by Microsoft BitNet team. The models use decoder-only architecture with last-token pooling and L2 normalization to produce dense text embeddings. They can be applied to a wide range of tasks, including text retrieval, clustering, semantic similarity, classification, bitext mining, and reranking. They achieve competitive performance on public benchmarks while maintaining excellent inference and storage efficiency. **Developed by:** BitNet Team, Microsoft Research **Model type:** BitNet b1.58 based Text Embeddings **Language(s):** Multilingual (see Supported Languages) **License:** MIT License Model Sources **Repository:** https://github.com/microsoft/BitNet **Paper:** BitNet Text Embeddings **Paper:** Multilingual E5 Text Embeddings: A Technical Report | Model | Weights | Parameters | Embedding Dimension | Max Tokens | MTEB v2 Mean | |---|---|---|---|---|---| | bitnet-embeddings-270m | 1.58-bit | 270M | 640 | 32,768 | 66.26 | | harrier-oss-v1-270m | bf16 | 270M | 640 | 32,768 | 66.5 | | bitnet-embeddings-0.6b | 1.58-bit | 0.6B | 1,024 | 32,768 | 67.49 | | harrier-oss-v1-0.6b | bf16 | 0.6B | 1,024 | 32,768 | 69.0 | Model Details **Architecture**: Transformer-based, modified with BitLinear layers (BitNet framework). Uses Rotary Position Embeddings (RoPE). Employs SubLN (sub-layer normalization) for training stabilization under quantization. No bias terms in linear or normalization layers. **Quantization**: Native 1.58-bit weights and 8-bit activations (W1.58A8). Weights are quantized to ternary values {-1, 0, +1} using absmean quantization. Activations are quantized to 8-bit integers using absmax quantization (per-token). Trained from scratch with this quantization scheme, not post-training quantized. **Context Length**: 32,768 tokens. **Pooling Strategy**: Last-token (EOS) pooling followed by L2 normalization. **Training Pipeline**: 1. **BitNet Conversion*