SAE-Res-Qwen3.5-2B-Base-W32K-L0_50

Qwen-Scope: Decoding Intelligence, Unleashing Potential We are excited to introduce Qwen-Scope, an interpretability module trained on the Qwen3 and Qwen3.5 series models. Specifically, we integrated and trained Sparse Autoencoders (SAEs) within Qwen’s hidden layers. By implementing sparsity constraints, we can automatically extract data features that are highly decoupled, low-redundancy, and significantly more interpretable. Qwen-Scope can be used not only to analyze the internal mechanisms of Qwen’s behavior but also holds immense potential for model optimization. Application scenarios include steerable inference control, evaluation sample distribution analysis and comparison, data classification and synthesis, and model training and optimization. See our technical report for more details. Model Details | Property | Value | |---|---| | Base model | Qwen3.5-2B-Base | | SAE width (`d_sae`) | 32768 | | Hidden size (`d_model`) | 2048 | | Expansion factor | 16× | | Top-K | 50 | | Hook point | Residual stream | | Layers covered | 0 – 23 (24 layers total) | | File format | PyTorch `.pt` dict | Architecture This is a **TopK SAE** — at each forward pass, exactly **50** features are kept non-zero. Each checkpoint file `layer{n}.sae.pt` is a Python `dict` with four tensors: | Key | Shape | Description | |---|---|---| | `W_enc` | `(32768, 2048)` | Encoder weight matrix | | `W_dec` | `(2048, 32768)` | Decoder weight matrix | | `b_enc` | `(32768,)` | Encoder bias | | `b_dec` | `(2048,)` | Decoder bias | Files This repository contains one SAE checkpoint per transformer layer (layers 0–23): Feature Activation Extraction End-to-end demo: run the base LLM, hook the residual stream at a chosen layer, and extract sparse SAE feature activations. For most of the situations, using SAEs trained on base models to explore the internal process of post-training checkpoints is also reasonable. Gradio Demo We also provide a gradio demo `app.py`. You can run it locally: Caution It is strictly pr