ESMC-6B

ESMC Model Details ESMC is a state-of-the-art protein language model that has learned the rules of protein biology from training on billions of protein sequences. ESMC provides representations of proteins enabling novel AI applications from therapeutic protein engineering to unlocking basic insights into protein biology across life. The ESMC 6B model has 6 billion parameters, with 80 layers and 2.37e23 training flops. We additionally release overtrained 300M and 600M parameter variants of ESMC for local inference and finetuning. The ESMFold2 structure prediction models are trained on top of a frozen ESMC 6B language model. ESMFold2 is a state-of-the-art model for protein structure prediction and design that defines a new frontier for speed and accuracy. The ESMC sparse autoencoder, `ESMC-6B-sae-layer60-k64-codebook16384`, is built on the ESMC 6B model and provides human-interpretable, agent-generated feature descriptions. See the ESMC SAE overview card for the full set of ESMC SAE variants. To run this model with the Biohub Platform API, visit the Biohub Platform. Read more about ESMC in our paper here. Example Usage Install `esm` from GitHub (a PyPI release is coming soon): By default, the model returns only the final layer representations. To return hidden states from **all transformer layers**, set: For detailed usage, refer to the Usage section below. Citation Model Architecture ESMC is based on the transformer architecture. It features Pre-LN, rotary embeddings, and SwiGLU activations. No biases are used in linear layers or layer norms. Parameters ESMC was trained at multiple scales: | Model | Parameters | Layers | Training FLOPs | | :---- | ----: | ----: | ----: | | **ESMC-300M** | 300M | 30 | 1.26e22 | | **ESMC-600M** | 600M | 36 | 2.17e22 | | **ESMC-6B** | 6B | 80 | 2.37e23 | ![][pal] Model Variants | Model Variant | Description | URL | | :---- | :---- | :---- | | ESMC 300M | Smallest variant, publicly released. | https://huggingface.co/biohub/ESMC-300M | |