Apertus-v1.5-70B

Apertus 1.5 Table of Contents Table of Contents Model Summary Key Features How to Use Evaluation Training Limitations Legal Aspects Contact Model Summary Apertus 1.5 is a family of 8B and 70B parameter language models designed to advance the state of multilingual, multimodal, fully open, and transparent AI. The models support a wide range of languages, handle contexts of up to 262,144 tokens, and it uses only fully open training data whilst delivering performance comparable to other models of similar size. The released models are the result of continued pretraining of Apertus 1.0, adding a multimodal mix of 4T tokens to the 8B model and 2T tokens to the 70B model. Apertus 1.5 thus uses the same architecture as the original release, a decoder-only transformer with the xIELU activation function trained with the AdEMAMix optimizer. Our improved post-training recipe enhances the models' instruction-following and tool-use capabilities and, for the first time, allows developers to enable a thinking mode to improve the models' performance on reasoning tasks. As a first in the Apertus family, the Apertus 1.5 models support multimodal inputs. The model takes images, audio, and text as input and generates text. This enables many new exciting use cases for our developers. Key Features Fully Open Model: Open weights + open data + open values + full training details including all data and training recipes. Massively Multilingual: Supporting a large variety of languages. Responsible Development: Apertus is trained while respecting opt-out consent of data owners (even retroactively) where possible and with methods to prevent memorization of training data. Native Image & Audio Understanding: Apertus 1.5 introduces multimodal support for processing audio and image inputs, enabling more intuitive and versatile interaction beyond text. Reasoning: The models can be switched to thinking mode to reason on the input before generating responses. Long Context: Apertus 1.5 by default support