VTP-Base-f16d64

Towards Scalable Pre-training of Visual Tokenizers for Generation Jingfeng Yao 1 , Yuda Song 2 , Yucong Zhou 2 , Xinggang Wang 1,* 1 Huazhong University of Science and Technology 2 MiniMax Corresponding author: xgwang@hust.edu.cn Work still in Progress.*** News **[2025.12.16]** We have released our technical report and pretrained weights. Takeaways By integrating contrastive, self-supervised, and reconstruction learning, we have trained numerous visual tokenizers from scratch. We are seeking to unveil the novel scalability interlinking understanding, generation, and reconstruction. **Same FLOPs in DiT Training, VTP scaling helps better generation.** **Traditional auto-encoders CANNOT be scaled up for diffusion generative models.** **Understanding is the key driver for improving the learnability scaling.** **Parameter, data and training scalability can be seen while representation learning involved.** Get Checkpoints | Checkpoints | |-------| | | | | | | Weights will be released very soon. 🚀 Click Here to Quick Start Performance Model Understanding Reconstruction Generation Zero-shot Acc. Linear Probing rFID LightningDiT-XL 80ep nocfg FID-50K OpenCLIP 74.0 - - - CLIP 75.5 - - - SigLIP 80.5 - - - MAE - 85.9 - - DINOv2 - 86.7 - - UniTok 70.8 - 0.41 - VILA-U 73.3 - 1.80 - VA-VAE-f16d32 - - 0.28 4.29 VA-VAE-f16d64 - - 0.15 - RAE-f16d768 - 84.5 0.57 4.28 VTP-S-f16d64 (ours) 66.7 77.5 0.98 5.46 VTP-B-f16d64 (ours) 73.2 81.0 0.74 3.88 VTP-L-f16d64 (ours) 78.2 85.7 0.36 2.81 Introduction The quality of the latent space in visual tokenizers (e.g., VAEs) is crucial for modern generative models. However, the standard reconstruction-based training paradigm produces a latent space that is biased towards low-level information, leading to a foundation flaw: better pixel-level accuracy does not lead to higher-quality generation. This implies that pouring extensive compute into visual tokenizer pre-training translates poorly to improved performance in generation. We identify this as