The Design Space of Tri-Modal Masked Diffusion Models
AI Summary
Apple researchers introduce the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text, and audio-text data. The study systematically analyzes multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, introducing a novel stochastic differential equation (SDE) based reparameterization to decouple physical and logical batch sizes. A preliminary 3B model trained on 6.4T tokens demonstrates strong capabilities in text generation, text-to-image (T2I), and text-to-speech (T2S) tasks.
Key Takeaways
- First tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text, and audio-text.
- Introduces an SDE-based reparameterization that eliminates the need to tune optimal batch sizes by decoupling physical and logical batch sizes.
- Presents results from a 3B model scale trained on 6.4T tokens across text, T2I, and T2S tasks.
Why it matters
This work advances discrete diffusion models as strong alternatives to autoregressive language models for unified multimodal generation, providing practical insights into scaling behaviors, batch-size decoupling, and inference optimizations.
Conversation 0
Be the first to share your perspective.