Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning
AI Summary
The Open TTS Leaderboard addresses the lack of scalable evaluation for open-source and multilingual text-to-speech models. Traditional human-preference arenas cannot keep pace with thousands of model releases. This leaderboard utilizes objective metrics such as Word Error Rate (WER) and Character Error Rate (CER) via Qwen3 ASR for intelligibility, inverse real-time factor (RTFx) and time-to-first-audio (TTFA) for speed, and WavLM speaker similarity (SIM) for voice cloning. It also features a listening tab for community feedback and streaming performance metrics on H200 GPUs and CPUs.
Key Takeaways
- Evaluates intelligibility using Qwen3 ASR to compute WER and CER.
- Measures speed via RTFx for batched offline inference and TTFA for streaming latency on H200 GPUs and CPUs.
- Assesses voice cloning capabilities using WavLM speaker embeddings cosine similarity (SIM).
Why it matters
Traditional human-preference arenas are slow, expensive, and heavily skewed toward commercial API models, leaving open-source models underrepresented. The Open TTS Leaderboard provides fast, objective, and reproducible evaluations across multiple languages, voice cloning, and streaming latency, enabling engineers to select optimal models efficiently.
Conversation 0
Be the first to share your perspective.