Adding Benchmaxxer Repellant to the Open ASR Leaderboard
AI Summary
The Open ASR Leaderboard has integrated private evaluation datasets from Appen Inc. and DataoceanAI to prevent test-set contamination and benchmaxxing. By keeping these datasets private, the platform provides a more robust measure of real-world speech recognition performance across diverse accents and conversational settings. The default Average WER calculation still relies on public datasets, but users can toggle on the private data to view rankings with targeted metrics like Avg Scripted, Avg Conversational, Avg US, and Avg non-US.
Key Takeaways
- Appen Inc. and DataoceanAI provided high-quality English ASR datasets covering scripted and conversational speech.
- Default Average WER calculation excludes private datasets to maintain backward compatibility, but they can be enabled via a toggle.
- Targeted metrics highlight gaps between controlled, American accent settings and nuanced conversational or non-American conditions.
Why it matters
Leaderboard optimization without real-world generalization is a critical challenge in machine learning. Introducing private evaluation data helps prevent developers from overfitting models to public benchmarks, ensuring more trustworthy and practical ASR model evaluations.
Conversation 0
Be the first to share your perspective.