Putting sign language AI into users’ hands
AI Summary
Google DeepMind has introduced a sign-language-to-text (SL2T) model that enables real-time translation of American Sign Language (ASL) into English on mobile devices. By utilizing MediaPipe Holistic to track pose landmarks rather than raw video, the system ensures user privacy while bypassing traditional 'gloss' annotations. The model is trained on over 100,000 hours of data across 50+ sign languages, achieving a 70 BLEURT score on the FLEURS-ASL benchmark. It is currently integrated into Gboard and Live Transcribe on Pixel 11, with development guided by the AI Sign Language Advisory Committee to ensure accessibility and cultural sensitivity.
Key Takeaways
- The model processes geometric pose landmarks to protect privacy, discarding raw video frames immediately.
- Direct translation from landmarks to text avoids the limitations of intermediate gloss-based systems.
- Development involved a participatory governance model with the Deaf community to address ethical and practical deployment challenges.
Why it matters
This technology bridges a significant accessibility gap for the Deaf and hard of hearing community by providing a natural, efficient way to interact with digital interfaces, moving beyond the limitations of text-based input.
Conversation 0
Be the first to share your perspective.