Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
AI Summary
This article evaluates the performance of Large Language Models (LLMs) against human annotators for machine translation (MT) quality evaluation schemes like Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). Using a long-context test set of 70 language pairs and WMT data, the study investigates score and error span agreement across various domains. Results indicate that while LLM agreement with humans surpasses human-to-human agreement in certain tasks, both humans and models remain unreliable in most settings. Humans struggle with fine-grained MQM and low-resource languages, whereas LLMs face difficulties with minor errors, wrong language variants, and error span annotation, suggesting a need for human-LLM collaborative pipelines.
Key Takeaways
- Evaluates LLMs on Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA).
- Tests span 70 language pairs, long-context test sets, and WMT23/WMT25 data.
- Shows that both humans and LLMs face distinct challenges, pointing toward human-LLM collaborative annotation.
Why it matters
Evaluating machine translation accurately is critical for deploying reliable multilingual AI systems. Understanding the limitations of both LLMs and human annotators helps engineers design better collaborative annotation pipelines and quality assurance frameworks.
Conversation 0
Be the first to share your perspective.