Artificial intelligence news.
Straight from the original announcements.

Apple study: LLMs and humans both struggle with translation quality annotation

October 12, 2026By the AI News Bites Editorial TeamSource: Apple
Apple study: LLMs and humans both struggle with translation quality annotation
Image: Apple

Apple researchers report that large language models can sometimes agree with human annotators more than humans agree with each other when judging machine translation quality, but that both remain unreliable in most settings. The paper, from Hala Almaghout, Christian Federmann and Qin Gao, appears at EMNLP.

Key points
  • Apple researchers compared LLM and human annotators on MQM and ESA translation quality schemes.
  • Tests used a long-context set of 70 language pairs plus public WMT23 and WMT25 data.
  • LLM-human agreement exceeds human-human agreement for some tasks, but both vary widely and remain unreliable for most settings.
  • Humans struggle with fine-grained MQM and low-resource languages; LLMs struggle with minor errors, wrong language variants and error spans.
  • The authors point to human-LLM collaborative annotation pipelines as a possible improvement.

What the researchers tested

LLMs are widely seen as a more efficient and cost-effective alternative to human judgment for evaluating machine translation (MT), according to the paper. MT evaluation covers many language pairs, domains and levels of annotation granularity, so the authors argue that LLMs need to be thoroughly checked across all of these dimensions before they can reliably replace human evaluation.

The team examined two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). For each, they compared how closely LLM annotations agreed with those of human annotators. They looked at both score agreement and error span annotation agreement.

The data included a long-context test set covering 70 language pairs, as well as the publicly available WMT23 and WMT25 data.

Mixed results for models and humans

According to the paper, LLM agreement with human annotators exceeds the agreement between human annotators for some evaluation tasks. However, the results vary substantially across annotation schemes, language pairs and domains, and the authors say they remain unreliable for most settings.

The study also identifies where each group of annotators runs into trouble:

  • Humans are particularly challenged by fine-grained MQM annotations and by low-resource language pairs.
  • LLMs struggle with minor errors, wrong language variants and error span annotation.

The finding that humans disagree with each other is notable because human judgments are often treated as the reference point against which automated methods are measured.

What it means for MT evaluation

The authors say their results point to room for improvement in both human and LLM annotation. They suggest that human-LLM collaborative annotation pipelines could address the reliability issues the paper identifies, though they frame this as a possibility rather than a tested solution.

For teams that rely on LLMs to score translations, the study is a caution against assuming consistent quality across all language pairs, domains and annotation types. The paper does not show LLMs as a drop-in replacement for human evaluation, and it also shows that human annotation is not uniformly dependable, especially for fine-grained error labeling and low-resource languages.

Related news

Apple Paper Defines Theoretical Limits for Language Model Alignment
ResearchNovember 12, 2026

Apple Paper Defines Theoretical Limits for Language Model Alignment

Apple researchers have published a NeurIPS paper defining the maximum reward improvement language models can achieve under a fixed KL-divergence budget. The work also…

Apple Study Examines Why Gated State Space Models Struggle to Learn in Context
ResearchNovember 10, 2026

Apple Study Examines Why Gated State Space Models Struggle to Learn in Context

A paper published by Apple examines how gating affects State Space Models, finding that it can steer them toward memorization in model weights and delay or prevent…

Apple Study Questions Whether LLM Judges Reflect Human Writing Preferences
ResearchNovember 10, 2026

Apple Study Questions Whether LLM Judges Reflect Human Writing Preferences

Apple researchers have examined how reliably large language models can judge whether generated writing matches a user’s preferred style. The study finds that judge…