BLEU scores a candidate translation by how much its word sequences overlap with one reference translation; COMET and other neutral metrics replaced that surface comparison with a learned model of human quality judgements, which correlates far better with what people think. Both remain metrics of sentences in isolation. Neither sees whether your defined term was enforced, whether clause 8.3's cross-reference still resolves, whether a table survived, whether a figure was transcribed correctly, or whether the last two pages were translated at all. For document work, automatic metrics are a regression alarm, not an acceptance test.
This guide covers what each metric measures, where neural metrics genuinely improved matters, the failure classes no metric captures, and what to substitute when you are buying document translation rather than benchmarking a research system.
BLEU: N-Gram Overlap Against a Reference
BLEU compares a machine output against one or more human reference translations by counting matching n-grams — contiguous runs of one to four words — and combining those precisions with a penalty for output that is shorter than the reference. It produces a number between 0 and 100. Higher means more overlap.
That is the whole mechanism, and its virtues are real: it is cheap, deterministic, language-agnostic in implementation, and it was good enough to drive twenty years of research progress. If system B beats system A by ten BLEU points on the same test set, something genuinely improved.
The trouble starts when the number is asked to mean "quality" rather than "overlap". BLEU has no notion of meaning. A translation that renders a negation backwards but otherwise matches the reference word for word scores near-perfectly. A translation that is fully correct but phrased differently from the reference scores badly. Both outcomes are routine in professional prose, where any given sentence has many defensible renderings.
The Single-Reference Problem
Most evaluations use one reference translation, because references are expensive. This is the assumption that quietly breaks BLEU on real documents.
Take a governing-law clause. A competent translator can produce four or five renderings that a court would treat identically, differing in clause order, in whether a relative clause becomes a participle, in which of two accepted equivalents is used for a term of art. Exactly one of them is your reference. The other four are penalised as errors.
The effect is not random noise; it is biased. Systems whose output style resembles the reference translator's style score higher than systems that are equally correct but phrased differently. On legal and financial text, where sentences are long and structurally flexible, the penalty for legitimate variation swamps the signal from actual errors. You end up measuring stylistic similarity to one person and calling it accuracy.
The Improvement Neural Metrics Delivered
COMET, introduced in a 2020 framework paper for neural MT evaluation, takes a different approach: rather than matching strings, it encodes the source, the hypothesis and the reference with a pretrained multilingual language model and predicts the quality score a human would assign, having been trained on collected human judgements.
The improvement is substantial and well documented. Because the comparison happens in a semantic representation rather than on surface tokens, a correct paraphrase is no longer punished for being a paraphrase. Correlation with human ranking improved sharply, and the annual WMT shared tasks — which evaluate the metrics themselves against human judgements each year — have made neural metrics the default for system-level comparison.
There is also a practically useful variant: reference-free quality estimation, which scores source against output with no human reference at all. That turns evaluation into something you can run over an entire live workload rather than over a small test set, which changes what it is good for.
Where Neural Metrics Still Fall Short
Better correlation is not the same as fitness for purpose, and four gaps matter for buyers.
Segment-level reliability lags system-level reliability. Neural metrics rank systems over a corpus well; on a single sentence, the score is a noisy estimate. Do not reject a translation because one clause scored low.
Domain and training data drive behaviour. These models learn from human judgements collected on general-domain material. They are not calibrated to know that in your document "security" means collateral rather than safety, and they will not treat that substitution as the serious error it is.
Reference quality still constrains reference-based scores. A weak reference caps the score of a strong system.
And a subtler problem: metrics that correlate with average human preference reward fluency. Machine output that reads smoothly while dropping a qualifier is exactly the profile that a fluency-weighted metric under-penalises and a lawyer would call a defect. Practitioners describe this pattern constantly — a rant about DeepL on r/TranslationStudies captures the fluent-but-wrong failure mode better than any score does.
The Failure Classes No Metric Sees
This is the part that matters most and gets discussed least. Automatic metrics operate on aligned text segments. Your document is not a bag of aligned text segments.
Nothing in BLEU or COMET observes that merged cells in a rate table collapsed into one column, that clause numbering restarted after a section break, or that a cross-reference now points at the wrong heading — the field codes that make "see clause 12.4" self-updating are structure, not text.
Nothing checks numbers. Decimal commas, currency symbols, day-month versus month-day ordering, contract values: a metric scoring a sentence containing €1.250.000 against a reference containing €1,250,000 treats the difference as trivial. Your CFO will not, and our notes on translating financial reports without losing table formatting cover where figures go wrong.
Nothing verifies terminology enforcement, because a term substitution changes one token in a long sentence and barely moves the score. Nothing detects a skipped page, a dropped footnote or a text box left in the source language. Completeness is invisible to a metric computed over the segments that exist.
Reading WMT Results as a Buyer
The WMT campaigns are the best public source on which metrics track human judgement, and they are worth reading with a specific frame. They tell you which metric to prefer if you are comparing systems on general-domain corpora with human references available.
What they do not tell you is how a system will behave on your credit agreements, in your file formats, with your term base loaded. The test conditions are deliberately standardised; your conditions are deliberately not. A system's rank on a news or general-domain test set is weak evidence about its handling of a scanned annex with a three-level numbered list.
Use the findings the way a procurement lead should use any published benchmark: as a prior, not as a result. They narrow the field of plausible vendors. The decision comes from your own documents. Our guide to where automated and human translation each belong in regulated work sets out where that human judgement has to sit.
The Check Battery to Run Instead
Replace the single score with a small battery of checks, most of which are mechanical and none of which require a linguist.
Structural integrity. Does clause numbering match the source, do all cross-references resolve, do tables have the same row and column counts, do headers, footers and footnotes survive, is the page count plausible?
Numeric and date reconciliation. Extract every number, date, percentage and currency amount from both versions and diff them. This is a script, it runs in seconds, and it catches a category of error that costs real money.
Terminology compliance. Measure the enforcement rate of your approved term base directly: how many occurrences of each source term were rendered with the approved target term, and list every exception. This is the check that most closely predicts whether lawyers will trust the output.
Completeness. Segment counts, untranslated-string detection, text extracted from images and text boxes.
A human error typology on a small sample. Use MQM categories on a few thousand words rather than a metric on all of it.
Where Automatic Metrics Genuinely Earn Their Place
None of this means throwing the metrics away. They do two jobs well.
The first is regression monitoring. Once a pipeline is in production, a reference-free quality estimate computed over every job gives you a time series. When the average drops after a model update or a new document type appears, you find out from the chart rather than from a partner's email.
The second is routing. Quality estimation scores are a reasonable way to decide which segments get human eyes. Low-confidence clusters go to a reviewer; high-confidence routine text does not. That converts a noisy segment-level signal into something useful precisely because you are not treating it as a verdict.
Both uses share a property: the metric decides where to look, and a human decides what is true. Keep that division and the numbers stay honest. For pipelines wired into a document workflow, our notes on the document translation API for legal workflows cover where these checks attach.
Sources and Further Reading
WMT25 — the shared tasks that evaluate translation systems and metrics against human judgement
Microsoft — list of field codes in Word, the structures that break silently
A rant about DeepL, r/TranslationStudies — working translators on output that reads well and says the wrong thing
Related Reading
How to translate financial reports without losing table formatting
Automated versus human document translation in regulated industries
Last reviewed 24 August 2026 by the Bluente document engineering team, who build and test the pipeline described here. We update these guides when the underlying standards, regulations or file formats change.
A metric tells you the sentence looks right. Only a check tells you the document is. Try BluTranslate free.