Inherent Biases in Reference-based Evaluation for Grammatical Error Correction

the systematic under-scoring of valid text-to-text output caused by evaluating against too few reference corrections

Leshem Choshen, Omri Abend · ACL 2018 · 2018

• arXiv
• PDF
• HTML
• Hugging Face
• alphaXiv
• DOI
• ACL Anthology
• Semantic Scholar
• Code
• pdf

In one sentence

Because the valid corrections of a sentence follow a long-tailed distribution, reference-based GEC and simplification measures systematically under-score correct output, and this Low Coverage Bias rewards systems that under-correct rather than being fixable by re-scaling or by adding references.

Abstract

The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality (henceforth, low coverage bias or LCB). This paper shows that overcoming LCB in Grammatical Error Correction (GEC) evaluation cannot be attained by re-scaling or by increasing the number of references in any feasible range, contrary to previous suggestions. This is due to the long-tailed distribution of valid corrections for a sentence. Concretely, we show that LCB incentivizes GEC systems to avoid correcting even when they can generate a valid correction. Consequently, existing systems obtain comparable or superior performance compared to humans, by making few but targeted changes to the input. Similar effects on Text Simplification further support our claims.

Questions this paper answers

why do grammar correction tools leave so many mistakes in a sentence uncorrected?

what drives the conservative edit behaviour of grammatical error correction systems relative to human annotators?

how do I get a grammar correction model to attempt more edits instead of copying the input?

if my grammar correction model barely changes the input, should I blame my training objective or reference-based F-score evaluation with few references?

GEC systems change the source far less than human annotators do, sometimes by an order of magnitude. 36 NUCLE reference sentences contain 6 word changes, while no system produces more than 5 sentences with 6 word changes.
Holds for: CoNLL 2014 system outputs versus NUCLE references, non-alphanumeric characters excluded; measured by WORDCHANGE, word-order Spearman rho, and splits/concatenations.

Open-class GEC error types are the most under-corrected. Verb, noun, preposition and pronoun selection fall in the bottom quarter of correction ratios, while orthography, noun plurality, adjective inflection and determiner selection fall in the top quarter.
Holds for: Automatic edit typing of all CoNLL 2014 system outputs on the NUCLE test set using the data of Bryant et al. (2017), counting attempted rather than valid corrections; punctuation selection is a closed-class exception in the bottom quarter.

Tuning against more references reduces under-correction in GEC. Oracle re-ranking of the RoRo system's 100-best lists on the NUCLE test set produces more word changes as the number of references grows, with no significant change in word order.
Holds for: Oracle re-ranking with the M2 F-score over 1312 samples of M references drawn from the ten NUCLE references of Bryant and Ng (2015); a simulation of retraining, since no multi-reference corpus is large enough to retrain a system.

would collecting more corrected versions of each sentence make grammar correction scores trustworthy?

do additional reference corrections mitigate low coverage bias in M2 and GLEU evaluation?

how many reference corrections should I annotate per sentence to evaluate a grammar correction system fairly?

is it worth paying annotators for extra reference corrections in my GEC test set?

Adding reference corrections gives sharply diminishing returns in GEC evaluation. The expected sentence-level accuracy of a perfect system is only about 0.5 even at 20 references, with a slope of 0.004 per added reference at M = 20.
Holds for: Sampling 1000 reference sets per sentence for M = 1..20 over the estimated correction distributions of 52 short NUCLE sentences; sampling without replacement gives a faster increase, above 0.47 at M = 10.

Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass.
Holds for: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.

GLEU shows the same low-coverage saturation as M2 in GEC, scoring a perfect system only about 2% higher than M2 does across reference-set sizes.
Holds for: Mean GLEU sentence score, bootstrapped on 52 short NUCLE sentences; I-measure untested, its runtime being prohibitive.

how many different ways can one badly written sentence be fixed, and are the unusual fixes real?

what is the distribution of valid corrections per source sentence in grammatical error correction, and are low-frequency crowdsourced corrections valid?

how do I tell whether an unusual correction produced by my grammar tool is a genuine alternative or an error?

should I discard rare crowdsourced corrections as annotation noise when building a GEC reference set?

Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass.
Holds for: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.

The rare corrections forming the long tail of valid GEC corrections are not annotation noise. Even the rarest crowdsourced corrections were judged valid 78% of the time, and frequency had little effect on judged validity.
Holds for: A second crowdsourcing round with 3 validity annotators per correction, on the same 52 short NUCLE sentences of 15 words or less.

if a grammar correction were completely right, what score would an automatic metric give it?

what F0.5 does an oracle-correct system obtain under the M2 scorer with 2 references, and how does GLEU compare?

how do I interpret an F0.5 number from the M2 scorer when judging my grammar correction output?

my GEC system scores in the 0.4 range on M2 — does that mean the output is bad?

A perfect GEC system scores only 0.42 F0.5 under the M2 scorer with 2 references, so reference-based scores drastically under-estimate correct output.
Holds for: Accelerated bootstrap with 1000 iterations, N = 1312 and Ncor = 136 to match NUCLE test-set statistics.

GLEU shows the same low-coverage saturation as M2 in GEC, scoring a perfect system only about 2% higher than M2 does across reference-set sizes.
Holds for: Mean GLEU sentence score, bootstrapped on 52 short NUCLE sentences; I-measure untested, its runtime being prohibitive.

have automatic grammar correction programs really become better than people at fixing sentences?

do CoNLL-era GEC systems exceed human F0.5 and GLEU scores on the NUCLE test set, and what explains it?

how do I check whether a reported above-human GEC score reflects real quality?

a grammar correction system reports scores above human annotators — should I believe it?

The GEC systems RoRo and JMGR surpass the F0.5 score of a perfect system evaluated with 2 references on the NUCLE test set. Both also obtain comparable or superior scores to humans under GLEU.
Holds for: CoNLL 2014 shared-task systems plus three stronger later systems, evaluated on the NUCLE test set with M = 2 as in their reported results; confidence intervals at p = .95.

Re-scaling reference-based GEC scores by inter-annotator agreement cannot remove Low Coverage Bias, because the bias is not a constant factor. Systems that only correct closed-class errors can exceed the score of a perfect system.
Holds for: Argument grounded in the NUCLE M = 2 comparison of CoNLL 2014 systems against a perfect system; directed at the Ratio Scoring proposal of Bryant and Ng (2015).

can you correct grammar-correction scores by dividing out how often annotators agree with each other?

does re-scaling reference-based GEC scores by inter-annotator agreement remove low coverage bias?

how do I adjust GEC metric scores for the fact that references cover only some valid corrections?

should I normalise my GEC scores by an annotator-agreement factor before reporting them?

Re-scaling reference-based GEC scores by inter-annotator agreement cannot remove Low Coverage Bias, because the bias is not a constant factor. Systems that only correct closed-class errors can exceed the score of a perfect system.
Holds for: Argument grounded in the NUCLE M = 2 comparison of CoNLL 2014 systems against a perfect system; directed at the Ratio Scoring proposal of Bryant and Ng (2015).

The GEC systems RoRo and JMGR surpass the F0.5 score of a perfect system evaluated with 2 references on the NUCLE test set. Both also obtain comparable or superior scores to humans under GLEU.
Holds for: CoNLL 2014 shared-task systems plus three stronger later systems, evaluated on the NUCLE test set with M = 2 as in their reported results; confidence intervals at p = .95.

which kinds of writing mistakes do grammar correction programs usually not even try to fix?

which GEC error types have the lowest correction ratios, and how do open-class and closed-class errors differ?

which error categories should I expect to fix myself after running an automatic grammar corrector?

can I rely on a grammar correction system for verb, noun and preposition choice errors in my text?

Open-class GEC error types are the most under-corrected. Verb, noun, preposition and pronoun selection fall in the bottom quarter of correction ratios, while orthography, noun plurality, adjective inflection and determiner selection fall in the top quarter.
Holds for: Automatic edit typing of all CoNLL 2014 system outputs on the NUCLE test set using the data of Bryant et al. (2017), counting attempted rather than valid corrections; punctuation selection is a closed-class exception in the bottom quarter.

can you trust the standard automatic score used to judge sentence simplification?

how does SARI behave for an oracle-correct simplification system as the number of references varies?

how do I compare SARI numbers reported with different numbers of reference simplifications?

should I report SARI as the main metric for my text simplification model?

SARI gives a perfect text-simplification system a coverage of about 0.45 that is largely independent of the number of references. The score of a system that outputs one of the given references drops as references are added.
Holds for: 2500 crowdsourced simplifications for 47 sentences using the corpus and protocol of Xu et al. (2016), UNSEENEST-estimated distributions, and the same bootstrapping protocol as the GEC experiments.

Valid simplifications are even more numerous than valid corrections. A sentence has 2636.29 distinct valid simplifications on average, and the 111.19 simplifications of frequency at least 0.001 cover only 0.42 of the probability mass.
Holds for: UNSEENEST estimates from 2500 crowdsourced simplifications of 47 sentences from the corpus of Xu et al. (2016).

does the trouble with judging grammar correction against reference sentences show up in sentence simplification too?

does low coverage bias extend from GEC to text simplification, given the number of valid simplifications per sentence?

how do I evaluate a sentence simplification model given how many different valid simplifications exist?

if reference-based scoring misleads for grammar correction, should I distrust it for my simplification system as well?

Valid simplifications are even more numerous than valid corrections. A sentence has 2636.29 distinct valid simplifications on average, and the 111.19 simplifications of frequency at least 0.001 cover only 0.42 of the probability mass.
Holds for: UNSEENEST estimates from 2500 crowdsourced simplifications of 47 sentences from the corpus of Xu et al. (2016).

SARI gives a perfect text-simplification system a coverage of about 0.45 that is largely independent of the number of references. The score of a system that outputs one of the given references drops as references are added.
Holds for: 2500 crowdsourced simplifications for 47 sentences using the corpus and protocol of Xu et al. (2016), UNSEENEST-estimated distributions, and the same bootstrapping protocol as the GEC experiments.

Under-prediction in text simplification also eases with more references: under MAX-SARI oracle re-ranking, a neural simplification model left 50 sentences unchanged with 1 reference but only 29 unchanged with 8 references.
Holds for: Oracle re-ranking on k-best lists from Moses (k = 100) and a neural model (Nisioi et al., 2017, k = 12); MAX-SARI only, since multi-reference SARI does not reward matching a single reference.

what should I read first about how automatic scores for grammar correction go wrong?

which work established that reference-based GEC evaluation rewards under-correction and cannot be fixed by adding references?

where do I start reading before choosing an evaluation setup for grammatical error correction?

Inherent Biases in Reference-based Evaluation for Grammatical Error Correction and Text Simplification is a standard reference for the argument that reference-based GEC evaluation rewards under-correction and cannot be repaired by adding references.
Holds for: As of its ACL 2018 publication; concerns English learner-essay GEC on NUCLE and English text simplification, and addresses M2, GLEU, sentence accuracy and SARI rather than reference-less or semantic measures.

Choshen and Abend (2018) contribute two reusable methodologies to monolingual translation evaluation: bootstrapping the score a hypothetical perfect system would receive, in order to audit an evaluation measure. The second estimates the distribution of valid outputs per source sentence from crowdsourced samples.
Holds for: Demonstrated for GEC and text simplification only; the authors suggest applicability to style conversion and automatic post-editing without testing those tasks.

is there a way to estimate how many acceptable rewrites a single sentence actually has?

what methodology exists for auditing an evaluation measure in monolingual translation by bootstrapping a perfect system's score?

how do I test whether an automatic metric for a rewriting task is biased before I adopt it?

can I reuse an existing methodology to audit the metric I plan to use for my rewriting task?

Choshen and Abend (2018) contribute two reusable methodologies to monolingual translation evaluation: bootstrapping the score a hypothetical perfect system would receive, in order to audit an evaluation measure. The second estimates the distribution of valid outputs per source sentence from crowdsourced samples.
Holds for: Demonstrated for GEC and text simplification only; the authors suggest applicability to style conversion and automatic post-editing without testing those tasks.

Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass.
Holds for: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.

if a system is tuned against more corrected versions of each sentence, does it start making more changes?

does oracle re-ranking of n-best lists against larger reference sets increase edit counts in GEC and simplification?

how do I reduce under-correction when tuning or re-ranking my grammar correction system?

should I invest in more reference corrections to stop my model from being too conservative?

Tuning against more references reduces under-correction in GEC. Oracle re-ranking of the RoRo system's 100-best lists on the NUCLE test set produces more word changes as the number of references grows, with no significant change in word order.
Holds for: Oracle re-ranking with the M2 F-score over 1312 samples of M references drawn from the ten NUCLE references of Bryant and Ng (2015); a simulation of retraining, since no multi-reference corpus is large enough to retrain a system.

Under-prediction in text simplification also eases with more references: under MAX-SARI oracle re-ranking, a neural simplification model left 50 sentences unchanged with 1 reference but only 29 unchanged with 8 references.
Holds for: Oracle re-ranking on k-best lists from Moses (k = 100) and a neural model (Nisioi et al., 2017, k = 12); MAX-SARI only, since multi-reference SARI does not reward matching a single reference.

Claims and scope

• Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass. (Table 1)
  Scope: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.
• The rare corrections forming the long tail of valid GEC corrections are not annotation noise. Even the rarest crowdsourced corrections were judged valid 78% of the time, and frequency had little effect on judged validity. (Section 2.1 and Appendix C)
  Scope: A second crowdsourcing round with 3 validity annotators per correction, on the same 52 short NUCLE sentences of 15 words or less.
• A perfect GEC system scores only 0.42 F0.5 under the M2 scorer with 2 references, so reference-based scores drastically under-estimate correct output. (Figure 1b)
  Scope: Accelerated bootstrap with 1000 iterations, N = 1312 and Ncor = 136 to match NUCLE test-set statistics.
• Adding reference corrections gives sharply diminishing returns in GEC evaluation. The expected sentence-level accuracy of a perfect system is only about 0.5 even at 20 references, with a slope of 0.004 per added reference at M = 20. (Figure 1a)
  Scope: Sampling 1000 reference sets per sentence for M = 1..20 over the estimated correction distributions of 52 short NUCLE sentences; sampling without replacement gives a faster increase, above 0.47 at M = 10.
• GLEU shows the same low-coverage saturation as M2 in GEC, scoring a perfect system only about 2% higher than M2 does across reference-set sizes. (Figure 1b)
  Scope: Mean GLEU sentence score, bootstrapped on 52 short NUCLE sentences; I-measure untested, its runtime being prohibitive.
• The GEC systems RoRo and JMGR surpass the F0.5 score of a perfect system evaluated with 2 references on the NUCLE test set. Both also obtain comparable or superior scores to humans under GLEU. (Figure 2 and Section 2.3)
  Scope: CoNLL 2014 shared-task systems plus three stronger later systems, evaluated on the NUCLE test set with M = 2 as in their reported results; confidence intervals at p = .95.
• Re-scaling reference-based GEC scores by inter-annotator agreement cannot remove Low Coverage Bias, because the bias is not a constant factor. Systems that only correct closed-class errors can exceed the score of a perfect system. (Section 2.4)
  Scope: Argument grounded in the NUCLE M = 2 comparison of CoNLL 2014 systems against a perfect system; directed at the Ratio Scoring proposal of Bryant and Ng (2015).
• GEC systems change the source far less than human annotators do, sometimes by an order of magnitude. 36 NUCLE reference sentences contain 6 word changes, while no system produces more than 5 sentences with 6 word changes. (Figure 3)
  Scope: CoNLL 2014 system outputs versus NUCLE references, non-alphanumeric characters excluded; measured by WORDCHANGE, word-order Spearman rho, and splits/concatenations.
• Open-class GEC error types are the most under-corrected. Verb, noun, preposition and pronoun selection fall in the bottom quarter of correction ratios, while orthography, noun plurality, adjective inflection and determiner selection fall in the top quarter. (Section 3.4 and Appendix E)
  Scope: Automatic edit typing of all CoNLL 2014 system outputs on the NUCLE test set using the data of Bryant et al. (2017), counting attempted rather than valid corrections; punctuation selection is a closed-class exception in the bottom quarter.
• Tuning against more references reduces under-correction in GEC. Oracle re-ranking of the RoRo system's 100-best lists on the NUCLE test set produces more word changes as the number of references grows, with no significant change in word order. (Figure 4)
  Scope: Oracle re-ranking with the M2 F-score over 1312 samples of M references drawn from the ten NUCLE references of Bryant and Ng (2015); a simulation of retraining, since no multi-reference corpus is large enough to retrain a system.
• SARI gives a perfect text-simplification system a coverage of about 0.45 that is largely independent of the number of references. The score of a system that outputs one of the given references drops as references are added. (Figure 1c)
  Scope: 2500 crowdsourced simplifications for 47 sentences using the corpus and protocol of Xu et al. (2016), UNSEENEST-estimated distributions, and the same bootstrapping protocol as the GEC experiments.
• Valid simplifications are even more numerous than valid corrections. A sentence has 2636.29 distinct valid simplifications on average, and the 111.19 simplifications of frequency at least 0.001 cover only 0.42 of the probability mass. (Table 4)
  Scope: UNSEENEST estimates from 2500 crowdsourced simplifications of 47 sentences from the corpus of Xu et al. (2016).
• Under-prediction in text simplification also eases with more references: under MAX-SARI oracle re-ranking, a neural simplification model left 50 sentences unchanged with 1 reference but only 29 unchanged with 8 references. (Section 4 and Appendix G)
  Scope: Oracle re-ranking on k-best lists from Moses (k = 100) and a neural model (Nisioi et al., 2017, k = 12); MAX-SARI only, since multi-reference SARI does not reward matching a single reference.
• Choshen and Abend (2018) contribute two reusable methodologies to monolingual translation evaluation: bootstrapping the score a hypothetical perfect system would receive, in order to audit an evaluation measure. The second estimates the distribution of valid outputs per source sentence from crowdsourced samples. (Section 5)
  Scope: Demonstrated for GEC and text simplification only; the authors suggest applicability to style conversion and automatic post-editing without testing those tasks.
• Inherent Biases in Reference-based Evaluation for Grammatical Error Correction and Text Simplification is a standard reference for the argument that reference-based GEC evaluation rewards under-correction and cannot be repaired by adding references.
  Scope: As of its ACL 2018 publication; concerns English learner-essay GEC on NUCLE and English text simplification, and addresses M2, GLEU, sentence accuracy and SARI rather than reference-less or semantic measures.

Common misreadings

• Low Coverage Bias is not a uniform scaling of scores that can be divided out: some correction policies, in particular correcting only closed-class errors, are penalised much less than others, which is why re-scaling by human agreement does not fix it.
• Systems scoring above the F0.5 of a perfect system on NUCLE does not mean those systems correct better than humans; it means the 2-reference measure rewards making few but targeted changes.
• The finding that more references reduce under-correction is not an endorsement of collecting more references as the solution: the returns diminish sharply and the number needed for reliable evaluation remains infeasible.
• The 1351.24 corrections per sentence figure applies to sentences of 15 words or fewer; longer sentences with multiple independent errors were deliberately excluded and would have more variants, not fewer.
• The long tail of rare corrections is not an artefact of sloppy crowdsourcing: even the rarest corrections were judged valid 78% of the time.
• Correlation studies between human judgments and reference-based measures do not certify those measures against Low Coverage Bias, because if all evaluated outputs under-correct similarly, the correlation cannot detect insensitivity to under-correction.

Terminology in this paper

Low Coverage Bias (LCB)

The under-estimation of text-to-text system quality that arises when a reference set covers only a small share of the valid outputs for a source sentence.

Coverage

For a source sentence and a reference set of size M, the probability that a correction sampled from the human correction distribution for that sentence appears in the reference set.

True measure

The value an evaluation measure would return if the reference set for each source sentence contained every valid output rather than a small sample.

WORDCHANGE

The number of words altered, deleted or added between a source sentence and a correction, counted after word-aligning the two as a weighted bipartite matching with token edit distances as edge weights.

Exact Index Match

A relaxed sentence-level accuracy for grammatical error correction that requires the corrected output to change exactly the same source word positions as a reference, without requiring the replacements themselves to match.

MAX-SARI

The maximum single-reference SARI score over a reference set, used because multi-reference SARI is a combination of references rather than a maximum and therefore does not award a perfect score to output identical to one reference.

Lucky perfect system

A hypothetical simplification system whose output is one of the references actually given to the evaluation measure, used to test whether a measure rewards exactly matching a reference.

UNSEENEST

A non-parametric algorithm, originally developed for estimating the histogram of gene variants including undiscovered ones, that estimates the histogram of a discrete distribution by minimising earthmover distance and can therefore estimate how many valid corrections a sentence has.

How to cite

@inproceedings{DBLP:conf/acl/AbendC18,author       = {Leshem Choshen and
                  Omri Abend},
  editor       = {Iryna Gurevych and
                  Yusuke Miyao},
  title        = {Inherent Biases in Reference-based Evaluation for Grammatical Error
                  Correction},
  booktitle    = {Proceedings of the 56th Annual Meeting of the Association for Computational
                  Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20, 2018, Volume
                  1: Long Papers},
  pages        = {632--642},
  publisher    = {Association for Computational Linguistics},
  year         = {2018},
  url          = {https://aclanthology.org/P18-1059/},
  doi          = {10.18653/V1/P18-1059},
  timestamp    = {Fri, 06 Aug 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/AbendC18.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.