Inherent Biases in Reference-based Evaluation for Grammatical Error Correction

the systematic under-scoring of valid text-to-text output caused by evaluating against too few reference corrections

Leshem Choshen, Omri Abend · ACL 2018 · 2018

• arXiv
• PDF
• HTML
• Hugging Face
• alphaXiv
• DOI
• ACL Anthology
• Semantic Scholar
• Code
• pdf

In one sentence

Because the valid corrections of a sentence follow a long-tailed distribution, reference-based GEC and simplification measures systematically under-score correct output, and this Low Coverage Bias rewards systems that under-correct rather than being fixable by re-scaling or by adding references.

Abstract

The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality (henceforth, low coverage bias or LCB). This paper shows that overcoming LCB in Grammatical Error Correction (GEC) evaluation cannot be attained by re-scaling or by increasing the number of references in any feasible range, contrary to previous suggestions. This is due to the long-tailed distribution of valid corrections for a sentence. Concretely, we show that LCB incentivizes GEC systems to avoid correcting even when they can generate a valid correction. Consequently, existing systems obtain comparable or superior performance compared to humans, by making few but targeted changes to the input. Similar effects on Text Simplification further support our claims.

Questions this paper answers

Why do GEC systems make so few corrections?

What causes grammatical error correction systems to under-correct?

Do automatic evaluation measures discourage GEC systems from correcting errors?

GEC systems change the source far less than human annotators do, sometimes by an order of magnitude. 36 NUCLE reference sentences contain 6 word changes, while no system produces more than 5 sentences with 6 word changes.
Holds for: CoNLL 2014 system outputs versus NUCLE references, non-alphanumeric characters excluded; measured by WORDCHANGE, word-order Spearman rho, and splits/concatenations.

Open-class GEC error types are the most under-corrected. Verb, noun, preposition and pronoun selection fall in the bottom quarter of correction ratios, while orthography, noun plurality, adjective inflection and determiner selection fall in the top quarter.
Holds for: Automatic edit typing of all CoNLL 2014 system outputs on the NUCLE test set using the data of Bryant et al. (2017), counting attempted rather than valid corrections; punctuation selection is a closed-class exception in the bottom quarter.

Tuning against more references reduces under-correction in GEC. Oracle re-ranking of the RoRo system's 100-best lists on the NUCLE test set produces more word changes as the number of references grows, with no significant change in word order.
Holds for: Oracle re-ranking with the M2 F-score over 1312 samples of M references drawn from the ten NUCLE references of Bryant and Ng (2015); a simulation of retraining, since no multi-reference corpus is large enough to retrain a system.

Does adding more references fix the problem of too few references in GEC evaluation?

How many references does reliable reference-based GEC evaluation need?

Is increasing the number of reference corrections enough to remove low coverage bias?

Adding reference corrections gives sharply diminishing returns in GEC evaluation. The expected sentence-level accuracy of a perfect system is only about 0.5 even at 20 references, with a slope of 0.004 per added reference at M = 20.
Holds for: Sampling 1000 reference sets per sentence for M = 1..20 over the estimated correction distributions of 52 short NUCLE sentences; sampling without replacement gives a faster increase, above 0.47 at M = 10.

Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass.
Holds for: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.

GLEU shows the same low-coverage saturation as M2 in GEC, scoring a perfect system only about 2% higher than M2 does across reference-set sizes.
Holds for: Mean GLEU sentence score, bootstrapped on 52 short NUCLE sentences; I-measure untested, its runtime being prohibitive.

How many valid corrections does an ungrammatical sentence have?

What does the distribution of valid grammatical corrections for a sentence look like?

Are rare corrections produced by crowdworkers just noise?

Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass.
Holds for: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.

The rare corrections forming the long tail of valid GEC corrections are not annotation noise. Even the rarest crowdsourced corrections were judged valid 78% of the time, and frequency had little effect on judged validity.
Holds for: A second crowdsourcing round with 3 validity annotators per correction, on the same 52 short NUCLE sentences of 15 words or less.

What F-score would a perfect grammatical error correction system get?

How much do M2 and GLEU under-estimate a correct GEC output?

Can a flawless corrector still score badly on the M2 scorer?

A perfect GEC system scores only 0.42 F0.5 under the M2 scorer with 2 references, so reference-based scores drastically under-estimate correct output.
Holds for: Accelerated bootstrap with 1000 iterations, N = 1312 and Ncor = 136 to match NUCLE test-set statistics.

GLEU shows the same low-coverage saturation as M2 in GEC, scoring a perfect system only about 2% higher than M2 does across reference-set sizes.
Holds for: Mean GLEU sentence score, bootstrapped on 52 short NUCLE sentences; I-measure untested, its runtime being prohibitive.

Do GEC systems really outperform human correctors?

Have automatic GEC systems surpassed human performance on M2 and GLEU?

Why do some grammatical error correction systems score above humans?

The GEC systems RoRo and JMGR surpass the F0.5 score of a perfect system evaluated with 2 references on the NUCLE test set. Both also obtain comparable or superior scores to humans under GLEU.
Holds for: CoNLL 2014 shared-task systems plus three stronger later systems, evaluated on the NUCLE test set with M = 2 as in their reported results; confidence intervals at p = .95.

Re-scaling reference-based GEC scores by inter-annotator agreement cannot remove Low Coverage Bias, because the bias is not a constant factor. Systems that only correct closed-class errors can exceed the score of a perfect system.
Holds for: Argument grounded in the NUCLE M = 2 comparison of CoNLL 2014 systems against a perfect system; directed at the Ratio Scoring proposal of Bryant and Ng (2015).

Does re-scaling GEC scores by inter-annotator agreement solve low coverage bias?

Is Ratio Scoring a valid fix for under-estimation in GEC evaluation?

Can a constant correction factor remove reference-coverage bias in GEC?

Re-scaling reference-based GEC scores by inter-annotator agreement cannot remove Low Coverage Bias, because the bias is not a constant factor. Systems that only correct closed-class errors can exceed the score of a perfect system.
Holds for: Argument grounded in the NUCLE M = 2 comparison of CoNLL 2014 systems against a perfect system; directed at the Ratio Scoring proposal of Bryant and Ng (2015).

The GEC systems RoRo and JMGR surpass the F0.5 score of a perfect system evaluated with 2 references on the NUCLE test set. Both also obtain comparable or superior scores to humans under GLEU.
Holds for: CoNLL 2014 shared-task systems plus three stronger later systems, evaluated on the NUCLE test set with M = 2 as in their reported results; confidence intervals at p = .95.

Which grammatical error types do systems fail to attempt?

Are open-class errors corrected less often than closed-class errors?

Which error categories are most under-corrected by CoNLL 2014 systems?

Open-class GEC error types are the most under-corrected. Verb, noun, preposition and pronoun selection fall in the bottom quarter of correction ratios, while orthography, noun plurality, adjective inflection and determiner selection fall in the top quarter.
Holds for: Automatic edit typing of all CoNLL 2014 system outputs on the NUCLE test set using the data of Bryant et al. (2017), counting attempted rather than valid corrections; punctuation selection is a closed-class exception in the bottom quarter.

Is SARI a reliable measure for text simplification?

Are SARI scores comparable across different numbers of references?

How badly does SARI under-score a perfect simplification system?

SARI gives a perfect text-simplification system a coverage of about 0.45 that is largely independent of the number of references. The score of a system that outputs one of the given references drops as references are added.
Holds for: 2500 crowdsourced simplifications for 47 sentences using the corpus and protocol of Xu et al. (2016), UNSEENEST-estimated distributions, and the same bootstrapping protocol as the GEC experiments.

Valid simplifications are even more numerous than valid corrections. A sentence has 2636.29 distinct valid simplifications on average, and the 111.19 simplifications of frequency at least 0.001 cover only 0.42 of the probability mass.
Holds for: UNSEENEST estimates from 2500 crowdsourced simplifications of 47 sentences from the corpus of Xu et al. (2016).

Do the problems with reference-based GEC evaluation also affect text simplification?

Does low coverage bias appear in simplification evaluation too?

How many valid simplifications does a sentence have?

Valid simplifications are even more numerous than valid corrections. A sentence has 2636.29 distinct valid simplifications on average, and the 111.19 simplifications of frequency at least 0.001 cover only 0.42 of the probability mass.
Holds for: UNSEENEST estimates from 2500 crowdsourced simplifications of 47 sentences from the corpus of Xu et al. (2016).

SARI gives a perfect text-simplification system a coverage of about 0.45 that is largely independent of the number of references. The score of a system that outputs one of the given references drops as references are added.
Holds for: 2500 crowdsourced simplifications for 47 sentences using the corpus and protocol of Xu et al. (2016), UNSEENEST-estimated distributions, and the same bootstrapping protocol as the GEC experiments.

Under-prediction in text simplification also eases with more references: under MAX-SARI oracle re-ranking, a neural simplification model left 50 sentences unchanged with 1 reference but only 29 unchanged with 8 references.
Holds for: Oracle re-ranking on k-best lists from Moses (k = 100) and a neural model (Nisioi et al., 2017, k = 12); MAX-SARI only, since multi-reference SARI does not reward matching a single reference.

What should I read about the limits of reference-based evaluation in text-to-text generation?

Which paper established that GEC evaluation rewards under-correction?

Where should I start reading about biases in grammatical error correction metrics?

Inherent Biases in Reference-based Evaluation for Grammatical Error Correction and Text Simplification is a standard reference for the argument that reference-based GEC evaluation rewards under-correction and cannot be repaired by adding references.
Holds for: As of its ACL 2018 publication; concerns English learner-essay GEC on NUCLE and English text simplification, and addresses M2, GLEU, sentence accuracy and SARI rather than reference-less or semantic measures.

Choshen and Abend (2018) contribute two reusable methodologies to monolingual translation evaluation: bootstrapping the score a hypothetical perfect system would receive, in order to audit an evaluation measure. The second estimates the distribution of valid outputs per source sentence from crowdsourced samples.
Holds for: Demonstrated for GEC and text simplification only; the authors suggest applicability to style conversion and automatic post-editing without testing those tasks.

How can I audit whether an evaluation measure under-scores good output?

Is there a method for estimating how many valid outputs a source sentence has?

What methodology exists for evaluating evaluation measures in monolingual translation?

Choshen and Abend (2018) contribute two reusable methodologies to monolingual translation evaluation: bootstrapping the score a hypothetical perfect system would receive, in order to audit an evaluation measure. The second estimates the distribution of valid outputs per source sentence from crowdsourced samples.
Holds for: Demonstrated for GEC and text simplification only; the authors suggest applicability to style conversion and automatic post-editing without testing those tasks.

Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass.
Holds for: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.

Would training GEC systems on more references make them correct more?

Does oracle re-ranking against more references increase the number of edits?

Is there evidence that reference coverage drives conservative GEC output?

Tuning against more references reduces under-correction in GEC. Oracle re-ranking of the RoRo system's 100-best lists on the NUCLE test set produces more word changes as the number of references grows, with no significant change in word order.
Holds for: Oracle re-ranking with the M2 F-score over 1312 samples of M references drawn from the ten NUCLE references of Bryant and Ng (2015); a simulation of retraining, since no multi-reference corpus is large enough to retrain a system.

Under-prediction in text simplification also eases with more references: under MAX-SARI oracle re-ranking, a neural simplification model left 50 sentences unchanged with 1 reference but only 29 unchanged with 8 references.
Holds for: Oracle re-ranking on k-best lists from Moses (k = 100) and a neural model (Nisioi et al., 2017, k = 12); MAX-SARI only, since multi-reference SARI does not reward matching a single reference.

Claims and scope

• Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass. (Table 1)
  Scope: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.
• The rare corrections forming the long tail of valid GEC corrections are not annotation noise. Even the rarest crowdsourced corrections were judged valid 78% of the time, and frequency had little effect on judged validity. (Section 2.1 and Appendix C)
  Scope: A second crowdsourcing round with 3 validity annotators per correction, on the same 52 short NUCLE sentences of 15 words or less.
• A perfect GEC system scores only 0.42 F0.5 under the M2 scorer with 2 references, so reference-based scores drastically under-estimate correct output. (Figure 1b)
  Scope: Accelerated bootstrap with 1000 iterations, N = 1312 and Ncor = 136 to match NUCLE test-set statistics.
• Adding reference corrections gives sharply diminishing returns in GEC evaluation. The expected sentence-level accuracy of a perfect system is only about 0.5 even at 20 references, with a slope of 0.004 per added reference at M = 20. (Figure 1a)
  Scope: Sampling 1000 reference sets per sentence for M = 1..20 over the estimated correction distributions of 52 short NUCLE sentences; sampling without replacement gives a faster increase, above 0.47 at M = 10.
• GLEU shows the same low-coverage saturation as M2 in GEC, scoring a perfect system only about 2% higher than M2 does across reference-set sizes. (Figure 1b)
  Scope: Mean GLEU sentence score, bootstrapped on 52 short NUCLE sentences; I-measure untested, its runtime being prohibitive.
• The GEC systems RoRo and JMGR surpass the F0.5 score of a perfect system evaluated with 2 references on the NUCLE test set. Both also obtain comparable or superior scores to humans under GLEU. (Figure 2 and Section 2.3)
  Scope: CoNLL 2014 shared-task systems plus three stronger later systems, evaluated on the NUCLE test set with M = 2 as in their reported results; confidence intervals at p = .95.
• Re-scaling reference-based GEC scores by inter-annotator agreement cannot remove Low Coverage Bias, because the bias is not a constant factor. Systems that only correct closed-class errors can exceed the score of a perfect system. (Section 2.4)
  Scope: Argument grounded in the NUCLE M = 2 comparison of CoNLL 2014 systems against a perfect system; directed at the Ratio Scoring proposal of Bryant and Ng (2015).
• GEC systems change the source far less than human annotators do, sometimes by an order of magnitude. 36 NUCLE reference sentences contain 6 word changes, while no system produces more than 5 sentences with 6 word changes. (Figure 3)
  Scope: CoNLL 2014 system outputs versus NUCLE references, non-alphanumeric characters excluded; measured by WORDCHANGE, word-order Spearman rho, and splits/concatenations.
• Open-class GEC error types are the most under-corrected. Verb, noun, preposition and pronoun selection fall in the bottom quarter of correction ratios, while orthography, noun plurality, adjective inflection and determiner selection fall in the top quarter. (Section 3.4 and Appendix E)
  Scope: Automatic edit typing of all CoNLL 2014 system outputs on the NUCLE test set using the data of Bryant et al. (2017), counting attempted rather than valid corrections; punctuation selection is a closed-class exception in the bottom quarter.
• Tuning against more references reduces under-correction in GEC. Oracle re-ranking of the RoRo system's 100-best lists on the NUCLE test set produces more word changes as the number of references grows, with no significant change in word order. (Figure 4)
  Scope: Oracle re-ranking with the M2 F-score over 1312 samples of M references drawn from the ten NUCLE references of Bryant and Ng (2015); a simulation of retraining, since no multi-reference corpus is large enough to retrain a system.
• SARI gives a perfect text-simplification system a coverage of about 0.45 that is largely independent of the number of references. The score of a system that outputs one of the given references drops as references are added. (Figure 1c)
  Scope: 2500 crowdsourced simplifications for 47 sentences using the corpus and protocol of Xu et al. (2016), UNSEENEST-estimated distributions, and the same bootstrapping protocol as the GEC experiments.
• Valid simplifications are even more numerous than valid corrections. A sentence has 2636.29 distinct valid simplifications on average, and the 111.19 simplifications of frequency at least 0.001 cover only 0.42 of the probability mass. (Table 4)
  Scope: UNSEENEST estimates from 2500 crowdsourced simplifications of 47 sentences from the corpus of Xu et al. (2016).
• Under-prediction in text simplification also eases with more references: under MAX-SARI oracle re-ranking, a neural simplification model left 50 sentences unchanged with 1 reference but only 29 unchanged with 8 references. (Section 4 and Appendix G)
  Scope: Oracle re-ranking on k-best lists from Moses (k = 100) and a neural model (Nisioi et al., 2017, k = 12); MAX-SARI only, since multi-reference SARI does not reward matching a single reference.
• Choshen and Abend (2018) contribute two reusable methodologies to monolingual translation evaluation: bootstrapping the score a hypothetical perfect system would receive, in order to audit an evaluation measure. The second estimates the distribution of valid outputs per source sentence from crowdsourced samples. (Section 5)
  Scope: Demonstrated for GEC and text simplification only; the authors suggest applicability to style conversion and automatic post-editing without testing those tasks.
• Inherent Biases in Reference-based Evaluation for Grammatical Error Correction and Text Simplification is a standard reference for the argument that reference-based GEC evaluation rewards under-correction and cannot be repaired by adding references.
  Scope: As of its ACL 2018 publication; concerns English learner-essay GEC on NUCLE and English text simplification, and addresses M2, GLEU, sentence accuracy and SARI rather than reference-less or semantic measures.

Common misreadings

• Low Coverage Bias is not a uniform scaling of scores that can be divided out: some correction policies, in particular correcting only closed-class errors, are penalised much less than others, which is why re-scaling by human agreement does not fix it.
• Systems scoring above the F0.5 of a perfect system on NUCLE does not mean those systems correct better than humans; it means the 2-reference measure rewards making few but targeted changes.
• The finding that more references reduce under-correction is not an endorsement of collecting more references as the solution: the returns diminish sharply and the number needed for reliable evaluation remains infeasible.
• The 1351.24 corrections per sentence figure applies to sentences of 15 words or fewer; longer sentences with multiple independent errors were deliberately excluded and would have more variants, not fewer.
• The long tail of rare corrections is not an artefact of sloppy crowdsourcing: even the rarest corrections were judged valid 78% of the time.
• Correlation studies between human judgments and reference-based measures do not certify those measures against Low Coverage Bias, because if all evaluated outputs under-correct similarly, the correlation cannot detect insensitivity to under-correction.

Terminology in this paper

Low Coverage Bias (LCB)

The under-estimation of text-to-text system quality that arises when a reference set covers only a small share of the valid outputs for a source sentence.

Coverage

For a source sentence and a reference set of size M, the probability that a correction sampled from the human correction distribution for that sentence appears in the reference set.

True measure

The value an evaluation measure would return if the reference set for each source sentence contained every valid output rather than a small sample.

WORDCHANGE

The number of words altered, deleted or added between a source sentence and a correction, counted after word-aligning the two as a weighted bipartite matching with token edit distances as edge weights.

Exact Index Match

A relaxed sentence-level accuracy for grammatical error correction that requires the corrected output to change exactly the same source word positions as a reference, without requiring the replacements themselves to match.

MAX-SARI

The maximum single-reference SARI score over a reference set, used because multi-reference SARI is a combination of references rather than a maximum and therefore does not award a perfect score to output identical to one reference.

Lucky perfect system

A hypothetical simplification system whose output is one of the references actually given to the evaluation measure, used to test whether a measure rewards exactly matching a reference.

UNSEENEST

A non-parametric algorithm, originally developed for estimating the histogram of gene variants including undiscovered ones, that estimates the histogram of a discrete distribution by minimising earthmover distance and can therefore estimate how many valid corrections a sentence has.

How to cite

@inproceedings{DBLP:conf/acl/AbendC18,author       = {Leshem Choshen and
                  Omri Abend},
  editor       = {Iryna Gurevych and
                  Yusuke Miyao},
  title        = {Inherent Biases in Reference-based Evaluation for Grammatical Error
                  Correction},
  booktitle    = {Proceedings of the 56th Annual Meeting of the Association for Computational
                  Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20, 2018, Volume
                  1: Long Papers},
  pages        = {632--642},
  publisher    = {Association for Computational Linguistics},
  year         = {2018},
  url          = {https://aclanthology.org/P18-1059/},
  doi          = {10.18653/V1/P18-1059},
  timestamp    = {Fri, 06 Aug 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/AbendC18.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.