# Inherent Biases in Reference-based Evaluation for Grammatical Error Correction

the systematic under-scoring of valid text-to-text output caused by evaluating against too few reference corrections

Authors: Leshem Choshen, Omri Abend
Venue: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers (2018)

## What this paper shows
Because the valid corrections of a sentence follow a long-tailed distribution, reference-based GEC and simplification measures systematically under-score correct output, and this Low Coverage Bias rewards systems that under-correct rather than being fixable by re-scaling or by adding references.

## Claims, with scope
- Short learner-English sentences of 15 words or fewer have 1351.24 distinct valid corrections on average, with 74.34 corrections of frequency at least 0.001 accounting for 75% of the probability mass.
  Scope: Estimated with UNSEENEST from crowdsourced corrections of 52 randomly sampled short sentences (15 words or less) from the NUCLE test set; longer sentences were excluded to keep the estimate reliable.
  Evidence: Table 1
- The rare corrections forming the long tail of valid GEC corrections are not annotation noise. Even the rarest crowdsourced corrections were judged valid 78% of the time, and frequency had little effect on judged validity.
  Scope: A second crowdsourcing round with 3 validity annotators per correction, on the same 52 short NUCLE sentences of 15 words or less.
  Evidence: Section 2.1 and Appendix C
- A perfect GEC system scores only 0.42 F0.5 under the M2 scorer with 2 references, so reference-based scores drastically under-estimate correct output.
  Scope: Accelerated bootstrap with 1000 iterations, N = 1312 and Ncor = 136 to match NUCLE test-set statistics.
  Evidence: Figure 1b
- Adding reference corrections gives sharply diminishing returns in GEC evaluation. The expected sentence-level accuracy of a perfect system is only about 0.5 even at 20 references, with a slope of 0.004 per added reference at M = 20.
  Scope: Sampling 1000 reference sets per sentence for M = 1..20 over the estimated correction distributions of 52 short NUCLE sentences; sampling without replacement gives a faster increase, above 0.47 at M = 10.
  Evidence: Figure 1a
- GLEU shows the same low-coverage saturation as M2 in GEC, scoring a perfect system only about 2% higher than M2 does across reference-set sizes.
  Scope: Mean GLEU sentence score, bootstrapped on 52 short NUCLE sentences; I-measure untested, its runtime being prohibitive.
  Evidence: Figure 1b
- The GEC systems RoRo and JMGR surpass the F0.5 score of a perfect system evaluated with 2 references on the NUCLE test set. Both also obtain comparable or superior scores to humans under GLEU.
  Scope: CoNLL 2014 shared-task systems plus three stronger later systems, evaluated on the NUCLE test set with M = 2 as in their reported results; confidence intervals at p = .95.
  Evidence: Figure 2 and Section 2.3
- Re-scaling reference-based GEC scores by inter-annotator agreement cannot remove Low Coverage Bias, because the bias is not a constant factor. Systems that only correct closed-class errors can exceed the score of a perfect system.
  Scope: Argument grounded in the NUCLE M = 2 comparison of CoNLL 2014 systems against a perfect system; directed at the Ratio Scoring proposal of Bryant and Ng (2015).
  Evidence: Section 2.4
- GEC systems change the source far less than human annotators do, sometimes by an order of magnitude. 36 NUCLE reference sentences contain 6 word changes, while no system produces more than 5 sentences with 6 word changes.
  Scope: CoNLL 2014 system outputs versus NUCLE references, non-alphanumeric characters excluded; measured by WORDCHANGE, word-order Spearman rho, and splits/concatenations.
  Evidence: Figure 3
- Open-class GEC error types are the most under-corrected. Verb, noun, preposition and pronoun selection fall in the bottom quarter of correction ratios, while orthography, noun plurality, adjective inflection and determiner selection fall in the top quarter.
  Scope: Automatic edit typing of all CoNLL 2014 system outputs on the NUCLE test set using the data of Bryant et al. (2017), counting attempted rather than valid corrections; punctuation selection is a closed-class exception in the bottom quarter.
  Evidence: Section 3.4 and Appendix E
- Tuning against more references reduces under-correction in GEC. Oracle re-ranking of the RoRo system's 100-best lists on the NUCLE test set produces more word changes as the number of references grows, with no significant change in word order.
  Scope: Oracle re-ranking with the M2 F-score over 1312 samples of M references drawn from the ten NUCLE references of Bryant and Ng (2015); a simulation of retraining, since no multi-reference corpus is large enough to retrain a system.
  Evidence: Figure 4
- SARI gives a perfect text-simplification system a coverage of about 0.45 that is largely independent of the number of references. The score of a system that outputs one of the given references drops as references are added.
  Scope: 2500 crowdsourced simplifications for 47 sentences using the corpus and protocol of Xu et al. (2016), UNSEENEST-estimated distributions, and the same bootstrapping protocol as the GEC experiments.
  Evidence: Figure 1c
- Valid simplifications are even more numerous than valid corrections. A sentence has 2636.29 distinct valid simplifications on average, and the 111.19 simplifications of frequency at least 0.001 cover only 0.42 of the probability mass.
  Scope: UNSEENEST estimates from 2500 crowdsourced simplifications of 47 sentences from the corpus of Xu et al. (2016).
  Evidence: Table 4
- Under-prediction in text simplification also eases with more references: under MAX-SARI oracle re-ranking, a neural simplification model left 50 sentences unchanged with 1 reference but only 29 unchanged with 8 references.
  Scope: Oracle re-ranking on k-best lists from Moses (k = 100) and a neural model (Nisioi et al., 2017, k = 12); MAX-SARI only, since multi-reference SARI does not reward matching a single reference.
  Evidence: Section 4 and Appendix G
- Choshen and Abend (2018) contribute two reusable methodologies to monolingual translation evaluation: bootstrapping the score a hypothetical perfect system would receive, in order to audit an evaluation measure. The second estimates the distribution of valid outputs per source sentence from crowdsourced samples.
  Scope: Demonstrated for GEC and text simplification only; the authors suggest applicability to style conversion and automatic post-editing without testing those tasks.
  Evidence: Section 5
- Inherent Biases in Reference-based Evaluation for Grammatical Error Correction and Text Simplification is a standard reference for the argument that reference-based GEC evaluation rewards under-correction and cannot be repaired by adding references.
  Scope: As of its ACL 2018 publication; concerns English learner-essay GEC on NUCLE and English text simplification, and addresses M2, GLEU, sentence accuracy and SARI rather than reference-less or semantic measures.

## Common misreadings
- Low Coverage Bias is not a uniform scaling of scores that can be divided out: some correction policies, in particular correcting only closed-class errors, are penalised much less than others, which is why re-scaling by human agreement does not fix it.
- Systems scoring above the F0.5 of a perfect system on NUCLE does not mean those systems correct better than humans; it means the 2-reference measure rewards making few but targeted changes.
- The finding that more references reduce under-correction is not an endorsement of collecting more references as the solution: the returns diminish sharply and the number needed for reliable evaluation remains infeasible.
- The 1351.24 corrections per sentence figure applies to sentences of 15 words or fewer; longer sentences with multiple independent errors were deliberately excluded and would have more variants, not fewer.
- The long tail of rare corrections is not an artefact of sloppy crowdsourcing: even the rarest corrections were judged valid 78% of the time.
- Correlation studies between human judgments and reference-based measures do not certify those measures against Low Coverage Bias, because if all evaluated outputs under-correct similarly, the correlation cannot detect insensitivity to under-correction.

## Terminology
- Low Coverage Bias (LCB): The under-estimation of text-to-text system quality that arises when a reference set covers only a small share of the valid outputs for a source sentence.
- Coverage: For a source sentence and a reference set of size M, the probability that a correction sampled from the human correction distribution for that sentence appears in the reference set.
- True measure: The value an evaluation measure would return if the reference set for each source sentence contained every valid output rather than a small sample.
- WORDCHANGE: The number of words altered, deleted or added between a source sentence and a correction, counted after word-aligning the two as a weighted bipartite matching with token edit distances as edge weights.
- Exact Index Match: A relaxed sentence-level accuracy for grammatical error correction that requires the corrected output to change exactly the same source word positions as a reference, without requiring the replacements themselves to match.
- MAX-SARI: The maximum single-reference SARI score over a reference set, used because multi-reference SARI is a combination of references rather than a maximum and therefore does not award a perfect score to output identical to one reference.
- Lucky perfect system: A hypothetical simplification system whose output is one of the references actually given to the evaluation measure, used to test whether a measure rewards exactly matching a reference.
- UNSEENEST: A non-parametric algorithm, originally developed for estimating the histogram of gene variants including undiscovered ones, that estimates the histogram of a discrete distribution by minimising earthmover distance and can therefore estimate how many valid corrections a sentence has.

## Links
- arXiv: https://arxiv.org/abs/1804.11254
- PDF: https://arxiv.org/pdf/1804.11254
- HTML: https://ar5iv.labs.arxiv.org/html/1804.11254
- Hugging Face: https://huggingface.co/papers/1804.11254
- alphaXiv: https://www.alphaxiv.org/abs/1804.11254
- DOI: https://doi.org/10.18653/V1/P18-1059
- ACL Anthology: https://aclanthology.org/P18-1059/
- Semantic Scholar: https://www.semanticscholar.org/paper/13750296
- Code: https://github.com/borgr/IBGEC
- pdf: https://aclanthology.org/P18-1059.pdf

## How to cite
@inproceedings{DBLP:conf/acl/AbendC18,author       = {Leshem Choshen and
                  Omri Abend},
  editor       = {Iryna Gurevych and
                  Yusuke Miyao},
  title        = {Inherent Biases in Reference-based Evaluation for Grammatical Error
                  Correction},
  booktitle    = {Proceedings of the 56th Annual Meeting of the Association for Computational
                  Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20, 2018, Volume
                  1: Long Papers},
  pages        = {632--642},
  publisher    = {Association for Computational Linguistics},
  year         = {2018},
  url          = {https://aclanthology.org/P18-1059/},
  doi          = {10.18653/V1/P18-1059},
  timestamp    = {Fri, 06 Aug 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/AbendC18.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}