Learning to combine Grammatical Error Corrections

Yoav Kantor, Yoav Katz, Leshem Choshen, Edo Cohen-Karlik, Naftali Liberman, Assaf Toledo, Amir Menczel, Noam Slonim · BEA@ACL 2019 · 2019

In one sentence

A black-box combination method for Grammatical Error Correction that splits two systems' edits into agreed and disagreed subsets, then solves a convex program per error type to pick which subsets to keep so that F-0.5 is maximized directly.

Abstract

The field of Grammatical Error Correction (GEC) has produced various systems to deal with focused phenomena or general text editing. We propose an automatic way to combine black-box systems. Our method automatically detects the strength of a system or the combination of several systems per error type, improving precision and recall while optimizing F-score directly. We show consistent improvement over the best standalone system in all the configurations tested. This approach also outperforms average ensembling of different RNN models with random initializations. In addition, we analyze the use of BERT for GEC - reporting promising results on this end. We also present a spellchecker created for this task which outperforms standard spellcheckers tested on the task of spellchecking. This paper describes a system submission to Building Educational Applications 2019 Shared Task: Grammatical Error Correction. Combining the output of top BEA 2019 shared task systems using our approach, currently holds the highest reported score in the open phase of the BEA 2019 shared task, improving F-0.5 score by 3.7 points over the best result reported.

Questions this paper answers

how can several grammar checking programs be pooled into one better corrector without touching their code?
what work introduces system-agnostic combination of GEC outputs using only M2 edit files?
how do I combine the outputs of several grammatical error correction tools that I only have text output from?
I have three grammar correctors and no access to their models, is there published research telling me how to fuse their corrections?
The BEA 2019 system paper 'Learning to combine Grammatical Error Corrections' introduces automatic, system-agnostic combination for Grammatical Error Correction. It learns per error type which systems' edits to keep from outputs alone, not from model internals or hand-written pipelines.
Holds for: As of 2019; earlier GEC combination work existed but was ad-hoc — pipelining, per-phenomenon assignment, or rescoring hybrids tailored to the specific systems used — and required manual adjustment for each new set of systems.
Framing GEC system combination as a convex program over per-error-type edit-subset selection variables makes it possible to optimize the corpus-level F-beta score directly, using only M2 edit files from each system.
Holds for: Requires a dev set with gold M2 annotations from the same distribution as the test data, and an error-type classifier such as ERRANT; combining more than 2 systems is iterative.
does pooling the corrections of several grammar checkers beat just using the strongest one?
what F-0.5 does learned per-error-type edit selection reach over BEA 2019 shared task system outputs?
how much score can I gain on the BEA 2019 test set by combining published system outputs instead of training a better model?
if I already have the best available grammar correction system, is it worth combining it with weaker ones?
Combining four released BEA 2019 shared task system outputs as black boxes reaches F-0.5 72.84 on the test set, a 3.7 point gain over the best standalone system (UEDIN-MS, 69.47). Combining only UEDIN-MS with Kakao&Brain reaches 73.18.
Holds for: BEA 2019 shared task test set, open phase; selection variables fitted on the entire shared task dev set; the four systems were UEDIN-MS, Kakao&Brain, Shuyao and CAMB-CUED, whose outputs the teams released after the test phase.
Combining BEA 2019 shared task systems raises precision from 72.28 to 78.74 on the test set, a 6.5 point improvement over the best standalone system, while recall falls from 60.12 to 56.04.
Holds for: BEA 2019 test set, 4 combined systems (UEDIN-MS, Kakao&Brain, Shuyao, CAMB-CUED); optimization targets F-0.5, which weights precision over recall.
when you have several neural models correcting the same sentence, is picking corrections per error type better than averaging the models?
does per-error-type edit selection outperform decoder-side average ensembling of 4 RNN Nematus GEC models?
how should I combine 4 neural grammar correction checkpoints, ensemble decoding or output-level edit selection?
should I ensemble my grammar correction models at decoding time or combine their corrected sentences afterwards?
Combining 4 RNN Nematus models by error-type edit selection reaches F-0.5 0.3508 against 0.3122 for Nematus's built-in average ensembling of the same 4 models, a gain of almost 4 points.
Holds for: 4 RNN-based Nematus models differing only in random initialization, scored on one random half of the W&I dev set with selection variables fitted on the other half.
when several grammar correction tools are pooled, does accuracy go up only because fewer corrections are made?
does per-error-type combination of neural GEC, spellchecking and masked-LM systems raise precision and recall together?
how do I add a spellchecker and a weak corrector to my neural grammar correction models without losing recall?
will adding a weak grammar correction component to my pipeline drag down the corrections I already get right?
Iteratively combining 4 Nematus models, a spellchecker and a BERT-based system raises F-0.5 from 0.3429 for the best standalone Nematus to 0.4051. Precision and recall improve together, from 0.4839 to 0.5029 and from 0.1583 to 0.2278.
Holds for: W&I dev set, restricted track submission; the spellchecker alone scores F-0.5 0.1242 and the BERT system 0.0135, yet both still add to the combination.
Adding LanguageTool, Grammarly and JamSpell as black boxes to the restricted-track combination raises F-0.5 from 0.4051 to 0.4375, about 9 points above the best standalone off-the-shelf system (Grammarly, 0.3612).
Holds for: W&I dev set; Grammarly outputs were collected manually through its free web interface taking the top suggestion per correction, since it exposes no programmatic API.
do consumer grammar checking tools add anything to a research grammar correction system?
what does adding off-the-shelf checkers as black boxes contribute to a restricted-track GEC combination?
how do I get extra F-0.5 out of commercial grammar checkers alongside my own trained corrector?
is it worth paying for a commercial grammar checker to plug into my correction pipeline?
Adding LanguageTool, Grammarly and JamSpell as black boxes to the restricted-track combination raises F-0.5 from 0.4051 to 0.4375, about 9 points above the best standalone off-the-shelf system (Grammarly, 0.3612).
Holds for: W&I dev set; Grammarly outputs were collected manually through its free web interface taking the top suggestion per correction, since it exposes no programmatic API.
can one grammar checker be improved just by ignoring the mistake categories it handles badly?
how much F-0.5 does per-error-type filtering of a single GEC system's edits recover?
how do I raise a single grammar corrector's score without retraining it?
if I only have one grammar correction tool, is discarding its weak error types worth doing?
Discarding a single system's predictions on error types where it performs poorly gives small F-0.5 gains. LanguageTool rises from 0.2107 to 0.2355, Grammarly from 0.3627 to 0.3754 and Nematus from 0.373 to 0.3761.
Holds for: W&I dev set, filtering fitted on one random half and reported on the other; gains are minor for the two stronger systems and largest for the weakest one.
if two grammar checkers suggest the same fix, is that fix more likely to be right?
how does precision on common edits compare with standalone precision for neural GEC and a commercial checker on R:OTHER?
how do I decide when to keep only the corrections that two systems agree on?
should I only apply the grammar corrections my two tools both propose?
Edits proposed by both Nematus and Grammarly are far more precise than edits from either alone: for R:OTHER, precision is 0.67 on common edits versus 0.17 and 0.28 standalone. The optimizer therefore keeps only the agreed edits for that error type.
Holds for: W&I dev set, the 10 most frequent error types over the Nematus–Grammarly pair; agreement does not help every type, and for R:PUNCT and R:VERB:TENSE the intersection subset is dropped entirely.
is the improvement from pooling two grammar checkers a fluke of how the tuning data was split?
how much does the F-0.5 gain from a 2-system GEC combination vary across random dev-set fold partitions?
how do I check that a gain from combining two correction systems is not an artefact of my dev-set split?
can I trust the gain I measure from combining two grammar correctors on one dev split?
Combining Nematus with Grammarly gives an average F-0.5 improvement of 6.2 points over the better of the two systems. The standard deviation across 10 different random dev-set fold partitions is 0.28 points.
Holds for: W&I dev set split randomly in two, selection variables fitted on one half and scored on the other, repeated over 10 partitions.
which spelling correction tool fixes learner writing mistakes best?
how does a frequency-and-Levenshtein spellchecker score on R:SPELL edits against JamSpell, Norvig and Enchant?
how do I get the strongest spelling correction component for a grammatical error correction pipeline?
should I build my own word-frequency spellchecker or install an existing one for correcting learner English?
A heuristic spellchecker built from Gutenberg word counts, a LibreOffice dictionary and Levenshtein-distance-1 candidates reaches F-0.5 0.6378 on R:SPELL edits, above Norvig (0.5882), JamSpell (0.5599) and Enchant (0.3544).
Holds for: R:SPELL error category only on the W&I dev set; over all error categories the same spellchecker scores F-0.5 0.1198, below JamSpell's 0.1593.
can a pretrained language model fix grammar mistakes on its own by guessing masked words?
what F-0.5 does iterative masked-LM querying of BERT reach as a standalone GEC system?
how well can I correct grammatical errors by repeatedly masking tokens and taking BERT's predictions?
should I expect a masked language model alone to work as a grammar corrector?
Iteratively querying BERT as a masked language model to propose GEC edits scores F-0.5 0.0135 as a standalone system. Confidence thresholds between 0.6 and 0.98 gave unsatisfying results, and fine-tuning the masked LM on synthetic errors did not help.
Holds for: W&I dev set; the submitted variant was restricted to replacement edits inside predefined interchangeable word sets, ignoring insertions and deletions and excluding R:PUNCT; the system still contributed when combined with others.
can artificially corrupted sentences stand in for real annotated writing errors when training a corrector?
how does synthetic error generation over in-domain W&I sentences compare with real W&I training data for a Nematus GEC model?
how do I train a grammar correction model when I have little annotated learner data, and does the source text domain matter?
should I generate millions of synthetic errors from generic text or a smaller amount from in-domain sentences?
Nematus trained on synthetic errors generated over in-domain W&I gold sentences reaches F-0.5 0.1919, close to the 0.232 obtained from the real W&I train set. 7,000,000 synthetic Gutenberg sentences reach only 0.1294.
Holds for: W&I dev set; errors were generated by applying W&I corrections backwards while matching the observed distribution of edits per sentence; more Gutenberg data did not help (650,000 sentences scored 0.1483).
does piling on more learner-writing corpora help a grammar correction model, or does it swamp the data you care about?
what happens to Nematus GEC F-0.5 when Lang8, FCE and NUCLE are added to W&I with and without upsampling?
how do I mix extra grammatical error correction corpora into my training data without losing target-domain accuracy?
should I add Lang8, FCE and NUCLE to my W&I training set, and do I need to upsample W&I?
Adding Lang8, FCE and NUCLE to W&I training data only helps Nematus when W&I is upsampled 10 times. F-0.5 goes from 0.232 (W&I alone) to 0.225 (plus Lang8 and FCE) to 0.333 (upsampled W&I plus all three).
Holds for: Transformer Nematus with the WMT17 recommended hyperparameters, evaluated on the W&I dev set under the BEA 2019 restricted-track data.

Claims and scope

Common misreadings

Terminology in this paper

selection variable
A value between 0 and 1 per (error type, edit subset) pair in the combination method, giving the probability that an edit of that error type falling in that subset is kept in the merged output; in practice rounded to 0 or 1.
edit subset
A partition of two systems' proposed corrections into edits unique to system 1, edits unique to system 2, and edits both systems propose, so that every edit belongs to exactly one subset.
iterative combination
Extending pairwise system combination to N systems by combining two systems, then treating that merged output as a system to combine with the next, which avoids the tiny statistics of 2^N subsets but can overfit the development set.
M2 file
The standard Grammatical Error Correction annotation format listing, for each source sentence, the corrections applied to it, and sufficient to compare a system's edits against gold edits.
black-box combination
Merging grammatical error correction systems using only their corrected output text, with no access to model parameters, probabilities or training data.

How to cite

@inproceedings{DBLP:conf/bea/KantorKCCLTMS19,author       = {Yoav Kantor and
                  Yoav Katz and
                  Leshem Choshen and
                  Edo Cohen{-}Karlik and
                  Naftali Liberman and
                  Assaf Toledo and
                  Amir Menczel and
                  Noam Slonim},
  editor       = {Helen Yannakoudakis and
                  Ekaterina Kochmar and
                  Claudia Leacock and
                  Nitin Madnani and
                  Ildik{\'{o}} Pil{\'{a}}n and
                  Torsten Zesch},
  title        = {Learning to combine Grammatical Error Corrections},
  booktitle    = {Proceedings of the Fourteenth Workshop on Innovative Use of {NLP}
                  for Building Educational Applications, BEA@ACL 2019, Florence, Italy,
                  August 2, 2019},
  pages        = {139--148},
  publisher    = {Association for Computational Linguistics},
  year         = {2019},
  url          = {https://doi.org/10.18653/v1/w19-4414},
  doi          = {10.18653/V1/W19-4414},
  timestamp    = {Fri, 06 Aug 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/bea/KantorKCCLTMS19.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.