Classifying Syntactic Errors in Learner Language

Leshem Choshen; Dmitry Nikolaev; Yevgeni Berzak; Omri Abend

doi:10.18653/V1/2020.CONLL-1.7

Classifying Syntactic Errors in Learner Language

syntactic error classification from Universal Dependencies parses of a learner sentence and its correction

Leshem Choshen, Dmitry Nikolaev, Yevgeni Berzak, Omri Abend · CoNLL 2020 · 2020

In one sentence

SErCl classifies a grammatical error by the pair of Universal Dependencies labels — POS tags, dependency edge labels or morphological features — that change between the learner's span and its correction, giving an error taxonomy that needs no hand-written categories and transfers across languages.

Abstract

We present a method for classifying syntactic errors in learner language, namely errors whose correction alters the morphosyntactic structure of a sentence. The methodology builds on the established Universal Dependencies syntactic representation scheme, and provides complementary information to other error-classification systems. Unlike existing error classification methods, our method is applicable across languages, which we showcase by producing a detailed picture of syntactic errors in learner English and learner Russian. We further demonstrate the utility of the methodology for analyzing the outputs of leading Grammatical Error Correction (GEC) systems.

Questions this paper answers

is there a way to label grammar mistakes in learner writing that works for more than one language?
can a learner-error taxonomy be derived from Universal Dependencies annotation instead of hand-designed error categories?
how do I get error types for learner text in a language with no hand-built error taxonomy?
should I build my own error category list for my learner corpus, or reuse a syntax-based one?: SErCl derives a learner-error taxonomy from an existing syntactic representation framework, Universal Dependencies, rather than from hand-designed error categories, which makes the same taxonomy usable across languages without per-language rules.
Holds for: As of publication in 2020; demonstrated on learner English (TLE, NUCLE, Lang8, W&I) and learner Russian (RULEC) only, and the earlier automatic classifier ERRANT requires new rules per language.; About 60% of the English learner errors that ERRANT dumps into its residual OTHER category receive a POS-based syntactic-error class from SErCl. Of 842 OTHER edits in W&I, 504 (59.9%) change the POS tag between source and correction.
Holds for: W&I learner English, using ERRANT's own edit spans and UDPipe parses; ERRANT sends about 25% of its predictions to OTHER in NUCLE and Lang8 and about 15% in W&I and TLE.
how many of the learner mistakes that automatic error taggers leave unclassified can actually be given a type?
what share of ERRANT OTHER edits receive a syntactic error class from POS-change typing?
how do I get informative labels for the edits ERRANT dumps into OTHER?
if I already tag my learner corpus with ERRANT, is a syntax-based taxonomy worth adding on top?: About 60% of the English learner errors that ERRANT dumps into its residual OTHER category receive a POS-based syntactic-error class from SErCl. Of 842 OTHER edits in W&I, 504 (59.9%) change the POS tag between source and correction.
Holds for: W&I learner English, using ERRANT's own edit spans and UDPipe parses; ERRANT sends about 25% of its predictions to OTHER in NUCLE and Lang8 and about 15% in W&I and TLE.
can you trust an automatic parser on sentences written by language learners, which are full of mistakes?
does substituting UDPipe parses for gold treebank dependency annotation shift the distribution of extracted syntactic error types?
how do I extract syntactic error types from a learner corpus that has no manual dependency annotation?
do I need gold-parsed learner data, or is running a parser over my own corpus good enough?: Replacing manual dependency annotation with a UDPipe parser barely changes the distribution of syntactic error types in the TLE learner corpus. Normalising by tokens per POS, class frequencies shift 0.4% on average, with Pearson r=0.998.
Holds for: TLE learner English, the only corpus with manual UD; UDPipe parses; non-lexical tags X, INTJ and SYM are excluded because the parser inflates them.
do error types built from grammar annotation line up with the categories human annotators use?
how much do POS-change syntactic error types overlap with the NUCLE error taxonomy's manual categories?
how do I tell whether a syntax-derived error type corresponds to a familiar hand-annotated error category?
can I interpret syntax-derived error types using the NUCLE categories my team already knows?: SErCl types sit largely inside single categories of NUCLE's hand-built taxonomy. On average 62% of a given syntactic-error type's instances fall in its maximally overlapping NUCLE category, and 82% in one of the top three.
Holds for: NUCLE training set, relocation errors excluded because edits do not distinguish relocation from deletion; SErCl types with fewer than 30 occurrences omitted from the comparison matrix.
are automatic grammar correction tools worse at mistakes that change a sentence's structure?
what recall upper bound does the BEA2019 winning system UEDIN-MS reach on gold syntactic errors compared with its overall recall?
how do I find out which part of my error correction system's recall gap comes from structural errors?
if my users mostly make structure-changing mistakes, can I expect a top shared-task system to fix them?: Syntactic errors are harder for grammatical error correction systems than other errors. The BEA2019 winner UEDIN-MS changes only 2686 of 4790 gold syntactic errors, a 56% recall upper bound, against its reported overall recall of 60%.
Holds for: W&I development set, BEA2019 system outputs; the bound counts any change of the right type, correct or not; the ~40% SE recall estimate assumes SE precision equals the reported overall 72%.; Recall upper bounds of UEDIN-MS vary widely across syntactic error types, from 38% on ADJ→ADV and 44% on ADJ→NOUN up to 61-63% on determiners, adjectives and pronouns as source POS.
Holds for: W&I development set, UEDIN-MS output; upper bounds computed as predicted changes divided by gold changes per type, and the rare NUM→DET and PART→DET types are set aside.
which kinds of learner mistakes do automatic correction tools handle worst?
how do recall upper bounds for UEDIN-MS and Grammarly vary across individual syntactic error types?
how do I work out which error types to target when improving a grammar correction system?
which learner error types should I not rely on existing correction systems for?: Recall upper bounds of UEDIN-MS vary widely across syntactic error types, from 38% on ADJ→ADV and 44% on ADJ→NOUN up to 61-63% on determiners, adjectives and pronouns as source POS.
Holds for: W&I development set, UEDIN-MS output; upper bounds computed as predicted changes divided by gold changes per type, and the rare NUM→DET and PART→DET types are set aside.; Grammarly detects and validly corrects almost none of most syntactic error types, with 0% recall on 7 of the 15 examined types including missing adverbs, missing verbs and VERB→ADJ replacements. Its best cases are superfluous determiners (41%) and PART→DET (67%, from 6 instances).
Holds for: TLE-derived selected SE types, manually annotated by whether Grammarly detected the edit at all and whether at least one offered correction was valid; product version as evaluated at the time of the 2020 study.
do grammar correction systems that make more changes score better in competitions?
is the number of syntactic-error edits a BEA2019 system makes correlated with its shared-task ranking?
should I make my grammar correction system less conservative to move up the leaderboard?
if I tune my system to edit more aggressively, will its shared-task rank improve?: The number of syntactic changes a BEA2019 grammatical error correction system makes is unrelated to its shared-task rank. The partial-order Kendall tau between rank and number of SE changes, overall and per source POS, is 0.
Holds for: 5 BEA2019 systems (UEDIN-MS, KAKAO&BRAIN, SHUYAO, CAMB-CUED, AIP-TOHOKU) on the W&I development set; ranks 1, 2, 5, 8 and 9 only, so a small sample.
does a commercial writing assistant catch mistakes where the wrong kind of word was used?
what is Grammarly's recall on syntactic error types where correction changes the token's POS tag?
how do I know which learner error types a commercial proofreader will leave uncorrected?
can I hand my students Grammarly and expect it to fix missing verbs and adverbs?: Grammarly detects and validly corrects almost none of most syntactic error types, with 0% recall on 7 of the 15 examined types including missing adverbs, missing verbs and VERB→ADJ replacements. Its best cases are superfluous determiners (41%) and PART→DET (67%, from 6 instances).
Holds for: TLE-derived selected SE types, manually annotated by whether Grammarly detected the edit at all and whether at least one offered correction was valid; product version as evaluated at the time of the 2020 study.
do stronger language learners make fewer mistakes where a word of the wrong type is used?
does the share of POS-preserving corrections increase monotonically with CEFR proficiency level in W&I, and how do native writers compare?
how do I check whether my learners' error profiles shift as their proficiency rises?
can I use POS-changing error rates as a proficiency signal for my learners?: In the W&I corpus, the share of words whose POS tag survives correction rises with learner proficiency from level A to level C for every POS tag. Native writers generally change fewer tags than advanced learners, but the trend is mixed.
Holds for: W&I training set, proficiency levels A-C plus LOCNESS native text, automatic UD parses; native counts are smaller, so borderline native-vs-C differences should not be read as significant.
which noun endings do people learning Russian get wrong most often?
which noun case substitutions dominate the case confusion matrix in learner Russian corrections?
how do I find out which Russian case contrasts to drill with my students?
should my Russian learner-error tooling prioritise genitive case confusions?: Learners of Russian most often replace the genitive with accusative or nominative on nouns: 132 accusative-for-genitive and 163 nominative-for-genitive corrections, the two largest cells of the noun case confusion matrix.
Holds for: RULEC learner Russian, UDPipe morphological features on automatic parses; nouns only, adjective case agreement is more symmetric (27 accusative-for-genitive vs 19 converse).
what kinds of verb mistakes do learners of Russian make, and do the mix-ups go both ways?
are aspect substitutions in learner Russian symmetric, and how does that compare with voice substitutions?
how do I tell whether a Russian verb error type needs direction-specific handling?
for Russian learner verb errors, do I need to model both directions of the confusion separately?: Aspect errors in learner Russian are near-symmetric while voice errors are not. Perfective was corrected to imperfective 210 times against 223 the other way, but active-for-middle voice occurs 108 times against 45 for the converse.
Holds for: RULEC learner Russian, UD morphological features from UDPipe parses; verbal features only.
when labelling learner mistakes, do grammatical relations tell you anything that word categories do not?
how strongly associated are POS-based and dependency-edge-label-based syntactic error types in learner English?
should I type learner errors by dependency relation labels or by POS tags?
is it worth the extra work to build a dependency-edge error taxonomy for my learner corpus?: POS-based and dependency-edge-based syntactic error types carry largely the same information in learner English: Cramer's V between them is 0.78 for additions and deletions and 0.76 for replacements in TLE.
Holds for: TLE with manual UD annotation, 4584 extracted syntactic errors (2042 additions, 1048 deletions, 1495 replacements); English only, so the redundancy may not hold for morphologically richer languages.
which learner mistakes are missed by an approach that defines errors as changes to a word's grammatical label?
which error classes fall outside a taxonomy keyed on morphosyntactic label change, and are agreement errors covered?
how do I know whether agreement and determiner-choice errors will show up in a label-change error taxonomy?
if my main concern is agreement errors, will a POS-change-based taxonomy cover them?: SErCl covers only errors whose correction changes a morphosyntactic label, leaving agreement errors and inappropriate determiners that keep the representative token's label outside the taxonomy unless UD morphological features are used.
Holds for: By construction: an edit is a syntactic error only when the source and target representative tokens' labels differ, so non-SEs fall on the diagonal of the confusion matrix; in TLE 44.4% of errors are POS-based SEs.

Claims and scope

About 60% of the English learner errors that ERRANT dumps into its residual OTHER category receive a POS-based syntactic-error class from SErCl. Of 842 OTHER edits in W&I, 504 (59.9%) change the POS tag between source and correction. (Section 3.3)
Scope: W&I learner English, using ERRANT's own edit spans and UDPipe parses; ERRANT sends about 25% of its predictions to OTHER in NUCLE and Lang8 and about 15% in W&I and TLE.
Replacing manual dependency annotation with a UDPipe parser barely changes the distribution of syntactic error types in the TLE learner corpus. Normalising by tokens per POS, class frequencies shift 0.4% on average, with Pearson r=0.998. (Section 3.1)
Scope: TLE learner English, the only corpus with manual UD; UDPipe parses; non-lexical tags X, INTJ and SYM are excluded because the parser inflates them.
SErCl types sit largely inside single categories of NUCLE's hand-built taxonomy. On average 62% of a given syntactic-error type's instances fall in its maximally overlapping NUCLE category, and 82% in one of the top three. (Table 24 and Section 3.2)
Scope: NUCLE training set, relocation errors excluded because edits do not distinguish relocation from deletion; SErCl types with fewer than 30 occurrences omitted from the comparison matrix.
Syntactic errors are harder for grammatical error correction systems than other errors. The BEA2019 winner UEDIN-MS changes only 2686 of 4790 gold syntactic errors, a 56% recall upper bound, against its reported overall recall of 60%. (Table 6 and Section 5.1)
Scope: W&I development set, BEA2019 system outputs; the bound counts any change of the right type, correct or not; the ~40% SE recall estimate assumes SE precision equals the reported overall 72%.
Recall upper bounds of UEDIN-MS vary widely across syntactic error types, from 38% on ADJ→ADV and 44% on ADJ→NOUN up to 61-63% on determiners, adjectives and pronouns as source POS. (Table 5 and Table 6)
Scope: W&I development set, UEDIN-MS output; upper bounds computed as predicted changes divided by gold changes per type, and the rare NUM→DET and PART→DET types are set aside.
The number of syntactic changes a BEA2019 grammatical error correction system makes is unrelated to its shared-task rank. The partial-order Kendall tau between rank and number of SE changes, overall and per source POS, is 0. (Table 14 and Section 5.1)
Scope: 5 BEA2019 systems (UEDIN-MS, KAKAO&BRAIN, SHUYAO, CAMB-CUED, AIP-TOHOKU) on the W&I development set; ranks 1, 2, 5, 8 and 9 only, so a small sample.
Grammarly detects and validly corrects almost none of most syntactic error types, with 0% recall on 7 of the 15 examined types including missing adverbs, missing verbs and VERB→ADJ replacements. Its best cases are superfluous determiners (41%) and PART→DET (67%, from 6 instances). (Table 7)
Scope: TLE-derived selected SE types, manually annotated by whether Grammarly detected the edit at all and whether at least one offered correction was valid; product version as evaluated at the time of the 2020 study.
In the W&I corpus, the share of words whose POS tag survives correction rises with learner proficiency from level A to level C for every POS tag. Native writers generally change fewer tags than advanced learners, but the trend is mixed. (Table 3)
Scope: W&I training set, proficiency levels A-C plus LOCNESS native text, automatic UD parses; native counts are smaller, so borderline native-vs-C differences should not be read as significant.
Learners of Russian most often replace the genitive with accusative or nominative on nouns: 132 accusative-for-genitive and 163 nominative-for-genitive corrections, the two largest cells of the noun case confusion matrix. (Table 4)
Scope: RULEC learner Russian, UDPipe morphological features on automatic parses; nouns only, adjective case agreement is more symmetric (27 accusative-for-genitive vs 19 converse).
Aspect errors in learner Russian are near-symmetric while voice errors are not. Perfective was corrected to imperfective 210 times against 223 the other way, but active-for-middle voice occurs 108 times against 45 for the converse. (Section 4.2.2)
Scope: RULEC learner Russian, UD morphological features from UDPipe parses; verbal features only.
POS-based and dependency-edge-based syntactic error types carry largely the same information in learner English: Cramer's V between them is 0.78 for additions and deletions and 0.76 for replacements in TLE. (Section 4.1)
Scope: TLE with manual UD annotation, 4584 extracted syntactic errors (2042 additions, 1048 deletions, 1495 replacements); English only, so the redundancy may not hold for morphologically richer languages.
SErCl derives a learner-error taxonomy from an existing syntactic representation framework, Universal Dependencies, rather than from hand-designed error categories, which makes the same taxonomy usable across languages without per-language rules. (Section 1 and Section 6)
Scope: As of publication in 2020; demonstrated on learner English (TLE, NUCLE, Lang8, W&I) and learner Russian (RULEC) only, and the earlier automatic classifier ERRANT requires new rules per language.
SErCl covers only errors whose correction changes a morphosyntactic label, leaving agreement errors and inappropriate determiners that keep the representative token's label outside the taxonomy unless UD morphological features are used. (Section 2 and Section 4.1)
Scope: By construction: an edit is a syntactic error only when the source and target representative tokens' labels differ, so non-SEs fall on the diagonal of the confusion matrix; in TLE 44.4% of errors are POS-based SEs.

Common misreadings

The 56% figure for UEDIN-MS is an upper bound on recall, not measured recall: it counts every change the system made of the right type regardless of whether the correction was valid.
SErCl does not aim to reproduce human error annotation; the comparison with NUCLE checks that the automatic types carry the manual taxonomy's information, not that the two agree label for label.
Finding that 60% of ERRANT's OTHER edits get a SErCl class is not a claim that SErCl is better than ERRANT overall — ERRANT remains the more informative source for spelling and word-order errors, which the parser handles poorly.
The claim of cross-linguistic applicability rests on demonstrations in learner English and learner Russian; no result is reported for any other language.
Advanced learners scoring above natives on the least error-prone POS tags is a mixed trend on small native data, not evidence that natives are worse writers.

Terminology in this paper

syntactic error (SE): A grammatical error whose correction changes a morphological feature, a POS tag or a dependency label of the edited span's representative token; errors that leave those labels intact are non-syntactic errors.
representative token: In an edit span analysed against a dependency parse, the node of the span's sub-forest closest to the tree root, with the leftmost token used to break ties; its labels before and after correction define the error type.
recall upper bound: For a grammatical error correction system and an error type, the number of edits of that type the system makes divided by the number in the gold standard, counting changes whether or not they are correct.

How to cite

@inproceedings{DBLP:conf/conll/ChoshenNBA20,author       = {Leshem Choshen and
                  Dmitry Nikolaev and
                  Yevgeni Berzak and
                  Omri Abend},
  editor       = {Raquel Fern{\'{a}}ndez and
                  Tal Linzen},
  title        = {Classifying Syntactic Errors in Learner Language},
  booktitle    = {Proceedings of the 24th Conference on Computational Natural Language
                  Learning, CoNLL 2020, Online, November 19-20, 2020},
  pages        = {97--107},
  publisher    = {Association for Computational Linguistics},
  year         = {2020},
  url          = {https://doi.org/10.18653/v1/2020.conll-1.7},
  doi          = {10.18653/V1/2020.CONLL-1.7},
  timestamp    = {Sat, 30 Sep 2023 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/conll/ChoshenNBA20.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.