SERRANT: a syntactic classifier for English Grammatical Error Types

an automatic English grammatical-error-type classifier that falls back from ERRANT's readable labels to SErCl's "what changed into what" syntactic labels

Leshem Choshen, Matanel Oren, Dmitry Nikolaev, Omri Abend · arXiv · 2021

In one sentence

SERRANT is an automatic classifier of English grammatical error types that returns ERRANT's human-readable categories by default and substitutes SErCl's syntactic "source-tag → target-tag" types wherever ERRANT's label is uninformative, such as Other, Morph, or a POS-changing edit.

Abstract

SERRANT is a system and code for automatic classification of English grammatical errors that combines SErCl and ERRANT. SERRANT uses ERRANT's annotations when they are informative and those provided by SErCl otherwise.

Questions this paper answers

is there a tool that labels what kind of mistake each correction in a learner's English sentence fixes?
which automatic edit-type classifier assigns grammatical error categories to aligned source-target edits in English GEC data?
how do I get error-type labels for the edits between an ungrammatical sentence and its correction?
should I use SERRANT instead of ERRANT to tag error types in my GEC corpus?
SERRANT is a released system and code library that unifies the two existing automatic English grammatical-error-type classifiers, ERRANT and SErCl, into a single taxonomy and a single output format.
Holds for: English only, as of the 2021 release; the library exposes ERRANT-only, SErCl-only and combined annotators, and the SErCl component compares POS tags rather than full morphological features.
SERRANT returns ERRANT's edit type in the default case, keeping ERRANT's R, M and U prefixes for replacement, missing and unnecessary edits. ERRANT's sub-classifications are preserved so users can group similar classes or ignore them.
Holds for: Edits whose ERRANT type is judged informative; the 8 enumerated special cases of Section 2 override this default.
what should I read first about how English writing mistakes are sorted into categories?
which paper unifies the ERRANT and SErCl grammatical error type taxonomies into one scheme?
where do I start if I want to compare error categories across several grammatical error correction datasets?
which error-taxonomy paper is worth reading before I pick an annotation scheme for my correction data?
SERRANT is a released system and code library that unifies the two existing automatic English grammatical-error-type classifiers, ERRANT and SErCl, into a single taxonomy and a single output format.
Holds for: English only, as of the 2021 release; the library exposes ERRANT-only, SErCl-only and combined annotators, and the SErCl component compares POS tags rather than full morphological features.
SERRANT addresses the fact that grammatical-error taxonomies always differ across datasets of different languages and mostly differ even across datasets of the same language. Automatic edit-type classifiers are therefore the instrument of choice whenever more than one dataset is used.
Holds for: The unification SERRANT provides is between the 2 English classifiers ERRANT and SErCl; cross-language taxonomy differences are stated as motivation and are not resolved by the released English-only system.
what is wrong with existing English error categories that made a new labelling system worth building?
which ERRANT error types were found uninformative or inconsistent, and where does SERRANT substitute SErCl syntactic types?
how do I avoid uninformative Other and Morph labels when tagging edit types in learner text?
if ERRANT already labels my edits, why would I switch to SERRANT?
SERRANT's design rule is informativeness rather than fidelity to either source taxonomy. ERRANT is the default because it is more human-readable when accurate, and SErCl is used exactly where ERRANT's categories were shown to be uninformative or inconsistent.
Holds for: The judgement that specific ERRANT categories are uninformative or inconsistent comes from the earlier SErCl study (Choshen et al., 2020); SERRANT itself reports no quantitative comparison of the two taxonomies.
SERRANT replaces ERRANT's Other category, which signals failure to find an informative type, with SErCl's source-to-target syntactic type. Edits involving Intj, Num, Sym, X and Punct POS tags stay Other.
Holds for: Proper nouns are also treated as unreliable because the parser uses Propn as a fallback for misspelled words, so only the Propn → Propn type is kept.
SERRANT replaces ERRANT's Morph type with SErCl types and re-exposes the lost information through a "WC" suffix. The suffix marks edits where the POS is unchanged but the lemma differs, so consume → eat becomes Verb:WC while eat → ate does not.
Holds for: Unreliable POS tags listed for the Other case are excluded, but Adj → Propn and Propn → Adj edits such as China → Chinese are kept.
how is a correction labelled when the word's part of speech changes, like a noun becoming a verb?
how are POS-changing edits and auxiliary-versus-main-verb edits typed in SERRANT compared with ERRANT?
how do I get an explicit source-to-target part-of-speech tag for a grammatical error edit such as "these" corrected to "their"?
will an error-type classifier tell me a noun was replaced by a verb rather than calling it a verb form error?
SERRANT gives POS-changing edits an explicit source-to-target type instead of ERRANT's misleading label. A noun corrected to a verb becomes Noun → Verb rather than Verb:Form, and pronoun/determiner swaps such as these → their become Pron → Det or Det → Pron.
Holds for: English edits classified with a UD parser; correctness of the source and target tags depends on the parser's POS assignment.
SERRANT separates auxiliaries from main verbs, marking edits as Aux where SErCl would, whereas ERRANT's Verb type covers both Aux and Verb edits.
Holds for: English edits; the distinction relies on the UD/spaCy analysis of the span.
how are tense mistakes told apart from mood or modal-verb mistakes when English corrections are labelled?
how is ERRANT's Verb:Tense category decomposed into tense, aspect and modality in SERRANT?
how do I separate modal verb errors from tense errors when I tag verb edits in learner writing?
can I count modal-verb errors separately from tense errors with an off-the-shelf error type classifier?
SERRANT breaks up ERRANT's Verb:Tense category, which lumps tense, aspect and mood together. It keeps Verb:Tense only for be, have or "will" forms, adds a "Modal" suffix when both wordforms are modal verbs, and otherwise falls back to SErCl's type.
Holds for: The modal list has 9 members: can, could, may, might, shall, should, will, would and must; other tense, aspect and mood edits fall back to SErCl annotation.
how do error labels tell apart picking the wrong word from getting the word's form wrong?
what does the WC suffix encode, and how does SERRANT refine ERRANT's Morph type?
how do I distinguish lemma-changing edits from inflectional edits when annotating error types?
can I find out whether consume to eat is treated differently from eat to ate in an error-type label?
SERRANT replaces ERRANT's Morph type with SErCl types and re-exposes the lost information through a "WC" suffix. The suffix marks edits where the POS is unchanged but the lemma differs, so consume → eat becomes Verb:WC while eat → ate does not.
Holds for: Unreliable POS tags listed for the Other case are excluded, but Adj → Propn and Propn → Adj edits such as China → Chinese are kept.
how is a fix that only capitalises a word, like apple to Apple, categorised?
how does SERRANT handle ERRANT's Orth type for non-sentence-initial words corrected into proper nouns?
how do I tell apart pure spelling-case fixes from proper-noun corrections in my error-type counts?
will an error type classifier call apple to Apple an orthography error or something more specific?
For ERRANT's Orth type, SERRANT substitutes SErCl's X → Propn annotation when a non-sentence-initial word is corrected into a proper noun, as in "He founded apple" → "He founded Apple". Such capitalisation can change morphosyntax or meaning.
Holds for: Non-sentence-initial words changed into Propn only; other orthographic edits, such as a missing whitespace, keep ERRANT's Orth type.
how are corrections that span more than one word marked in error category labels?
what does the MW suffix indicate in a SERRANT error type, and when is it appended?
how do I identify multi-word edits in error-type annotated grammatical error correction data?
SERRANT adds an "MW" suffix to mark edits where either the source or the correction spans multiple words. The suffix is added only when the edit does not already carry a named type such as Verb:Tense.
Holds for: Multi-word edits after edit extraction; edits already assigned a named type are left without the suffix, and the Section 3 examples contain no multiword errors.
are there accuracy numbers comparing English error-category labellers, or only example sentences?
does the SERRANT work include a quantitative evaluation against ERRANT and SErCl, or qualitative illustration only?
before I trust SERRANT's error types, is there a measured comparison with ERRANT or just worked examples?
SERRANT's output is illustrated on made-up sentences and on level-A learner sentences from the W&I corpus, for example "my cook → cooking" annotated R:Morph:Noun and "it → that" annotated R:Pron → Det.
Holds for: Qualitative examples only, with no multiword errors and no accuracy or agreement measurements; the model outputs the M2 format, rendered visually in the examples.
SERRANT's design rule is informativeness rather than fidelity to either source taxonomy. ERRANT is the default because it is more human-readable when accurate, and SErCl is used exactly where ERRANT's categories were shown to be uninformative or inconsistent.
Holds for: The judgement that specific ERRANT categories are uninformative or inconsistent comes from the earlier SErCl study (Choshen et al., 2020); SERRANT itself reports no quantitative comparison of the two taxonomies.
does an automatic grammar-mistake labelling tool for English writing work on other languages too, or only English?
is SERRANT's SErCl-based syntactic taxonomy cross-lingual in the released implementation?
how do I tag error types across corpora in several languages if a tool covers only English?
can I run SERRANT on my non-English learner corpus?
SERRANT is a released system and code library that unifies the two existing automatic English grammatical-error-type classifiers, ERRANT and SErCl, into a single taxonomy and a single output format.
Holds for: English only, as of the 2021 release; the library exposes ERRANT-only, SErCl-only and combined annotators, and the SErCl component compares POS tags rather than full morphological features.
SERRANT addresses the fact that grammatical-error taxonomies always differ across datasets of different languages and mostly differ even across datasets of the same language. Automatic edit-type classifiers are therefore the instrument of choice whenever more than one dataset is used.
Holds for: The unification SERRANT provides is between the 2 English classifiers ERRANT and SErCl; cross-language taxonomy differences are stated as motivation and are not resolved by the released English-only system.

Claims and scope

Common misreadings

Terminology in this paper

edit
In grammatical error correction, an erroneous part of a sentence paired with its correction.
taxonomy
A set of grammatical error edit types used to categorise corrections.
SErCl type
An error type written as the morphosyntactic annotation of the learner span followed by that of the corrected span, e.g. Noun → Verb, with None on either side for insertions and deletions, and abbreviated to a single tag when the two sides are identical.
WC suffix
A marker in a SERRANT error type indicating that the part of speech did not change but the lemma did, so the error is one of word choice rather than of morphosyntax (consume → eat is Verb:WC).
MW suffix
A marker in a SERRANT error type indicating that the source or the correction spans more than one word, added only when the edit has no already-named type.
Modal suffix
A marker in a SERRANT error type indicating that both the original and the corrected wordform are modal verbs, separating modality errors from tense errors.
Other type
In ERRANT, the category assigned when no informative error type can be found for an edit.

How to cite

@article{DBLP:journals/corr/abs-2104-02310,author       = {Leshem Choshen and
                  Matanel Oren and
                  Dmitry Nikolaev and
                  Omri Abend},
  title        = {{SERRANT:} a syntactic classifier for English Grammatical Error Types},
  journal      = {CoRR},
  volume       = {abs/2104.02310},
  year         = {2021},
  url          = {https://arxiv.org/abs/2104.02310},
  eprinttype    = {arXiv},
  eprint       = {2104.02310},
  timestamp    = {Mon, 25 Oct 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2104-02310.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.