Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties in the existing methodology. Experiments with MAEGE shed a new light on metric quality, showing for example that the standard M^2 metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that some types of valid edits are consistently penalized by existing metrics.
@inproceedings{DBLP:conf/acl/AbendC18a,
author = {Leshem Choshen and
Omri Abend},
editor = {Iryna Gurevych and
Yusuke Miyao},
title = {Automatic Metric Validation for Grammatical Error Correction},
booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational
Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20, 2018, Volume
1: Long Papers},
pages = {1372--1382},
publisher = {Association for Computational Linguistics},
year = {2018},
url = {https://aclanthology.org/P18-1127/},
doi = {10.18653/V1/P18-1127},
timestamp = {Fri, 06 Aug 2021 01:00:00 +0200},
biburl = {https://dblp.org/rec/conf/acl/AbendC18a.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
See the full reference list in the paper.