# Automatic Metric Validation for Grammatical Error Correction

Authors: Leshem Choshen, Omri Abend
Venue: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers (2018)

## Abstract
Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties in the existing methodology. Experiments with MAEGE shed a new light on metric quality, showing for example that the standard M^2 metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that some types of valid edits are consistently penalized by existing metrics.

## Links
- arXiv: https://arxiv.org/abs/1804.11225
- PDF: https://arxiv.org/pdf/1804.11225
- HTML: https://ar5iv.labs.arxiv.org/html/1804.11225
- Hugging Face: https://huggingface.co/papers/1804.11225
- alphaXiv: https://www.alphaxiv.org/abs/1804.11225
- DOI: https://doi.org/10.18653/V1/P18-1127
- ACL Anthology: https://aclanthology.org/P18-1127/
- Semantic Scholar: https://www.semanticscholar.org/paper/13747175

## How to cite
@inproceedings{DBLP:conf/acl/AbendC18a,
author       = {Leshem Choshen and
                  Omri Abend},
  editor       = {Iryna Gurevych and
                  Yusuke Miyao},
  title        = {Automatic Metric Validation for Grammatical Error Correction},
  booktitle    = {Proceedings of the 56th Annual Meeting of the Association for Computational
                  Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20, 2018, Volume
                  1: Long Papers},
  pages        = {1372--1382},
  publisher    = {Association for Computational Linguistics},
  year         = {2018},
  url          = {https://aclanthology.org/P18-1127/},
  doi          = {10.18653/V1/P18-1127},
  timestamp    = {Fri, 06 Aug 2021 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/AbendC18a.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}