Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

Leshem Choshen; Omri Abend

doi:10.18653/V1/K19-1028

Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

Leshem Choshen, Omri Abend · CoNLL 2019 · 2019

In one sentence

The Transformer with learned positional embeddings shows no locality bias, yet long-distance dependencies still hurt it — so Choshen and Abend extract MT challenge sets automatically from dependency parses and word alignments, producing sets thousands of sentences large for German-English and English-German.

Abstract

We show that the state-of-the-art Transformer MT model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model. Since most dependencies are short-distance, common evaluation metrics will be little influenced by how well systems perform on them. We therefore propose an automatic approach for extracting challenge sets rich with long-distance dependencies, and argue that evaluation using this methodology provides a complementary perspective on system performance. To support our claim, we compile challenge sets for English-German and German-English, which are much larger than any previously released challenge set for MT. The extracted sets are large enough to allow reliable automatic evaluation, which makes the proposed approach a scalable and practical solution for evaluating MT performance on the long-tail of syntactic phenomena.

Questions this paper answers

do machine translation models care whether nearby words stay near each other?
does a self-attention encoder exhibit a locality bias when source token order is permuted?
how do I test whether a translation model relies on source word order being monotonic?
if my language pair has very different word order, will a Transformer or an LSTM encoder suffer more?: A Transformer with learned positional embeddings scores 24.81 BLEU on regular German-English data and 24.87 BLEU when the same fixed permutation is applied to every source sentence, showing no locality bias.
Holds for: German-English WMT2015 sentences of length 18 only (130,983 sentences, 1,000 held out), one fixed permutation sigma, 5 runs in the Regular setting and 5 in Permuted; highest test BLEU over epochs reported.; Sine positional embeddings leave the Transformer with a residual locality bias: BLEU drops 1.18 points, from 25.08 to 23.90, when source tokens are permuted. A BiLSTM (Nematus) drops far more, 2.65 points from 22.32 to 19.67.
Holds for: German-English, fixed-length-18 sentences from WMT2015 (130,983 sentences, comparable to a low-resource setting), single training run each for SinePE and Nematus in both the Regular and Permuted conditions.
does it matter whether a translation model learns its position information or uses a fixed formula?
do sinusoidal versus learned positional embeddings differ in the locality bias they induce in NMT?
which positional embedding should I pick when training translation for a language pair with heavy reordering?
should I switch my Transformer from sinusoidal to learned position embeddings?: Sine positional embeddings leave the Transformer with a residual locality bias: BLEU drops 1.18 points, from 25.08 to 23.90, when source tokens are permuted. A BiLSTM (Nematus) drops far more, 2.65 points from 22.32 to 19.67.
Holds for: German-English, fixed-length-18 sentences from WMT2015 (130,983 sentences, comparable to a low-resource setting), single training run each for SinePE and Nematus in both the Regular and Permuted conditions.; Choshen and Abend argue that the choice between learned and sine positional embeddings is not neutral. Learned embeddings are preferable when a locality bias is undesirable, such as for highly divergent language pairs.
Holds for: Based on permutation experiments on one German-English setting with 18-token sentences; the two embedding types are comparable in BLEU under normal, unpermuted training.
do today's translation systems still get sentences wrong when related words are far apart?
are long-distance dependencies still a bottleneck for Transformer NMT quality?
how do I find out whether my translation model breaks on separable verbs and other split constructions?
can I trust a Transformer to translate sentences where a verb and its particle are separated?: Long-distance dependencies remain hard for the Transformer even though it shows no locality bias. On German-English News, BLEU drops from 28.23 on the full test set to 22.68 on the reordering challenge set and 27.46 on the verb-particle set.
Holds for: Transformer trained on WMT2015, evaluated on challenge sets extracted from newstest2013 and the Books corpus with minimum head-dependent distance d>=1; the same drop pattern appears for Nematus and for English-German.; Translation quality falls as the head-dependent distance grows. Across 10 phenomenon-model-language combinations, 9 show a negative Spearman correlation between minimum distance and BLEU, including -1 for German reflexive verbs with the Transformer.
Holds for: Books challenge sets, minimum distances of 1, 2 and 3 against an unrestricted control; Transformer and Nematus; English verb-particle constructions with the Transformer are the single positive correlation (0.73).
does translation get worse the further apart the connected words in a sentence are?
how does head-dependent distance correlate with translation quality for reflexive verbs and verb particles?
how do I measure the effect of dependency length on my model's translation quality?
should I expect more translation errors in my data as the gap between a verb and its dependent grows?: Translation quality falls as the head-dependent distance grows. Across 10 phenomenon-model-language combinations, 9 show a negative Spearman correlation between minimum distance and BLEU, including -1 for German reflexive verbs with the Transformer.
Holds for: Books challenge sets, minimum distances of 1, 2 and 3 against an unrestricted control; Transformer and Nematus; English verb-particle constructions with the Transformer are the single positive correlation (0.73).; Manual annotation of German-English Transformer output confirms the distance effect: 60% of lexical long-distance dependencies are translated correctly at distance 1, 54% at distance 2 and 38% at distance 5.
Holds for: 180 German source sentences from Books, distances of exactly 1, 2 and 5, judged by 2 annotators at kappa=0.79, after removing extraction errors.
can test sets that target a specific grammar construction be built without hand-writing examples?
can MT challenge sets be extracted automatically from dependency parses and word alignments?
how do I build a large phenomenon-specific evaluation set for translation without manual annotation?
is automatic extraction going to give me enough examples to evaluate one construction with BLEU?: Choshen and Abend's CoNLL 2019 work introduces automatic extraction of MT challenge sets from dependency parses and word alignments. The extracted sets are large enough that phenomenon-specific evaluation can use standard automatic metrics such as BLEU and RIBES instead of manual inspection.
Holds for: As of 2019, when earlier MT challenge sets for French-English and English-Swedish were hand-compiled and manually scored; demonstrated only for German-English and English-German, and requires a Universal Dependencies parser for the source language.; Automatic extraction yields German-English challenge sets of 7,584 verb-particle and 8,122 reflexive-verb sentences at minimum distance 1. Previously released MT challenge sets were compiled by hand at about 10 examples per phenomenon.
Holds for: Extracted from the Books corpus (51K sentence pairs) and newstest2013 (3K) for German-English and English-German; English-German sets are much smaller, e.g. 191 preposition-stranding sentences at distance 1.
if examples are pulled out of a corpus by a parser, how often do they really contain the construction you wanted?
what is the precision of parser-and-alignment-based extraction of reflexive verb and preposition-stranding sentences?
how do I check that an automatically built challenge set is clean enough to report scores on?
can I rely on parser-extracted long-distance-dependency sentences without checking them by hand?: Automatic parser-based extraction of long-distance-dependency sentences is accurate enough for evaluation: 85% of extracted German sentences, 87% of English News sentences and 86% of English Books sentences genuinely contain the target phenomenon.
Holds for: Manual check of 180 German and 81 English sentences by 2 proficient annotators (the paper's authors); per-type precision in English ranges from 1.00 for particles on Books down to 0.60 for preposition stranding on Books.
are sentences with far-apart words harder just because they are longer sentences?
is source sentence length a confound in long-distance-dependency challenge set evaluation?
how do I separate the effect of dependency distance from the effect of sentence length on BLEU?
before I blame my model on long-distance dependencies, how do I rule out that it is just long sentences?: Transformer BLEU on every German-English challenge set is lower than on any of the 100 length-matched Books corpora sampled per set and per d. Correlations between a sampled corpus's average sentence length and Transformer BLEU are only 0.06, 0.09 and 0.03.
Holds for: Books corpus, Transformer, d values 0-3, length-matched samples drawn within 1 token per sentence; the three correlations are for samples of 1,000, 100 and 10 sentences; English-German trends are similar but less pronounced.
which study first showed how to build grammar-targeted translation test sets from a corpus automatically?
what work established automatic construction of phenomenon-specific challenge sets for MT evaluation?
where should I start reading if I want to evaluate translation on specific syntactic phenomena?: Choshen and Abend's CoNLL 2019 work introduces automatic extraction of MT challenge sets from dependency parses and word alignments. The extracted sets are large enough that phenomenon-specific evaluation can use standard automatic metrics such as BLEU and RIBES instead of manual inspection.
Holds for: As of 2019, when earlier MT challenge sets for French-English and English-Swedish were hand-compiled and manually scored; demonstrated only for German-English and English-German, and requires a Universal Dependencies parser for the source language.
does a word-overlap score like BLEU really show whether word order was translated right?
do BLEU and RIBES agree on reordering-focused challenge sets for German-English?
which metric should I report when I am evaluating reordering rather than lexical choice?
should I add a reordering-sensitive metric alongside BLEU for my word-order experiments?: RIBES, a reordering-sensitive metric, confirms the BLEU trend on the reordering challenge sets: the Transformer scores 0.79 versus 0.82 baseline on German News and 0.54 versus 0.57 on German Books.
Holds for: Reordering challenge sets extracted with FastAlign at alignment index difference d>=5, Transformer only, News and Books domains with German and English as source.; Long-distance dependencies remain hard for the Transformer even though it shows no locality bias. On German-English News, BLEU drops from 28.23 on the full test set to 22.68 on the reordering challenge set and 27.46 on the verb-particle set.
Holds for: Transformer trained on WMT2015, evaluated on challenge sets extracted from newstest2013 and the Books corpus with minimum head-dependent distance d>=1; the same drop pattern appears for Nematus and for English-German.
which German and English grammar constructions were collected into the long-distance dependency test sets, and how many sentences each?
what phenomena and set sizes do the automatically extracted German-English verb-particle and reflexive-verb challenge sets cover?
where can I get a ready-made large test set for separable verbs or reflexive verbs in German-English translation?
are the released challenge sets big enough for me to compare two systems on one construction?: Automatic extraction yields German-English challenge sets of 7,584 verb-particle and 8,122 reflexive-verb sentences at minimum distance 1. Previously released MT challenge sets were compiled by hand at about 10 examples per phenomenon.
Holds for: Extracted from the Books corpus (51K sentence pairs) and newstest2013 (3K) for German-English and English-German; English-German sets are much smaller, e.g. 191 preposition-stranding sentences at distance 1.

Claims and scope

A Transformer with learned positional embeddings scores 24.81 BLEU on regular German-English data and 24.87 BLEU when the same fixed permutation is applied to every source sentence, showing no locality bias. (Table 1)
Scope: German-English WMT2015 sentences of length 18 only (130,983 sentences, 1,000 held out), one fixed permutation sigma, 5 runs in the Regular setting and 5 in Permuted; highest test BLEU over epochs reported.
Sine positional embeddings leave the Transformer with a residual locality bias: BLEU drops 1.18 points, from 25.08 to 23.90, when source tokens are permuted. A BiLSTM (Nematus) drops far more, 2.65 points from 22.32 to 19.67. (Table 1)
Scope: German-English, fixed-length-18 sentences from WMT2015 (130,983 sentences, comparable to a low-resource setting), single training run each for SinePE and Nematus in both the Regular and Permuted conditions.
Long-distance dependencies remain hard for the Transformer even though it shows no locality bias. On German-English News, BLEU drops from 28.23 on the full test set to 22.68 on the reordering challenge set and 27.46 on the verb-particle set. (Table 4)
Scope: Transformer trained on WMT2015, evaluated on challenge sets extracted from newstest2013 and the Books corpus with minimum head-dependent distance d>=1; the same drop pattern appears for Nematus and for English-German.
Translation quality falls as the head-dependent distance grows. Across 10 phenomenon-model-language combinations, 9 show a negative Spearman correlation between minimum distance and BLEU, including -1 for German reflexive verbs with the Transformer. (Table 5)
Scope: Books challenge sets, minimum distances of 1, 2 and 3 against an unrestricted control; Transformer and Nematus; English verb-particle constructions with the Transformer are the single positive correlation (0.73).
Manual annotation of German-English Transformer output confirms the distance effect: 60% of lexical long-distance dependencies are translated correctly at distance 1, 54% at distance 2 and 38% at distance 5. (Table 8)
Scope: 180 German source sentences from Books, distances of exactly 1, 2 and 5, judged by 2 annotators at kappa=0.79, after removing extraction errors.
Automatic parser-based extraction of long-distance-dependency sentences is accurate enough for evaluation: 85% of extracted German sentences, 87% of English News sentences and 86% of English Books sentences genuinely contain the target phenomenon. (Table 7)
Scope: Manual check of 180 German and 81 English sentences by 2 proficient annotators (the paper's authors); per-type precision in English ranges from 1.00 for particles on Books down to 0.60 for preposition stranding on Books.
Automatic extraction yields German-English challenge sets of 7,584 verb-particle and 8,122 reflexive-verb sentences at minimum distance 1. Previously released MT challenge sets were compiled by hand at about 10 examples per phenomenon. (Table 3)
Scope: Extracted from the Books corpus (51K sentence pairs) and newstest2013 (3K) for German-English and English-German; English-German sets are much smaller, e.g. 191 preposition-stranding sentences at distance 1.
RIBES, a reordering-sensitive metric, confirms the BLEU trend on the reordering challenge sets: the Transformer scores 0.79 versus 0.82 baseline on German News and 0.54 versus 0.57 on German Books. (Table 6)
Scope: Reordering challenge sets extracted with FastAlign at alignment index difference d>=5, Transformer only, News and Books domains with German and English as source.
Transformer BLEU on every German-English challenge set is lower than on any of the 100 length-matched Books corpora sampled per set and per d. Correlations between a sampled corpus's average sentence length and Transformer BLEU are only 0.06, 0.09 and 0.03. (Section 4.3)
Scope: Books corpus, Transformer, d values 0-3, length-matched samples drawn within 1 token per sentence; the three correlations are for samples of 1,000, 100 and 10 sentences; English-German trends are similar but less pronounced.
Choshen and Abend's CoNLL 2019 work introduces automatic extraction of MT challenge sets from dependency parses and word alignments. The extracted sets are large enough that phenomenon-specific evaluation can use standard automatic metrics such as BLEU and RIBES instead of manual inspection. (Section 2.2)
Scope: As of 2019, when earlier MT challenge sets for French-English and English-Swedish were hand-compiled and manually scored; demonstrated only for German-English and English-German, and requires a Universal Dependencies parser for the source language.
Choshen and Abend argue that the choice between learned and sine positional embeddings is not neutral. Learned embeddings are preferable when a locality bias is undesirable, such as for highly divergent language pairs. (Section 3.2)
Scope: Based on permutation experiments on one German-English setting with 18-token sentences; the two embedding types are comparable in BLEU under normal, unpermuted training.

Common misreadings

The absence of a locality bias in the Transformer does not mean it handles long-distance dependencies well: BLEU still drops consistently on the extracted challenge sets, and quality falls further as dependency distance grows.
The Transformer's lower BLEU than Nematus on the Books corpus is not evidence that self-attention generalises worse out of domain; the two models were trained on different data and the experiments were not designed to compare architectures.
Extracting challenge sets from dependency parses does not presuppose that MT systems internally build syntactic representations; parses are used only as a way to find sentences likely to be hard to translate.
The difficulty of the challenge sets is not an artefact of longer sentences: length-matched control corpora sampled from Books all score higher in BLEU than the German-English challenge sets.
Automatic extraction is not perfectly precise; about 14-15% of extracted sentences lack the target phenomenon, and preposition-stranding precision is the lowest at 0.60 on Books.

Terminology in this paper

locality bias: The inductive assumption in a translation model that source words close together correspond to target words close together, so that arbitrarily distant alignments are dispreferred.
reordering LDD: A long-distance dependency in which source and target words largely correspond one-to-one but are ordered very differently, detected by word alignments whose source and target indices differ by at least d.
lexical LDD: A long-distance dependency in which how a word or contiguous expression is translated depends on non-adjacent source words, such as a reflexive verb, a verb-particle construction or a stranded preposition.
PerPosEmb: A control setting in which the source tokens stay in order but their positional embeddings are permuted, so token identity and position information are decoupled.

How to cite

@inproceedings{DBLP:conf/conll/ChoshenA19,author       = {Leshem Choshen and
                  Omri Abend},
  editor       = {Mohit Bansal and
                  Aline Villavicencio},
  title        = {Automatically Extracting Challenge Sets for Non-Local Phenomena in
                  Neural Machine Translation},
  booktitle    = {Proceedings of the 23rd Conference on Computational Natural Language
                  Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019},
  pages        = {291--303},
  publisher    = {Association for Computational Linguistics},
  year         = {2019},
  url          = {https://doi.org/10.18653/v1/K19-1028},
  doi          = {10.18653/V1/K19-1028},
  timestamp    = {Thu, 05 Aug 2021 17:36:17 +0200},
  biburl       = {https://dblp.org/rec/conf/conll/ChoshenA19.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.