On the Weaknesses of Reinforcement Learning for Neural Machine Translation

RL fine-tuning concentrates probability mass on already-likely tokens instead of on the correct ones

Leshem Choshen, Lior Fox, Zohar Aizenbud, Omri Abend · ICLR 2020 · 2020

In one sentence

Reinforcement learning for neural machine translation mostly sharpens the pretrained model's output distribution rather than teaching it new translations: Contrastive MRT provably does not optimize expected reward, and Reinforce only promotes a target token that the pretrained model already ranks 2nd or 3rd.

Abstract

Reinforcement learning (RL) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). However, little is known about what and how these methods learn in the context of MT. We prove that one of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. In fact, our results suggest that RL practices in MT are likely to improve performance only where the pre-trained parameters are already close to yielding the correct translation. Our findings further suggest that observed gains may be due to effects unrelated to the training signal, but rather from changes in the shape of the distribution curve.

Questions this paper answers

does minimum risk training really push a translation model toward higher expected reward?
is the contrastive MRT objective used in NMT a consistent estimator of expected reward, and does it inherit Reinforce's rank limitation?
how do I tell whether minimum risk training will actually optimize the sentence-level score I care about?
should I trust minimum risk training over plain policy gradient for fine-tuning my translation model?
Contrastive Minimum Risk Training, the estimation method commonly called MRT in machine translation, does not optimize the expected reward. In a 3-outcome counterexample the expected reward is uniquely maximized at theta = 0.25, while CMRT converges to theta ≈ 0.295 or to theta = 0.
Holds for: Explicit counterexample with a 3-value distribution family, sample size k = 2 and a sufficiently small learning rate; maximizing the sampled objective E[R~] instead also fails, peaking at theta ≈ 0.32.
Contrastive MRT shows the same rank limitation as Reinforce in controlled simulations: it makes the target token top-ranked when it started 2nd, but struggles when it started 3rd or below. Only a small peakiness effect appears with CMRT, unlike with Reinforce.
Holds for: 1-layer softmax simulation with alpha = 0.005, k = 20 and 50K update steps, averaged over 100 trials, sampling with replacement; deduplicated sampling gives similar results.
are the translation quality gains from reinforcement learning actually coming from the reward signal?
do BLEU improvements from Reinforce fine-tuning of an NMT system survive replacing the sentence-level reward with a constant, and what does a zero-mean baseline do?
how can I check whether my RL fine-tuning run is learning from the reward or just sharpening the output distribution?
if I fine-tune my translation model with reinforcement learning, will the BLEU bump be worth attributing to the reward?
Fine-tuning a pretrained Transformer NMT system with a constant reward of 1 raises BLEU on news2014 from 30.31 to 30.72. Fine-tuning with an expected-BLEU reward gives essentially the same gain, reaching 30.73.
Holds for: WMT2015 German-English data, 6-layer Transformer, Reinforce with k = 1, learning rate and baseline retuned on the development set; other reward functions gave 30.73-30.84.
Subtracting a constant baseline that makes the expected reward zero disallows learning in the NMT Reinforce experiments. That is surprising, since Reinforce generally converges faster with rewards centered on zero, and it points to a positive expected reward driving the observed gain.
Holds for: Transformer NMT on WMT2015 German-English with an expected-BLEU reward, learning rate and baseline retuned on the development set; reported as a discussion observation rather than a full ablation table.
RL fine-tuning of a full NMT system makes its conditional distributions markedly peakier, shifting the modes' probability mass upward. Average entropy drops from 3.45 in the pretrained model to 2.82 after RL.
Holds for: Transformer pretrained on WMT2015 German-English, Reinforce with an expected-BLEU reward and k = 1; measured on 1000 contexts sampled independently from each model.
how nearly right does a translation model have to already be for reinforcement learning to fix a word?
what pretrained rank must the reference token hold for Reinforce fine-tuning to make it the mode, and how often is it outside the top 3 in a real NMT system?
how do I work out which of my translation model's errors policy gradient fine-tuning could realistically repair?
my model puts the correct word around rank 5 in many contexts, is reinforcement learning going to promote it?
In controlled softmax simulations, Reinforce makes the target token the mode within 100K steps only when the pretrained model already ranked it 2nd; when it starts 3rd or 4th the target does not become the mode, and below rank 4 its probability barely rises even after 1M steps.
Holds for: 1-layer softmax over a 30715-token vocabulary initialized from 1000 real Transformer logit vectors, deterministic noise-free reward, learning rate 0.1 (no rate below 0.1 improved expected reward); results averaged over 100 sampled contexts.
In a real NMT system, the target token is not among the pretrained model's top 3 choices in about half of the contexts where the pretrained model is wrong. Those contexts fall outside the range where RL fine-tuning can plausibly promote the target.
Holds for: 1000 contexts sampled from a Transformer pretrained on WMT2015 German-English; the pretrained model already ranks the target first in about two thirds of all contexts.
After Reinforce fine-tuning of an NMT system, more target tokens are ranked 1st and fewer 2nd, but no consistent shift of probability mass occurs across the other of the first 10 ranks. Any movement from ranks below 1000 up to ranks 10-1000 involves probabilities too small to change the system's output.
Holds for: Target-token ranks compared between a pretrained and a reinforced Transformer on WMT2015 German-English, with an expected-BLEU reward and k = 1.
why do a translation model's word probabilities become more concentrated after reinforcement learning?
how does Reinforce fine-tuning shift probability mass onto the mode and lower the conditional entropy of an NMT system?
how do I detect distribution sharpening rather than genuine learning in an RL-fine-tuned generation model?
will reinforcement learning fine-tuning make my translation system's predictions more confident even without a useful reward?
A single Reinforce step with a constant reward is overwhelmingly more likely to increase than to decrease the probability of the most probable token and of the top-10 tokens. Average entropy falls from 2.9 to 2.85 after that one step.
Holds for: Controlled 1-layer softmax simulation, 10000 pretrained distributions sampled from a Transformer's newstest2013 logits, one update step each, constant reward r = 1 with positive expected reward.
RL fine-tuning of a full NMT system makes its conditional distributions markedly peakier, shifting the modes' probability mass upward. Average entropy drops from 3.45 in the pretrained model to 2.82 after RL.
Holds for: Transformer pretrained on WMT2015 German-English, Reinforce with an expected-BLEU reward and k = 1; measured on 1000 contexts sampled independently from each model.
is there a paper arguing that reinforcement learning does not really improve machine translation?
what critical analysis of reward-based fine-tuning for NMT attributes the reported gains to peakiness rather than reward optimization?
what should I read before committing to reward-based fine-tuning for a text generation system?
"On the Weaknesses of Reinforcement Learning for Neural Machine Translation" is a critical analysis of RL fine-tuning for text generation. It argues that reported gains from Reinforce, MRT and GAN-based training come from distribution sharpening and from tokens the pretrained model nearly got right.
Holds for: As of its 2020 publication; conclusions are drawn from German-English Transformer NMT and from softmax simulations matched to it, and are argued to extend to other discrete, high-dimensional generation tasks by analogy rather than by experiment.
what would make reinforcement learning work better for training translation and other text generators?
which exploration strategies are recommended over on-policy sampling for reward-based fine-tuning in a vocabulary-sized action space?
how do I get useful exploration when fine-tuning an already peaky pretrained language generator with a reward?
if on-policy Reinforce is a dead end for my translation model, what direction should I try instead?
"On the Weaknesses of Reinforcement Learning for Neural Machine Translation" argues that off-policy sampling, parameter-space exploration and diversity- or multi-goal RL are the promising route for text generation. On-policy sampling from a peaky pretrained policy cannot explore a vocabulary-sized action space with near-universally zero reward.
Holds for: A direction proposed in the discussion and conclusion, not evaluated in the paper; framed for machine translation where RL tunes a pretrained model.
is it a bad idea to clip gradients when training a translation model with a reward signal?
does gradient clipping violate the assumptions of Reinforce when fine-tuning an NMT system?
how should I set gradient clipping for policy gradient fine-tuning of a translation model?
I clip gradients by default in my NMT training script, should I turn that off for reinforcement learning fine-tuning?
Gradient clipping should be avoided when fine-tuning a translation model with Reinforce, because it violates Reinforce's assumptions and is expected to slow convergence further.
Holds for: Reinforce fine-tuning of a pretrained NMT model; in the paper's own Transformer setup gradient clipping of size 5 was applied during pretraining but not during RL training.
how much did translation quality actually improve when a Transformer system was fine-tuned with a reward?
what BLEU did Reinforce fine-tuning of a pretrained Transformer reach on the news2014 test set, and what did a constant reward reach?
how big a BLEU gain should I expect from reward-based fine-tuning of a pretrained Transformer translation system?
Fine-tuning a pretrained Transformer NMT system with a constant reward of 1 raises BLEU on news2014 from 30.31 to 30.72. Fine-tuning with an expected-BLEU reward gives essentially the same gain, reaching 30.73.
Holds for: WMT2015 German-English data, 6-layer Transformer, Reinforce with k = 1, learning rate and baseline retuned on the development set; other reward functions gave 30.73-30.84.
does minimum risk training run into the same problem as policy gradients when the right word starts low in the ranking?
in controlled softmax simulations, does contrastive MRT promote the target token to the mode from rank 3 or below the way Reinforce fails to?
if the correct token sits at rank 3 in my model, would switching from Reinforce to MRT help?
Contrastive MRT shows the same rank limitation as Reinforce in controlled simulations: it makes the target token top-ranked when it started 2nd, but struggles when it started 3rd or below. Only a small peakiness effect appears with CMRT, unlike with Reinforce.
Holds for: 1-layer softmax simulation with alpha = 0.005, k = 20 and 50K update steps, averaged over 100 trials, sampling with replacement; deduplicated sampling gives similar results.
In controlled softmax simulations, Reinforce makes the target token the mode within 100K steps only when the pretrained model already ranked it 2nd; when it starts 3rd or 4th the target does not become the mode, and below rank 4 its probability barely rises even after 1M steps.
Holds for: 1-layer softmax over a 30715-token vocabulary initialized from 1000 real Transformer logit vectors, deterministic noise-free reward, learning rate 0.1 (no rate below 0.1 improved expected reward); results averaged over 100 sampled contexts.

Claims and scope

Common misreadings

Terminology in this paper

PkE (peakiness effect)
The tendency of policy gradient fine-tuning to increase the probability mass of the already most probable tokens, and lower the output distribution's entropy, regardless of whether those tokens are the rewarding ones.
CMRT (Contrastive MRT)
The estimation method usually called Minimum Risk Training in neural machine translation, which reweights a sampled set S by normalized probabilities P(y)^alpha restricted to S, as distinct from applying Reinforce to minimize risk.
Simulated Reward setting
A controlled simulation reward that gives 2 to the designated target token, 1 to each of the 10 initially highest-scoring tokens, and 0 otherwise, standing in for a decent but sub-optimal pretrained model.
Constant Reward setting
A controlled simulation in which every token receives reward 1, used to isolate changes in the output distribution that cannot come from the reward signal.

How to cite

@inproceedings{DBLP:conf/iclr/ChoshenFAA20,author       = {Leshem Choshen and
                  Lior Fox and
                  Zohar Aizenbud and
                  Omri Abend},
  title        = {On the Weaknesses of Reinforcement Learning for Neural Machine Translation},
  booktitle    = {8th International Conference on Learning Representations, {ICLR} 2020,
                  Addis Ababa, Ethiopia, April 26-30, 2020},
  publisher    = {OpenReview.net},
  year         = {2020},
  url          = {https://openreview.net/forum?id=H1eCw3EKvH},
  timestamp    = {Thu, 21 Jan 2021 00:00:00 +0100},
  biburl       = {https://dblp.org/rec/conf/iclr/ChoshenFAA20.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.