# Semantics-aware Attention Improves Neural Machine Translation

Authors: Aviv Slobodkin, Leshem Choshen, Omri Abend
Venue: Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, *SEM@NAACL-HLT 2022, Seattle, WA, USA, July 14-15, 2022 (2022)

## Abstract
The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs.

## Links
- arXiv: https://arxiv.org/abs/2110.06920
- PDF: https://arxiv.org/pdf/2110.06920
- HTML: https://ar5iv.labs.arxiv.org/html/2110.06920
- Hugging Face: https://huggingface.co/papers/2110.06920
- alphaXiv: https://www.alphaxiv.org/abs/2110.06920
- DOI: https://doi.org/10.18653/V1/2022.STARSEM-1.3
- ACL Anthology: https://aclanthology.org/2022.starsem-1.3/
- Semantic Scholar: https://www.semanticscholar.org/paper/237258842
- Publisher: https://doi.org/10.18653/v1/2022.starsem-1.3

## How to cite
@inproceedings{DBLP:conf/starsem/SlobodkinCA22,
author       = {Aviv Slobodkin and
                  Leshem Choshen and
                  Omri Abend},
  editor       = {Vivi Nastase and
                  Ellie Pavlick and
                  Mohammad Taher Pilehvar and
                  Jos{\'{e}} Camacho{-}Collados and
                  Alessandro Raganato},
  title        = {Semantics-aware Attention Improves Neural Machine Translation},
  booktitle    = {Proceedings of the 11th Joint Conference on Lexical and Computational
                  Semantics, *SEM@NAACL-HLT 2022, Seattle, WA, USA, July 14-15, 2022},
  pages        = {28--43},
  publisher    = {Association for Computational Linguistics},
  year         = {2022},
  url          = {https://doi.org/10.18653/v1/2022.starsem-1.3},
  doi          = {10.18653/V1/2022.STARSEM-1.3},
  timestamp    = {Mon, 26 Sep 2022 16:58:59 +0200},
  biburl       = {https://dblp.org/rec/conf/starsem/SlobodkinCA22.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}