Data Contamination Report from the 2024 CONDA Shared Task

Oscar Sainz, Iker Garc\'\ia-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D'Amico-Wong, Melissa Dell, Run-Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Suryansh Sharma, Emily Silcock, Kateryna Solonko, David Stap, Mihai Surdeanu, Yu-Min Tseng, Vishaal Udandarao, Zengzhi Wang, Ruijie Xu, Jinglin Yang · CoRR · 2024

Abstract

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community.

How to cite

@article{DBLP:journals/corr/abs-2407-21530,
author       = {Oscar Sainz and
                  Iker Garc{\'{\i}}a{-}Ferrero and
                  Alon Jacovi and
                  Jon Ander Campos and
                  Yanai Elazar and
                  Eneko Agirre and
                  Yoav Goldberg and
                  Wei{-}Lin Chen and
                  Jenny Chim and
                  Leshem Choshen and
                  Luca D'Amico{-}Wong and
                  Melissa Dell and
                  Run{-}Ze Fan and
                  Shahriar Golchin and
                  Yucheng Li and
                  Pengfei Liu and
                  Bhavish Pahwa and
                  Ameya Prabhu and
                  Suryansh Sharma and
                  Emily Silcock and
                  Kateryna Solonko and
                  David Stap and
                  Mihai Surdeanu and
                  Yu{-}Min Tseng and
                  Vishaal Udandarao and
                  Zengzhi Wang and
                  Ruijie Xu and
                  Jinglin Yang},
  title        = {Data Contamination Report from the 2024 {CONDA} Shared Task},
  journal      = {CoRR},
  volume       = {abs/2407.21530},
  year         = {2024},
  url          = {https://doi.org/10.48550/arXiv.2407.21530},
  doi          = {10.48550/ARXIV.2407.21530},
  eprinttype    = {arXiv},
  eprint       = {2407.21530},
  timestamp    = {Wed, 21 Aug 2024 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2407-21530.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.