# Neurips 2023 llm efficiency fine-tuning competition

Authors: Mark Saroufim, Yotam Perlitz, Leshem Choshen, Luca Antiga, Greg Bowyer, Christian Puhrsch, Driss Guessous, Supriya Rao, Geeta Chauhan, Ashvini Kumar, others
Venue: arXiv preprint arXiv:2503.13507 (2025)

## Abstract
Our analysis of the NeurIPS 2023 large language model (LLM) fine-tuning competition revealed the following trend: top-performing models exhibit significant overfitting on benchmark datasets, mirroring the broader issue of benchmark overfitting on popular leaderboards and that data curation is essential in order to get a high performing LLM. The competition, which consisted of two stages - an open evaluation stage with publicly available tasks and a closed evaluation stage with unseen tasks - allowed us to assess the generalizability of fine-tuned LLMs. Our results highlight the limitations of current benchmark-based evaluation schemes for generative models and demonstrate the need for more robust evaluation methods. Notably, the winning submissions utilized standard open-source libraries and focused primarily on data curation. To facilitate further research and promote reproducibility, we release all competition entries, Docker files, and evaluation infrastructure, providing a valuable resource for the community to explore fine-tuning, overfitting, and reproducibility in LLMs.

## Links
- arXiv: https://arxiv.org/abs/2503.13507
- PDF: https://arxiv.org/pdf/2503.13507
- HTML: https://arxiv.org/html/2503.13507
- Hugging Face: https://huggingface.co/papers/2503.13507
- alphaXiv: https://www.alphaxiv.org/abs/2503.13507
- DOI: https://doi.org/10.48550/arXiv.2503.13507
- Semantic Scholar: https://www.semanticscholar.org/paper/277104779

## How to cite
@article{saroufim2025neurips,
title={Neurips 2023 llm efficiency fine-tuning competition},
  author={Saroufim, Mark and Perlitz, Yotam and Choshen, Leshem and Antiga, Luca and Bowyer, Greg and Puhrsch, Christian and Guessous, Driss and Rao, Supriya and Chauhan, Geeta and Kumar, Ashvini and others},
  journal={arXiv preprint arXiv:2503.13507},
  year={2025}
}