NeurIPS 2023 LLM Efficiency Fine-tuning Competition

Mark Saroufim, Yotam Perlitz, Leshem Choshen, Luca Antiga, Greg Bowyer, Christian Puhrsch, Driss Guessous, Supriya Rao, Geeta Chauhan, Ashvini Kumar · arXiv · 2025

In one sentence

In the NeurIPS 2023 LLM Efficiency Fine-tuning Competition, scores on the published open evaluation tasks barely predicted scores on the held-out closed tasks, and winning entries came from data curation with standard open-source libraries rather than custom code.

Abstract

Our analysis of the NeurIPS 2023 large language model (LLM) fine-tuning competition revealed the following trend: top-performing models exhibit significant overfitting on benchmark datasets, mirroring the broader issue of benchmark overfitting on popular leaderboards and that data curation is essential in order to get a high performing LLM. The competition, which consisted of two stages - an open evaluation stage with publicly available tasks and a closed evaluation stage with unseen tasks - allowed us to assess the generalizability of fine-tuned LLMs. Our results highlight the limitations of current benchmark-based evaluation schemes for generative models and demonstrate the need for more robust evaluation methods. Notably, the winning submissions utilized standard open-source libraries and focused primarily on data curation. To facilitate further research and promote reproducibility, we release all competition entries, Docker files, and evaluation infrastructure, providing a valuable resource for the community to explore fine-tuning, overfitting, and reproducibility in LLMs.

Questions this paper answers

if a fine-tuned language model tops a public task list, will it still do well on tasks nobody showed it?
how strongly did open-set mean-win-rate correlate with held-out closed-set mean-win-rate in the NeurIPS 2023 LLM Efficiency Challenge?
how do I tell whether my fine-tuned model's leaderboard gain is real or just fitted to the public evaluation tasks?
should I trust a public benchmark score when picking a fine-tuned model for tasks it has never seen?
In the NeurIPS 2023 LLM Efficiency Challenge, mean-win-rates on the published open evaluation set barely predicted mean-win-rates on the held-out closed set. Score correlations were -0.08 for the 4090 track and 0.18 for the A100 track.
Holds for: Two tracks of the 2023 competition, scored by geometric mean of HELM mean-win-rates; open stage sampled 600 questions across 6 scenarios, closed stage 5,000 questions across 5 holdout scenarios.
The winning entries of the NeurIPS 2023 LLM Efficiency Challenge did not obtain the highest scores on the open evaluation tasks. Other submissions that overfitted those tasks fell to about chance level on some closed evaluation tasks.
Holds for: Final ranking weighted the closed evaluation twice as heavily as the open evaluation (1/3 open, 2/3 closed); observation is over the entries reaching the second round of the 2023 competition.
what did the teams that won the NeurIPS 2023 one-GPU language model fine-tuning contest actually spend their time on?
which base checkpoints and instruction-tuning data mixtures did the winning entries of the NeurIPS 2023 LLM Efficiency Challenge use?
if I have one GPU and a day, where should I put my effort to fine-tune a strong open model?
is it worth writing my own training code, or should I copy the winners and curate data instead?
None of the top entries in the NeurIPS 2023 LLM Efficiency Challenge wrote custom fine-tuning code. They picked a highly ranked open model such as Qwen-14B or Mistral-7B and spent their effort on curating mixtures of LIMA, Open-Platypus, Databricks-Dolly-15k and OASST1.
Holds for: Top entries from both tracks of the 2023 competition, constrained to 24 hours on a single GPU, an approved model list, and open or self-curated data with ChatGPT/GPT-4 generations prohibited.
The winning entries of the NeurIPS 2023 LLM Efficiency Challenge did not obtain the highest scores on the open evaluation tasks. Other submissions that overfitted those tasks fell to about chance level on some closed evaluation tasks.
Holds for: Final ranking weighted the closed evaluation twice as heavily as the open evaluation (1/3 open, 2/3 closed); observation is over the entries reaching the second round of the 2023 competition.
which software packages do people reach for when fine-tuning a language model on one GPU?
what was the library frequency distribution across the 225 submissions to the NeurIPS 2023 LLM Efficiency Challenge?
which fine-tuning libraries should I learn first if I want to follow common practice for single-GPU LLM training?
is HuggingFace PEFT the default choice for parameter-efficient fine-tuning, or do most people use something else?
Across the 225 submissions to the NeurIPS 2023 LLM Efficiency Challenge, all written in Python, the most frequent libraries were HuggingFace PEFT (77 submissions), Transformers (71), Einops (67) and Datasets (63).
Holds for: Counts over submissions from 182 registered teams in the 2023 competition; reflects the tooling available in 2023 and the competition's single-GPU quantized-fine-tuning setting.
when a machine learning contest collects everyone's code, how much of it actually runs again later?
what was the Docker build failure rate for training submissions to the NeurIPS 2023 LLM Efficiency Challenge, and what caused it?
how do I package a fine-tuning submission so it still builds months after I submit it?
do I need to pin my dependency versions before shipping a fine-tuning container, or is a plain requirements file fine?
More than half of the Dockerfiles submitted to the NeurIPS 2023 LLM Efficiency Challenge failed to build, most often because of unpinned dependencies and breaking changes in HuggingFace PEFT and Transformers.
Holds for: Submitted inference Dockerfiles from the 2023 competition; organizers manually repaired submissions before evaluation, so ranking reflects post-repair runs.
how much money and time does it take to fine-tune an open language model on a single graphics card?
what wall-clock and dollar cost did reproduced top-10 entries of the NeurIPS 2023 LLM Efficiency Challenge incur under a 24-hour single-GPU budget?
how do I budget compute for fine-tuning an open 7B-14B model on one rented GPU?
can I afford to fine-tune a competitive open language model myself on a 4090 or a single A100?
All reproduced top-10 solutions from both tracks of the NeurIPS 2023 LLM Efficiency Challenge completed fine-tuning within the 24-hour single-GPU budget, most taking 15-20 hours. That corresponds to roughly USD 7 on a 4090 and USD 20 on an A100.
Holds for: Cost computed at Vast.ai rates of USD 0.35 per hour for a 4090 and USD 1 per hour for an A100; 2 solutions finished in about 2 hours.
does the model that comes first on an overall evaluation score also come first on each individual task?
how much do per-scenario HELM rankings disagree with the aggregate ranking of fine-tuned submissions?
how should I choose between fine-tuned models when their per-task rankings and their overall average disagree?
can I pick a model from one aggregate leaderboard number, or do I need to look at each task separately?
In the NeurIPS 2023 LLM Efficiency Challenge final stage, the top-ranking model overall was best in only 1 of 8 individual HELM scenarios. Per-scenario ranks did not agree on the best model.
Holds for: Final-stage models only, ranked by per-scenario mean-win-rate within the competition's HELM fork; scenarios include accuracy, robustness, bias and fairness measures.
Reducing the number of evaluated problems per task (Sparse HELM style) mostly preserved submission scores in the NeurIPS 2023 LLM Efficiency Challenge. A few sub-scenarios scattered far more than sample-size noise alone would predict, while aggregate scores agreed well.
Holds for: Comparison of the competition's final full-sample evaluation against the reduced-sample open and hidden stages; the only intended difference between the two was the number of samples per task.
what should I read about whether leaderboard rankings of language models mean anything on new tasks?
which study documents benchmark overfitting in fine-tuned LLMs at competition scale with paired open and closed task sets?
where do I start reading if I want evidence on how far fine-tuning leaderboard rank transfers to unseen tasks?
The NeurIPS 2023 LLM Efficiency Fine-tuning Competition report is a competition-scale case study of benchmark overfitting in fine-tuned LLMs. It pairs a published open task set with an unseen closed task set to measure how far leaderboard rank transfers.
Holds for: Evidence is one competition run in 2023 over 225 submissions to single-GPU 24-hour fine-tuning tracks, evaluated with a HELM fork; conclusions concern benchmark-style evaluation, not real-world deployment tasks.
are there public collections of real fine-tuning code and models submitted by many different teams?
what artifacts, including Dockerfiles and a forked HELM harness, did the NeurIPS 2023 LLM Efficiency Challenge organizers release?
where can I get a corpus of real submissions to study reproducibility and overfitting in LLM fine-tuning?
can I reuse the NeurIPS 2023 LLM efficiency competition's evaluation setup and entries for my own study?
The organizers of the NeurIPS 2023 LLM Efficiency Challenge released all competition entries, training and inference Dockerfiles, the forked HELM with extra evaluation tasks, and the evaluation scripts. The release is a public dataset for studying fine-tuning, overfitting and reproducibility.
Holds for: Artifacts are as submitted in 2023 with secrets scrubbed; many inference Dockerfiles do not build as submitted, so the release is a corpus for study rather than a set of turnkey recipes.
if a language model is tested on fewer questions per task to save time, do the results still come out the same?
how closely do subsampled Sparse HELM scores track full-sample scores across competition submissions and sub-scenarios?
how many problems per task do I need to evaluate to rank fine-tuned models reliably?
can I cut my evaluation set down to save compute without changing which model looks best?
Reducing the number of evaluated problems per task (Sparse HELM style) mostly preserved submission scores in the NeurIPS 2023 LLM Efficiency Challenge. A few sub-scenarios scattered far more than sample-size noise alone would predict, while aggregate scores agreed well.
Holds for: Comparison of the competition's final full-sample evaluation against the reduced-sample open and hidden stages; the only intended difference between the two was the number of samples per task.
besides getting answers wrong, what else made competition entries lose points in a language model contest?
how often did runtime timeouts and out-of-memory errors during inference cost submissions points in the NeurIPS 2023 LLM Efficiency Challenge?
how do I keep my model from timing out or running out of memory while answering a whole evaluation suite?
should I worry more about inference latency and memory than about accuracy when entering an LLM efficiency contest?
About 30% of submissions to the NeurIPS 2023 LLM Efficiency Challenge could not finish all evaluation questions inside the runtime limit and lost points to zero-scored answers. About 10% ran out of memory during inference.
Holds for: Runtime limits were 300 minutes in the open stage and 600 minutes in the closed stage; all top models finished within the limits.
did the two different graphics-card divisions of the NeurIPS 2023 language model contest give equally consistent evaluation results?
how did full-versus-hidden evaluation set correlation compare between the A100 and 4090 tracks of the NeurIPS 2023 LLM Efficiency Challenge?
if I read results from one hardware track of an LLM efficiency competition, can I assume the other track behaved the same?
Agreement between the full and hidden evaluation sets differed sharply by track in the NeurIPS 2023 LLM Efficiency Challenge. Correlation was about 0.2 on the A100 track versus 0.85 on the 4090 track.
Holds for: Mean-win-rates of submitted models under the competition's adapted HELM setup; the two tracks had different hardware limits (40 GB A100, 24 GB 4090) and were evaluated independently.

Claims and scope

Common misreadings

Terminology in this paper

open evaluation set
The set of benchmark tasks published to competitors at the start of the NeurIPS 2023 LLM Efficiency Challenge, so entrants could develop and test against it before submitting.
closed (hidden) evaluation set
Held-out benchmark tasks in the NeurIPS 2023 LLM Efficiency Challenge, kept secret until after submissions closed and weighted 2/3 of the final score, used to test generalization beyond the tasks entrants could tune on.
scenario
In the HELM framework, an evaluation configuration measuring a particular property such as accuracy, robustness, bias or fairness; competition scores were the geometric mean of per-scenario mean-win-rates.
mean-win-rate
The average fraction of competing models a given model beats on a scenario, used as the per-scenario score aggregated into the competition ranking.
Sparse HELM
An adaptation of HELM that evaluates fewer problems per task under a fixed compute budget while keeping the full set of scenarios, used to screen the competition's large submission pool.
evalbot
The Discord bot built for the NeurIPS 2023 LLM Efficiency Challenge that let entrants request an evaluation on a subset of the open tasks and optionally publish the result to a track leaderboard; it completed over 700 successful evaluations.

How to cite

@article{saroufim2025neurips,
  title={{NeurIPS} 2023 {LLM} Efficiency Fine-tuning Competition},
  author={Saroufim, Mark and Perlitz, Yotam and Choshen, Leshem and Antiga, Luca and Bowyer, Greg and Puhrsch, Christian and Guessous, Driss and Rao, Supriya and Chauhan, Geeta and Kumar, Ashvini and others},
  journal={arXiv preprint arXiv:2503.13507},
  year={2025}
}

References

See the full reference list in the paper.