Efficient multi-prompt evaluation of LLMs

estimating an LLM's whole performance distribution over many prompt templates from a single-prompt-sized evaluation budget

Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, Mikhail Yurochkin · NeurIPS 2024 · 2024

In one sentence

PromptEval fits an Item Response Theory model over prompt templates and examples, borrowing strength across both to estimate an LLM's full performance distribution across 100+ prompts — and its quantiles — for the cost of 2 to 4 single-prompt evaluations.

Abstract

Most popular benchmarks for comparing LLMs rely on a limited set of prompt templates, which may not fully capture the LLMs' abilities and can affect the reproducibility of results on leaderboards. Many recent works empirically verify prompt sensitivity and advocate for changes in LLM evaluation. In this paper, we consider the problem of estimating the performance distribution across many prompt variants instead of finding a single prompt to evaluate with. We introduce PromptEval, a method for estimating performance across a large set of prompts borrowing strength across prompts and examples to produce accurate estimates under practical evaluation budgets. The resulting distribution can be used to obtain performance quantiles to construct various robust performance metrics (e.g., top 95% quantile or median). We prove that PromptEval consistently estimates the performance distribution and demonstrate its efficacy empirically on three prominent LLM benchmarks: MMLU, BIG-bench Hard, and LMentry; for example, PromptEval can accurately estimate performance quantiles across 100 prompt templates on MMLU with a budget equivalent to two single-prompt evaluations. Moreover, we show how PromptEval can be useful in LLM-as-a-judge and best prompt identification applications.

Questions this paper answers

how many test runs does it take to find out how much a model's score swings across 100 different prompt wordings?
what evaluation budget does PromptEval need to estimate performance quantiles over a pool of 100 MMLU prompt templates?
how do I measure the spread of MMLU accuracy over many prompt formats without running the whole benchmark once per format?
I can afford one or two full MMLU runs — can I still get performance quantiles across 100 prompt templates?
PromptEval estimates performance quantiles across 100 MMLU prompt templates with a budget of 1 to 4 times a single-prompt MMLU evaluation. A brute-force sweep would need 100 full evaluations.
Holds for: MMLU with 57 subjects and ~14k examples, 15 open-source LLMs, 100 generated templates per task, estimation done within each task; errors averaged over tasks, LLMs and 5 sampling seeds.
Central quantiles such as the median of the across-prompt performance distribution are estimated accurately by PromptEval with 200 evaluations, giving more than 100x compute saving in most cases. Extreme quantiles like the 5th and 95th need larger budgets.
Holds for: MMLU, BBH and LMentry; quantiles 5/25/50/75/95 evaluated at budgets of 200 to 1600 evaluations; the 100x figure is relative to fully evaluating every template on every example.
if I only test each prompt wording on a few questions, is fitting a model better than just averaging the scores I saw?
does the PromptEval item-response estimator beat per-template empirical averaging for estimating the across-prompt performance distribution, and which covariates matter?
how do I get the distribution of scores over prompt templates from a sparse evaluation matrix more accurately than by averaging each template's observed correctness?
is it worth fitting a Rasch-style model over prompts, or should I just average the correctness scores I already sampled per template?
PromptEval estimates the whole performance distribution across prompt templates with a much lower Wasserstein-1 error than the natural baseline of averaging observed correctness per template, on MMLU, BIG-bench Hard and LMentry.
Holds for: Budgets of 200, 400, 800 and 1600 total evaluations, which are 0.81% (MMLU), 1.15% (BBH) and 0.88% (LMentry) of all prompt-by-example evaluations; all methods sampled with the same two-way balanced sampler.
Even the covariate-free Rasch version of PromptEval beats per-template averaging on both distribution and quantile estimation, and adding prompt covariates improves it further, with pre-trained sentence-transformer embeddings robust across all three benchmarks.
Holds for: MMLU, BBH and LMentry; discrete hand-coded template covariates fail to help on LMentry; fine-tuned BERT embeddings help most for extreme quantiles at small budgets on MMLU but require full evaluation data from a held-out set of LLMs to train.
is it harder to pin down the best-case and worst-case prompt score than the typical one?
which quantiles of the across-prompt performance distribution require larger evaluation budgets, central or extreme?
how many evaluations do I need if I want the 95th percentile of accuracy over prompt templates rather than the median?
I want to report a worst-case prompt score on a small budget — is that as reliable as reporting the median?
Central quantiles such as the median of the across-prompt performance distribution are estimated accurately by PromptEval with 200 evaluations, giving more than 100x compute saving in most cases. Extreme quantiles like the 5th and 95th need larger budgets.
Holds for: MMLU, BBH and LMentry; quantiles 5/25/50/75/95 evaluated at budgets of 200 to 1600 evaluations; the 100x figure is relative to fully evaluating every template on every example.
how should prompt wordings be represented as features when predicting how well each one will do?
which prompt-template covariates give the best performance-distribution estimates in PromptEval: one-hot, hand-coded discrete features, or sentence-transformer embeddings?
do I need to fine-tune an encoder to embed my prompt templates, or will off-the-shelf sentence embeddings do?
should I bother engineering prompt features at all, or is the covariate-free version good enough for my evaluation?
Even the covariate-free Rasch version of PromptEval beats per-template averaging on both distribution and quantile estimation, and adding prompt covariates improves it further, with pre-trained sentence-transformer embeddings robust across all three benchmarks.
Holds for: MMLU, BBH and LMentry; discrete hand-coded template covariates fail to help on LMentry; fine-tuned BERT embeddings help most for extreme quantiles at small budgets on MMLU but require full evaluation data from a held-out set of LLMs to train.
is there any proof that estimating a model's score spread over prompts from a small sample gets the right answer?
are the PromptEval distribution and quantile estimators shown to be consistent as the numbers of prompt templates and examples grow?
how do I justify reporting a quantile of across-prompt performance estimated from a sparse sample of prompt-example pairs?
can I trust a quantile of prompt-template performance estimated from partial evaluations, or is it only an empirical heuristic?
PromptEval's estimators of the across-prompt performance distribution and its quantiles are proved consistent as the numbers of prompts and examples grow. The quantile error goes to 0 in probability for every p, and so does the Wasserstein-1 distance between estimated and true distribution.
Holds for: Proved for linear/affine prompt and example functions with uniformly bounded covariates, a correctly specified Bernoulli model with a consistent maximum-likelihood estimator, and a number of unseen examples per prompt growing fast enough relative to the number of prompts.
does the wording of the instructions given to an AI grader change which chatbot comes out on top?
how sensitive are AlpacaEval 2.0 LLM-as-a-judge rankings to the judge prompt template, and can that sensitivity be quantified on a small budget?
how do I check whether my judge model's verdicts and rankings survive a change of judge prompt without rerunning every pairwise comparison?
my leaderboard uses an LLM judge with one prompt — should I worry the ranking flips with a different judge prompt?
On AlpacaEval 2.0 with 100 prompt templates given to a GPT-4o-mini judge, the ranking of 4 similarly capable LLMs changes under 36% of the judge templates.
Holds for: Cohere Command, Qwen1.5-7B-Chat, Mistral-7B-Instruct-v0.2 and LLaMa-2-70B-Chat; only the judge's prompt varies.
PromptEval reconstructs the distribution of LLM-as-a-judge scores over 100 judge prompt templates from about 2% of the evaluations (1.6k of 80.5k), with much lower Wasserstein-1 error than per-template averaging.
Holds for: AlpacaEval 2.0 with GPT-4o-mini as judge and 4 evaluated LLMs; AlpacaEval scores binarized at a threshold of 1/2 to fit the model, but not binarized at test time.
can the same tool that measures score spread across prompts also pick the single best prompt to use?
how does PromptEval compare with TRIPLE-GSE on regret for best-prompt identification across one-hot, discrete-feature and embedding covariates?
how do I find the highest-scoring prompt in a candidate pool while spending as few evaluations as possible?
I have a pool of candidate prompts and a small eval budget — which selection method gives me the least regret?
For best-prompt identification on MMLU, BBH and LMentry, PromptEval attains lower regret than the TRIPLE-GSE baseline for each of the 3 covariate types: one-hot, discrete features and embeddings. It also beats TRIPLE-GSE with an MLP performance predictor.
Holds for: Fixed, predefined candidate template pools only, not dynamically generated prompts; PromptEval is coupled with a sequential-elimination bandit; regret is the best template's performance minus the chosen template's performance.
how much does an LLM's MMLU score move if you only change the prompt format?
what is the max-minus-min MMLU accuracy spread across 100 prompt templates at the subject level, and is the model ranking stable?
how do I tell whether an MMLU score difference between two models is real or just a prompt-format artefact?
can I compare models on a single-prompt MMLU number, or will a different template change who wins?
On MMLU, most LLMs show an average max-minus-min accuracy spread of around 10% across 100 prompt templates at the subject level. Spreads averaged over all 57 subjects are small, and Llama-3-70B-Instruct stays the best model under every template.
Holds for: 15 open-source LLMs; 100 templates generated by swapping separators, spaces and operators, so no semantic rewrites; spread measured within individual MMLU subjects.
is there one prompt format that works best for every model and every topic?
what is the rank agreement, measured by Kendall's W, among LLMs ranking MMLU prompt templates within and across subjects?
can I tune one prompt template on a few models and subjects and reuse it everywhere on MMLU?
if I find a great prompt format for one model, should I expect it to transfer to my other models and subjects?
No prompt template is broadly best on MMLU: Kendall's W across LLMs ranking templates is at most about 0.25 within any subject. Within a single model, only Gemma-7B-it (0.45) and Mistral-7B-v0.1 (0.35) show appreciable agreement across subjects.
Holds for: 15 open-source LLMs and 57 MMLU subjects with 100 templates; Kendall's W ranges from 0 (no agreement) to 1 (perfect agreement); the remaining 13 models fall below 0.29.
does the advantage of fitting a model over averaging shrink if I only have a handful of prompt wordings?
how does the size of the prompt-template pool affect PromptEval's gain over per-template averaging on MMLU?
I can only write about 20 prompt variations per task — is a fitted estimator still worth using over averaging?
with a small prompt pool, should I invest in PromptEval or just average the per-template scores?
Cutting the pool of prompt templates by a factor of 5, for example to 20 variations on MMLU, leaves PromptEval ahead of per-template averaging but narrows the gap. The method's advantage therefore grows with the number of templates evaluated.
Holds for: Same budgets and benchmarks as the main experiments, with 20% of each task's prompt variations retained at random.
which work suggests scoring language models by their whole range of results over many prompt wordings instead of one chosen prompt?
what paper reframes prompt-sensitive LLM benchmarking as estimating the performance distribution over a prompt-template pool for leaderboard statistics?
where should I start reading if I want to build a leaderboard whose numbers do not depend on one prompt template?
I want a prompt-robust evaluation protocol for my leaderboard — what should I read first?
PromptEval reframes prompt-sensitive LLM evaluation as estimating the whole distribution of performance over a pool of prompt templates, rather than searching for one prompt to report. Leaderboards can then quote robust statistics such as the median or the 95% quantile.
Holds for: Assumes a predefined pool of templates is given; choosing that pool and handling dynamically generated prompt candidates are left open, as stated in Appendix A.
is there a public dataset of many language models scored on MMLU under many different prompt wordings?
which large-scale study measures prompt sensitivity of open-source LLMs on MMLU across 100 templates and releases the evaluation data?
where can I get existing multi-prompt MMLU evaluation results instead of running 100 templates myself?
can I reuse released multi-prompt MMLU data for my own prompt-sensitivity analysis?
The PromptEval paper contributes a large-scale prompt-sensitivity study of 15 open-source LLMs on MMLU across 100 prompt templates, with the evaluation data released. Prior sensitivity analyses had covered SuperNaturalInstructions, BIG-bench and LMentry but not MMLU.
Holds for: As of publication in 2024; covers 15 open-source models and MMLU's 57 subjects with templates generated by feature swaps, so it does not speak to closed API models or to semantically rewritten instructions.

Claims and scope

Common misreadings

Terminology in this paper

PromptEval
A procedure that fits a Bernoulli/IRT correctness model on a sparse sample of prompt-template-by-example evaluations, predicts every template's score with the X-pIRT estimator, and returns the empirical distribution and quantile function of scores across the template pool.
X-pIRT
An extension of the Performance-IRT estimator in which prompt and example parameters are functions of covariates (template embeddings or discrete text features) rather than free per-item parameters; a template's score is the average of its observed correctness scores and its model-predicted correctness on unseen examples.
Two-way balanced sampling
A sampling scheme for choosing which prompt-example pairs to evaluate under a fixed budget, which repeatedly picks the least-evaluated prompt template and then its least-evaluated example, so every template and every example is observed a roughly equal number of times.
Performance spread
The difference between the best and worst accuracy an LLM attains over a set of prompt templates on the same task, used as a measure of prompt sensitivity.
Kendall's W
A rank-agreement coefficient from 0 (no agreement) to 1 (perfect agreement), used to measure whether different subjects or different LLMs order prompt templates the same way.

How to cite

@inproceedings{poloefficient,
  title={Efficient multi-prompt evaluation of LLMs},
  author={Polo, Felipe Maia and Xu, Ronald and Weber, Lucas and Silva, M{\'\i}rian and Bhardwaj, Onkar and Choshen, Leshem and de Oliveira, Allysson Flavio Melo and Sun, Yuekai and Yurochkin, Mikhail},
  booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
  year={2024},
}

References

See the full reference list in the paper.