Label-Efficient Model Selection for Text Generation

selecting which examples to have annotated by clustering the difference between two models' outputs, so fewer preference labels are needed to tell which model is better

Shir Ashury-Tahan, Ariel Gera, Benjamin Sznajder, Leshem Choshen, Liat Ein-Dor, Eyal Shnarch · ACL 2024 · 2024

• arXiv
• PDF
• HTML
• Hugging Face
• alphaXiv
• DOI
• Semantic Scholar
• Publisher

In one sentence

DiffUse picks which examples to send to a human or LLM judge by embedding both models' outputs, subtracting them, clustering the difference vectors and annotating one representative per cluster, so far fewer preference labels are needed to identify the better text generation model.

Abstract

Model selection for a given target task can be costly, as it may entail extensive annotation of the quality of outputs of different models. We introduce DiffUse, an efficient method to make an informed decision between candidate text generation models based on preference annotations. DiffUse reduces the required amount of annotations, thus saving valuable time and resources in performing evaluation. DiffUse intelligently selects instances by clustering embeddings that represent the semantic differences between model outputs. Thus, it is able to identify a subset of examples that are more informative for preference decisions. Our method is model-agnostic, and can be applied to any text generation model for selecting between models, prompts and configurations. Moreover, we propose a practical iterative approach for dynamically determining how many instances to annotate. In a series of experiments over hundreds of model pairs, we demonstrate that DiffUse can dramatically reduce the required number of annotations -- by up to 75% -- while maintaining high evaluation reliability.

Questions this paper answers

how can I tell which of two text generators is better without paying someone to read every test output?

what approach gives label-efficient pairwise model selection for generation tasks under a fixed preference-annotation budget?

how do I pick between two summarization or QA systems while annotating only a small subset of the test set?

I have two candidate generation models and a small annotation budget — can I still trust which one I pick?

DiffUse addresses label-efficient model selection for text generation: choosing between two generation models or prompts under a fixed preference-annotation budget. Earlier example-selection work had treated classification and question answering rather than generation.
Holds for: Pairwise comparison only, with no theoretical guarantee for a single comparison; picking from more than 2 candidates is left to future work. As of the ACL 2024 publication.

With the iterative stopping rule at risk threshold p=0.2, DiffUse reaches a decision after 28.77 annotations on NarrativeQA versus 118.69 for random sampling, and 85.25 versus 160.97 on open-book NaturalQuestions.
Holds for: HELM v0.2.2 data, 666 model pairs per scenario, Algorithm 1 with n=5 and maximum budget N=200; reference-based metrics stand in for the preference oracle.

DiffUse requires no existing annotations and no assumptions about the models, tasks, prompts or hyper-parameters, needing only the two models' generated outputs on unlabeled test inputs. Item-response-theory selection methods instead need fully annotated data for a set of existing models.
Holds for: Requires running inference with both models over the pool of examples to be clustered, in the range of hundreds; only worthwhile when the oracle's cost far exceeds that inference cost, as with paid APIs or human annotators.

how many judged examples does difference-based selection save compared with picking test examples at random?

what annotation-budget reduction does DiffUse achieve over random sampling at matched model-preference success rates?

how few preference judgments do I need to reliably rank two generation models against each other?

is selecting examples by output-difference clustering actually cheaper than random sampling for my comparison?

With the iterative stopping rule at risk threshold p=0.2, DiffUse reaches a decision after 28.77 annotations on NarrativeQA versus 118.69 for random sampling, and 85.25 versus 160.97 on open-book NaturalQuestions.
Holds for: HELM v0.2.2 data, 666 model pairs per scenario, Algorithm 1 with n=5 and maximum budget N=200; reference-based metrics stand in for the preference oracle.

At risk threshold p=0.2 the iterative DiffUse procedure returns the correct winner on 89.64% of NarrativeQA model pairs against 43.03% for random sampling, and on 86.50% versus 80.32% for CNN/DailyMail.
Holds for: 666 HELM model pairs per scenario, Algorithm 1 with n=5, N=200; gains are much smaller on closed-book NaturalQuestions, where winning distances are tiny.

how do I know when I have annotated enough examples to call a winner between two models?

how does the iterative DiffUse stopping rule at risk threshold p=0.2 trade off annotation count, decision accuracy and inconclusive verdicts?

how do I decide when to stop labeling and declare one of two generation models the winner?

if I use a 0.2 risk threshold stopping rule for a two-model comparison, how often will the verdict be wrong or inconclusive?

At risk threshold p=0.2 the iterative DiffUse procedure returns the correct winner on 89.64% of NarrativeQA model pairs against 43.03% for random sampling, and on 86.50% versus 80.32% for CNN/DailyMail.
Holds for: 666 HELM model pairs per scenario, Algorithm 1 with n=5, N=200; gains are much smaller on closed-book NaturalQuestions, where winning distances are tiny.

When iterative DiffUse picks the wrong model at p=0.2, the pair's average winning distance is 0.06 on both CNN/DailyMail and NarrativeQA. Correctly decided pairs have far larger gaps, 0.30 and 0.34, so the cost of an error is limited.
Holds for: 666 HELM model pairs per scenario, Algorithm 1 with p=0.2, n=5, N=200; winning distance is defined over reference-based metric scores, not human preferences.

On closed-book NaturalQuestions, iterative DiffUse at p=0.2 annotates 189.05 examples and still returns an inconclusive verdict for 93.41% of model pairs, because the test winning distances in that scenario are very small.
Holds for: HELM closed-book NaturalQuestions, 666 model pairs, Algorithm 1 with n=5 and maximum budget N=200; random sampling is no better, at 191.74 annotations and 95.53% inconclusive.

can a small hand-picked sample tell me how much better one model is, not just which one is better?

is the winning-distance estimate from difference-vector clustering unbiased, and why does the selection favour the test winner?

how do I estimate the size of the quality gap between two generation models from a small annotated subset?

should I report the performance gap between two models from a DiffUse-selected sample, or only the winner?

DiffUse produces a winning-distance estimate biased in favour of the true test winner, largest at small annotation budgets and dissipating as more examples are labeled. Random selection, by contrast, deviates from the test winning distance by 0 on average.
Holds for: Aggregated across all 666 model pairs on XSum, budgets from 5 to 200 examples; the bias means DiffUse should not be used to report the size of the performance gap.

Output pairs whose difference vector has a larger norm are more likely to carry the preference label matching the overall test winner. Hierarchical clustering exploits this: often over half the vectors fall into a single low-norm cluster that contributes just 1 annotated example.
Holds for: Difference vectors from Sentence-BERT all-MiniLM-L6-v2 embeddings, binned into 50 equal-count bins per model pair, aggregated over NarrativeQA model pairs.

which test examples tell you the most when comparing two models that write different answers?

does the norm of the output difference vector predict agreement with the overall test-set preference?

how do I find the handful of examples worth annotating when comparing two generation models?

should I spend my annotation budget on the examples where two models' outputs diverge most?

Output pairs whose difference vector has a larger norm are more likely to carry the preference label matching the overall test winner. Hierarchical clustering exploits this: often over half the vectors fall into a single low-norm cluster that contributes just 1 annotated example.
Holds for: Difference vectors from Sentence-BERT all-MiniLM-L6-v2 embeddings, binned into 50 equal-count bins per model pair, aggregated over NarrativeQA model pairs.

Annotating the examples with the highest difference-vector norm, without clustering, is inconsistent across the 6 HELM generation scenarios and does not match DiffUse. Norm alone selects outliers that do not represent the space of output differences.
Holds for: Max-norm baseline across the 6 HELM generation scenarios and 666 model pairs, budgets 5-200; norm ranking still carries useful signal, it is the loss of diversity that hurts.

is it enough to label only the examples where two models' answers differ the most?

does a max-norm difference-vector baseline match clustering-based selection across HELM generation scenarios?

can I skip clustering and just rank test examples by how far apart the two models' outputs are?

should I use a simple largest-difference heuristic instead of clustering output differences?

Annotating the examples with the highest difference-vector norm, without clustering, is inconsistent across the 6 HELM generation scenarios and does not match DiffUse. Norm alone selects outliers that do not represent the space of output differences.
Holds for: Max-norm baseline across the 6 HELM generation scenarios and 666 model pairs, budgets 5-200; norm ranking still carries useful signal, it is the loss of diversity that hurts.

would grouping the questions themselves work as well as grouping the ways two models' answers differ?

does clustering task-input embeddings match clustering output difference vectors for pairwise generation model selection?

should I select evaluation examples from input-space embeddings or from differences between the two models' outputs?

I already have input embeddings for my test set — can I just cluster those to choose which examples to annotate?

Clustering the embeddings of task inputs instead of output difference vectors does not consistently beat random sampling across the 6 HELM scenarios and 666 model pairs. The gains of DiffUse therefore come specifically from representing differences between model outputs.
Holds for: 6 HELM generation scenarios, 666 model pairs, budgets 5-200; contrasts with active-learning style input-space selection.

Output pairs whose difference vector has a larger norm are more likely to carry the preference label matching the overall test winner. Hierarchical clustering exploits this: often over half the vectors fall into a single low-norm cluster that contributes just 1 annotated example.
Holds for: Difference vectors from Sentence-BERT all-MiniLM-L6-v2 embeddings, binned into 50 equal-count bins per model pair, aggregated over NarrativeQA model pairs.

does the choice of clustering algorithm change how well the example selection works?

how sensitive is DiffUse's model-preference success rate to hierarchical Euclidean, hierarchical cosine and k-means clustering, and to representative selection?

which parts of a difference-vector selection pipeline do I have to tune before comparing two models?

do I need to tune the clustering algorithm and representative rule, or will any reasonable choice do?

Swapping DiffUse's clustering algorithm among 3 options (hierarchical with Euclidean or cosine distance, or k-means) changes model-preference success rates only slightly, and all configurations beat random selection. The rule for picking a cluster representative matters as little.
Holds for: 6 HELM generation scenarios, 666 model pairs, budgets 5-200; representative rules compared are random, nearest centroid by Euclidean or cosine distance, and maximum norm.

Clustering the embeddings of task inputs instead of output difference vectors does not consistently beat random sampling across the 6 HELM scenarios and 666 model pairs. The gains of DiffUse therefore come specifically from representing differences between model outputs.
Holds for: 6 HELM generation scenarios, 666 model pairs, budgets 5-200; contrasts with active-learning style input-space selection.

can the same cheap-labeling trick pick the better of two prompts for one model?

does difference-vector selection extend from model pairs to few-shot prompt variants of a fixed model?

how do I choose between two few-shot prompt templates with as few preference annotations as possible?

I am comparing two prompt variants of one model — can I use DiffUse instead of annotating a full test set?

DiffUse also identifies the better of two few-shot prompt variants for a single fixed model using far fewer annotations than random selection, consistently across 111 unique prompt pairs per scenario.
Holds for: 3 prompt variants for each of 37 HELM models, over the 6 generation scenarios with 3 reference-based metrics each; variants differ only in the few-shot exemplars before the input.

were the reported savings measured with real human judgments or with automatic scores?

what preference oracle backed the DiffUse experiments, and does it cover LLM-as-judge or human annotation?

before I rely on DiffUse for human-annotated preference comparisons, what was it actually validated against?

All DiffUse results use HELM reference-based automatic metrics as the preference oracle, with 3 metrics per each of 6 scenarios simulating different oracle types. The method is therefore not demonstrated on real human or LLM preference judgments.
Holds for: HELM v0.2.2 core scenarios, 6 generation tasks, 37 models, 666 model pairs, 800 of 1000 examples sampled per run, 10 seeds, budgets 5-200; a stated limitation of the work.

DiffUse addresses label-efficient model selection for text generation: choosing between two generation models or prompts under a fixed preference-annotation budget. Earlier example-selection work had treated classification and question answering rather than generation.
Holds for: Pairwise comparison only, with no theoretical guarantee for a single comparison; picking from more than 2 candidates is left to future work. As of the ACL 2024 publication.

when is it not worth trying to cut annotation costs for comparing two models?

what does DiffUse assume about the compared models and available labels, and in which scenarios does it fail to reach a verdict?

how do I know whether cheap difference-based selection will work for the two models I want to compare?

do I need any existing labeled evaluation data or model assumptions to use DiffUse, and when will it not pay off?

DiffUse requires no existing annotations and no assumptions about the models, tasks, prompts or hyper-parameters, needing only the two models' generated outputs on unlabeled test inputs. Item-response-theory selection methods instead need fully annotated data for a set of existing models.
Holds for: Requires running inference with both models over the pool of examples to be clustered, in the range of hundreds; only worthwhile when the oracle's cost far exceeds that inference cost, as with paid APIs or human annotators.

On closed-book NaturalQuestions, iterative DiffUse at p=0.2 annotates 189.05 examples and still returns an inconclusive verdict for 93.41% of model pairs, because the test winning distances in that scenario are very small.
Holds for: HELM closed-book NaturalQuestions, 666 model pairs, Algorithm 1 with n=5 and maximum budget N=200; random sampling is no better, at 191.74 annotations and 95.53% inconclusive.

what should I read first about cutting the cost of comparing two language models?

which work introduced label-efficient example selection for pairwise evaluation of text generation rather than classification?

where do I start if I want to reduce the annotation cost of my LLM comparisons?

is there a paper I can point my team to on cheap pairwise evaluation of generation models?

DiffUse addresses label-efficient model selection for text generation: choosing between two generation models or prompts under a fixed preference-annotation budget. Earlier example-selection work had treated classification and question answering rather than generation.
Holds for: Pairwise comparison only, with no theoretical guarantee for a single comparison; picking from more than 2 candidates is left to future work. As of the ACL 2024 publication.

DiffUse requires no existing annotations and no assumptions about the models, tasks, prompts or hyper-parameters, needing only the two models' generated outputs on unlabeled test inputs. Item-response-theory selection methods instead need fully annotated data for a set of existing models.
Holds for: Requires running inference with both models over the pool of examples to be clustered, in the range of hundreds; only worthwhile when the oracle's cost far exceeds that inference cost, as with paid APIs or human annotators.

Claims and scope

• With the iterative stopping rule at risk threshold p=0.2, DiffUse reaches a decision after 28.77 annotations on NarrativeQA versus 118.69 for random sampling, and 85.25 versus 160.97 on open-book NaturalQuestions. (Table 2)
  Scope: HELM v0.2.2 data, 666 model pairs per scenario, Algorithm 1 with n=5 and maximum budget N=200; reference-based metrics stand in for the preference oracle.
• At risk threshold p=0.2 the iterative DiffUse procedure returns the correct winner on 89.64% of NarrativeQA model pairs against 43.03% for random sampling, and on 86.50% versus 80.32% for CNN/DailyMail. (Table 2)
  Scope: 666 HELM model pairs per scenario, Algorithm 1 with n=5, N=200; gains are much smaller on closed-book NaturalQuestions, where winning distances are tiny.
• On closed-book NaturalQuestions, iterative DiffUse at p=0.2 annotates 189.05 examples and still returns an inconclusive verdict for 93.41% of model pairs, because the test winning distances in that scenario are very small. (Table 2)
  Scope: HELM closed-book NaturalQuestions, 666 model pairs, Algorithm 1 with n=5 and maximum budget N=200; random sampling is no better, at 191.74 annotations and 95.53% inconclusive.
• When iterative DiffUse picks the wrong model at p=0.2, the pair's average winning distance is 0.06 on both CNN/DailyMail and NarrativeQA. Correctly decided pairs have far larger gaps, 0.30 and 0.34, so the cost of an error is limited. (Table 2)
  Scope: 666 HELM model pairs per scenario, Algorithm 1 with p=0.2, n=5, N=200; winning distance is defined over reference-based metric scores, not human preferences.
• DiffUse produces a winning-distance estimate biased in favour of the true test winner, largest at small annotation budgets and dissipating as more examples are labeled. Random selection, by contrast, deviates from the test winning distance by 0 on average. (Figure 5)
  Scope: Aggregated across all 666 model pairs on XSum, budgets from 5 to 200 examples; the bias means DiffUse should not be used to report the size of the performance gap.
• Output pairs whose difference vector has a larger norm are more likely to carry the preference label matching the overall test winner. Hierarchical clustering exploits this: often over half the vectors fall into a single low-norm cluster that contributes just 1 annotated example. (Figure 10)
  Scope: Difference vectors from Sentence-BERT all-MiniLM-L6-v2 embeddings, binned into 50 equal-count bins per model pair, aggregated over NarrativeQA model pairs.
• Annotating the examples with the highest difference-vector norm, without clustering, is inconsistent across the 6 HELM generation scenarios and does not match DiffUse. Norm alone selects outliers that do not represent the space of output differences. (Figure 16)
  Scope: Max-norm baseline across the 6 HELM generation scenarios and 666 model pairs, budgets 5-200; norm ranking still carries useful signal, it is the loss of diversity that hurts.
• Clustering the embeddings of task inputs instead of output difference vectors does not consistently beat random sampling across the 6 HELM scenarios and 666 model pairs. The gains of DiffUse therefore come specifically from representing differences between model outputs. (Figure 14)
  Scope: 6 HELM generation scenarios, 666 model pairs, budgets 5-200; contrasts with active-learning style input-space selection.
• Swapping DiffUse's clustering algorithm among 3 options (hierarchical with Euclidean or cosine distance, or k-means) changes model-preference success rates only slightly, and all configurations beat random selection. The rule for picking a cluster representative matters as little. (Figure 12)
  Scope: 6 HELM generation scenarios, 666 model pairs, budgets 5-200; representative rules compared are random, nearest centroid by Euclidean or cosine distance, and maximum norm.
• DiffUse also identifies the better of two few-shot prompt variants for a single fixed model using far fewer annotations than random selection, consistently across 111 unique prompt pairs per scenario. (Figure 17)
  Scope: 3 prompt variants for each of 37 HELM models, over the 6 generation scenarios with 3 reference-based metrics each; variants differ only in the few-shot exemplars before the input.
• DiffUse addresses label-efficient model selection for text generation: choosing between two generation models or prompts under a fixed preference-annotation budget. Earlier example-selection work had treated classification and question answering rather than generation. (Section 8)
  Scope: Pairwise comparison only, with no theoretical guarantee for a single comparison; picking from more than 2 candidates is left to future work. As of the ACL 2024 publication.
• DiffUse requires no existing annotations and no assumptions about the models, tasks, prompts or hyper-parameters, needing only the two models' generated outputs on unlabeled test inputs. Item-response-theory selection methods instead need fully annotated data for a set of existing models. (Section 8)
  Scope: Requires running inference with both models over the pool of examples to be clustered, in the range of hundreds; only worthwhile when the oracle's cost far exceeds that inference cost, as with paid APIs or human annotators.
• All DiffUse results use HELM reference-based automatic metrics as the preference oracle, with 3 metrics per each of 6 scenarios simulating different oracle types. The method is therefore not demonstrated on real human or LLM preference judgments. (Section 5.1)
  Scope: HELM v0.2.2 core scenarios, 6 generation tasks, 37 models, 666 model pairs, 800 of 1000 examples sampled per run, 10 seeds, budgets 5-200; a stated limitation of the work.

Common misreadings

• The 75% reduction in annotations is a reduction relative to random sampling of the examples needed to reach the same reliability, not an absolute guarantee that 25% of a test set suffices for any model pair; on closed-book NaturalQuestions the full budget of 200 examples still leaves most comparisons inconclusive.
• DiffUse is a deliberately biased selector, not a better estimator of model quality: its winning-distance estimate leans toward the true winner at small budgets, so it should not be used to report how much better one model is.
• The hypergeometric stopping threshold is a heuristic proxy for risk, not a statistical guarantee — the distribution's assumptions are violated because DiffUse selects non-randomly, though the measured error rate stayed below the chosen threshold on every dataset tested.
• The reported preference labels come from HELM reference-based automatic metrics simulating oracles, so the DiffUse results do not directly establish behavior with human annotators or an LLM judge.
• DiffUse compares 2 candidates at a time; it is not evaluated on ranking a larger pool of models, which the paper leaves to future work.

Terminology in this paper

difference vector

The elementwise subtraction of the sentence embedding of one model's output from the other model's output on the same input, used as a representation of how the two models disagree on that example.

test winning model

The model whose outputs are preferred by the oracle on more examples of the full test set — the ground truth an annotation-budgeted comparison is trying to recover.

test winning distance

The absolute difference between the two models' win probabilities over the full test set, i.e. the size of the performance gap.

success rate

The fraction of selection runs in which the winner computed from the annotated subset equals the test winning model, aggregated over model pairs and random seeds.

oracle

Whatever judges which of two generated outputs is better — a human annotator, an LLM judge, or a reference-based automatic metric standing in for one.

How to cite

@inproceedings{DBLP:conf/acl/Ashury-TahanGSC24,author       = {Shir Ashury{-}Tahan and
                  Ariel Gera and
                  Benjamin Sznajder and
                  Leshem Choshen and
                  Liat Ein{-}Dor and
                  Eyal Shnarch},
  editor       = {Lun{-}Wei Ku and
                  Andre Martins and
                  Vivek Srikumar},
  title        = {Label-Efficient Model Selection for Text Generation},
  booktitle    = {Proceedings of the 62nd Annual Meeting of the Association for Computational
                  Linguistics (Volume 1: Long Papers), {ACL} 2024, Bangkok, Thailand,
                  August 11-16, 2024},
  pages        = {8384--8402},
  publisher    = {Association for Computational Linguistics},
  year         = {2024},
  url          = {https://aclanthology.org/2024.acl-long.456},
  timestamp    = {Mon, 26 Aug 2024 16:40:51 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/Ashury-TahanGSC24.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.