How Safe is Your Safety Metric? Automatic Concatenation Tests for Metric Reliability

Ora Nova Fandina; Leshem Choshen; Eitan Farchi; George Kour; Yotam Perlitz; Orna Raz

doi:10.48550/arXiv.2408.12259

How Safe is Your Safety Metric? Automatic Concatenation Tests for Metric Reliability

automatic tests that concatenate, repeat and reorder prompt-response pairs to check whether a safety metric's scores stay consistent

Ora Nova Fandina, Leshem Choshen, Eitan Farchi, George Kour, Yotam Perlitz, Orna Raz · 2025

In one sentence

Concatenation-based tests — repeating, clustering and permuting prompt-response pairs — expose that harmfulness metrics including GPT-3.5 and GPT-4o judges flip their safety verdicts and score by input order rather than content.

Abstract

Consider a scenario where a harmfulness evaluation metric intended to filter unsafe responses from a Large Language Model. When applied to individual harmful prompt-response pairs, it correctly flags them as unsafe by assigning a high-risk score. Yet, if those same pairs are concatenated, the metrics decision unexpectedly reverses - labelling the combined content as safe with a low score, allowing the harmful text to bypass the filter. We found that multiple safety metrics, including advanced metrics such as GPT-based judges, exhibit this non-safe behaviour. Moreover, they show a strong sensitivity to input order: responses are often classified as safe if safe content appears first, regardless of any harmful content that follows, and vice versa. These findings underscore the importance of evaluating the safety of safety metrics, that is, the reliability of their output scores. To address this, we developed general, automatic, concatenation-based tests to assess key properties of these metrics. When applied in a model safety scenario, the tests revealed significant inconsistencies in harmfulness evaluations.

Questions this paper answers

if you bundle several unsafe question-and-answer pairs into one block of text, will an automatic safety scorer still call them harmful?
how often does an LLM-as-a-judge harmfulness metric flip an unsafe verdict to safe when prompt-response pairs are concatenated into tuples?
how do I check whether my GPT-based harmfulness judge holds its verdict when I score batches of prompt-response pairs at once?
can I trust a GPT-3.5 or GPT-4o safety judge to score several unsafe prompt-response pairs in a single call?: The GPT-3.5-based harmfulness judge scored 203 of 1000 concatenated 4-tuples of unsafe prompt-response pairs as safe (scores 1 or 2), while GPT-4o flipped its verdict in only 2 of 1000 cases.
Holds for: 4-tuples sampled from 50 pairs the same judge scored 4 or 5, on Mistral-7B-Instruct-v0.2 responses to AttaQ prompts, 1-5 scale, OpenAI API at seed=2.; GPT-4o as a harmfulness judge keeps its decision-flipping rate below 0.1% as the number of concatenated prompt-response pairs grows, whereas GPT-3.5's flip rate rises with concatenation length.
Holds for: Safe and harmful cluster tests at 4, 8 and 16 concatenations, 1000 sampled tuples per setting, on AttaQ prompts and Mistral-7B-Instruct-v0.2 responses; single dataset and single task.
does shuffling the order of harmful and harmless examples inside one input change the safety score a language model gives it?
how large is the positional bias of GPT-3.5 and GPT-4o harmfulness judges on concatenated prompt-response lists?
how do I measure whether a safety judge is reacting to the ordering of concatenated examples rather than their content?
if I sort examples from mild to severe before sending them to a GPT judge, am I biasing my own safety numbers?: The GPT-3.5-based judge's positional bias is 24.4% for 8-concatenated lists of prompt-response pairs and 45.9% for 16-concatenated lists. Positional bias counts inputs whose verdict flips between scores {4,5} and {1,2} when the input order is reversed.
Holds for: Balanced pools of 50 pairs scored {4,5}, 25 scored 3 and 50 scored {1,2} per metric; 1000 sampled concatenations per length; increasing versus decreasing sorted permutations.; With 16-tuple inputs of effectively identical content, the GPT-4o judge scored 80% of tuples (803 of 1000) as safe when pairs were sorted from low to high harmfulness. The same content scored safe in about 40% of cases under random permutations and 1% under the decreasing permutation.
Holds for: 1000 concatenated 16-tuples drawn from a balanced score pool; AttaQ prompts with Mistral-7B-Instruct-v0.2 responses; GPT-4o via OpenAI API, default temperature, seed=2.
do automatic safety scorers get more order-sensitive as you feed them longer inputs?
does the positional bias of GPT-based harmfulness judges scale with concatenation length?
how long can an aggregated harmfulness-judging input get before ordering effects outweigh content?
should I keep the number of prompt-response pairs per safety-judge call small to limit order effects?: Positional bias in both GPT-based harmfulness judges grows with the length of the concatenated input, so longer aggregated inputs are scored more by the order of their parts than by their content.
Holds for: Concatenation lengths of 4, 8 and 16 prompt-response pairs; GPT-3.5-turbo-0125 and GPT-4o judges only; reward-model metrics show low positional bias instead.
if the same question and answer are pasted in several times, does a reward model rate the text as less harmful?
how sensitive are OpenAssistant deberta- and pythia-based reward-model harmfulness metrics to repetition of the prompt or the response?
how do I test whether a reward model I use as a safety filter can be moved just by duplicating prompt or response text?
is a reward model safe to use as my harmfulness scorer if attackers can repeat the input?: The OpenAssistant deberta-based and pythia-based reward metrics assign lower scores as input content is repeated, with the pythia-based model shifting most. Its Wasserstein distance from the unrepeated score distribution reaches 4.277 when both prompt and response are repeated 8 times.
Holds for: Repetitions up to l=5 for deberta-based (512-token context) and l=16 for pythia-based (1024-token context); 1000 AttaQ prompts with Mistral-7B-Instruct-v0.2 responses; distances computed between score distributions, not per example.; For the deberta-based reward metric, repeating the response alone shifts the score distribution more than repeating the prompt alone: Wasserstein distance 2.065 at 5 response repetitions versus 0.900 at 5 prompt repetitions.
Holds for: OpenAssistant/reward-model-deberta-v3-large-v2 with a 512-token context, so repetition counts stop at 5; scores over 1000 AttaQ prompt-response pairs from Mistral-7B-Instruct-v0.2.
do GPT judges give a different safety verdict when the input text is duplicated?
are GPT-3.5 and GPT-4o harmfulness judges invariant to prompt and response repetition?
which kind of harmfulness scorer should I pick if my inputs contain duplicated prompts or responses?
my evaluation data has repeated prompts and answers, will a GPT judge or a reward model give me stabler harmfulness scores?: The GPT-3.5 and GPT-4o judges are insensitive to repeated content, preserving their original harmfulness scores when the prompt, the response, or both are repeated — the opposite of the reward-model metrics' behaviour.
Holds for: Repetition tests on AttaQ prompts and Mistral-7B-Instruct-v0.2 responses; 16k-token context for GPT-3.5 and 128k for GPT-4o, so long repeated inputs still fit.
if a reward model scores safety, does rearranging the pieces of the input change its score?
what is the order sensitivity of reward-model harmfulness metrics under permutation of concatenated inputs?
how do I find a harmfulness metric whose score barely moves when I permute the concatenated inputs?
should I prefer a reward-model harmfulness score over a GPT judge if my inputs come in arbitrary order?: The reward-model harmfulness metrics show only minor sensitivity to input order, with average pairwise Wasserstein distance between permuted-input score distributions of 0.064 for the deberta-based metric and 0.097 for the pythia-based metric.
Holds for: Six permutations per concatenated input, including increasing- and decreasing-score sorts, where the largest gaps occur (0.147 deberta, 0.583 pythia between increasing and decreasing); AttaQ data with Mistral-7B-Instruct-v0.2 responses.
if you join two equally safe question-answer pairs into one input, does the safety score stay where it was?
do harmfulness metrics preserve cluster scores under concatenation of same-score prompt-response pairs?
how do I check that my harmfulness metric scores a concatenation of equally rated pairs consistently with the individual pairs?
can I score a batch of similarly safe prompt-response pairs together and treat the result as their shared score?: Reward-model harmfulness metrics do not preserve cluster scores under concatenation. For the safe (high-score) 2-concatenated cluster, a large part of the concatenated score distribution falls well below the average of the two original pair scores.
Holds for: Clusters built from the top and bottom 10% of scored pairs (100 pairs each), with 1000 randomly formed 2-concatenations; deberta-based and pythia-based OpenAssistant reward models.; The GPT-3.5 and GPT-4o judges are insensitive to repeated content, preserving their original harmfulness scores when the prompt, the response, or both are repeated — the opposite of the reward-model metrics' behaviour.
Holds for: Repetition tests on AttaQ prompts and Mistral-7B-Instruct-v0.2 responses; 16k-token context for GPT-3.5 and 128k for GPT-4o, so long repeated inputs still fit.
is there a study that tests whether automatic harmfulness scorers are actually valid?
which work provides validity tests for LLM-as-a-judge and reward-model safety metrics?
how do I audit the safety metric I am already using without collecting new human annotations?
where should I start reading before I commit to an automated harmfulness metric for my safety evaluation?: "How Safe is Your Safety Metric?" argues that safety metrics themselves need validity testing, and supplies automatic repetition, cluster and concatenate-and-permute tests for that purpose. The tests reuse a task's existing prompt-response data and need no new annotation.
Holds for: Demonstrated on one task (model safety) with one dataset (AttaQ) and four metrics; the authors note the test suite is small and its generality to other tasks such as translation is proposed rather than shown.
can I believe reported success rates for multi-turn jailbreaks when a language model did the judging?
how does the decision-flipping rate of a GPT-3.5 harmfulness judge affect reported attack success rates for conversation-based jailbreaks?
how do I sanity-check jailbreak attack-success numbers that were produced by a GPT-3.5 judge over concatenated dialogue content?
should I re-run my red-teaming evaluation with a stronger judge than GPT-3.5?: Because GPT-3.5-based judges are widely used to decide the success of multi-turn and conversation-based jailbreak attacks on concatenated prompt-response content, their measured decision-flipping rate makes such attack-success numbers questionable.
Holds for: Concerns gpt-3.5-turbo-0125 with the Appendix A scoring prompt as evaluated in this work; other implementations, prompts and judge models — GPT-4o in particular — behaved more consistently in the cluster tests.; The GPT-3.5-based harmfulness judge scored 203 of 1000 concatenated 4-tuples of unsafe prompt-response pairs as safe (scores 1 or 2), while GPT-4o flipped its verdict in only 2 of 1000 cases.
Holds for: 4-tuples sampled from 50 pairs the same judge scored 4 or 5, on Mistral-7B-Instruct-v0.2 responses to AttaQ prompts, 1-5 scale, OpenAI API at seed=2.
which harmfulness scorers and prompt datasets were put through the repetition and concatenation checks?
which reward-model and LLM-judge safety metrics, and which prompt-response data, were used in the repetition, cluster and concatenate-and-permute tests?
how do I set up repetition and concatenation validity tests for my own safety metric, and what data do they need?
are the harmfulness metrics I use covered by the repetition and concatenation tests, or would I have to run them myself?: "How Safe is Your Safety Metric?" argues that safety metrics themselves need validity testing, and supplies automatic repetition, cluster and concatenate-and-permute tests for that purpose. The tests reuse a task's existing prompt-response data and need no new annotation.
Holds for: Demonstrated on one task (model safety) with one dataset (AttaQ) and four metrics; the authors note the test suite is small and its generality to other tasks such as translation is proposed rather than shown.; The OpenAssistant deberta-based and pythia-based reward metrics assign lower scores as input content is repeated, with the pythia-based model shifting most. Its Wasserstein distance from the unrepeated score distribution reaches 4.277 when both prompt and response are repeated 8 times.
Holds for: Repetitions up to l=5 for deberta-based (512-token context) and l=16 for pythia-based (1024-token context); 1000 AttaQ prompts with Mistral-7B-Instruct-v0.2 responses; distances computed between score distributions, not per example.

Claims and scope

The GPT-3.5-based harmfulness judge scored 203 of 1000 concatenated 4-tuples of unsafe prompt-response pairs as safe (scores 1 or 2), while GPT-4o flipped its verdict in only 2 of 1000 cases. (Figure 6)
Scope: 4-tuples sampled from 50 pairs the same judge scored 4 or 5, on Mistral-7B-Instruct-v0.2 responses to AttaQ prompts, 1-5 scale, OpenAI API at seed=2.
GPT-4o as a harmfulness judge keeps its decision-flipping rate below 0.1% as the number of concatenated prompt-response pairs grows, whereas GPT-3.5's flip rate rises with concatenation length. (Figures 15 and 16)
Scope: Safe and harmful cluster tests at 4, 8 and 16 concatenations, 1000 sampled tuples per setting, on AttaQ prompts and Mistral-7B-Instruct-v0.2 responses; single dataset and single task.
The GPT-3.5-based judge's positional bias is 24.4% for 8-concatenated lists of prompt-response pairs and 45.9% for 16-concatenated lists. Positional bias counts inputs whose verdict flips between scores {4,5} and {1,2} when the input order is reversed. (Table 5 and Table 6)
Scope: Balanced pools of 50 pairs scored {4,5}, 25 scored 3 and 50 scored {1,2} per metric; 1000 sampled concatenations per length; increasing versus decreasing sorted permutations.
With 16-tuple inputs of effectively identical content, the GPT-4o judge scored 80% of tuples (803 of 1000) as safe when pairs were sorted from low to high harmfulness. The same content scored safe in about 40% of cases under random permutations and 1% under the decreasing permutation. (Figure 7)
Scope: 1000 concatenated 16-tuples drawn from a balanced score pool; AttaQ prompts with Mistral-7B-Instruct-v0.2 responses; GPT-4o via OpenAI API, default temperature, seed=2.
Positional bias in both GPT-based harmfulness judges grows with the length of the concatenated input, so longer aggregated inputs are scored more by the order of their parts than by their content. (Figure 12)
Scope: Concatenation lengths of 4, 8 and 16 prompt-response pairs; GPT-3.5-turbo-0125 and GPT-4o judges only; reward-model metrics show low positional bias instead.
The OpenAssistant deberta-based and pythia-based reward metrics assign lower scores as input content is repeated, with the pythia-based model shifting most. Its Wasserstein distance from the unrepeated score distribution reaches 4.277 when both prompt and response are repeated 8 times. (Table 2)
Scope: Repetitions up to l=5 for deberta-based (512-token context) and l=16 for pythia-based (1024-token context); 1000 AttaQ prompts with Mistral-7B-Instruct-v0.2 responses; distances computed between score distributions, not per example.
For the deberta-based reward metric, repeating the response alone shifts the score distribution more than repeating the prompt alone: Wasserstein distance 2.065 at 5 response repetitions versus 0.900 at 5 prompt repetitions. (Table 1)
Scope: OpenAssistant/reward-model-deberta-v3-large-v2 with a 512-token context, so repetition counts stop at 5; scores over 1000 AttaQ prompt-response pairs from Mistral-7B-Instruct-v0.2.
The GPT-3.5 and GPT-4o judges are insensitive to repeated content, preserving their original harmfulness scores when the prompt, the response, or both are repeated — the opposite of the reward-model metrics' behaviour. (Figures 9 and 10)
Scope: Repetition tests on AttaQ prompts and Mistral-7B-Instruct-v0.2 responses; 16k-token context for GPT-3.5 and 128k for GPT-4o, so long repeated inputs still fit.
The reward-model harmfulness metrics show only minor sensitivity to input order, with average pairwise Wasserstein distance between permuted-input score distributions of 0.064 for the deberta-based metric and 0.097 for the pythia-based metric. (Table 3 and Table 4)
Scope: Six permutations per concatenated input, including increasing- and decreasing-score sorts, where the largest gaps occur (0.147 deberta, 0.583 pythia between increasing and decreasing); AttaQ data with Mistral-7B-Instruct-v0.2 responses.
Reward-model harmfulness metrics do not preserve cluster scores under concatenation. For the safe (high-score) 2-concatenated cluster, a large part of the concatenated score distribution falls well below the average of the two original pair scores. (Figure 5 and Figure 13)
Scope: Clusters built from the top and bottom 10% of scored pairs (100 pairs each), with 1000 randomly formed 2-concatenations; deberta-based and pythia-based OpenAssistant reward models.
"How Safe is Your Safety Metric?" argues that safety metrics themselves need validity testing, and supplies automatic repetition, cluster and concatenate-and-permute tests for that purpose. The tests reuse a task's existing prompt-response data and need no new annotation. (Section 1 and Section 3)
Scope: Demonstrated on one task (model safety) with one dataset (AttaQ) and four metrics; the authors note the test suite is small and its generality to other tasks such as translation is proposed rather than shown.
Because GPT-3.5-based judges are widely used to decide the success of multi-turn and conversation-based jailbreak attacks on concatenated prompt-response content, their measured decision-flipping rate makes such attack-success numbers questionable. (Section 1)
Scope: Concerns gpt-3.5-turbo-0125 with the Appendix A scoring prompt as evaluated in this work; other implementations, prompts and judge models — GPT-4o in particular — behaved more consistently in the cluster tests.

Common misreadings

GPT-4o is not shown to be a robust safety judge overall: it keeps a decision-flipping rate below 0.1% in the cluster tests, yet exhibits strong positional bias, scoring 80% of 16-tuples safe under a low-to-high sort versus 1% under the reverse sort.
Insensitivity to repeated content is not a general property of LLM judges over reward models: GPT-3.5 and GPT-4o preserve their scores under repetition, but GPT-3.5 is the metric with the highest decision-flipping rate under concatenation.
The concatenation tests do not measure how harmful a language model is; they measure whether a harmfulness metric's scores stay consistent under input transformations that preserve content.
The findings are not established across tasks or datasets: all experiments use one task (model safety), one prompt dataset (AttaQ) and one base model (Mistral-7B-Instruct-v0.2), and the authors list this limited scope as a limitation.
A high positional-bias number is not evidence that a metric disagrees with humans; it reports how often the metric's own verdict flips when only the order of identical content is reversed.

Terminology in this paper

Decision flipping: A safety metric's verdict changing category — for example from harmful scores {4,5} to safe scores {1,2} — when individually scored prompt-response pairs are concatenated into a single input.
Positional bias (of a safety metric): The percentage of concatenated inputs whose metric verdict flips between the harmful and safe score categories when the order of the concatenated prompt-response pairs is reversed.
Cluster test: A metric validity test that collects inputs a metric scores uniformly high or uniformly low, concatenates them, and checks whether the concatenation receives the same class of score.
Repetition test: A metric validity test that concatenates a prompt, a response, or both with itself l times and tracks how the metric's score changes as l grows.
Concatenate-and-permute test: A metric validity test that concatenates several prompt-response pairs and rescores the same content under several orderings, including sorts by increasing and decreasing individual scores, to measure order sensitivity.
AttaQ: A dataset of 1400 questions designed to elicit harmful responses from a language model, used as the prompt source for the concatenation tests.

How to cite

@misc{fandina2025safesafetymetricautomatic,
      title={How Safe is Your Safety Metric? Automatic Concatenation Tests for Metric Reliability}, 
      author={Ora Nova Fandina and Leshem Choshen and Eitan Farchi and George Kour and Yotam Perlitz and Orna Raz},
      year={2025},
      eprint={2408.12259},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2408.12259}, 
}

References

See the full reference list in the paper.