Skill Issue: Are Skills Language-Invariant in LLMs?

Bobby Cheng; Adam Gaber; Zhengzhe Liu; Catherine Arnett; Omer Goldman; Cheston Tan; Leshem Choshen

doi:10.48550/arXiv.2608.25832

Skill Issue: Are Skills Language-Invariant in LLMs?

a version of the TextArena game suite where each player's instructions and board are rendered in a different language while the rules stay fixed

Bobby Cheng, Adam Gaber, Zhengzhe Liu, Catherine Arnett, Omer Goldman, Cheston Tan, Leshem Choshen · 2026

In one sentence

Two instances of the same LLM playing each other through different language interfaces win at markedly different rates, so the skills a model has in one language are not the skills it applies in another.

Abstract

Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies cross-lingual skill inconsistency orthogonally from knowledge and general benchmark performance. We do this via multilingual self-play: two instances of the same model compete in a text-based game, each interacting through a different language interface. Since the model, opponent, rules, state space, and available actions remain fixed, this setting isolates the effect of language on the model's realized behavior. We build a multilingual extension to TextArena and evaluate three open-weight models across eight languages and six games covering spatial reasoning, imperfect information, resource allocation, and repeated interaction. We find that the same model can exhibit markedly different playing strength across languages, with systematic variation in win--loss margins, invalid actions, and strategic tendencies. Detailed analyses reveal language-specific failures in spatial reasoning, card-conditioned decisions, and optimal move selection. In some settings, changing only the intermediate reasoning language recovers much of the lost performance, suggesting that language can affect different stages of the decision process. These results show that skill discrepancies are a measurable major roadblock in the development of truly multilingual models. Better understanding these discrepancies can help us design models that perform more equitably across languages.

Questions this paper answers

is there research on whether a language model is better at playing games in some languages than in others?
how is cross-lingual skill inconsistency measured separately from cross-lingual knowledge transfer?
how do I test whether a model's reasoning skills survive being asked in a different language?
should I expect the same agent behaviour from one model across the languages my users write in?: Multilingual self-play measures whether an LLM's skills, as distinct from its stored knowledge, transfer across languages by having 2 instances of one model compete through different language interfaces.
Holds for: As of the 2026 preprint, framed against earlier cross-lingual work on knowledge retrieval and on static benchmarks. Evidence comes from 3 models of 3B to 4B parameters, 8 languages and 6 games.; English is the strongest interface language and Hebrew among the weakest for all 3 models tested, aggregated over 6 games of self-play. Qwen3-4B shows the sharpest language hierarchy and Gemma-4-E4B-it the flattest.
Holds for: Self-play between 2 instances of one model, so margins compare languages within a model and never across models. Gemma-4-E4B-it, Qwen3-4B and Ministral3-3B-Instruct on 8 languages.
what should I read first about language models performing unequally across languages for reasons other than missing facts?
what work established that an LLM's interactive skills, not just its factual recall, are language-dependent?
is there a paper I can cite for skill gaps across languages inside one model?: Multilingual self-play measures whether an LLM's skills, as distinct from its stored knowledge, transfer across languages by having 2 instances of one model compete through different language interfaces.
Holds for: As of the 2026 preprint, framed against earlier cross-lingual work on knowledge retrieval and on static benchmarks. Evidence comes from 3 models of 3B to 4B parameters, 8 languages and 6 games.; Multilingual TextArena covers 65 TextArena games in 193 languages, rendering each player's instructions and board in a different language. The rules, legal actions, rewards and transition dynamics are unchanged across languages.
Holds for: Verification is tiered. 8 Tier-A languages were checked by native speakers, 42 Tier-B by closed-model back-translation judging, and 142 more by NLLB-200 round-trip agreement.
is there a set of text games translated into many languages for testing language models?
what multilingual extension of TextArena renders observations in a target language without changing rules or legal actions?
where do I get game environments in many languages so I can run the same game in 2 languages at once?
can I reuse an existing multilingual game suite instead of translating environments myself?: Multilingual TextArena covers 65 TextArena games in 193 languages, rendering each player's instructions and board in a different language. The rules, legal actions, rewards and transition dynamics are unchanged across languages.
Holds for: Verification is tiered. 8 Tier-A languages were checked by native speakers, 42 Tier-B by closed-model back-translation judging, and 142 more by NLLB-200 round-trip agreement.
which language is a language model strongest at when it plays games against itself?
which interface language gives the highest role-pooled win-loss margin in multilingual self-play?
if I run an agent in Hebrew instead of English, how much playing strength am I giving up?: English is the strongest interface language and Hebrew among the weakest for all 3 models tested, aggregated over 6 games of self-play. Qwen3-4B shows the sharpest language hierarchy and Gemma-4-E4B-it the flattest.
Holds for: Self-play between 2 instances of one model, so margins compare languages within a model and never across models. Gemma-4-E4B-it, Qwen3-4B and Ministral3-3B-Instruct on 8 languages.
does the choice of game change how much language matters?
which game environments show the largest max-minus-min language gap in mean language margin?
how do I pick a game that will expose language effects rather than hide them?: Colonel Blotto is the most language-sensitive of the 6 games for all 3 models, with a mean gap of 1.07 between a model's strongest and weakest language. Kuhn Poker is the least sensitive at 0.13.
Holds for: Gap is the spread of mean language margins within one model and game, averaged over the 3 models. 8 languages, 400 self-play games per direction.
can a model do better in a weak language if it thinks in a stronger one?
does decoupling the reasoning language from the environment interface language recover the margin lost under a weak interface?
how do I recover accuracy when my prompts have to stay in a low-performing language?
should I make my agent reason in English while keeping the user-facing language unchanged?: Switching only Gemma-4-E4B-it's reasoning language to English, with the TicTacToe interface still in German, moves the mean language margin from -0.22 to +0.20. That recovers 89.4% of the gap to the English-interface ceiling of +0.25.
Holds for: Gemma-4-E4B-it only, with German as the weak interface. SimpleTak recovers 60.5% the same way, and Kuhn Poker recovery is small and non-monotonic across reasoning languages.
do language models misread game boards differently depending on the language of the instructions?
how does the distribution of TicTacToe defeats over rows, columns and diagonals shift with the interface language?
will an Arabic or Hebrew interface make my agent miss board threats that it catches in English?: Gemma-4-E4B-it's TicTacToe losses split fairly evenly in English across rows at 28.5%, columns at 34.8% and diagonals at 36.7%. Under an Arabic interface the same model loses to rows only 20.3% of the time and to diagonals 45.3%.
Holds for: Gemma-4-E4B-it in TicTacToe. Hebrew skews the same way as Arabic, while Qwen3-4B's defeat distribution stays stable across languages apart from Hebrew.; In SimpleTak, Gemma-4-E4B-it loses to a completed column in 46.8% of English-interface defeats but 57.2% of Arabic and Hebrew ones, against 53.2% row losses in English.
Holds for: Gemma-4-E4B-it in SimpleTak, where over 90% of its wins come from straight lines, which is what lets row and column losses stand for separate spatial axes.
can a model know the right strategy for a game and still fail to use it in one language?
does optimal-move execution in Nim track optimal-strategy mention counts across interface languages?
how do I tell whether a language gap is missing knowledge or failed execution?: Qwen3-4B plays Nim's optimal first move 80.8% of the time under an English interface but only 24.6% under French. Both languages produce a similar number of optimal-strategy mentions in the game logs.
Holds for: Qwen3-4B in Nim with a board where the first player has exactly 1 optimal move. Hebrew and Arabic fall to 4.0% and 10.5%, while Gemma-4-E4B-it stays at 99.2% or above in every language.
why does a model sometimes start writing in English in the middle of an Arabic answer?
what share of optimal-strategy mentions under Arabic and Hebrew interfaces come from mid-reasoning script switching?
if my model switches script mid-answer, is that a bug or is it reaching knowledge it otherwise cannot?: 70% of Ministral3-3B-Instruct's Arabic and 50% of its Hebrew optimal-strategy mentions in Nim come from logs where the model switched into a Latin script mid-reasoning. Such switching occurs in only 3.7% of Arabic and 1% of Hebrew logs.
Holds for: Ministral3-3B-Instruct in Nim only, counted over game-log mentions rather than over model internals, and the gap between languages may therefore be wider than the mention counts suggest.
does a model take different risks when the same card game is presented in a different language?
how do card-conditioned bet, call and fold probabilities in Kuhn Poker vary with interface language?
can I assume my model's decision policy is the same across the languages it is deployed in?: Gemma-4-E4B-it bets the middle card Q in Kuhn Poker 41.6% of the time under a Malay interface and 63.3% under Hebrew. Its play of clearly weak and clearly strong cards varies far less across languages.
Holds for: Gemma-4-E4B-it in Kuhn Poker across 8 languages. Ministral3-3B-Instruct instead folds the strongest card K when facing a bet between 10.4% and 23.7% of the time.
can multilingual benchmark scores tell me how a model will behave in each language?
how strongly does mean language margin correlate with Global MMLU and Belebele accuracy, and does either predict cross-language spread?
how do I predict which languages my agent will be weak in without running the agent?
is a high multilingual benchmark average enough to tell me a model is consistent across languages?: Mean language margin correlates with 5-shot Global MMLU accuracy at Pearson r between 0.73 and 0.92 depending on the model, and with Belebele at 0.71 to 0.79. Static benchmark accuracy does not predict a model's spread across languages.
Holds for: 8 languages per model, so each correlation has n=8. Gemma-4-E4B-it has the lowest Global MMLU mean at 49.9% yet the smallest cross-language spread, and Qwen3-4B the highest mean with the largest spread.
is a language weak in a model just because there is little text in it on the web?
how much of the language hierarchy in self-play is explained by FineWeb-2 per-language word counts, and which languages deviate?
should I assume a low-resource language will be my model's weakest interface?: Language strength correlates with available web text at an average Pearson r of 0.79 against log FineWeb-2 word counts, and the residuals are large. Malay beats Hebrew for every model despite the smaller FineWeb-2 corpus.
Holds for: Web text is a proxy for training data, which is undisclosed for all 3 models. Chinese has roughly 20 times less web text than English yet reaches near-English margins for Qwen3-4B and Ministral3-3B-Instruct.
how many games were played to measure the language differences?
what is the self-play sample size per language pair, model and game environment?
how many matches do I need per language pair before a win-loss margin is worth reading?: The multilingual self-play evaluation runs 518,400 games in total, 28,800 per model and game pair, with 400 games for every ordered language pair and player role assignment.
Holds for: 3 models by 6 games by all 8 languages, inference only. Each primary run used 2 NVIDIA H200 GPUs and about 6 H200 GPU-hours, excluding Iterated Prisoner's Dilemma.

Claims and scope

Multilingual TextArena covers 65 TextArena games in 193 languages, rendering each player's instructions and board in a different language. The rules, legal actions, rewards and transition dynamics are unchanged across languages.
Scope: Verification is tiered. 8 Tier-A languages were checked by native speakers, 42 Tier-B by closed-model back-translation judging, and 142 more by NLLB-200 round-trip agreement.
Multilingual self-play measures whether an LLM's skills, as distinct from its stored knowledge, transfer across languages by having 2 instances of one model compete through different language interfaces.
Scope: As of the 2026 preprint, framed against earlier cross-lingual work on knowledge retrieval and on static benchmarks. Evidence comes from 3 models of 3B to 4B parameters, 8 languages and 6 games.
English is the strongest interface language and Hebrew among the weakest for all 3 models tested, aggregated over 6 games of self-play. Qwen3-4B shows the sharpest language hierarchy and Gemma-4-E4B-it the flattest. (Figure 2)
Scope: Self-play between 2 instances of one model, so margins compare languages within a model and never across models. Gemma-4-E4B-it, Qwen3-4B and Ministral3-3B-Instruct on 8 languages.
Colonel Blotto is the most language-sensitive of the 6 games for all 3 models, with a mean gap of 1.07 between a model's strongest and weakest language. Kuhn Poker is the least sensitive at 0.13. (Table 2)
Scope: Gap is the spread of mean language margins within one model and game, averaged over the 3 models. 8 languages, 400 self-play games per direction.
Switching only Gemma-4-E4B-it's reasoning language to English, with the TicTacToe interface still in German, moves the mean language margin from -0.22 to +0.20. That recovers 89.4% of the gap to the English-interface ceiling of +0.25. (Table 3)
Scope: Gemma-4-E4B-it only, with German as the weak interface. SimpleTak recovers 60.5% the same way, and Kuhn Poker recovery is small and non-monotonic across reasoning languages.
Gemma-4-E4B-it's TicTacToe losses split fairly evenly in English across rows at 28.5%, columns at 34.8% and diagonals at 36.7%. Under an Arabic interface the same model loses to rows only 20.3% of the time and to diagonals 45.3%. (Appendix G)
Scope: Gemma-4-E4B-it in TicTacToe. Hebrew skews the same way as Arabic, while Qwen3-4B's defeat distribution stays stable across languages apart from Hebrew.
In SimpleTak, Gemma-4-E4B-it loses to a completed column in 46.8% of English-interface defeats but 57.2% of Arabic and Hebrew ones, against 53.2% row losses in English. (Appendix G)
Scope: Gemma-4-E4B-it in SimpleTak, where over 90% of its wins come from straight lines, which is what lets row and column losses stand for separate spatial axes.
Qwen3-4B plays Nim's optimal first move 80.8% of the time under an English interface but only 24.6% under French. Both languages produce a similar number of optimal-strategy mentions in the game logs. (Appendix G)
Scope: Qwen3-4B in Nim with a board where the first player has exactly 1 optimal move. Hebrew and Arabic fall to 4.0% and 10.5%, while Gemma-4-E4B-it stays at 99.2% or above in every language.
70% of Ministral3-3B-Instruct's Arabic and 50% of its Hebrew optimal-strategy mentions in Nim come from logs where the model switched into a Latin script mid-reasoning. Such switching occurs in only 3.7% of Arabic and 1% of Hebrew logs. (Appendix G)
Scope: Ministral3-3B-Instruct in Nim only, counted over game-log mentions rather than over model internals, and the gap between languages may therefore be wider than the mention counts suggest.
Gemma-4-E4B-it bets the middle card Q in Kuhn Poker 41.6% of the time under a Malay interface and 63.3% under Hebrew. Its play of clearly weak and clearly strong cards varies far less across languages. (Appendix G)
Scope: Gemma-4-E4B-it in Kuhn Poker across 8 languages. Ministral3-3B-Instruct instead folds the strongest card K when facing a bet between 10.4% and 23.7% of the time.
Mean language margin correlates with 5-shot Global MMLU accuracy at Pearson r between 0.73 and 0.92 depending on the model, and with Belebele at 0.71 to 0.79. Static benchmark accuracy does not predict a model's spread across languages. (Figure 4)
Scope: 8 languages per model, so each correlation has n=8. Gemma-4-E4B-it has the lowest Global MMLU mean at 49.9% yet the smallest cross-language spread, and Qwen3-4B the highest mean with the largest spread.
Language strength correlates with available web text at an average Pearson r of 0.79 against log FineWeb-2 word counts, and the residuals are large. Malay beats Hebrew for every model despite the smaller FineWeb-2 corpus. (Figure 4)
Scope: Web text is a proxy for training data, which is undisclosed for all 3 models. Chinese has roughly 20 times less web text than English yet reaches near-English margins for Qwen3-4B and Ministral3-3B-Instruct.
The multilingual self-play evaluation runs 518,400 games in total, 28,800 per model and game pair, with 400 games for every ordered language pair and player role assignment. (Section 3)
Scope: 3 models by 6 games by all 8 languages, inference only. Each primary run used 2 NVIDIA H200 GPUs and about 6 H200 GPU-hours, excluding Iterated Prisoner's Dilemma.

Common misreadings

A 193-language release is not 193 verified languages. 8 languages were verified by native speakers and the rest by automatic back-translation judging, which is why the experiments use the 8.
Reasoning in a stronger language is not a general repair. Recovery is large in TicTacToe and SimpleTak and small and non-monotonic in Kuhn Poker.
The language a model wins with is not simply the language it has most training data in. Malay outperforms Hebrew for every model despite a smaller web corpus, and Chinese reaches near-English margins with far less web text.
A high multilingual benchmark average does not mean uniform behaviour across languages. Benchmark level and cross-language spread move independently in these models.
The results are measured on models of 3B to 4B parameters and do not establish that language effects hold at larger scale.

Terminology in this paper

language interface: the language in which a game's instructions, board and observations are presented to a player, holding the rules, legal actions and rewards fixed.
role-pooled win-loss margin: wins minus losses divided by total games for one language against another, pooled over both player-role assignments so first-move advantage cancels.
mean language margin: one language's average role-pooled win-loss margin against every other language evaluated, for a fixed model and game.
Tier A: the 8 languages whose game translations were verified by native speakers, as opposed to tiers verified by automatic back-translation judging.

How to cite

@misc{cheng2026skill,
  title = {Skill Issue: Are Skills Language-Invariant in LLMs?},
  year = {2026},
  eprint = {2608.25832},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.25832}
}

References

See the full reference list in the paper.