Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs

Cross-LIngual Knowledge Editing — a 13-language fill-in-the-blank factual probing and editing dataset

Maxim Ifergan, Leshem Choshen, Roee Aharoni, Idan Szpektor, Omri Abend · Findings of NAACL 2025 · 2025

• arXiv
• PDF
• HTML
• Hugging Face
• alphaXiv
• DOI
• Semantic Scholar
• Publisher

In one sentence

Repurposing knowledge-editing methods (ROME, MEMIT, finetuning) as a probe, this work edits a fact in one language and measures whether the edit propagates to other languages, separating surface-level cross-lingual answer consistency from genuinely shared internal representations in 7B LLMs.

Abstract

The veracity of a factoid is largely independent of the language it is written in. However, language models are inconsistent in their ability to answer the same factual question across languages. This raises questions about how LLMs represent a given fact across languages. We explore multilingual factual knowledge through two aspects: the model's ability to answer a query consistently across languages, and the ability to ''store'' answers in a shared representation for several languages. We propose a methodology to measure the extent of representation sharing across languages by repurposing knowledge editing methods. We examine LLMs with various multilingual configurations using a new multilingual dataset. We reveal that high consistency does not necessarily imply shared representation, particularly for languages with different scripts. Moreover, we find that script similarity is a dominant factor in representation sharing. Finally, we observe that if LLMs could fully share knowledge across languages, their accuracy in their best-performing language could benefit an increase of up to 150\% on average. These findings highlight the need for improved multilingual knowledge representation in LLMs and suggest a path for the development of more robust and consistent multilingual LLMs.

Questions this paper answers

if a chatbot gives the same right answer in two languages, does that mean it learned the fact once?

is cross-lingual answer consistency a valid proxy for shared factual knowledge representation in multilingual LLMs?

how do I tell whether a multilingual model stores a fact once or separately per language rather than just checking that its answers agree?

can I trust matching answers across languages as evidence that my model has one shared copy of a fact?

In four 7B LLMs, the expected number of languages in which a fact is answered consistently exceeds the number of languages an edit to that fact propagates to. Cross-lingual answer agreement therefore does not imply a shared internal representation.
Holds for: BLOOM-7B, Qwen-7B, Llama-2-7B and Mistral-7B-v0.1 on CLIKE's 13 languages, with sharing measured via MEMIT edits on 500 known facts per language; the gap is largest for language pairs using different scripts.

Chinese, Japanese, Hebrew and Arabic show high cross-lingual answer consistency with each other in all models except Qwen. Edits made in one of them nevertheless show limited propagation to the others.
Holds for: Pairwise MEMIT-based generalization scores on CLIKE; measured on base pretrained 7B decoder-only models, not instruction-tuned ones.

Cyrillic languages (Russian, Ukrainian, Bulgarian) share more representation with each other than with Latin-script languages. Answer consistency runs the opposite way, being higher between Cyrillic and Latin than within Cyrillic.
Holds for: CLIKE facts, pairwise C and SR matrices computed with MEMIT on the four base 7B models; a within-versus-across-script comparison, not a claim about any single language pair.

which pairs of languages does a language model actually share facts between?

does script family predict cross-lingual knowledge representation sharing between languages in 7B LLMs?

how do I predict which target languages will benefit when a fact is learned in one language?

my target languages use different alphabets — should I expect factual knowledge to carry over between them?

Languages within the same script family exhibit the highest degree of cross-lingual knowledge representation sharing in every model examined, including Latin (English, French, Italian, Spanish) and Cyrillic (Russian, Ukrainian, Bulgarian) groups.
Holds for: Multilingual BLOOM-7B, bilingual Qwen-7B and English monolingual Llama-2-7B and Mistral-7B-v0.1 on CLIKE; Devanagari-script sharing (Hindi, Bengali) is high only for models that perform well on those languages (BLOOM, Mistral).

Cyrillic languages (Russian, Ukrainian, Bulgarian) share more representation with each other than with Latin-script languages. Answer consistency runs the opposite way, being higher between Cyrillic and Latin than within Cyrillic.
Holds for: CLIKE facts, pairwise C and SR matrices computed with MEMIT on the four base 7B models; a within-versus-across-script comparison, not a claim about any single language pair.

how can someone test whether a language model keeps one copy of a fact for many languages?

how does model editing serve as a causal probe of cross-lingual factual representation sharing, instead of activation similarity or neuron overlap?

how do I measure cross-lingual knowledge sharing in a multilingual model without relying on hidden-state similarity metrics?

is editing a fact in one language and querying the others a measurement I can rely on for my own models?

Editing-based probing measures cross-lingual knowledge representation sharing by editing a fact in one language and testing whether the false target propagates to the same fact queried in other languages. It gives a causal alternative to activation-similarity and neuron-overlap analyses.
Holds for: Earlier cross-lingual representation work used passive analyses that indicate connection but do not quantify how much knowledge is shared; as of publication in 2025, and demonstrated only on 7B decoder-only models with middle-layer editing methods.

Representation-sharing measurements agree across the three editing methods used as probes, with a 0.87 correlation between methods and locality scores averaging above 70%, indicating edits were specific rather than broadly destructive.
Holds for: ROME, MEMIT and a finetuning baseline applied through EasyEdit to 500 known facts per language; custom hyperparameters were tuned for BLOOM, defaults used elsewhere. Main results are reported with MEMIT.

how many facts does a 7B model know in some language but get wrong in the language it is best at?

what is the gap between union-over-languages accuracy and best-single-language accuracy on multilingual factual probing?

how much accuracy could I recover by getting a model to reuse facts it already knows in other languages?

is there enough headroom in cross-lingual knowledge transfer to be worth chasing for my multilingual QA system?

Four 7B LLMs answer 42.5% of CLIKE facts correctly in at least one language, but only 27.6% in their best-performing language and 11.8% averaged over all 13 languages. Fully shared knowledge would raise the best language by up to 53%.
Holds for: Exact-match accuracy with 3-shot demonstrations and greedy decoding, counting a fact as known if any of 3 paraphrases is answered correctly; the headroom is an upper bound assuming perfect sharing, not an achieved improvement.

does knowing a fact in Russian make a model more likely to know it in English than the other way round?

is cross-lingual factual knowledge transfer between Cyrillic and Latin scripts directionally asymmetric?

which direction should I probe or edit in when I want a fact to hold in both a Cyrillic and a Latin-script language?

if my model answers correctly in Ukrainian, can I assume it will answer correctly in English too?

Knowing a fact in a Cyrillic-script language implies roughly 40-60% probability of knowing it in a Latin-script language, while the reverse direction holds only about 10-20% of the time.
Holds for: Pairwise consistency by exact match on CLIKE across the four base 7B models; the attribution to Latin-script pretraining dominance is a hypothesis, not a tested mechanism.

do English-only 7B models share any factual knowledge with other languages, and do multilingual ones do better?

how do monolingual English and multilingual 7B pretraining regimes differ in within-script and cross-script representation sharing?

which base model should I start from if I need facts to be shared across languages written in different scripts?

should I pick a multilingual base model over an English-centric one for cross-lingual factual coverage?

English monolingual Mistral-7B shows the highest within-script-family pairwise consistency (54.7%) and representation sharing (37.6%) of any model tested. Its facts peak anomalously at being known and represented in exactly 4 languages, the 4 Latin-script ones.
Holds for: Mistral-7B-v0.1 and Llama-2-7B evaluated on CLIKE's 13 languages; within-script-family averages, which are not comparable to the cross-script averages reported for BLOOM.

Multilingual BLOOM-7B attains the highest cross-script pairwise averages of the models tested, at 36% consistency and 8.4% representation sharing, including 28% sharing from Italian to Hindi.
Holds for: Averages over language pairs from different script families on CLIKE, compared against Qwen-7B, Llama-2-7B and Mistral-7B-v0.1; BLOOM's overall retrieval accuracy is low.

what happens to a model's English facts when it is further trained on Chinese or Hebrew?

does continued pretraining for language extension produce cross-script factual representation sharing, and at what cost to source-language accuracy?

how do I add a new language to a 7B model without losing the facts it already answers in English?

if I continue pretraining my model on Hebrew, will its English factual accuracy suffer?

Chinese-Llama-2-7B and Hebrew-Mistral-7B gain large accuracy in their extended language, 142% and 600% relative increases, while losing English accuracy by 29% and 32% relative. English-to-extended-language representation sharing stays in the single digits, at 4% and 6%.
Holds for: Two language-extended 7B models compared to their Llama-2-7B and Mistral-7B-v0.1 bases, with expanded tokenizer vocabularies and continued pretraining in English plus the extended language.

which kinds of facts travel best between languages — dates, countries, or people's names?

how does relation type affect cross-lingual factual representation sharing for closed-category and numeric versus name-valued objects?

which relation types should I expect to need language-specific data for, and which will transfer?

my knowledge base is mostly person names — should I expect those facts to transfer across languages?

Relations with few possible answer categories (countries, instruments, continents, company developers) and numerical relations such as birth year and death year show the higher cross-lingual representation sharing. Name-valued relations such as book authors, movie directors and discoverers transfer less.
Holds for: CLIKE's relation types under MEMIT editing; sports type and religion are exceptions to the low-cardinality trend, and high-cardinality relations such as cities also transfer poorly.

is there a ready-made dataset of facts in many languages for testing and editing what a model knows?

which multilingual cloze-style factual probing and editing benchmark covers 13 languages across multiple scripts with paraphrased templates?

where do I get parallel fill-in-the-blank fact templates in many languages to run editing experiments?

can I use an existing multilingual fact-editing benchmark instead of translating my own probes?

CLIKE is a multilingual fill-in-the-blank factual probing and editing dataset of about 35k facts in 13 languages spanning 7 scripts. Each (subject, relation, object) triplet has 3 paraphrased templates per language, verified by professional translators or native speakers.
Holds for: English, French, Italian, Spanish, Russian, Ukrainian, Bulgarian, Hindi, Bengali, Chinese, Japanese, Hebrew and Arabic; Wikidata triplets from 14 SPARQL queries, keeping only triplets labelled in at least 8 of these languages.

where should someone start reading about how language models store facts in more than one language?

what work distinguishes cross-lingual consistency from cross-lingual knowledge representation sharing in multilingual LLMs?

what should I read first to understand multilingual factual knowledge and editing in LLMs?

Editing-based probing measures cross-lingual knowledge representation sharing by editing a fact in one language and testing whether the false target propagates to the same fact queried in other languages. It gives a causal alternative to activation-similarity and neuron-overlap analyses.
Holds for: Earlier cross-lingual representation work used passive analyses that indicate connection but do not quantify how much knowledge is shared; as of publication in 2025, and demonstrated only on 7B decoder-only models with middle-layer editing methods.

In four 7B LLMs, the expected number of languages in which a fact is answered consistently exceeds the number of languages an edit to that fact propagates to. Cross-lingual answer agreement therefore does not imply a shared internal representation.
Holds for: BLOOM-7B, Qwen-7B, Llama-2-7B and Mistral-7B-v0.1 on CLIKE's 13 languages, with sharing measured via MEMIT edits on 500 known facts per language; the gap is largest for language pairs using different scripts.

CLIKE is a multilingual fill-in-the-blank factual probing and editing dataset of about 35k facts in 13 languages spanning 7 scripts. Each (subject, relation, object) triplet has 3 paraphrased templates per language, verified by professional translators or native speakers.
Holds for: English, French, Italian, Spanish, Russian, Ukrainian, Bulgarian, Hindi, Bengali, Chinese, Japanese, Hebrew and Arabic; Wikidata triplets from 14 SPARQL queries, keeping only triplets labelled in at least 8 of these languages.

do different fact-editing techniques agree about how much knowledge crosses between languages?

are cross-lingual representation-sharing estimates stable across ROME, MEMIT and finetuning edits, and do locality scores show the edits stayed specific?

how do I check that my cross-lingual editing measurements are not an artefact of the editing algorithm or of collateral damage?

does my choice of editing method change the conclusions I draw about multilingual knowledge sharing?

Representation-sharing measurements agree across the three editing methods used as probes, with a 0.87 correlation between methods and locality scores averaging above 70%, indicating edits were specific rather than broadly destructive.
Holds for: ROME, MEMIT and a finetuning baseline applied through EasyEdit to 500 known facts per language; custom hyperparameters were tuned for BLOOM, defaults used elsewhere. Main results are reported with MEMIT.

Claims and scope

• In four 7B LLMs, the expected number of languages in which a fact is answered consistently exceeds the number of languages an edit to that fact propagates to. Cross-lingual answer agreement therefore does not imply a shared internal representation. (Figure 3)
  Scope: BLOOM-7B, Qwen-7B, Llama-2-7B and Mistral-7B-v0.1 on CLIKE's 13 languages, with sharing measured via MEMIT edits on 500 known facts per language; the gap is largest for language pairs using different scripts.
• Chinese, Japanese, Hebrew and Arabic show high cross-lingual answer consistency with each other in all models except Qwen. Edits made in one of them nevertheless show limited propagation to the others. (Figure 4)
  Scope: Pairwise MEMIT-based generalization scores on CLIKE; measured on base pretrained 7B decoder-only models, not instruction-tuned ones.
• Cyrillic languages (Russian, Ukrainian, Bulgarian) share more representation with each other than with Latin-script languages. Answer consistency runs the opposite way, being higher between Cyrillic and Latin than within Cyrillic. (Figure 4)
  Scope: CLIKE facts, pairwise C and SR matrices computed with MEMIT on the four base 7B models; a within-versus-across-script comparison, not a claim about any single language pair.
• Languages within the same script family exhibit the highest degree of cross-lingual knowledge representation sharing in every model examined, including Latin (English, French, Italian, Spanish) and Cyrillic (Russian, Ukrainian, Bulgarian) groups. (Figure 4)
  Scope: Multilingual BLOOM-7B, bilingual Qwen-7B and English monolingual Llama-2-7B and Mistral-7B-v0.1 on CLIKE; Devanagari-script sharing (Hindi, Bengali) is high only for models that perform well on those languages (BLOOM, Mistral).
• Knowing a fact in a Cyrillic-script language implies roughly 40-60% probability of knowing it in a Latin-script language, while the reverse direction holds only about 10-20% of the time. (Figure 4)
  Scope: Pairwise consistency by exact match on CLIKE across the four base 7B models; the attribution to Latin-script pretraining dominance is a hypothesis, not a tested mechanism.
• Four 7B LLMs answer 42.5% of CLIKE facts correctly in at least one language, but only 27.6% in their best-performing language and 11.8% averaged over all 13 languages. Fully shared knowledge would raise the best language by up to 53%. (Figure 2)
  Scope: Exact-match accuracy with 3-shot demonstrations and greedy decoding, counting a fact as known if any of 3 paraphrases is answered correctly; the headroom is an upper bound assuming perfect sharing, not an achieved improvement.
• English monolingual Mistral-7B shows the highest within-script-family pairwise consistency (54.7%) and representation sharing (37.6%) of any model tested. Its facts peak anomalously at being known and represented in exactly 4 languages, the 4 Latin-script ones. (Figure 3)
  Scope: Mistral-7B-v0.1 and Llama-2-7B evaluated on CLIKE's 13 languages; within-script-family averages, which are not comparable to the cross-script averages reported for BLOOM.
• Multilingual BLOOM-7B attains the highest cross-script pairwise averages of the models tested, at 36% consistency and 8.4% representation sharing, including 28% sharing from Italian to Hindi. (Figure 4)
  Scope: Averages over language pairs from different script families on CLIKE, compared against Qwen-7B, Llama-2-7B and Mistral-7B-v0.1; BLOOM's overall retrieval accuracy is low.
• Chinese-Llama-2-7B and Hebrew-Mistral-7B gain large accuracy in their extended language, 142% and 600% relative increases, while losing English accuracy by 29% and 32% relative. English-to-extended-language representation sharing stays in the single digits, at 4% and 6%. (Table 1)
  Scope: Two language-extended 7B models compared to their Llama-2-7B and Mistral-7B-v0.1 bases, with expanded tokenizer vocabularies and continued pretraining in English plus the extended language.
• Representation-sharing measurements agree across the three editing methods used as probes, with a 0.87 correlation between methods and locality scores averaging above 70%, indicating edits were specific rather than broadly destructive. (Section 4)
  Scope: ROME, MEMIT and a finetuning baseline applied through EasyEdit to 500 known facts per language; custom hyperparameters were tuned for BLOOM, defaults used elsewhere. Main results are reported with MEMIT.
• Relations with few possible answer categories (countries, instruments, continents, company developers) and numerical relations such as birth year and death year show the higher cross-lingual representation sharing. Name-valued relations such as book authors, movie directors and discoverers transfer less. (Figure 7)
  Scope: CLIKE's relation types under MEMIT editing; sports type and religion are exceptions to the low-cardinality trend, and high-cardinality relations such as cities also transfer poorly.
• Editing-based probing measures cross-lingual knowledge representation sharing by editing a fact in one language and testing whether the false target propagates to the same fact queried in other languages. It gives a causal alternative to activation-similarity and neuron-overlap analyses. (Section 5)
  Scope: Earlier cross-lingual representation work used passive analyses that indicate connection but do not quantify how much knowledge is shared; as of publication in 2025, and demonstrated only on 7B decoder-only models with middle-layer editing methods.
• CLIKE is a multilingual fill-in-the-blank factual probing and editing dataset of about 35k facts in 13 languages spanning 7 scripts. Each (subject, relation, object) triplet has 3 paraphrased templates per language, verified by professional translators or native speakers. (Section 3.1)
  Scope: English, French, Italian, Spanish, Russian, Ukrainian, Bulgarian, Hindi, Bengali, Chinese, Japanese, Hebrew and Arabic; Wikidata triplets from 14 SPARQL queries, keeping only triplets labelled in at least 8 of these languages.

How to cite

@inproceedings{ifergan2024beneath,
    title = "Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in {LLM}s",
    author = "Ifergan, Maxim  and
      Choshen, Leshem  and
      Aharoni, Roee  and
      Szpektor, Idan  and
      Abend, Omri",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-naacl.475/",
    pages = "4630--4644",
    ISBN = "979-8-89176-195-7"
}

References

See the full reference list in the paper.