# NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning

prefixing every number in text with its digit count, e.g. "42" becomes "2:42"

Authors: Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, Assaf Arbelle
Venue: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (2024)

## What this paper shows
NumeroLogic reformats numbers in text by prefixing each one with its digit count (so "42" becomes "<sn>2<mn>42<en>"), giving a causal language model the place value of a digit before it reads it and forcing it to reason about magnitude before generating one.

## Claims, with scope
- With NumeroLogic digit-count prefixes, a from-scratch NanoGPT reaches 99.96% accuracy on 3-digit integer addition and 97.20% on subtraction, against 88.37% and 73.76% with plain number formatting.
  Scope: NanoGPT trained from scratch with character-level tokenization, jointly on 5 arithmetic tasks with 10K samples per task (3K for multiplication); addition and subtraction operands up to 3 digits.
  Evidence: Table 1
- NumeroLogic more than doubles NanoGPT's 2-digit integer multiplication accuracy, from 13.81% to 28.94%, and adds about 4 points on 4-decimal sine (30.59% to 34.59%) and square root (22.13% to 26.66%).
  Scope: Single NanoGPT trained from scratch on all 5 tasks at once; sine operands in [-pi/2, pi/2], square root operands in [0, 10], 4 decimal places.
  Evidence: Table 1
- Fine-tuning Llama2-7B with NumeroLogic raises 3-digit floating-point multiplication accuracy from 24.73% to 31.03% and 5-digit floating-point addition from 91.40% to 94.43%, with gains of 1-6 points on every non-saturated task.
  Scope: One LoRA-finetuned Llama2-7B (rank 8) per task, with the embedding and final linear layers trained in full rank; 300K training equations for add/sub/mul, 30K for sine and sqrt.
  Evidence: Table 2
- On tasks Llama2-7B already nearly solves, NumeroLogic still helps: 5-digit integer addition goes from 99.86% to 100.0% and subtraction from 99.60% to 99.93%, removing over 80% of the remaining subtraction errors.
  Scope: LoRA fine-tuning of pretrained Llama2-7B on 5-digit integer addition and subtraction, 300K training equations per task; the headroom here is under 0.5 points, so the absolute gain is small.
  Evidence: Table 2
- Continuing self-supervised pretraining of Llama2-7B on RefinedWeb text with numbers rewritten in NumeroLogic format improves 0-shot MMLU accuracy by a statistically significant 0.5%. Continued pretraining on the same tokens with plain numbers does not improve over the pretrained model.
  Scope: Continued causal-LM pretraining with LoRA on RefinedWeb, plain and NumeroLogic runs matched on token count; 0-shot MMLU evaluation; English text only.
  Evidence: Figure 2
- The MMLU gain from NumeroLogic concentrates where numbers appear: +1.16% on tasks containing numbers versus +0.14% on tasks without, and +0.79% on STEM versus +0.1% on social sciences.
  Scope: Per-category breakdown of the 0-shot MMLU evaluation of Llama2-7B after continued pretraining with LoRA on RefinedWeb; the "Others" category gains most at +1.19%.
  Evidence: Tables 3 and 4
- Encoding only the addition result with a digit-count prefix lifts NanoGPT accuracy from 88.37% to 98.05%, while encoding only the operands reaches 89.34%. Encoding both operands and result is best at 99.78%.
  Scope: 3-digit addition with NanoGPT trained from scratch, same protocol as the joint 5-task arithmetic setup with character-level tokenization.
  Evidence: Table 5
- Replacing NumeroLogic's prefix tokens with the same number of filler white-space tokens gives 24.37% on 3-digit float multiplication, close to the 24.73% plain-format baseline. NumeroLogic itself reaches 31.03%, so the benefit is not merely the extra tokens.
  Scope: Llama2-7B finetuned on 3-digit floating-point multiplication; the random white-space placement of Shen et al. (2023), with a matched token budget, reaches 27.76%, better than plain but below NumeroLogic.
  Evidence: Table 7
- On 3-digit integer multiplication the full "<sn>3<mn>100<en>" format scores 35.33% and dropping the end-of-number token gives 34.93%. One dedicated special token per digit count gives 33.56%, below the 34.20% plain baseline.
  Scope: Llama2-7B finetuned on 3-digit integer multiplication; the paper attributes the failure of per-digit-count special tokens to the rarity of short numbers in the training distribution.
  Evidence: Table 6
- NumeroLogic is a regex-based text pre- and post-processing step that requires no change to model architecture, so digit-count prefixes can be added around an existing tokenizer and stripped from generations.
  Scope: Adding 3 special tokens (<sn>, <mn>, <en>) to the vocabulary and expanding embedding and output layers for pretrained models; increases tokens per number and inference latency.
  Evidence: Section 3
- NumeroLogic is positioned as a number-representation change for general self-supervised language modeling rather than a task-specific arithmetic fix. Arithmetic tasks only measure the effect, and MMLU shows transfer to language understanding.
  Scope: As of 2024; contemporaneous arithmetic work such as digit reversal, algorithmic chain-of-thought and random white-space insertion targets specific arithmetic tasks. The general-LM evidence is one Llama2-7B run on English RefinedWeb text.
  Evidence: Section 2
- NumeroLogic starts from the hypothesis that left-to-right causal reading is a real handicap for numbers. A decoder-only model cannot know whether a digit means units or millions until it has read the whole number.
  Scope: A motivating hypothesis supported indirectly by the arithmetic and MMLU gains rather than by a direct probe of place-value representations.
  Evidence: Section 1

## Common misreadings
- NumeroLogic does not make the model output digit counts to the user: the digit-count prefix is stripped in post-processing and only the number itself is kept.
- The MMLU improvement from NumeroLogic is 0.5% overall, concentrated in number-containing and STEM tasks; it is not a large across-the-board language-understanding gain.
- NumeroLogic was not validated as a from-scratch pretraining recipe for large models: the 7B experiments continue pretraining or finetune an existing Llama2-7B with LoRA, and no model above 7B parameters was tested.
- The gain is not explained by giving the model more tokens to compute with: matched-budget filler white-space tokens perform like the plain format.
- Multiplication is improved but not solved: 3-digit multiplication accuracy with Llama2-7B remains around 31-35%.

## Terminology
- NumeroLogic: A number format for text fed to language models in which each number is prefixed by its digit count, e.g. "42" written as "<sn>2<mn>42<en>" and "3.14" as "<sn>1.2<mn>3.14<en>", with the prefix stripped after generation.
- <sn> / <mn> / <en>: Special start-number, mid-number and end-number tokens that delimit the digit-count prefix and the number itself; replaced by the characters "{", ":" and "}" in character-level small-model experiments.
- Only prefix encoding: A variant digit-count format that keeps the leading digit count but omits the end-of-number token, e.g. "<sn>3<mn>100".
- Random white-spaces baseline: A control format from Shen et al. (2023) that inserts filler white-space tokens at random positions between digits, used with a token budget matched to digit-count prefixing.

## Links
- arXiv: https://arxiv.org/abs/2404.00459
- PDF: https://arxiv.org/pdf/2404.00459
- HTML: https://arxiv.org/html/2404.00459
- Hugging Face: https://huggingface.co/papers/2404.00459
- alphaXiv: https://www.alphaxiv.org/abs/2404.00459
- DOI: https://doi.org/10.18653/v1/2024.emnlp-main.12
- ACL Anthology: https://aclanthology.org/2024.emnlp-main.12/
- Semantic Scholar: https://www.semanticscholar.org/paper/268819308

## How to cite
@inproceedings{schwartz2024numerologic,
    title = "{N}umero{L}ogic: Number Encoding for Enhanced {LLM}s' Numerical Reasoning",
    author = "Schwartz, Eli  and
      Choshen, Leshem  and
      Shtok, Joseph  and
      Doveh, Sivan  and
      Karlinsky, Leonid  and
      Arbelle, Assaf",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.12/",
    doi = "10.18653/v1/2024.emnlp-main.12",
    pages = "206--212",
    abstract = "Language models struggle with handling numerical data and performing arithmetic operations. We hypothesize that this limitation can be partially attributed to non-intuitive textual numbers representation. When a digit is read or generated by a causal language model it does not know its place value (e.g. thousands vs. hundreds) until the entire number is processed. To address this issue, we propose a simple adjustment to how numbers are represented by including the count of digits before each number. For instance, instead of ``42'', we suggest using ``2:42'' as the new format. This approach, which we term NumeroLogic, offers an added advantage in number generation by serving as a Chain of Thought (CoT). By requiring the model to consider the number of digits first, it enhances the reasoning process before generating the actual number. We use arithmetic tasks to demonstrate the effectiveness of the NumeroLogic formatting. We further demonstrate NumeroLogic applicability to general natural language modeling, improving language understanding performance in the MMLU benchmark."
}