NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning

prefixing every number in text with its digit count, e.g. "42" becomes "2:42"

Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, Assaf Arbelle · EMNLP 2024 · 2024

In one sentence

NumeroLogic reformats numbers in text by prefixing each one with its digit count (so "42" becomes "<sn>2<mn>42<en>"), giving a causal language model the place value of a digit before it reads it and forcing it to reason about magnitude before generating one.

Abstract

Language models struggle with handling numerical data and performing arithmetic operations. We hypothesize that this limitation can be partially attributed to non-intuitive textual numbers representation. When a digit is read or generated by a causal language model it does not know its place value (e.g. thousands vs. hundreds) until the entire number is processed. To address this issue, we propose a simple adjustment to how numbers are represented by including the count of digits before each number. For instance, instead of ``42'', we suggest using ``2:42'' as the new format. This approach, which we term NumeroLogic, offers an added advantage in number generation by serving as a Chain of Thought (CoT). By requiring the model to consider the number of digits first, it enhances the reasoning process before generating the actual number. We use arithmetic tasks to demonstrate the effectiveness of the NumeroLogic formatting. We further demonstrate NumeroLogic applicability to general natural language modeling, improving language understanding performance in the MMLU benchmark.

Questions this paper answers

is there a simple way to rewrite numbers in text so a language model handles them better?
what effect does a digit-count prefix on numeric tokens have on arithmetic and numeric task accuracy?
how do I improve a language model's number handling without touching its architecture or tokenizer?
should I reformat the numbers in my training data instead of changing my model?
NumeroLogic is a regex-based text pre- and post-processing step that requires no change to model architecture, so digit-count prefixes can be added around an existing tokenizer and stripped from generations.
Holds for: Adding 3 special tokens (<sn>, <mn>, <en>) to the vocabulary and expanding embedding and output layers for pretrained models; increases tokens per number and inference latency.
With NumeroLogic digit-count prefixes, a from-scratch NanoGPT reaches 99.96% accuracy on 3-digit integer addition and 97.20% on subtraction, against 88.37% and 73.76% with plain number formatting.
Holds for: NanoGPT trained from scratch with character-level tokenization, jointly on 5 arithmetic tasks with 10K samples per task (3K for multiplication); addition and subtraction operands up to 3 digits.
Fine-tuning Llama2-7B with NumeroLogic raises 3-digit floating-point multiplication accuracy from 24.73% to 31.03% and 5-digit floating-point addition from 91.40% to 94.43%, with gains of 1-6 points on every non-saturated task.
Holds for: One LoRA-finetuned Llama2-7B (rank 8) per task, with the embedding and final linear layers trained in full rank; 300K training equations for add/sub/mul, 30K for sine and sqrt.
why is reading a number one digit at a time from the left hard for a text-generating model?
why does causal left-to-right decoding create a place-value ambiguity when a model reads digit sequences?
how do I let a language model know a number's magnitude before it reads all the digits?
is my model's arithmetic error rate partly caused by not knowing digit place value until the number ends?
NumeroLogic starts from the hypothesis that left-to-right causal reading is a real handicap for numbers. A decoder-only model cannot know whether a digit means units or millions until it has read the whole number.
Holds for: A motivating hypothesis supported indirectly by the arithmetic and MMLU gains rather than by a direct probe of place-value representations.
does rewriting numbers still help a 7-billion-parameter model, or only small models trained from scratch?
do digit-count prefixes yield gains when fine-tuning Llama2-7B on floating-point arithmetic tasks?
how much accuracy can I gain on multi-digit float arithmetic by fine-tuning Llama2-7B with digit-count-prefixed numbers?
my model already gets 5-digit addition almost right, is there anything left for number reformatting to fix?
Fine-tuning Llama2-7B with NumeroLogic raises 3-digit floating-point multiplication accuracy from 24.73% to 31.03% and 5-digit floating-point addition from 91.40% to 94.43%, with gains of 1-6 points on every non-saturated task.
Holds for: One LoRA-finetuned Llama2-7B (rank 8) per task, with the embedding and final linear layers trained in full rank; 300K training equations for add/sub/mul, 30K for sine and sqrt.
On tasks Llama2-7B already nearly solves, NumeroLogic still helps: 5-digit integer addition goes from 99.86% to 100.0% and subtraction from 99.60% to 99.93%, removing over 80% of the remaining subtraction errors.
Holds for: LoRA fine-tuning of pretrained Llama2-7B on 5-digit integer addition and subtraction, 300K training equations per task; the headroom here is under 0.5 points, so the absolute gain is small.
can changing how numbers are written in the training text help a model on general knowledge questions, not just sums?
does continued self-supervised pretraining with digit-count-prefixed numbers transfer to 0-shot MMLU accuracy?
how do I tell whether a number-formatting change helps beyond arithmetic benchmarks?
is continued pretraining with reformatted numbers worth the compute if I care about MMLU rather than arithmetic?
Continuing self-supervised pretraining of Llama2-7B on RefinedWeb text with numbers rewritten in NumeroLogic format improves 0-shot MMLU accuracy by a statistically significant 0.5%. Continued pretraining on the same tokens with plain numbers does not improve over the pretrained model.
Holds for: Continued causal-LM pretraining with LoRA on RefinedWeb, plain and NumeroLogic runs matched on token count; 0-shot MMLU evaluation; English text only.
The MMLU gain from NumeroLogic concentrates where numbers appear: +1.16% on tasks containing numbers versus +0.14% on tasks without, and +0.79% on STEM versus +0.1% on social sciences.
Holds for: Per-category breakdown of the 0-shot MMLU evaluation of Llama2-7B after continued pretraining with LoRA on RefinedWeb; the "Others" category gains most at +1.19%.
could the improvement just come from adding more tokens around each number rather than from the digit count itself?
is the digit-count prefix gain separable from the effect of additional tokens, as tested against whitespace filler control tokens?
how do I check that a formatting gain comes from the digit-count information and not from extra tokens giving the model more compute?
if I just pad numbers with filler tokens, do I get the same benefit as prefixing the digit count?
Replacing NumeroLogic's prefix tokens with the same number of filler white-space tokens gives 24.37% on 3-digit float multiplication, close to the 24.73% plain-format baseline. NumeroLogic itself reaches 31.03%, so the benefit is not merely the extra tokens.
Holds for: Llama2-7B finetuned on 3-digit floating-point multiplication; the random white-space placement of Shen et al. (2023), with a matched token budget, reaches 27.76%, better than plain but below NumeroLogic.
when numbers in an equation are labelled with their length, does it matter whether the inputs or the answer get labelled?
is the digit-count encoding gain attributable to input comprehension of operands or to a chain-of-thought effect on the generated result?
where should I put digit-count prefixes in an arithmetic training example, on the operands, the result, or both?
if I can only reformat one side of my equations, should I pick the operands or the answer?
Encoding only the addition result with a digit-count prefix lifts NanoGPT accuracy from 88.37% to 98.05%, while encoding only the operands reaches 89.34%. Encoding both operands and result is best at 99.78%.
Holds for: 3-digit addition with NanoGPT trained from scratch, same protocol as the joint 5-task arithmetic setup with character-level tokenization.
which way of writing a number's length in text works best for a language model?
how do full start-and-end delimited digit-count formats compare with dropping the end-of-number token or using one dedicated special token per digit count?
how do I pick a number format for digit-count encoding, and do I need to add special tokens to the vocabulary?
is adding one new special token per digit count to my tokenizer worth it, or should I stick with plain text prefixes?
On 3-digit integer multiplication the full "<sn>3<mn>100<en>" format scores 35.33% and dropping the end-of-number token gives 34.93%. One dedicated special token per digit count gives 33.56%, below the 34.20% plain baseline.
Holds for: Llama2-7B finetuned on 3-digit integer multiplication; the paper attributes the failure of per-digit-count special tokens to the rarity of short numbers in the training distribution.
what should I read about changing the way numbers are written for language models rather than adding reasoning steps?
which work treats numeric representation as a general self-supervised language modeling change rather than a task-specific arithmetic intervention?
where do I start reading if I want to improve numeric handling in language models by representation rather than prompting?
NumeroLogic is positioned as a number-representation change for general self-supervised language modeling rather than a task-specific arithmetic fix. Arithmetic tasks only measure the effect, and MMLU shows transfer to language understanding.
Holds for: As of 2024; contemporaneous arithmetic work such as digit reversal, algorithmic chain-of-thought and random white-space insertion targets specific arithmetic tasks. The general-LM evidence is one Llama2-7B run on English RefinedWeb text.
NumeroLogic starts from the hypothesis that left-to-right causal reading is a real handicap for numbers. A decoder-only model cannot know whether a digit means units or millions until it has read the whole number.
Holds for: A motivating hypothesis supported indirectly by the arithmetic and MMLU gains rather than by a direct probe of place-value representations.
what does it cost to prefix every number in the text with its digit count?
what overhead does digit-count number encoding add in sequence length, preprocessing and vocabulary changes?
how do I add and later strip digit-count prefixes around an existing tokenizer without retraining anything else?
will digit-count number formatting force me to change my model or my serving stack?
NumeroLogic is a regex-based text pre- and post-processing step that requires no change to model architecture, so digit-count prefixes can be added around an existing tokenizer and stripped from generations.
Holds for: Adding 3 special tokens (<sn>, <mn>, <en>) to the vocabulary and expanding embedding and output layers for pretrained models; increases tokens per number and inference latency.
how much better does a small model trained from scratch get at arithmetic when numbers carry their digit count?
what accuracy do NanoGPT models reach on integer addition, subtraction, multiplication, sine and square root with digit-count prefixed numbers?
how do I improve a small from-scratch transformer on multiplication and on 4-decimal function tasks like sine and square root?
I train tiny models on arithmetic from scratch, will digit-count prefixes help on multiplication as much as on addition?
NumeroLogic more than doubles NanoGPT's 2-digit integer multiplication accuracy, from 13.81% to 28.94%, and adds about 4 points on 4-decimal sine (30.59% to 34.59%) and square root (22.13% to 26.66%).
Holds for: Single NanoGPT trained from scratch on all 5 tasks at once; sine operands in [-pi/2, pi/2], square root operands in [0, 10], 4 decimal places.
With NumeroLogic digit-count prefixes, a from-scratch NanoGPT reaches 99.96% accuracy on 3-digit integer addition and 97.20% on subtraction, against 88.37% and 73.76% with plain number formatting.
Holds for: NanoGPT trained from scratch with character-level tokenization, jointly on 5 arithmetic tasks with 10K samples per task (3K for multiplication); addition and subtraction operands up to 3 digits.

Claims and scope

Common misreadings

Terminology in this paper

NumeroLogic
A number format for text fed to language models in which each number is prefixed by its digit count, e.g. "42" written as "<sn>2<mn>42<en>" and "3.14" as "<sn>1.2<mn>3.14<en>", with the prefix stripped after generation.
<sn> / <mn> / <en>
Special start-number, mid-number and end-number tokens that delimit the digit-count prefix and the number itself; replaced by the characters "{", ":" and "}" in character-level small-model experiments.
Only prefix encoding
A variant digit-count format that keeps the leading digit count but omits the end-of-number token, e.g. "<sn>3<mn>100".
Random white-spaces baseline
A control format from Shen et al. (2023) that inserts filler white-space tokens at random positions between digits, used with a token budget matched to digit-count prefixing.

How to cite

@inproceedings{schwartz2024numerologic,
    title = "{N}umero{L}ogic: Number Encoding for Enhanced {LLM}s' Numerical Reasoning",
    author = "Schwartz, Eli  and
      Choshen, Leshem  and
      Shtok, Joseph  and
      Doveh, Sivan  and
      Karlinsky, Leonid  and
      Arbelle, Assaf",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.12/",
    doi = "10.18653/v1/2024.emnlp-main.12",
    pages = "206--212",
    abstract = "Language models struggle with handling numerical data and performing arithmetic operations. We hypothesize that this limitation can be partially attributed to non-intuitive textual numbers representation. When a digit is read or generated by a causal language model it does not know its place value (e.g. thousands vs. hundreds) until the entire number is processed. To address this issue, we propose a simple adjustment to how numbers are represented by including the count of digits before each number. For instance, instead of ``42'', we suggest using ``2:42'' as the new format. This approach, which we term NumeroLogic, offers an added advantage in number generation by serving as a Chain of Thought (CoT). By requiring the model to consider the number of digits first, it enhances the reasoning process before generating the actual number. We use arithmetic tasks to demonstrate the effectiveness of the NumeroLogic formatting. We further demonstrate NumeroLogic applicability to general natural language modeling, improving language understanding performance in the MMLU benchmark."
}

References

See the full reference list in the paper.