Numerologic: Number Encoding for Enhanced LLMs' Numerical Reasoning

Numerologic: Number Encoding for Enhanced LLMs' Numerical Reasoning

30 Mar 2024 | Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, Assaf Arbelle
NumeroLogic is a method to enhance the numerical reasoning capabilities of large language models (LLMs). The main issue is that LLMs struggle with numerical tasks due to the non-intuitive textual representation of numbers. When a model reads a number, it does not know the place value of each digit until the entire number is processed. To address this, NumeroLogic introduces a prefix indicating the number of digits before the actual number. For example, "42" becomes "2:42". This format helps the model understand the place value of digits before processing the full number, acting as a Chain of Thought (CoT) to improve reasoning. The method was tested on various arithmetic tasks, including addition, subtraction, multiplication, sine, and square root. Results showed significant improvements in accuracy, especially for smaller models like NanoGPT. For larger models like Llama2-7B, the method also improved performance, with some tasks being almost perfectly solved. Additionally, NumeroLogic was applied to general language modeling tasks, showing improvements in the MMLU benchmark, particularly in tasks requiring numerical understanding. Experiments demonstrated that NumeroLogic enhances numerical abilities across different model sizes and tasks. It was implemented through text preprocessing and post-processing using regular expressions, without altering the model's architecture. The method also showed benefits in self-supervised pretraining, improving performance on general language understanding tasks. Ablation studies indicated that encoding operands rather than results had a stronger effect on performance. Different encoding formats were tested, but the full NumeroLogic encoding provided the best results. Overall, NumeroLogic offers a simple yet effective solution for improving numerical reasoning in LLMs without requiring architectural changes.NumeroLogic is a method to enhance the numerical reasoning capabilities of large language models (LLMs). The main issue is that LLMs struggle with numerical tasks due to the non-intuitive textual representation of numbers. When a model reads a number, it does not know the place value of each digit until the entire number is processed. To address this, NumeroLogic introduces a prefix indicating the number of digits before the actual number. For example, "42" becomes "2:42". This format helps the model understand the place value of digits before processing the full number, acting as a Chain of Thought (CoT) to improve reasoning. The method was tested on various arithmetic tasks, including addition, subtraction, multiplication, sine, and square root. Results showed significant improvements in accuracy, especially for smaller models like NanoGPT. For larger models like Llama2-7B, the method also improved performance, with some tasks being almost perfectly solved. Additionally, NumeroLogic was applied to general language modeling tasks, showing improvements in the MMLU benchmark, particularly in tasks requiring numerical understanding. Experiments demonstrated that NumeroLogic enhances numerical abilities across different model sizes and tasks. It was implemented through text preprocessing and post-processing using regular expressions, without altering the model's architecture. The method also showed benefits in self-supervised pretraining, improving performance on general language understanding tasks. Ablation studies indicated that encoding operands rather than results had a stronger effect on performance. Different encoding formats were tested, but the full NumeroLogic encoding provided the best results. Overall, NumeroLogic offers a simple yet effective solution for improving numerical reasoning in LLMs without requiring architectural changes.
Reach us at info@study.space
Understanding NumeroLogic%3A Number Encoding for Enhanced LLMs%E2%80%99 Numerical Reasoning