MetaRuleGPT: Recursive Numerical Reasoning of Language Models Trained with Simple Rules
^††thanks: * Equal Contribution ^††thanks: ${\dagger}$ Corresponding Author

In our research, the model exhibited exceptional logical reasoning and generalization capabilities while tackling three complex tasks: high-digit(10 or more digits) addition and subtraction, as well as vector cross-product computations. To facilitate these tasks, we created specialized rule datasets for training. For instance, during the training for addition calculations, the model was guided to acquire essential knowledge, including single-digit addition rules, carry rules, digit mapping rules(between numbers and strings), and fundamental computation rules in Fig. 2. By mastering these foundational concepts, and following meticulous pre-training [21], our model was able to flexibly apply and integrate these rules, enabling it to accurately perform complex mathematical operations, including high-digit addition and subtraction.

We extended our model’s capabilities to perform vector cross-product computations since it had mastered the rules of addition and subtraction. To achieve this goal, we introduced rules for vector representation and cross-product computation into the model to realize vector cross-product calculations. This means that once the model learns these new rules, it could combine the newly acquired rules with existing numerical computation rules to perform vector cross-product calculations. During the process of vector cross-product computation, the model needs to handle a large amount of complex derivation. Through gradual derivation, combining the right-hand rule for cross-products with basic numerical computation rules, the model gains the ability to compute vector cross-products. This strategy showcases the model’s deep logical reasoning and strong generalization ability to solve more complex mathematical operations by learning and integrating various rules in TABLE. I.

We designed the overall computation rule dataset for training, covering a wide range of arithmetic operations from the most basic single-digit arithmetic tasks to various complex arithmetic rules. This dataset, carefully planned, only contains the most basic single-digit operations, including alignment rules, carry rules, borrow rules, basic computation rules, and composite rules. Each arithmetic expression involves $2$ to $10$ operational steps, involving a series of mathematical computation operations, such as addition $(+)$, subtraction $(-)$, and vector cross-product operations $(\times)$. Our constructed dataset contains approximately $20,000$ records.

To closely mimic the natural process of humans solving mathematical problems, we did not directly solve each complex arithmetic expression but adopted an iterative and stepwise strategy. Through this method, our model breaks down complex expressions into a series of simpler and basic computational steps, reasoning out the final answer step by step. This approach enables the language model to have a deeper understanding and more effective application of specific rules during the learning process, allowing for flexible combination and application of these rules in problem-solving. MetaRuleGPT’s proficiency in mathematical tasks stems primarily from its mastery of core computation rules rather than mere memorization of specific cases.

Focusing on arithmetic tasks, we developed a language model based on the Transformer architecture, aimed at solving mathematical problems, which we refer to as the MetaRuleGPT language model. The model architecture, as shown in the Fig. 3, includes several key components: the MetaRuleGPT pre-trained model, the RefeedFormatter (formatting tool), and VeriGate (verification gate). We designed a self-iteration method that enables the model to simplify complex problems through continuous iteration, and finally obtain the correct answer within a limited number of steps.

We have trained a language model based on the Transformer[11] architecture. To flexibly adjust the model’s parameter size and internal structure, we designed and implemented a custom Transformer model. Given the limited vocabulary involved in the problems we address, we adopted a single-byte-based training method, which offers clear advantages over traditional word-based or character-based methods.

Byte-based language models provide a flexible and effective means for handling multilingual text and unknown characters. Fig. 4 is an example of using the Transformer model to train vector cross product calculation rules. By processing each character individually, the model can ensure more accurate learning of the rules, laying a solid foundation for solving complex logical tasks.

To demonstrate how the model operates, we use a simple addition example. When the input “Input: 78 + 263” is provided, it is processed sequentially through the Mapping Rule, Compute Rule, Align Rule, and Carry/Borrow Rule to derive the computation result. Fig. 5 illustrates the internal structure of our model and explains how the initial input is transformed into the final result.

1. 1.

   First, the model structurally processes our input question, where “78 + 263” under the Mapping Rule becomes:

   $$a_{1}=7,b_{1}=8,c_{1}=2,d_{1}=6,e_{1}=3.$$

   The expression “$a_{1}b_{1}+c_{1}d_{1}e_{1}$” through the Align Rule becomes:

   $$c_{1}|(a_{1}+d_{1})|(b_{1}+e_{1}).$$

   Through alignment, a combination of mapping rules produces the intermediate output:“$2|(7+6)|(8+3)$”.

2. 2.

   Similarly, for “$2|(7+6)|(8+3)$”, a combination of the Mapping Rule and single-digit addition rule (Add Sub-rule) produces the intermediate output: “$2|13|11$”.

3. 3.

   When “$2|13|11$” is input, the model invokes the Carry Rule and the Mapping Rule to perform digit carry operations, producing an intermediate output: “$(2+1)|(3+1)|1$”.

4. 4.

   “$(2+1)|(3+1)|1$” as a new input, again applying the Mapping Rule and Compute Rule, leads to the final computation result: “$3|4|1$”.

5. 5.

   “$3|4|1$” as the final input stage, our model invokes the formatting rules and uses special symbols for marking. Finally, the result is formatted using VeriGate to output: “$Output:341$”.

In summary, MetaRuleGPT utilizes a combination of rules to align, carry, and output the final result during the computation process.

To demonstrate the exceptional accuracy and generalization capability of our MetaRuleGPT model in reasoning tasks, we designed two experiments: numerical arithmetic tasks and vector cross-product computation tasks. These experiments not only tested the model’s basic computational ability but also its ability to solve complex problems, providing a solid foundation for comprehensively evaluating the model’s performance in logical reasoning. Furthermore, to further prove the advantages of MetaRuleGPT, we compared it with several well-known large language models, including ChatGPT-3.5, ChatGPT-4.0 [23], Alibaba’s QWen [24], Google’s Palm [25, 26], Llama2 [27] and Mathematical Mastery Model Goat[28], demonstrating the superior performance of MetaRuleGPT.

Current large language models exhibit certain limitations in handling mathematically rigorous problems, partly due to a lack of deep understanding of mathematical logic. In contrast, our model, built from the ground up on the fundamental principles of mathematics, demonstrates higher precision in solving math challenges. To validate this advantage, we designed diverse test datasets and prepared a comprehensive computation dataset to highlight the significant advantage of our model in mathematical reasoning. Through these validations, we demonstrate that MetaRuleGPT not only precisely grasps and applies the basic logic of mathematics but also exhibits powerful generalization in problem-solving.

In the domain of arithmetic tasks, we constructed a diverse training dataset containing a wide range of arithmetic operations. To comprehensively evaluate our model’s computational accuracy and generalization ability, we designed an evaluation dataset containing $8,000$ test cases shown in Table. II, entirely non-overlapping with the training set. This dataset covers various types of numerical operations, including but not limited to perfect decimal addition, reverse magnitude subtraction, misplaced subtraction, and addition and subtraction operations based on randomly generated numbers.

To compare with other models focusing on mathematical calculations, we used an additional substantial number of randomly generated dataset sampled from a logarithmic space¹¹1Our dataset is available at https://www.scidb.cn/en/detail?dataSetId=04575028fb8d4bfabeeba5825c8f57fc. This sampling approach ensures that the numbers are equally likely to originate from different orders of magnitude, similar to the method employed by [29].

In evaluating the model’s performance, we consider not only the accuracy of the computed results but also the difference ratio between the computed and correct answers. A smaller absolute difference ratio indicates that the model’s output is closer to the true value, suggesting potential for improvement through parameter tuning. Conversely, a difference ratio greater than 1 implies that the model struggles with such problems or faces significant challenges. Assuming the number of correctly predicted quantities is TP and the total number of predictions is N, then accuracy can be defined as: Accuracy $=\frac{TP}{N}\times 100\%$

Suppose our model’s computation result is y, the actual computation result N numbers in total, then our final overall difference ratio can be defined as:
DifferenceRatio $=\frac{1}{N}\sum_{i=0}^{N}\left|\frac{y_{i}-\hat{y_{i}}}{\max(y_{i},\hat{y_{i}})}\right|$

To test our model’s mathematical reasoning and generalization capabilities, we conducted comparisons using well-known language models such as the currently leading ChatGPT-4.0, ChatGPT-3.5, Alibaba’s QWen, Google’s Palm, Llama2 and Goat. Through such comparisons, we can comprehensively assess the performance differences between models and evaluate MetaRuleGPT’s capabilities in mathematical reasoning tasks. A comprehensive comparison is shown in Fig. 6 and Table. III.

Using the specific test datasets, we previously organized to invoke and test with the aforementioned large language models, preserving and comparing the computational results of each model. We conducted a series of detailed experiments and evaluations, and the results are shown in Table. IV, V, VI, IX and VII.

Finally, in order to compare with the public dataset, we selected some subsets of gsm8k[30] and simplified the natural language part into mathematical formulas for comparison. The results are shown in VIII.

To demonstrate our model’s capability in handling complex logical problems, we have selected vector cross product calculation, a challenging mathematical task, as a test case. This test not only verifies the model’s computational accuracy but also compares its performance with those of leading large language models. Table. X presents the detailed accuracy comparison results of various models on the vector cross product calculation dataset.

Although existing language models have demonstrated powerful capabilities, they still face challenges in terms of controllability. In particular, most models struggle to precisely answer questions within a controlled range, often resulting in significant deviations, which is a crucial issue that current language models need to address. Conversely, MetaRuleGPT’s rule-based execution ensures relative reliability and better controllability.

Experiments show that existing language models struggle with high-digit calculations and complex computational tasks due to limitations in understanding and the difficulty of learning a unified representation for text and numbers. MetaRuleGPT addresses these challenges by precisely completing complex tasks and demonstrating the generalization potential of language models in numerical calculations through rule learning.