Enhancing Neural Mathematical Reasoning by Abductive Combination with Symbolic Library

Saxton et al. (Saxton et al., 2019) introduced a mathematics dataset that contains a variety of math problems, including algebra, arithmetic, numerical comparison, numerical factorization, calculus, measurement, and probability. Each problem is a question-answer pair, where the question is like Let q(m) = m**3 + 2. Let r(c) = -4*c**3 - 9. What is 18*q(f) + 4*r(f)? and the answer is like 2*f**3. Although there may be many forms of answer sequences with the same mathematical meaning, the evaluation criterion is character-by-character (i.e., each question is scored by either 0 or 1 according to whether the answer matches the correct answer character-by-character). The dataset is procedurally generated and consists of 56 modules, and each module provides 2M per-generated training samples and 10k interpolation samples. Extrapolation samples are also provided for an additional measure of algebraic generalization.

Sympy (Meurer et al., 2017) is a mathematical symbolic computing library, which contains about 300+ mathematical functions. Although many mathematical engines can be used, we adopt Sympy because it can conveniently get all the appropriate mathematical functions, easily exclude non-mathematical functions, and support direct access to the docstrings of mathematical functions.

A mathematics dataset was released in (Saxton et al., 2019) that analyzes the reasoning and generalization ability of popular reasoning neural architectures such as recurrent neural architectures and attention-augmented architectures (i.e., Transformer (Vaswani et al., 2017)). The results show that the learned models did not do mathematical reasoning well, particularly for the extrapolation zone. (Schlag et al., 2019) incorporates the tensor-product representation technique within the Transformer to better support the explicit representation of relation structure. They achieved improved results than the vanilla Transformer architecture without introducing any domain knowledge.

Program format is a typical way to represent both of the discrete domain knowledge and the solution structure of mathematical problems. Amini et al. (Amini et al., 2019) released a dataset of math word problems that are densely annotated with programs by crowd-sourcing. Based on the dataset, they proposed a sequence-to-program model with automatic problem categorization. Comparing with their method, our approach applying to the dataset without annotated programs, and moreover, we use both the neural network and the discrete symbolic system for prediction.

Abductive learning (Dai et al., 2019) was recently proposed for connecting a perception module with an abductive logical reasoning module using consistency optimization. The perception module generates output, the reasoning module checks and corrects the logical consistency, and the consistency information is used to update the perception module to generate logically more consistent output. This constitutes a forward cycle. Our approach is inspired by the above abductive learning framework, while we are addressing a different domain.

We define the program based on a domain-specific language (DSL) instead of arbitrary Turing-complete languages to reduce the search space of programs. Every word in the DSL is called an operator. All available operators form an operator space. The relationship between adjacent operators is appropriately restricted, such as argc operators must be followed by math operators, the number of optional variables must be no less than argc, and argc must be an available number of arguments to the mathematical operator.

The neural network model we use is a modified version of the original Transformer (Vaswani et al., 2017), with a shared transformer encoder $\theta^{enc}$ and two separate transformer decoders $\theta_{a}^{dec}$ and $\theta_{p}^{dec}$. We use the encoder $\theta^{enc}$ with hidden states $\mathbf{h}^{enc}$ to encode the problem $\mathbf{x}$. The decoders $\theta_{a}^{dec}$ and $\theta_{p}^{dec}$ take the shared hidden states $\mathbf{h}^{enc}$ and auto-regressively generates the answer sequence and program sequence respectively. During training, the decoders receive the shifted targets while during inference, we use the previously generated symbols with the highest probability. We treat the question and answer as a sequence of characters just like (Vaswani et al., 2017) and treat the question as a sequence of operators. The overall training loss is the weighted sum of the answer decoding loss and the program decoding loss:

During the search, the maximum number of sampled programs for each problem is $N_{w}=100k$ on the first iteration and $N_{n}=1k$ on the other iterations. The number of iterations $I$ is set to 5.

We extract a character-level vocabulary of 72 symbols and an operator-level vocabulary of 380 symbols, both including START, END, and PADDING symbols.

Our transformer-like model parameters $\theta^{enc}$, $\theta_{a}^{dec}$, $\theta_{p}^{dec}$ are set to an embedding size of $512$, with $8$ attentional heads, and intermediate feed-forward dimension of 2048. The answer decoder $\theta_{a}^{dec}$ is with layers of $6$ while the program decoder $\theta_{p}^{dec}$ is with layers of $2$. We train our model via the Adam optimizer  (Kingma & Ba, 2014) with a learning rate of $8\times 10^{-5}$, $\beta_{1}=0.9$, $\beta_{2}=0.995$, $\epsilon=10^{-9}$. We use a batch size of $1024$, with absolute gradient value clipping of $0.1$. We trained our model on one server with 8 V100 Nvidia GPUs for 12 days. During the search process, the parameters configuration of our program-generated model is the same as the above model.

At the inference, answers and programs are generated by sequential decoding. If the predicted program is none or fails to run successfully, the neural model answer is used as the final result.

Table 1 presents the overall performance on the dataset. We can see that our model significantly outperforms the previous state-of-the-art by up to $7.8\%$ absolute improvement on the interpolation test dataset and $24.8\%$ absolute improvement on the extrapolation test dataset. Our program-augmented model dramatically improves the performance of the model, especially for generalizing the model to areas not previously seen. For a more detailed comparison, Fig. 1 shows the test performance on extrapolation modules.

Table 2 shows the performance of the 5 iterations of ABL-Sym together with the random search performance. ABL-Sym shows clearly better than random search. In the first iteration, it used an average of 20% fewer search times than the random search strategy but found 76% more programs, which mainly due to the warm-up strategy and curriculum search strategy. These strategies allow us to search more programs faster within the maximum search limit. After the first iteration, the model we learned as a better program generator generated better candidate programs, so we searched an additional 8% of the programs with negligible search times. Compared to the second iteration, the number of programs searched in the next few iterations increased by only a litter bit. This is because most programs that can be searched are also almost searched. Still, $57.7\%$ programs were not found.

ABL-Sym can find many programs of compositional or complex problems (e.g., Calculate the common denominator of 25/13728 and 121/1248. the program found is pos7 argc1 denom pos5 argc1 denom argc2 lcm), but random search strategy was failed.