17 research outputs found
Explaining Math Word Problem Solvers
Automated math word problem solvers based on neural networks have
successfully managed to obtain 70-80\% accuracy in solving arithmetic word
problems. However, it has been shown that these solvers may rely on superficial
patterns to obtain their equations. In order to determine what information math
word problem solvers use to generate solutions, we remove parts of the input
and measure the model's performance on the perturbed dataset. Our results show
that the model is not sensitive to the removal of many words from the input and
can still manage to find a correct answer when given a nonsense question. This
indicates that automatic solvers do not follow the semantic logic of math word
problems, and may be overfitting to the presence of specific words
Unbiased Math Word Problems Benchmark for Mitigating Solving Bias
In this paper, we revisit the solving bias when evaluating models on current
Math Word Problem (MWP) benchmarks. However, current solvers exist solving bias
which consists of data bias and learning bias due to biased dataset and
improper training strategy. Our experiments verify MWP solvers are easy to be
biased by the biased training datasets which do not cover diverse questions for
each problem narrative of all MWPs, thus a solver can only learn shallow
heuristics rather than deep semantics for understanding problems. Besides, an
MWP can be naturally solved by multiple equivalent equations while current
datasets take only one of the equivalent equations as ground truth, forcing the
model to match the labeled ground truth and ignoring other equivalent
equations. Here, we first introduce a novel MWP dataset named UnbiasedMWP which
is constructed by varying the grounded expressions in our collected data and
annotating them with corresponding multiple new questions manually. Then, to
further mitigate learning bias, we propose a Dynamic Target Selection (DTS)
Strategy to dynamically select more suitable target expressions according to
the longest prefix match between the current model output and candidate
equivalent equations which are obtained by applying commutative law during
training. The results show that our UnbiasedMWP has significantly fewer biases
than its original data and other datasets, posing a promising benchmark for
fairly evaluating the solvers' reasoning skills rather than matching nearest
neighbors. And the solvers trained with our DTS achieve higher accuracies on
multiple MWP benchmarks. The source code is available at
https://github.com/yangzhch6/UnbiasedMWP
Expression Syntax Information Bottleneck for Math Word Problems
Math Word Problems (MWP) aims to automatically solve mathematical questions
given in texts. Previous studies tend to design complex models to capture
additional information in the original text so as to enable the model to gain
more comprehensive features. In this paper, we turn our attention in the
opposite direction, and work on how to discard redundant features containing
spurious correlations for MWP. To this end, we design an Expression Syntax
Information Bottleneck method for MWP (called ESIB) based on variational
information bottleneck, which extracts essential features of expression syntax
tree while filtering latent-specific redundancy containing syntax-irrelevant
features. The key idea of ESIB is to encourage multiple models to predict the
same expression syntax tree for different problem representations of the same
problem by mutual learning so as to capture consistent information of
expression syntax tree and discard latent-specific redundancy. To improve the
generalization ability of the model and generate more diverse expressions, we
design a self-distillation loss to encourage the model to rely more on the
expression syntax information in the latent space. Experimental results on two
large-scale benchmarks show that our model not only achieves state-of-the-art
results but also generates more diverse solutions. The code is available.Comment: This paper has been accepted by SIGIR 2022. The code can be found at
https://github.com/menik1126/math_ESI
SMART: A Situation Model for Algebra Story Problems via Attributed Grammar
Solving algebra story problems remains a challenging task in artificial
intelligence, which requires a detailed understanding of real-world situations
and a strong mathematical reasoning capability. Previous neural solvers of math
word problems directly translate problem texts into equations, lacking an
explicit interpretation of the situations, and often fail to handle more
sophisticated situations. To address such limits of neural solvers, we
introduce the concept of a \emph{situation model}, which originates from
psychology studies to represent the mental states of humans in problem-solving,
and propose \emph{SMART}, which adopts attributed grammar as the representation
of situation models for algebra story problems. Specifically, we first train an
information extraction module to extract nodes, attributes, and relations from
problem texts and then generate a parse graph based on a pre-defined attributed
grammar. An iterative learning strategy is also proposed to improve the
performance of SMART further. To rigorously study this task, we carefully
curate a new dataset named \emph{ASP6.6k}. Experimental results on ASP6.6k show
that the proposed model outperforms all previous neural solvers by a large
margin while preserving much better interpretability. To test these models'
generalization capability, we also design an out-of-distribution (OOD)
evaluation, in which problems are more complex than those in the training set.
Our model exceeds state-of-the-art models by 17\% in the OOD evaluation,
demonstrating its superior generalization ability
Non-Autoregressive Math Word Problem Solver with Unified Tree Structure
Existing MWP solvers employ sequence or binary tree to present the solution
expression and decode it from given problem description. However, such
structures fail to handle the variants that can be derived via mathematical
manipulation, e.g., and can both be
possible valid solutions for a same problem but formulated as different
expression sequences or trees. The multiple solution variants depicting
different possible solving procedures for the same input problem would raise
two issues: 1) making it hard for the model to learn the mapping function
between the input and output spaces effectively, and 2) wrongly indicating
\textit{wrong} when evaluating a valid expression variant. To address these
issues, we introduce a unified tree structure to present a solution expression,
where the elements are permutable and identical for all the expression
variants. We propose a novel non-autoregressive solver, named \textit{MWP-NAS},
to parse the problem and deduce the solution expression based on the unified
tree. For evaluating the possible expression variants, we design a path-based
metric to evaluate the partial accuracy of expressions of a unified tree. The
results from extensive experiments conducted on Math23K and MAWPS demonstrate
the effectiveness of our proposed MWP-NAS. The codes and checkpoints are
available at: \url{https://github.com/mengqunhan/MWP-NAS}.Comment: Accepted at EMNLP202
ComSearch: Equation Searching with Combinatorial Strategy for Solving Math Word Problems with Weak Supervision
17th Conference of the European Chapter of the Association for Computational Linguistics, May 2-6, 2023Previous studies have introduced a weakly-supervised paradigm for solving math word problems requiring only the answer value annotation. While these methods search for correct value equation candidates as pseudo labels, they search among a narrow sub-space of the enormous equation space. To address this problem, we propose a novel search algorithm with combinatorial strategy ComSearch, which can compress the search space by excluding mathematically equivalent equations. The compression allows the searching algorithm to enumerate all possible equations and obtain high-quality data. We investigate the noise in the pseudo labels that hold wrong mathematical logic, which we refer to as the false-matching problem, and propose a ranking model to denoise the pseudo labels. Our approach holds a flexible framework to utilize two existing supervised math word problem solvers to train pseudo labels, and both achieve state-of-the-art performance in the weak supervision task
ATHENA: Mathematical Reasoning with Thought Expansion
Solving math word problems depends on how to articulate the problems, the
lens through which models view human linguistic expressions. Real-world
settings count on such a method even more due to the diverse practices of the
same mathematical operations. Earlier works constrain available thinking
processes by limited prediction strategies without considering their
significance in acquiring mathematical knowledge. We introduce Attention-based
THought Expansion Network Architecture (ATHENA) to tackle the challenges of
real-world practices by mimicking human thought expansion mechanisms in the
form of neural network propagation. A thought expansion recurrently generates
the candidates carrying the thoughts of possible math expressions driven from
the previous step and yields reasonable thoughts by selecting the valid
pathways to the goal. Our experiments show that ATHENA achieves a new
state-of-the-art stage toward the ideal model that is compelling in variant
questions even when the informativeness in training examples is restricted.Comment: EMNLP 2023 (main); 13 pages, 5 figures, 8 table
An Expression Tree Decoding Strategy for Mathematical Equation Generation
Generating mathematical equations from natural language requires an accurate
understanding of the relations among math expressions. Existing approaches can
be broadly categorized into token-level and expression-level generation. The
former treats equations as a mathematical language, sequentially generating
math tokens. Expression-level methods generate each expression one by one.
However, each expression represents a solving step, and there naturally exist
parallel or dependent relations between these steps, which are ignored by
current sequential methods. Therefore, we integrate tree structure into the
expression-level generation and advocate an expression tree decoding strategy.
To generate a tree with expression as its node, we employ a layer-wise parallel
decoding strategy: we decode multiple independent expressions (leaf nodes) in
parallel at each layer and repeat parallel decoding layer by layer to
sequentially generate these parent node expressions that depend on others.
Besides, a bipartite matching algorithm is adopted to align multiple
predictions with annotations for each layer. Experiments show our method
outperforms other baselines, especially for these equations with complex
structures.Comment: Accepted to EMNLP-2023, camera-ready versio