3,479 research outputs found
Unbiased Math Word Problems Benchmark for Mitigating Solving Bias
In this paper, we revisit the solving bias when evaluating models on current
Math Word Problem (MWP) benchmarks. However, current solvers exist solving bias
which consists of data bias and learning bias due to biased dataset and
improper training strategy. Our experiments verify MWP solvers are easy to be
biased by the biased training datasets which do not cover diverse questions for
each problem narrative of all MWPs, thus a solver can only learn shallow
heuristics rather than deep semantics for understanding problems. Besides, an
MWP can be naturally solved by multiple equivalent equations while current
datasets take only one of the equivalent equations as ground truth, forcing the
model to match the labeled ground truth and ignoring other equivalent
equations. Here, we first introduce a novel MWP dataset named UnbiasedMWP which
is constructed by varying the grounded expressions in our collected data and
annotating them with corresponding multiple new questions manually. Then, to
further mitigate learning bias, we propose a Dynamic Target Selection (DTS)
Strategy to dynamically select more suitable target expressions according to
the longest prefix match between the current model output and candidate
equivalent equations which are obtained by applying commutative law during
training. The results show that our UnbiasedMWP has significantly fewer biases
than its original data and other datasets, posing a promising benchmark for
fairly evaluating the solvers' reasoning skills rather than matching nearest
neighbors. And the solvers trained with our DTS achieve higher accuracies on
multiple MWP benchmarks. The source code is available at
https://github.com/yangzhch6/UnbiasedMWP
Non-Autoregressive Math Word Problem Solver with Unified Tree Structure
Existing MWP solvers employ sequence or binary tree to present the solution
expression and decode it from given problem description. However, such
structures fail to handle the variants that can be derived via mathematical
manipulation, e.g., and can both be
possible valid solutions for a same problem but formulated as different
expression sequences or trees. The multiple solution variants depicting
different possible solving procedures for the same input problem would raise
two issues: 1) making it hard for the model to learn the mapping function
between the input and output spaces effectively, and 2) wrongly indicating
\textit{wrong} when evaluating a valid expression variant. To address these
issues, we introduce a unified tree structure to present a solution expression,
where the elements are permutable and identical for all the expression
variants. We propose a novel non-autoregressive solver, named \textit{MWP-NAS},
to parse the problem and deduce the solution expression based on the unified
tree. For evaluating the possible expression variants, we design a path-based
metric to evaluate the partial accuracy of expressions of a unified tree. The
results from extensive experiments conducted on Math23K and MAWPS demonstrate
the effectiveness of our proposed MWP-NAS. The codes and checkpoints are
available at: \url{https://github.com/mengqunhan/MWP-NAS}.Comment: Accepted at EMNLP202
- β¦