94 research outputs found
Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule
Progress in neural grammatical error correction (GEC) is hindered by the lack
of annotated training data. Sufficient amounts of high-quality manually
annotated data are not available, so recent research has relied on generating
synthetic data, pretraining on it, and then fine-tuning on real datasets;
performance gains have been achieved either by ensembling or by using huge
pretrained models such as XXL-T5 as the backbone. In this work, we explore an
orthogonal direction: how to use available data more efficiently. First, we
propose auxiliary tasks that exploit the alignment between the original and
corrected sentences, such as predicting a sequence of corrections. We formulate
each task as a sequence-to-sequence problem and perform multi-task training.
Second, we discover that the order of datasets used for training and even
individual instances within a dataset may have important effects on the final
performance, so we set out to find the best training schedule. Together, these
two ideas lead to significant improvements, producing results that improve
state of the art with much smaller models; in particular, we outperform the
best models based on T5-XXL (11B parameters) with a BART-based model (400M
parameters).Comment: EMNLP 202
Machine Learning for SAT: Restricted Heuristics and New Graph Representations
Boolean satisfiability (SAT) is a fundamental NP-complete problem with many
applications, including automated planning and scheduling. To solve large
instances, SAT solvers have to rely on heuristics, e.g., choosing a branching
variable in DPLL and CDCL solvers. Such heuristics can be improved with machine
learning (ML) models; they can reduce the number of steps but usually hinder
the running time because useful models are relatively large and slow. We
suggest the strategy of making a few initial steps with a trained ML model and
then releasing control to classical heuristics; this simplifies cold start for
SAT solving and can decrease both the number of steps and overall runtime, but
requires a separate decision of when to release control to the solver.
Moreover, we introduce a modification of Graph-Q-SAT tailored to SAT problems
converted from other domains, e.g., open shop scheduling problems. We validate
the feasibility of our approach with random and industrial SAT problems
- …