1,269 research outputs found
Weighted Automata Extraction from Recurrent Neural Networks via Regression on State Spaces
We present a method to extract a weighted finite automaton (WFA) from a
recurrent neural network (RNN). Our algorithm is based on the WFA learning
algorithm by Balle and Mohri, which is in turn an extension of Angluin's
classic \lstar algorithm. Our technical novelty is in the use of
\emph{regression} methods for the so-called equivalence queries, thus
exploiting the internal state space of an RNN to prioritize counterexample
candidates. This way we achieve a quantitative/weighted extension of the recent
work by Weiss, Goldberg and Yahav that extracts DFAs. We experimentally
evaluate the accuracy, expressivity and efficiency of the extracted WFAs.Comment: AAAI 2020. We are preparing to distribute the implementatio
MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning
Training deep networks and tuning hyperparameters on large datasets is
computationally intensive. One of the primary research directions for efficient
training is to reduce training costs by selecting well-generalizable subsets of
training data. Compared to simple adaptive random subset selection baselines,
existing intelligent subset selection approaches are not competitive due to the
time-consuming subset selection step, which involves computing model-dependent
gradients and feature embeddings and applies greedy maximization of submodular
objectives. Our key insight is that removing the reliance on downstream model
parameters enables subset selection as a pre-processing step and enables one to
train multiple models at no additional cost. In this work, we propose MILO, a
model-agnostic subset selection framework that decouples the subset selection
from model training while enabling superior model convergence and performance
by using an easy-to-hard curriculum. Our empirical results indicate that MILO
can train models faster and tune hyperparameters
faster than full-dataset training or tuning without
compromising performance
Neurogenetic Programming Framework for Explainable Reinforcement Learning
Automatic programming, the task of generating computer programs compliant
with a specification without a human developer, is usually tackled either via
genetic programming methods based on mutation and recombination of programs, or
via neural language models. We propose a novel method that combines both
approaches using a concept of a virtual neuro-genetic programmer: using
evolutionary methods as an alternative to gradient descent for neural network
training}, or scrum team. We demonstrate its ability to provide performant and
explainable solutions for various OpenAI Gym tasks, as well as inject expert
knowledge into the otherwise data-driven search for solutions.Comment: Source code is available at https://github.com/vadim0x60/cib
Neural Networks Reduction via Lumping
The increasing size of recently proposed Neural Networks makes it hard to implement them on embedded devices, where memory, battery and computational power are a non-trivial bottleneck. For this reason during the last years network compression literature has been thriving and a large number of solutions has been published to reduce both the number of operations and the parameters involved with the models. Unfortunately, most of these reducing techniques are actually heuristic methods and usually require at least one re-training step to recover the accuracy. The need of procedures for model reduction is well-known also in the fields of Verification and Performances Evaluation, where large efforts have been devoted to the definition of quotients that preserve the observable underlying behaviour. In this paper we try to bridge the gap between the most popular and very effective network reduction strategies and formal notions, such as lumpability, introduced for verification and evaluation of Markov Chains. Elaborating on lumpability we propose a pruning approach that reduces the number of neurons in a network without using any data or fine-tuning, while completely preserving the exact behaviour. Relaxing the constraints on the exact definition of the quotienting method we can give a formal explanation of some of the most common reduction techniques
Deep Multitask Learning for Semantic Dependency Parsing
We present a deep neural architecture that parses sentences into three
semantic dependency graph formalisms. By using efficient, nearly arc-factored
inference and a bidirectional-LSTM composed with a multi-layer perceptron, our
base system is able to significantly improve the state of the art for semantic
dependency parsing, without using hand-engineered features or syntax. We then
explore two multitask learning approaches---one that shares parameters across
formalisms, and one that uses higher-order structures to predict the graphs
jointly. We find that both approaches improve performance across formalisms on
average, achieving a new state of the art. Our code is open-source and
available at https://github.com/Noahs-ARK/NeurboParser.Comment: Proceedings of ACL 201
- …