48,914 research outputs found
Gradient-based Inference for Networks with Output Constraints
Practitioners apply neural networks to increasingly complex problems in
natural language processing, such as syntactic parsing and semantic role
labeling that have rich output structures. Many such structured-prediction
problems require deterministic constraints on the output values; for example,
in sequence-to-sequence syntactic parsing, we require that the sequential
outputs encode valid trees. While hidden units might capture such properties,
the network is not always able to learn such constraints from the training data
alone, and practitioners must then resort to post-processing. In this paper, we
present an inference method for neural networks that enforces deterministic
constraints on outputs without performing rule-based post-processing or
expensive discrete search. Instead, in the spirit of gradient-based training,
we enforce constraints with gradient-based inference (GBI): for each input at
test-time, we nudge continuous model weights until the network's unconstrained
inference procedure generates an output that satisfies the constraints. We
study the efficacy of GBI on three tasks with hard constraints: semantic role
labeling, syntactic parsing, and sequence transduction. In each case, the
algorithm not only satisfies constraints but improves accuracy, even when the
underlying network is state-of-the-art.Comment: AAAI 201
Multi-Sample Online Learning for Probabilistic Spiking Neural Networks
Spiking Neural Networks (SNNs) capture some of the efficiency of biological
brains for inference and learning via the dynamic, online, event-driven
processing of binary time series. Most existing learning algorithms for SNNs
are based on deterministic neuronal models, such as leaky integrate-and-fire,
and rely on heuristic approximations of backpropagation through time that
enforce constraints such as locality. In contrast, probabilistic SNN models can
be trained directly via principled online, local, update rules that have proven
to be particularly effective for resource-constrained systems. This paper
investigates another advantage of probabilistic SNNs, namely their capacity to
generate independent outputs when queried over the same input. It is shown that
the multiple generated output samples can be used during inference to robustify
decisions and to quantify uncertainty -- a feature that deterministic SNN
models cannot provide. Furthermore, they can be leveraged for training in order
to obtain more accurate statistical estimates of the log-loss training
criterion, as well as of its gradient. Specifically, this paper introduces an
online learning rule based on generalized expectation-maximization (GEM) that
follows a three-factor form with global learning signals and is referred to as
GEM-SNN. Experimental results on structured output memorization and
classification on a standard neuromorphic data set demonstrate significant
improvements in terms of log-likelihood, accuracy, and calibration when
increasing the number of samples used for inference and training.Comment: Submitte
Connections Between Adaptive Control and Optimization in Machine Learning
This paper demonstrates many immediate connections between adaptive control
and optimization methods commonly employed in machine learning. Starting from
common output error formulations, similarities in update law modifications are
examined. Concepts in stability, performance, and learning, common to both
fields are then discussed. Building on the similarities in update laws and
common concepts, new intersections and opportunities for improved algorithm
analysis are provided. In particular, a specific problem related to higher
order learning is solved through insights obtained from these intersections.Comment: 18 page
- …