4 research outputs found
Globally Gated Deep Linear Networks
Recently proposed Gated Linear Networks present a tractable nonlinear network
architecture, and exhibit interesting capabilities such as learning with local
error signals and reduced forgetting in sequential learning. In this work, we
introduce a novel gating architecture, named Globally Gated Deep Linear
Networks (GGDLNs) where gating units are shared among all processing units in
each layer, thereby decoupling the architectures of the nonlinear but unlearned
gatings and the learned linear processing motifs. We derive exact equations for
the generalization properties in these networks in the finite-width
thermodynamic limit, defined by , where P
and N are the training sample size and the network width respectively. We find
that the statistics of the network predictor can be expressed in terms of
kernels that undergo shape renormalization through a data-dependent matrix
compared to the GP kernels. Our theory accurately captures the behavior of
finite width GGDLNs trained with gradient descent dynamics. We show that kernel
shape renormalization gives rise to rich generalization properties w.r.t.
network width, depth and L2 regularization amplitude. Interestingly, networks
with sufficient gating units behave similarly to standard ReLU networks.
Although gatings in the model do not participate in supervised learning, we
show the utility of unsupervised learning of the gating parameters.
Additionally, our theory allows the evaluation of the network's ability for
learning multiple tasks by incorporating task-relevant information into the
gating units. In summary, our work is the first exact theoretical solution of
learning in a family of nonlinear networks with finite width. The rich and
diverse behavior of the GGDLNs suggests that they are helpful analytically
tractable models of learning single and multiple tasks, in finite-width
nonlinear deep networks
Top-Down Processing: Top-Down Network Combines Back-Propagation with Attention
Early neural network models relied exclusively on bottom-up processing going
from the input signals to higher-level representations. Many recent models also
incorporate top-down networks going in the opposite direction. Top-down
processing in deep learning models plays two primary roles: learning and
directing attention. These two roles are accomplished in current models through
distinct mechanisms. While top-down attention is often implemented by extending
the model's architecture with additional units that propagate information from
high to low levels of the network, learning is typically accomplished by an
external learning algorithm such as back-propagation. In the current work, we
present an integration of the two functions above, which appear unrelated,
using a single unified mechanism. We propose a novel symmetric bottom-up
top-down network structure that can integrate standard bottom-up networks with
a symmetric top-down counterpart, allowing each network to guide and influence
the other. The same top-down network is being used for both learning, via
back-propagating feedback signals, and at the same time also for top-down
attention, by guiding the bottom-up network to perform a selected task. We show
that our method achieves competitive performance on a standard multi-task
learning benchmark. Yet, we rely on standard single-task architectures and
optimizers, without any task-specific parameters. Additionally, our learning
algorithm addresses in a new way some neuroscience issues that arise in
biological modeling of learning in the brain