259,513 research outputs found
Adversarial Training Using Feedback Loops
Deep neural networks (DNN) have found wide applicability in numerous fields
due to their ability to accurately learn very complex input-output relations.
Despite their accuracy and extensive use, DNNs are highly susceptible to
adversarial attacks due to limited generalizability. For future progress in the
field, it is essential to build DNNs that are robust to any kind of
perturbations to the data points. In the past, many techniques have been
proposed to robustify DNNs using first-order derivative information of the
network.
This paper proposes a new robustification approach based on control theory. A
neural network architecture that incorporates feedback control, named Feedback
Neural Networks, is proposed. The controller is itself a neural network, which
is trained using regular and adversarial data such as to stabilize the system
outputs. The novel adversarial training approach based on the feedback control
architecture is called Feedback Looped Adversarial Training (FLAT). Numerical
results on standard test problems empirically show that our FLAT method is more
effective than the state-of-the-art to guard against adversarial attacks
A Globally Optimal Energy-Efficient Power Control Framework and its Efficient Implementation in Wireless Interference Networks
This work develops a novel power control framework for energy-efficient power
control in wireless networks. The proposed method is a new branch-and-bound
procedure based on problem-specific bounds for energy-efficiency maximization
that allow for faster convergence. This enables to find the global solution for
all of the most common energy-efficient power control problems with a
complexity that, although still exponential in the number of variables, is much
lower than other available global optimization frameworks. Moreover, the
reduced complexity of the proposed framework allows its practical
implementation through the use of deep neural networks. Specifically, thanks to
its reduced complexity, the proposed method can be used to train an artificial
neural network to predict the optimal resource allocation. This is in contrast
with other power control methods based on deep learning, which train the neural
network based on suboptimal power allocations due to the large complexity that
generating large training sets of optimal power allocations would have with
available global optimization methods. As a benchmark, we also develop a novel
first-order optimal power allocation algorithm. Numerical results show that a
neural network can be trained to predict the optimal power allocation policy.Comment: submitte
CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics
Advances in optical and electrophysiological recording technologies have made
it possible to record the dynamics of thousands of neurons, opening up new
possibilities for interpreting and controlling large neural populations in
behaving animals. A promising way to extract computational principles from
these large datasets is to train data-constrained recurrent neural networks
(dRNNs). Performing this training in real-time could open doors for research
techniques and medical applications to model and control interventions at
single-cell resolution and drive desired forms of animal behavior. However,
existing training algorithms for dRNNs are inefficient and have limited
scalability, making it a challenge to analyze large neural recordings even in
offline scenarios. To address these issues, we introduce a training method
termed Convex Optimization of Recurrent Neural Networks (CORNN). In studies of
simulated recordings, CORNN attained training speeds ~100-fold faster than
traditional optimization approaches while maintaining or enhancing modeling
accuracy. We further validated CORNN on simulations with thousands of cells
that performed simple computations such as those of a 3-bit flip-flop or the
execution of a timed response. Finally, we showed that CORNN can robustly
reproduce network dynamics and underlying attractor structures despite
mismatches between generator and inference models, severe subsampling of
observed neurons, or mismatches in neural time-scales. Overall, by training
dRNNs with millions of parameters in subminute processing times on a standard
computer, CORNN constitutes a first step towards real-time network reproduction
constrained on large-scale neural recordings and a powerful computational tool
for advancing the understanding of neural computation.Comment: Accepted at NeurIPS 202
ABC: Aggregation before Communication, a Communication Reduction Framework for Distributed Graph Neural Network Training and Effective Partition
Graph Neural Networks(GNNs) are a family of neural models tailored for
graph-structure data and have shown superior performance in learning
representations for graph-structured data. However, training GNNs on large
graphs remains challenging and a promising direction is distributed GNN
training, which is to partition the input graph and distribute the workload
across multiple machines. The key bottleneck of the existing distributed GNNs
training framework is the across-machine communication induced by the
dependency on the graph data and aggregation operator of GNNs. In this paper,
we study the communication complexity during distributed GNNs training and
propose a simple lossless communication reduction method, termed the
Aggregation before Communication (ABC) method. ABC method exploits the
permutation-invariant property of the GNNs layer and leads to a paradigm where
vertex-cut is proved to admit a superior communication performance than the
currently popular paradigm (edge-cut). In addition, we show that the new
partition paradigm is particularly ideal in the case of dynamic graphs where it
is infeasible to control the edge placement due to the unknown stochastic of
the graph-changing process
Balanced Training for Sparse GANs
Over the past few years, there has been growing interest in developing larger
and deeper neural networks, including deep generative models like generative
adversarial networks (GANs). However, GANs typically come with high
computational complexity, leading researchers to explore methods for reducing
the training and inference costs. One such approach gaining popularity in
supervised learning is dynamic sparse training (DST), which maintains good
performance while enjoying excellent training efficiency. Despite its potential
benefits, applying DST to GANs presents challenges due to the adversarial
nature of the training process. In this paper, we propose a novel metric called
the balance ratio (BR) to study the balance between the sparse generator and
discriminator. We also introduce a new method called balanced dynamic sparse
training (ADAPT), which seeks to control the BR during GAN training to achieve
a good trade-off between performance and computational cost. Our proposed
method shows promising results on multiple datasets, demonstrating its
effectiveness.Comment: Accepted at NeurIPS 2023
(https://neurips.cc/virtual/2023/poster/70078). Our code will be released at
https://github.com/YiteWang/ADAP
- …