23,961 research outputs found
The difference between memory and prediction in linear recurrent networks
Recurrent networks are trained to memorize their input better, often in the
hopes that such training will increase the ability of the network to predict.
We show that networks designed to memorize input can be arbitrarily bad at
prediction. We also find, for several types of inputs, that one-node networks
optimized for prediction are nearly at upper bounds on predictive capacity
given by Wiener filters, and are roughly equivalent in performance to randomly
generated five-node networks. Our results suggest that maximizing memory
capacity leads to very different networks than maximizing predictive capacity,
and that optimizing recurrent weights can decrease reservoir size by half an
order of magnitude
full-FORCE: A Target-Based Method for Training Recurrent Networks
Trained recurrent networks are powerful tools for modeling dynamic neural
computations. We present a target-based method for modifying the full
connectivity matrix of a recurrent network to train it to perform tasks
involving temporally complex input/output transformations. The method
introduces a second network during training to provide suitable "target"
dynamics useful for performing the task. Because it exploits the full recurrent
connectivity, the method produces networks that perform tasks with fewer
neurons and greater noise robustness than traditional least-squares (FORCE)
approaches. In addition, we show how introducing additional input signals into
the target-generating network, which act as task hints, greatly extends the
range of tasks that can be learned and provides control over the complexity and
nature of the dynamics of the trained, task-performing network.Comment: 20 pages, 8 figure
Optimal modularity and memory capacity of neural reservoirs
The neural network is a powerful computing framework that has been exploited
by biological evolution and by humans for solving diverse problems. Although
the computational capabilities of neural networks are determined by their
structure, the current understanding of the relationships between a neural
network's architecture and function is still primitive. Here we reveal that
neural network's modular architecture plays a vital role in determining the
neural dynamics and memory performance of the network of threshold neurons. In
particular, we demonstrate that there exists an optimal modularity for memory
performance, where a balance between local cohesion and global connectivity is
established, allowing optimally modular networks to remember longer. Our
results suggest that insights from dynamical analysis of neural networks and
information spreading processes can be leveraged to better design neural
networks and may shed light on the brain's modular organization
Echo State Networks for Proactive Caching in Cloud-Based Radio Access Networks with Mobile Users
In this paper, the problem of proactive caching is studied for cloud radio
access networks (CRANs). In the studied model, the baseband units (BBUs) can
predict the content request distribution and mobility pattern of each user,
determine which content to cache at remote radio heads and BBUs. This problem
is formulated as an optimization problem which jointly incorporates backhaul
and fronthaul loads and content caching. To solve this problem, an algorithm
that combines the machine learning framework of echo state networks with
sublinear algorithms is proposed. Using echo state networks (ESNs), the BBUs
can predict each user's content request distribution and mobility pattern while
having only limited information on the network's and user's state. In order to
predict each user's periodic mobility pattern with minimal complexity, the
memory capacity of the corresponding ESN is derived for a periodic input. This
memory capacity is shown to be able to record the maximum amount of user
information for the proposed ESN model. Then, a sublinear algorithm is proposed
to determine which content to cache while using limited content request
distribution samples. Simulation results using real data from Youku and the
Beijing University of Posts and Telecommunications show that the proposed
approach yields significant gains, in terms of sum effective capacity, that
reach up to 27.8% and 30.7%, respectively, compared to random caching with
clustering and random caching without clustering algorithm.Comment: Accepted in the IEEE Transactions on Wireless Communication
- …