469 research outputs found
Comparison between Oja's and BCM neural networks models in finding useful projections in high-dimensional spaces
This thesis presents the concept of a neural network starting from its corresponding biological model, paying particular attention to the learning algorithms proposed by Oja and Bienenstock Cooper & Munro. A brief introduction to Data Analysis is then performed, with particular reference to the Principal Components Analysis and Singular Value Decomposition.
The two previously introduced algorithms are then dealt with more thoroughly, going to study in particular their connections with data analysis. Finally, it is proposed to use the Singular Value Decomposition as a method for obtaining stationary points in the BCM algorithm, in the case of linearly dependent inputs
An Implicit Form of Krasulina's k-PCA Update without the Orthonormality Constraint
We shed new insights on the two commonly used updates for the online -PCA
problem, namely, Krasulina's and Oja's updates. We show that Krasulina's update
corresponds to a projected gradient descent step on the Stiefel manifold of the
orthonormal -frames, while Oja's update amounts to a gradient descent step
using the unprojected gradient. Following these observations, we derive a more
\emph{implicit} form of Krasulina's -PCA update, i.e. a version that uses
the information of the future gradient as much as possible. Most interestingly,
our implicit Krasulina update avoids the costly QR-decomposition step by
bypassing the orthonormality constraint. We show that the new update in fact
corresponds to an online EM step applied to a probabilistic -PCA model. The
probabilistic view of the updates allows us to combine multiple models in a
distributed setting. We show experimentally that the implicit Krasulina update
yields superior convergence while being significantly faster. We also give
strong evidence that the new update can benefit from parallelism and is more
stable w.r.t. tuning of the learning rate
- …