Autoencoders are unsupervised machine learning circuits whose learning goal
is to minimize a distortion measure between inputs and outputs. Linear
autoencoders can be defined over any field and only real-valued linear
autoencoder have been studied so far. Here we study complex-valued linear
autoencoders where the components of the training vectors and adjustable
matrices are defined over the complex field with the L2 norm. We provide
simpler and more general proofs that unify the real-valued and complex-valued
cases, showing that in both cases the landscape of the error function is
invariant under certain groups of transformations. The landscape has no local
minima, a family of global minima associated with Principal Component Analysis,
and many families of saddle points associated with orthogonal projections onto
sub-space spanned by sub-optimal subsets of eigenvectors of the covariance
matrix. The theory yields several iterative, convergent, learning algorithms, a
clear understanding of the generalization properties of the trained
autoencoders, and can equally be applied to the hetero-associative case when
external targets are provided. Partial results on deep architecture as well as
the differential geometry of autoencoders are also presented. The general
framework described here is useful to classify autoencoders and identify
general common properties that ought to be investigated for each class,
illuminating some of the connections between information theory, unsupervised
learning, clustering, Hebbian learning, and autoencoders.Comment: Final version, journal ref adde