Factor Analysis and Alternating Minimization

In this paper we make a first attempt at understanding how to build an optimal approximate normal factor analysis model. The criterion we have chosen to evaluate the distance between different models is the I-divergence between the corresponding normal laws. The algorithm that we propose for the construction of the best approximation is of an the alternating minimization kind

Approximate Nonnegative Matrix Factorization via Alternating Minimization

In this paper we consider the Nonnegative Matrix Factorization (NMF) problem: given an (elementwise) nonnegative matrix $V \in \R_+^{m\times n}$ find, for assigned $k$, nonnegative matrices $W\in\R_+^{m\times k}$ and $H\in\R_+^{k\times n}$ such that $V=WH$. Exact, non trivial, nonnegative factorizations do not always exist, hence it is interesting to pose the approximate NMF problem. The criterion which is commonly employed is I-divergence between nonnegative matrices. The problem becomes that of finding, for assigned $k$, the factorization $WH$ closest to $V$ in I-divergence. An iterative algorithm, EM like, for the construction of the best pair $(W, H)$ has been proposed in the literature. In this paper we interpret the algorithm as an alternating minimization procedure \`a la Csisz\'ar-Tusn\'ady and investigate some of its stability properties. NMF is widespreading as a data analysis method in applications for which the positivity constraint is relevant. There are other data analysis methods which impose some form of nonnegativity: we discuss here the connections between NMF and Archetypal Analysis. An interesting system theoretic application of NMF is to the problem of approximate realization of Hidden Markov Models

Consistent Estimation of the Order for Markov and Hidden Markov Chains

The structural parameters of many statistical models can be estimated maximizing a penalized version of the likelihood function. We use this idea to construct strongly consistent estimators of the order for Markov Chains and Hidden Markov Chain models. The specification of the penalty term requires precise information on the rate of growth of the maximized likelihood ratio. For Markov chain models we determine the rate using the Law of the Iterated Logarithm. For Hidden Markov chain models we find an upper bound to the rate using results from Information Theory. We give sufficient conditions on the penalty term to avoid overestimation and underestimation of the order. Examples of penalty terms that generate strongly consitent estimators are also given

Order Determination for Probabilistic Functions of Finite Markov Chains

Let {Y sub t} be a stationary stochastic process with values in the finite set YY. We model {Y sub t} as a probabilistic function of a finite state Markov Chain {X sub t} i.e. X sub t is such that: P[Y sub t | X sup t, Y sup t-1] = P[Y sub t | X sub t] Define the cardinality of the state space of {X sub t} as the order of the model. The problem is to determine the order given the observations {y sub 1, y sub 2, y sub T}. We show that under mild conditions on the probability distribution function P sub Y (.) of {Y sub t} the order is identifiable and can be consistently determined from the data

Approximation of nonnegative systems by moving averages of fixed order

Contains fulltext : 206990.pdf (preprint version ) (Open Access