Sample Complexity of Dictionary Learning and other Matrix Factorizations

Many modern tools in machine learning and signal processing, such as sparse dictionary learning, principal component analysis (PCA), non-negative matrix factorization (NMF), $K$-means clustering, etc., rely on the factorization of a matrix obtained by concatenating high-dimensional vectors from a training collection. While the idealized task would be to optimize the expected quality of the factors over the underlying distribution of training vectors, it is achieved in practice by minimizing an empirical average over the considered collection. The focus of this paper is to provide sample complexity estimates to uniformly control how much the empirical average deviates from the expected cost function. Standard arguments imply that the performance of the empirical predictor also exhibit such guarantees. The level of genericity of the approach encompasses several possible constraints on the factors (tensor product structure, shift-invariance, sparsity \ldots), thus providing a unified perspective on the sample complexity of several widely used matrix factorization schemes. The derived generalization bounds behave proportional to $\sqrt{\log(n)/n}$ w.r.t.\ the number of samples $n$ for the considered matrix factorization techniques.Comment: to appea

A lower complexity K best algorithm for multiple input and multiple output detection

This paper presents Multiple Input Multiple Output (MIMO) detection steps using tree search based method known as the ‘K’ best algorithm. This low complexity algorithm is based on probabilistic approach of sphere decoding with self adjustable capability depending on the levels (root, branch, leaf etc.) of a tree. While the tree was searched to estimate the transmitted symbols level by level, the algorithm took into account the effect of the undetected symbols in the search criteria. Simulation results showed that the proposed method reduced complexity (in terms of the average number of visited nodes) about 10% for higher (medium to high) signal to noise ratio (SNR) values without degrading the system BER performance

Distributive Stochastic Learning for Delay-Optimal OFDMA Power and Subband Allocation

In this paper, we consider the distributive queue-aware power and subband allocation design for a delay-optimal OFDMA uplink system with one base station, $K$ users and $N_F$ independent subbands. Each mobile has an uplink queue with heterogeneous packet arrivals and delay requirements. We model the problem as an infinite horizon average reward Markov Decision Problem (MDP) where the control actions are functions of the instantaneous Channel State Information (CSI) as well as the joint Queue State Information (QSI). To address the distributive requirement and the issue of exponential memory requirement and computational complexity, we approximate the subband allocation Q-factor by the sum of the per-user subband allocation Q-factor and derive a distributive online stochastic learning algorithm to estimate the per-user Q-factor and the Lagrange multipliers (LM) simultaneously and determine the control actions using an auction mechanism. We show that under the proposed auction mechanism, the distributive online learning converges almost surely (with probability 1). For illustration, we apply the proposed distributive stochastic learning framework to an application example with exponential packet size distribution. We show that the delay-optimal power control has the {\em multi-level water-filling} structure where the CSI determines the instantaneous power allocation and the QSI determines the water-level. The proposed algorithm has linear signaling overhead and computational complexity $\mathcal O(KN)$, which is desirable from an implementation perspective.Comment: To appear in Transactions on Signal Processin

Probing variability patterns of the Fe K line complex in bright nearby AGNs

The unprecedented sensitivity of current X-ray telescopes allows for the first time to address the issue of the Fe K line complex variability patterns in bright, nearby AGNs. We examine XMM-Newton observations of the brightest sources of the FERO sample of radio-quiet type 1 AGNs with the aim of characterizing the temporal behaviour of Fe K complex features. A systematic mapping of residual flux above and below the continuum in the 4-9 keV range is performed in the time vs energy domain, with the purpose of identifying interesting spectral features in the three energy bands: 5.4-6.1 keV, 6.1-6.8 keV and 6.8-7.2 keV, corresponding respectively to the redshifted, rest frame and blueshifted or highly ionized Fe Kalpha line bands. The variability significance is assessed by extracting light curves and comparing them with MonteCarlo simulations. The time-averaged profile of the Fe K complex revealed spectral complexity in several observations. Red- and blue-shifted components (either in emission or absorption) were observed in 30 out of 72 observations, with an average ~90 eV for emission and ~ -30 eV for absorption features. We detected significant line variability (with confidence levels ranging between 90% and 99.7%) within at least one of the above energy bands in 26 out of 72 observations on time scales of ~6-30 ks. Reliability of these features has been carefully calculated using this sample and has been assessed at ~3sigma confidence level. This work increases the currently scanty number of detections of variable, energy shifted, Fe lines and confirms the reliability of the claimed detections. We found that the distribution of detected features is peaked at high variability significances in the red- and blue-shifted energy bands, suggesting an origin in a relativistically modified accretion flow.Comment: Accepted for publication in Astronomy & Astrophysic

Necessary and Sufficient Conditions on Sparsity Pattern Recovery

The problem of detecting the sparsity pattern of a k-sparse vector in R^n from m random noisy measurements is of interest in many areas such as system identification, denoising, pattern recognition, and compressed sensing. This paper addresses the scaling of the number of measurements m, with signal dimension n and sparsity-level nonzeros k, for asymptotically-reliable detection. We show a necessary condition for perfect recovery at any given SNR for all algorithms, regardless of complexity, is m = Omega(k log(n-k)) measurements. Conversely, it is shown that this scaling of Omega(k log(n-k)) measurements is sufficient for a remarkably simple ``maximum correlation'' estimator. Hence this scaling is optimal and does not require more sophisticated techniques such as lasso or matching pursuit. The constants for both the necessary and sufficient conditions are precisely defined in terms of the minimum-to-average ratio of the nonzero components and the SNR. The necessary condition improves upon previous results for maximum likelihood estimation. For lasso, it also provides a necessary condition at any SNR and for low SNR improves upon previous work. The sufficient condition provides the first asymptotically-reliable detection guarantee at finite SNR.Comment: Submitted to IEEE Transactions on Information Theor

Fast and Robust Parametric Estimation of Jointly Sparse Channels

We consider the joint estimation of multipath channels obtained with a set of receiving antennas and uniformly probed in the frequency domain. This scenario fits most of the modern outdoor communication protocols for mobile access or digital broadcasting among others. Such channels verify a Sparse Common Support property (SCS) which was used in a previous paper to propose a Finite Rate of Innovation (FRI) based sampling and estimation algorithm. In this contribution we improve the robustness and computational complexity aspects of this algorithm. The method is based on projection in Krylov subspaces to improve complexity and a new criterion called the Partial Effective Rank (PER) to estimate the level of sparsity to gain robustness. If P antennas measure a K-multipath channel with N uniformly sampled measurements per channel, the algorithm possesses an O(KPNlogN) complexity and an O(KPN) memory footprint instead of O(PN^3) and O(PN^2) for the direct implementation, making it suitable for K << N. The sparsity is estimated online based on the PER, and the algorithm therefore has a sense of introspection being able to relinquish sparsity if it is lacking. The estimation performances are tested on field measurements with synthetic AWGN, and the proposed algorithm outperforms non-sparse reconstruction in the medium to low SNR range (< 0dB), increasing the rate of successful symbol decodings by 1/10th in average, and 1/3rd in the best case. The experiments also show that the algorithm does not perform worse than a non-sparse estimation algorithm in non-sparse operating conditions, since it may fall-back to it if the PER criterion does not detect a sufficient level of sparsity. The algorithm is also tested against a method assuming a "discrete" sparsity model as in Compressed Sensing (CS). The conducted test indicates a trade-off between speed and accuracy.Comment: 11 pages, 9 figures, submitted to IEEE JETCAS special issue on Compressed Sensing, Sep. 201