Learning computationally efficient dictionaries and their implementation as fast transforms

Dictionary learning is a branch of signal processing and machine learning that aims at finding a frame (called dictionary) in which some training data admits a sparse representation. The sparser the representation, the better the dictionary. The resulting dictionary is in general a dense matrix, and its manipulation can be computationally costly both at the learning stage and later in the usage of this dictionary, for tasks such as sparse coding. Dictionary learning is thus limited to relatively small-scale problems. In this paper, inspired by usual fast transforms, we consider a general dictionary structure that allows cheaper manipulation, and propose an algorithm to learn such dictionaries --and their fast implementation-- over training data. The approach is demonstrated experimentally with the factorization of the Hadamard matrix and with synthetic dictionary learning experiments

Strategies to learn computationally efficient and compact dictionaries

International audienceDictionary learning is a branch of signal processing and machine learning that aims at expressing some given training data matrix as the multiplication of two factors: one dense matrix called dictionary and one sparse matrix being the representation of the data in the dictionary. The sparser the representation, the better the dictionary. However, manipulating the dictionary as a dense matrix can be computationally costly both in the learning process and later in the usage of this dictionary, thus limiting dictionary learning to relatively small-scale problems. In this paper we consider a general structure of dictionary allowing faster manipulation, and give an algorithm to learn such dictionaries over training data, as well as preliminary results showing the interest of our approach

Flexible Multi-layer Sparse Approximations of Matrices and Applications

The computational cost of many signal processing and machine learning techniques is often dominated by the cost of applying certain linear operators to high-dimensional vectors. This paper introduces an algorithm aimed at reducing the complexity of applying linear operators in high dimension by approximately factorizing the corresponding matrix into few sparse factors. The approach relies on recent advances in non-convex optimization. It is first explained and analyzed in details and then demonstrated experimentally on various problems including dictionary learning for image denoising, and the approximation of large matrices arising in inverse problems

Proceedings of the second "international Traveling Workshop on Interactions between Sparse models and Technology" (iTWIST'14)

The implicit objective of the biennial "international - Traveling Workshop on Interactions between Sparse models and Technology" (iTWIST) is to foster collaboration between international scientific teams by disseminating ideas through both specific oral/poster presentations and free discussions. For its second edition, the iTWIST workshop took place in the medieval and picturesque town of Namur in Belgium, from Wednesday August 27th till Friday August 29th, 2014. The workshop was conveniently located in "The Arsenal" building within walking distance of both hotels and town center. iTWIST'14 has gathered about 70 international participants and has featured 9 invited talks, 10 oral presentations, and 14 posters on the following themes, all related to the theory, application and generalization of the "sparsity paradigm": Sparsity-driven data sensing and processing; Union of low dimensional subspaces; Beyond linear and convex inverse problem; Matrix/manifold/graph sensing/processing; Blind inverse problems and dictionary learning; Sparsity and computational neuroscience; Information theory, geometry and randomness; Complexity/accuracy tradeoffs in numerical methods; Sparsity? What's next?; Sparse machine learning and inference.Comment: 69 pages, 24 extended abstracts, iTWIST'14 website: http://sites.google.com/site/itwist1

High Dimensional Dependent Data Analysis for Neuroimaging.

This dissertation contains three projects focusing on two major high-dimensional problems for dependent data, particularly neuroimaging data: multiple testing and estimation of large covariance/precision matrices. Project 1 focuses on the multiple testing problem. Traditional voxel-level false discovery rate (FDR) controlling procedures for neuroimaging data often ignore the spatial correlations among neighboring voxels, thus suffer from substantial loss of efficiency in reducing the false non-discovery rate. We extend the one-dimensional hidden Markov chain based local-significance-index procedure to three-dimensional hidden Markov random field (HMRF). To estimate model parameters, a generalized EM algorithm is proposed for maximizing the penalized likelihood. Simulations show increased efficiency of the proposed approach over commonly used FDR controlling procedures. We apply the method to the comparison between patients with mild cognitive impairment and normal controls in the ADNI FDG-PET imaging study. Project 2 considers estimating large covariance and precision matrices from temporally dependent observations, in particular, the resting-state functional MRI (rfMRI) data in brain functional connectivity studies. Existing work on large covariance and precision matrices is primarily for i.i.d. observations. The rfMRI data from the Human Connectome Project, however, are shown to have long-range memory. Assuming a polynomial-decay-dominated temporal dependence, we obtain convergence rates for the generalized thresholding estimation of covariance and correlation matrices, and for the constrained $ell_1$ minimization and the $ell_1$ penalized likelihood estimation of precision matrix. Properties of sparsistency and sign-consistency are also established. We apply the considered methods to estimating the functional connectivity from single-subject rfMRI data. Project 3 extends Project 2 to multiple independent samples of temporally dependent observations. This is motivated by the group-level functional connectivity analysis using rfMRI data, where each subject has a sample of temporally dependent image observations. We use different concentration inequalities to obtain faster convergence rates than those in Project 2 of the considered estimators for multi-sample data. The new proof allows more general within-sample temporal dependence. We also discuss a potential way of improving the convergence rates by using a weighted sample covariance matrix. We apply the considered methods to the functional connectivity estimation for the ADHD-200 rfMRI data.PhDBiostatisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/133198/1/haishu_1.pd