Kung, H. T.

Lin, Tsung-Han

Harvard University - DASH 

 Computing Sparse Representations in O(N log N) Time  (Article begins on next page)The Harvard community has made this article openly available.Please share how this access benefits you. Your story matters.Citation Lin, Tsung-Han and H. T. Kung. 2013. Computing SparseRepresentations in O(N log N) Time. Abstract. Paper presented atSignal Processing with Adaptive Sparse StructuredRepresentations (SPARS 2013), École Polytechnique Fédérale deLausanne, Switzerland, July 8-11.Published Version http://spars2013.epfl.ch/data/_uploaded/flash/Paper%2059.pdfAccessed February 19, 2015 5:14:51 PM ESTCitable Link http://nrs.harvard.edu/urn-3:HUL.InstRepos:12667563Terms of Use This article was downloaded from Harvard University's DASHrepository, and is made available under the terms and conditionsapplicable to Open Access Policy Articles, as set forth athttp://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#OAPComputing Sparse Representations in O(N logN) TimeTsung-Han Lin and H. T. KungSchool of Engineering and Applied Sciences, Harvard Universityfthlin, htkg@eecs.harvard.eduI. INTRODUCTIONMachine learning concerns forming representations of input obser-vations to facilitate tasks such as classiﬁcation. A recent insight indeep learning [1] is to use a deep architecture that stacks multiplelevels of nonlinear operations in an inference hierarchy to extractdifferent layers of abstractions. Deep learning is a promising directionand has attained state-of-the-art performance in some applicationareas such as computer vision and speech recognition. In this work,we aim to improve the computational efﬁciency of deep learning.A key step in deep learning is to represent input signals aslayers of sparse representations. In image processing, this meansto progressively describe objects using features of larger spatialscales. For example, in the bottom layer objects can be representedby edges of different widths, lengths and orientations over smallregions. In higher layers objects may be represented by shapes suchas squares, triangles and so on over large regions. The representationsare computed based on some designed or learned dictionary. At eachlayer an input signal is represented using just a few dictionary atoms.We consider the orthogonal matching pursuit (OMP) algorithm [5],which forms sparse representations by greedily selecting representingdictionary atoms. The computation cost of OMP is proportional to thedimensions of the dictionary. Suppose that the input signal is an M1vector x, and the dictionary has N M1 atoms di, i = 1;:::;N. Thebulk of the sparse representation computation amounts to computingN correlations between x and di for all i. Thus the total computationcost is O(MN).Note that N is usually governed by the characteristics of a givenmachine learning task. For example, if the task is to classify objectswith a large number of categories, N tends to be large for an increasedchance of representing x with just a few dictionary atoms. On theother hand, with an easier task such as differentiating between only afew different looking objects, a relatively small N may be sufﬁcientto derive discernible representations. M, however, is driven by theinput signal size and the size of intermediate sparse representationscomputed in the hierarchy. This means M can be large especiallyin higher layers of the learning framework. For example, in imageprocessing, sparse representations for small local regions of animage are formed in the bottom layer. The representations are thenaggregated and vectorized over a larger neighborhood as the inputsignal for the next layer, which can easily be very long.We show that this O(MN) cost can be reduced to O(N logN),a complexity independent of the signal or representation size M.This means that the computation cost is only dictated by the desiredclassiﬁcation resolution.II. PERFORMING OMP IN THE COMPRESSED DOMAINWe propose to move the computations of OMP to a lower di-mensional subspace, i.e., a compressed domain, thereby signiﬁcantlyreducing the data size and computation cost. We ﬁrst state our resultsfor a signal x that can be exactly represented using K atoms froma normalized M  N dictionary D, meaning that x = Df, where fis a sparse vector. We deﬁne a linear mapping  : RM ! Rc M withc M < M and call the reduced-dimension problem of recovering ffrom x = Df the compressed domain recovery problem.Theorem 1. For any given  2 (0;1), OMP solves the compresseddomain recovery problem, where c M can be as small as O(logN=2),if the dictionary D is sufﬁciently incoherent, i.e.,  <12K 1  where is mutual coherence of D deﬁned as the maximum absolute innerproduct between any two atoms.The proof of Theorem 1 makes use of Johnson-Lindenstrausslemma [4] that there exists a  which embeds the N dictionary atomsin M-dimensional space to an O(logN=2)-dimensional subspace ina way that distances between atoms are nearly preserved subject to asmall distortion factor . Consequently, the inner product between anytwo atoms is also preserved up to  in the subspace [3]. This meansthat the coherence of the compressed dictionary D will not bechanged by more than  from that of D. Theorem 1 then follows froma sufﬁcient condition for the working of OMP that mutual coherencefor the dictionary needs to be sufﬁciently low [5].For constant  and K, by Theorem 1, c M can be O(logN). Thusthe computation cost of OMP is now O(N logN).III. COMPRESSED DOMAIN OMP IN IMAGE CLASSIFICATIONWe have tested compressed domain OMP for image classiﬁcation.Our implementation is similar to that in [2], and has two layers. Inthe ﬁrst layer, the sparse representations for local 6x6 patches arecomputed. The representations computed in layer 1 are aggregated inlayer 2 over 16x16 patches and re-encoded to form higher-level sparserepresentations. These representations are then aggregated over thewhole image to form an image-level representation. The dictionariesat layer 1 and 2 are set to have dimensions 36108 and 22681000,and the sparsity K is set to 5 and 10, respectively.We use 20 object categories from the Caltech-101 data set. Afterobtaining the image representations, a linear SVM is trained with30 images for each category, and we evaluate the classiﬁcationperformance over 655 images. We compress the dictionary at layer2 using random Bernoulli matrices with 2x and 10x compressionratios, giving dictionaries of size 1134  1000 and 226  1000.Without any compression, the classiﬁcation accuracy is 78.9%. With2x and 10x compression, the classiﬁcation accuracy is 78% and 75%,respectively. This suggests that the computation cost of OMP can bereduced by 10x with less than 4% decrease in classiﬁcation accuracy.REFERENCES[1] Y. Bengio. Learning deep architectures for AI. Foundations and Trendsin Machine Learning, 2(1):1–127, 2009.[2] L. Bo, X. Ren, and D. Fox. Hierarchical matching pursuit for imageclassiﬁcation: Architecture and fast algorithms. In NIPS, 2011.[3] M. Davenport, P. Boufounos, M. Wakin, and R. Baraniuk. Signalprocessing with compressive measurements. Selected Topics in SignalProcessing, IEEE Journal of, 4(2):445–460, 2010.[4] W. Johnson and J. Lindenstrauss. Extensions of lipschitz mappings intoa hilbert space. Contemporary mathematics, 26(189-206):1, 1984.[5] J. Tropp. Greed is good: Algorithmic results for sparse approximation.Information Theory, IEEE Transactions on, 50(10):2231–2242, 2004.

Computing Sparse Representations in O(N log N) Time

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.421.3089

Computing Sparse Representations in O(N log N) Time

Abstract

Similar works

Full text

Available Versions

Harvard University - DASH