In this paper a new Bayesian model for sparse linear regression with a
spatio-temporal structure is proposed. It incorporates the structural
assumptions based on a hierarchical Gaussian process prior for spike and slab
coefficients. We design an inference algorithm based on Expectation Propagation
and evaluate the model over the real data.Comment: SPARS 201

Isupova, Olga

Kuzin, Danil

Mihaylova, Lyudmila

English

arXiv

In this paper a new Bayesian model for sparse linear regression with a spatio-temporal structure is proposed. It incorporates the structural assumptions based on a hierarchical Gaussian process prior for spike and slab coefficients. We design an inference algorithm based on Expectation Propagation and evaluate the model over the real data

OPUS

        Citation for published version:Kuzin, D, Isupova, O & Mihaylova, L 2017 'Structured Sparse Modelling with Hierarchical GP'.Publication date:2017Link to publicationUniversity of BathAlternative formatsIf you require this document in an alternative format, please contact:openaccess@bath.ac.ukGeneral rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Download date: 12. Jan. 2023Structured Sparse Modelling with Hierarchical GPDanil Kuzin, Olga Isupova, Lyudmila MihaylovaThe University of Sheffield, Sheffield, UKEmail: dkuzin1@sheffield.ac.uk, oisupova1@sheffield.ac.uk, l.s.mihaylova@sheffield.ac.ukAbstract—In this paper a new Bayesian model for sparse linearregression with a spatio-temporal structure is proposed. It incorporatesthe structural assumptions based on a hierarchical Gaussian process priorfor spike and slab coefficients. We design an inference algorithm basedon Expectation Propagation and evaluate the model over the real data.I. INTRODUCTIONSparse regression problems arise often in various applications, e.g.,model selection, compressive sensing, EEG source localisation andgene modelling [1], [2]. One of the Bayesian approaches to force thecoefficients being zeros is the spike and slab prior [3]: each componentis modelled as a mixture of spike, that is the delta-function in zero,and slab, that is some vague distribution. Following the Bayesianapproach, latent variables that are indicators of spikes are added tothe model [4] and the relevant distribution is placed over them [5].In this model each component is modelled to be spike or slabindependently. However, in many applications non-zero elements tendto appear in groups forming an unknown structure: wavelet coefficientsof images are usually organised in trees [6], chromosomes have aspatial structure along the genome [2].We propose an extension of the spike and slab model by imposing ahierarchical Gaussian process (GP) prior on the latent variables. Suchhierarchical prior allows to model spatial structural dependencies forcoefficients that can evolve in time. The new model is flexible asspatial and temporal dependencies are decoupled by different levelsof the hierarchical GP prior.II. PROPOSED MODELThe observations yt ∈ RK are collected with the design matrixX ∈ RK×N from the unknown coefficients βt ∈ RN at every timemoment t ∈ [1, . . . , T ], with independent noise:yt ∼ N (yt|Xβt, σ2I). (1)We consider the case when K < N , therefore the problem of recoveryof βt from yt is ill-posed and regularisation is required.a) Sparsity: The vectors βt are assumed to be sparse, that isimplemented in the model using the spike and slab approach:βit ∼ ωitδ0(βit) + (1− ωit)N (βit|0, σ2β), (2)where ωit are the latent indicators of spike and slab.b) Spatial clustering: Non-zero elements in βt are assumed to beclustered in groups at every timestamp. Therefore spatial dependenciesfor the positions of spikes in βit are modelled with the GP:ωit ∼ Ber(ωit|Φ(γit)), Φ(·) is the standard Gaussian cdf (3)γt ∼ N (γt|µt,Σ0), Σ0(i, j) = αΣ exp(− (i− j)22l2Σ). (4)Olga Isupova and Lyudmila Mihaylova acknowledge the support from theEC Seventh Framework Programme [FP7 2013-2017] TRAcking in compleXsensor systems (TRAX) Grant agreement no.: 607400.GPs specify prior over an unknown structure. This is particularlyuseful as it allows to avoid a specification of any structural patterns —structural modelling is governed only by the GP covariance function.c) Temporal evolution: Clusters of spikes in βt are assumed toevolve in time. This evolution is addressed with the hierarchical GPdynamic system model [7]. The mean for the spatial GP changes overtime according to the top-level temporal GP:µt ∼ N (µt|µt−1,W), W(i, j) = αW exp(− (i− j)22l2W). (5)This allows to implicitly specify the prior over the transition functionof the structure. The rate of the evolution is controlled with thetop-level GP covariance function.The exact posterior of the parameters is intractable, thereforeapproximate inference methods are required. Inference is based onExpectation Propagation [8] in this paper.The structural assumptions in sparse models are studied in theliterature. The group lasso [9] provides sparse solutions for predefinedgroups of coefficients. Group constraints for sparse models includesmooth relevance vector machines [10], Boltzmann machine prior [11];spatio-temporal coupling of the parameters [12], [13]. In [14] a spatio-temporal structure is modelled with a one-level GP prior. In contrast tothat model the new one introduces an additional level of a GP prior fortemporal dependencies, therefore the temporal and spatial structuresare decoupled, adding flexibility to the model. The high-level GPcontrols the change of spike groups in time while the low-level GPallows the local changes within each group.III. NUMERICAL EXPERIMENTSThe performance of the proposed hierarchical GP algorithm iscompared with the one-level GP prior introduced in [14]. We applyboth algorithms to the problem of object detection in video sequences.The Convoy dataset [15] is used where the frame difference is appliedfor moving object detection. The sparse observations are obtained asyt = Xβt, where X is the matrix with i.i.d. Gaussian elements,the number of observations is ≈ 40% of the dimension of thehidden signal βt. This procedure corresponds to compressive sensingobservations [16].The reconstruction results on the sample frames are presented inFigure 1. The obtained performance measures values can be foundin Table I. The F-measure compares binary masks computed bythresholding the true vectors β and the posterior estimates β̂. NMSErepresents normalised mean squared error between β and β̂. Theproposed algorithm shows better results in terms of both measures.IV. CONCLUSIONIn this work we propose a new approach for spatial structuremodelling in sparse models that allows to capture complex temporalevolution of data patterns. We also develop an efficient inferencemethod based on EP. The numerical experiments show superiority ofthe proposed model over the current state-of-the-art.arXiv:1704.08727v1  [stat.ML]  27 Apr 2017(a) Original frame (b) Frame with subtracted background (c) Hierarchical GP reconstruction (d) One-level GP reconstructionFig. 1. Sample frame with reconstruction results from sparse measurements. (a): the original non-compressed frame; (b): object detection results based onnon-compressed frame difference (static background frame is subtracted from the original frame); (c): reconstruction of compressed object detection resultsbased on the proposed hierarchical GPs method; (d): reconstruction of the compressed object detection results based on the one-level GP method. (b) representsthe true βt, while (c) and (d) are posterior estimates β̂t obtained by the reconstruction algorithms.TABLE IPERFORMANCE OF SIGNAL RECONSTRUCTION FOR BACKGROUNDSUBTRACTION IN VIDEOMeasure Hierarchical GP One-level GPNMSE 0.0153 0.0392F-measure 0.9870 0.9403REFERENCES[1] J. Mairal, F. Bach, and J. Ponce, “Sparse modeling for image and visionprocessing,” Foundations and Trends in Computer Graphics and Vision,vol. 8, no. 2-3, pp. 85–283, 2014.[2] T. Hastie, R. Tibshirani, and M. Wainwright, Statistical Learning withSparsity: The Lasso and Generalizations. CRC Press, 2015.[3] T. J. Mitchell and J. J. Beauchamp, “Bayesian variable selection in linearregression,” Journal of the American Statistical Association, vol. 83, no.404, pp. 1023–1032, 1988.[4] N. G. Polson and J. G. Scott, “Shrink globally, act locally: SparseBayesian regularization and prediction,” Bayesian Statistics, vol. 9, pp.501–538, 2010.[5] K. P. Murphy, Machine learning: a probabilistic perspective. MIT press,2012.[6] S. Mallat, A Wavelet Tour of Signal Processing, Third Edition: The SparseWay, 3rd ed. Academic Press, 2008.[7] M. Deisenroth and S. Mohamed, “Expectation propagation in Gaus-sian process dynamical systems,” in Advances in Neural InformationProcessing Systems 25, 2012, pp. 2609–2617.[8] T. P. Minka, “Expectation propagation for approximate Bayesian infer-ence,” in Proceedings of the 17th Conference on Uncertainty in ArtificialIntelligence, 2001, pp. 362–369.[9] M. Yuan and Y. Lin, “Model selection and estimation in regression withgrouped variables,” Journal of the Royal Statistical Society: Series B(Statistical Methodology), vol. 68, no. 1, pp. 49–67, 2006.[10] A. Schmolck, “Smooth relevance vector machines,” Ph.D. dissertation,University of Exeter, 2008.[11] A. Drémeau, C. Herzet, and L. Daudet, “Boltzmann machine andmean-field approximation for structured sparse decompositions,” IEEETransactions on Signal Processing, vol. 60, no. 7, pp. 3425–3438, 2012.[12] M. A. Van Gerven, B. Cseke, F. P. De Lange, and T. Heskes, “EfficientBayesian multivariate fMRI analysis using a sparsifying spatio-temporalprior,” NeuroImage, vol. 50, no. 1, pp. 150–161, 2010.[13] A. Wu, M. Park, O. O. Koyejo, and J. W. Pillow, “Sparse Bayesianstructure learning with dependent relevance determination priors,” inAdvances in Neural Information Processing Systems 27, 2014, pp. 1628–1636.[14] M. R. Andersen, A. Vehtari, O. Winther, and L. K. Hansen, “Bayesianinference for spatio-temporal spike and slab priors,” arXiv preprintarXiv:1509.04752, 2015.[15] G. Warnell, S. Bhattacharya, R. Chellappa, and T. Basar, “Adaptive-rate compressive sensing using side information,” IEEE Transactions onImage Processing, vol. 24, no. 11, pp. 3846–3857, 2015.[16] V. Cevher, A. Sankaranarayanan, M. F. Duarte, D. Reddy, and R. G.Baraniuk, “Compressive sensing for background subtraction,” in Pro-ceedings of 10th European Conference on Computer Vision, 2008, pp.155–168.

Structured Sparse Modelling with Hierarchical GP

Abstract

Similar works

Full text

Available Versions

OPUS