Cluster Variation Method in Statistical Physics and Probabilistic Graphical Models

The cluster variation method (CVM) is a hierarchy of approximate variational techniques for discrete (Ising--like) models in equilibrium statistical mechanics, improving on the mean--field approximation and the Bethe--Peierls approximation, which can be regarded as the lowest level of the CVM. In recent years it has been applied both in statistical physics and to inference and optimization problems formulated in terms of probabilistic graphical models. The foundations of the CVM are briefly reviewed, and the relations with similar techniques are discussed. The main properties of the method are considered, with emphasis on its exactness for particular models and on its asymptotic properties. The problem of the minimization of the variational free energy, which arises in the CVM, is also addressed, and recent results about both provably convergent and message-passing algorithms are discussed.Comment: 36 pages, 17 figure

Terrain analysis using radar shape-from-shading

This paper develops a maximum a posteriori (MAP) probability estimation framework for shape-from-shading (SFS) from synthetic aperture radar (SAR) images. The aim is to use this method to reconstruct surface topography from a single radar image of relatively complex terrain. Our MAP framework makes explicit how the recovery of local surface orientation depends on the whereabouts of terrain edge features and the available radar reflectance information. To apply the resulting process to real world radar data, we require probabilistic models for the appearance of terrain features and the relationship between the orientation of surface normals and the radar reflectance. We show that the SAR data can be modeled using a Rayleigh-Bessel distribution and use this distribution to develop a maximum likelihood algorithm for detecting and labeling terrain edge features. Moreover, we show how robust statistics can be used to estimate the characteristic parameters of this distribution. We also develop an empirical model for the SAR reflectance function. Using the reflectance model, we perform Lambertian correction so that a conventional SFS algorithm can be applied to the radar data. The initial surface normal direction is constrained to point in the direction of the nearest ridge or ravine feature. Each surface normal must fall within a conical envelope whose axis is in the direction of the radar illuminant. The extent of the envelope depends on the corrected radar reflectance and the variance of the radar signal statistics. We explore various ways of smoothing the field of surface normals using robust statistics. Finally, we show how to reconstruct the terrain surface from the smoothed field of surface normal vectors. The proposed algorithm is applied to various SAR data sets containing relatively complex terrain structure

From Approximations to Decisions

Bayesian models capture the intrinsic variability of a data-generating process as a posterior distribution over the parameters of the model for the process. Decisions that are optimal for a user-defined loss are obtained by minimizing expectation of the loss over the posterior. Because posterior inference is often intractable approximations of the posterior are obtained either via sampling with Monte Carlo Markov chain methods or through variational methods which minimize a discrepancy measure between an approximation and the true posterior. Probabilistic programming offers practitioners tools that combine easy model specification with automatic approximate inference techniques. However, these techniques do not yet accommodate posterior calibrations that yield decisions that are optimal for the expected posterior loss. This thesis develops efficient and flexible variational approximations as well as density function transformations for flexible modeling of skewed data for use in probabilistic programs. It also proposes extensions to the Bayesian decision framework and a suite of automatic loss-sensitive inference techniques for decision-making under posterior approximations. Briefly, we make four concrete contributions: First, we exploit importance sampling to approximate the objective gradient and show how to speed up convergence in stochastic gradient and stochastic average gradient descent for variational inference. Next, we propose a new way to model skewed data in probabilistic programs by prescribing an improved version of the Lambert W distribution amenable to gradient-based inference. Lastly, we propose two new techniques to better integrate decision-making into probabilistic programs - a gradient-based optimization routine for the loss-calibrated variational objective, specifically for the challenging case of continuous losses, and an amalgamation of learning theory and Bayesian decision theory that utilizes a separate decision-making module to map the posterior to decisions minimizing the empirical risk.Tilastollisia koneoppimismalleja käytetään nykyisin laajalti eri sovelluksissa tietoaineistojen analysointiin, ennustustehtäviin ja päätöksenteon tukena. Eräs keskeinen haaste näille malleille on kohinaisiin havaintoihin liittyvän epävarmuuden huomioiminen. Bayesilainen päättely tarjoaa siihen perustellun tavan. Bayesilaiseen päättelyyn perustuvien koneoppimismallien avulla voidaan luotettavammin tehdä perusteltuja päätöksiä jotka huomioivat mallin epävarmuudet ja eri vaihtoehtoihin liittyvät hyödyt ja kustannukset. Bayesilaisten mallien toteuttamiseen voidaan käyttää todennäköisyysohjelmointia, jossa erityisellä kuvauskielellä kirjoitetun mallin päättelyyn käytetään malliriippumattomia ja laskennallisesti tehokkaita mutta likiarvoisia päättelyalgoritmeja. Tässä väitöskirjassa kehitetään todennäköisyysohjelmoinnin tarpeisiin aiempaa tehokkaampia päättelyalgoritmeja sekä työkaluja vinojen todennäköisyysjakaumien käsittelyyn. Lisäksi työssä keskitytään Bayesilaisten mallien käyttöön päätösongelmissa. Työssä osoitetaan kuinka likiarvoisen päättelyn pohjalta tehdyt päätökset eivät välttämättä ole optimaalisia ja esitetään tälle ongelmalle kaksi ratkaisua. Ensimmäisessä muokataan itse päättelyalgoritmia siten että mallin avulla lopulta tehtävät päätökset huomioidaan jo päättelyvaiheessa ja osoitetaan, että näin pystytään parantamaan ennusteiden ja päätösten luotettavuutta. Toinen ratkaisu puolestaan korjaa päätöksentekovaiheessa likiarvoisesta päättelystä johtuvia virheitä ja soveltuu käytettäväksi kaikkien päättelyalgoritmien kanssa

Data Science: Measuring Uncertainties

With the increase in data processing and storage capacity, a large amount of data is available. Data without analysis does not have much value. Thus, the demand for data analysis is increasing daily, and the consequence is the appearance of a large number of jobs and published articles. Data science has emerged as a multidisciplinary field to support data-driven activities, integrating and developing ideas, methods, and processes to extract information from data. This includes methods built from different knowledge areas: Statistics, Computer Science, Mathematics, Physics, Information Science, and Engineering. This mixture of areas has given rise to what we call Data Science. New solutions to the new problems are reproducing rapidly to generate large volumes of data. Current and future challenges require greater care in creating new solutions that satisfy the rationality for each type of problem. Labels such as Big Data, Data Science, Machine Learning, Statistical Learning, and Artificial Intelligence are demanding more sophistication in the foundations and how they are being applied. This point highlights the importance of building the foundations of Data Science. This book is dedicated to solutions and discussions of measuring uncertainties in data analysis problems

Structural damage assessment through parametric and nonparametric models

The main purpose of Structural Health Monitoring (SHM) is the assessment of structural conditions in aerospace, mechanical and civil systems. In structural engineering, damage is defined as any permanent change in the structural and geometric properties of a system caused by an external action. Vibration-based damage assessment methods rely on the use of sensors that record the structural dynamic response of a system that is determined by its structural and geometric properties. External disturbances and environmental conditions in which the system operates cause fluctuations of these properties and might hide the change in signature induced by damage. To handle the uncertainties in the determination of the structure’s characteristics, a statistical pattern recognition approach is presented in this thesis. Any statistical approach relies on the statistics of some features that provide a compact representation of the structural properties and that are sensitive to damage. Such features are called damage sensitive features and are extracted from the dynamic response of the structure: their statistical distribution is then analyzed to assess the occurrence of damage. This dissertation focuses on the analysis of the statistical distribution of damage sensitive features which are extracted through parametric and nonparametric algorithms. Cepstral coefficients are features defined in the field of acoustics and, in this thesis, they have been adapted to SHM analyses in order to develop compact damage sensitive features whose extraction requires a low computational effort. In this thesis, cepstral coefficients have been mathematically transformed through a Principal Component Analysis in order to generate damage sensitive features that are barely sensitive to measurement noise, environmental conditions and different excitation sources. In an attempt to develop an automated strategy for structural damage assessment, the search for damage sensitive features has been extended to the estimation of structural mode characteristics obtained through an output-only version of the Inner Product Vector methodology, e.g. considering only the structural response time histories. This new damage assessment procedure requires low computational effort and is capable to identify both the presence of damage and its location. However, one of the critical points of the proposed procedure consists in the manual evaluation of the spectral content of the dynamic responses that requires the user’s intervention. To automatize this procedure, a Bayesian clustering algorithm and a classifier have been successfully implemented and tested. Finally, the robustness of Bayesian regression algorithms to overfitting led us to consider their applicability to the field of system identification in order to provide a reliable estimate of the structural modal parameters that can be used as damage sensitive fea- tures. In fact, one of the main problems of system identification algorithms is that they rely on a regression algorithm that tends to overfit data producing unreliable results. Results provided by the Bayesian regression based system identification algorithm are obtained and compared with the ones coming from standard system identification algorithms