Active Learning of Multi-Index Function Models

We consider the problem of actively learning \textit{multi-index} functions of the form $f(x) = g(Ax)= \sum_{i=1}^k g_i(a_i^Tx)$ from point evaluations of $f$. We assume that the function $f$ is defined on an $\ell_2$-ball in \Real^d, $g$ is twice continuously differentiable almost everywhere, and $A \in \mathbb{R}^{k \times d}$ is a rank $k$ matrix, where $k \ll d$. We propose a randomized, active sampling scheme for estimating such functions with uniform approximation guarantees. Our theoretical developments leverage recent techniques from low rank matrix recovery, which enables us to derive an estimator of the function $f$ along with sample complexity bounds. We also characterize the noise robustness of the scheme, and provide empirical evidence that the high-dimensional scaling of our sample complexity bounds are quite accurate

Learning Patterns with Kernels and Learning Kernels from Patterns

A major technique in learning involves the identification of patterns and their use to make predictions. In this work, we examine the symbiotic relationship between patterns and Gaussian process regression (GPR), which is mathematically equivalent to kernel interpolation. We introduce techniques where GPR can be used to learn patterns in denoising and mode (signal) decomposition. Additionally, we present the kernel flow (KF) algorithm which learns a kernels from patterns in the data with methodology inspired by cross validation. We further show how the KF algorithm can be applied to artificial neural networks (ANNs) to make improvements to learning patterns in images. In our denoising and mode decomposition examples, we show how kernels can be constructed to estimate patterns that may be hidden due to data corruption. In other words, we demonstrate how to learn patterns with kernels. Donoho and Johnstone proposed a near-minimax method for reconstructing an unknown smooth function u from noisy data u + ζ by translating the empirical wavelet coefficients of u + ζ towards zero. We consider the situation where the prior information on the unknown function u may not be the regularity of u, but that of ℒu where ℒ is a linear operator, such as a partial differential equation (PDE) or a graph Laplacian. We show that a near-minimax approximation of u can be obtained by truncating the ℒ-gamblet (operator-adapted wavelet) coefficients of u + ζ. The recovery of u can be seen to be precisely a Gaussian conditioning of u + ζ on measurement functions with length scale dependent on the signal-to-noise ratio. We next introduce kernel mode decomposition (KMD), which has been designed to learn the modes vi = ai(t)yi(θi(t)) of a (possibly noisy) signal Σivi when the amplitudes ai, instantaneous phases θi, and periodic waveforms yi may all be unknown. GPR with Gabor wavelet-inspired kernels is used to estimate ai, θi, and yi. We show near machine precision recovery under regularity and separation assumptions on the instantaneous amplitudes ai and frequencies &#729;θi. GPR and kernel interpolation require the selection of an appropriate kernel modeling the data. We present the KF algorithm, which is a numerical-approximation approach to this selection. The main principle the method utilizes is that a "good" kernel is able to make accurate predictions with small subsets of a training set. In this way, we learn a kernel from patterns. In image classification, we show that the learned kernels are able to classify accurately using only one training image per class and show signs of unsupervised learning. Furthermore, we introduce the combination of the KF algorithm with conventional neural-network training. This combination is able to train the intermediate-layer outputs of the network simultaneously with the final-layer output. We test the proposed method on Convolutional Neural Networks (CNNs) and Wide Residual Networks (WRNs) without alteration of their structure or their output classifier. We report reduced test errors, decreased generalization gaps, and increased robustness to distribution shift without significant increase in computational complexity relative to standard CNN and WRN training (with Drop Out and Batch Normalization). As a whole, this work highlights the interplay between kernel techniques with pattern recognition and numerical approximation.</p

Ehdolliset normalisoivat virtaukset kuvien käänteisongelmissa

Learning-based methods have provided powerful tools for solving classification and regression -related problems yielding far superior results to classical handcrafted rule-based models. These models have proven to be efficient in multiple domains in many different fields. However, many common problems are inherently illposed and lack a unique answer hence requiring a regularization pass or alternatively a probabilistic framework for successful modeling. While many different families of models capable of learning distributions given samples exist, they commonly resort to approximations or surrogate training objectives. In this thesis we solve image-related inverse problems with a family of probabilistic models known as conditional normalizing flows. A normalizing flow consists of repeated applications of invertible transformations on a simple prior distribution rendering it into a more complex distribution with direct and tractable probability density evaluation and efficient sampling. We show that a conditional normalizing flow is able to provide plausible, high-quality samples with visible benign variance from a conditional distribution in image super resolution, denoising and colorization tasks. We quantify the success of the model as well as its shortcomings and inspect how it internally addresses the conversion of white noise into a realistic image.Havainnoista oppimiseen optimoinnin avulla perustuvat mallit kykenevät ratkaisemaan monia ongelmia huomattavasti tehokkaammin, kuin klassiset staattisiin päätössääntöihin perustuvat mallit. Perinteisesti mallit antavat yleensä kuitenkin vain yhden vastauksen, vaikka useilla ongelmilla saattaa olla monta keskenään yhtä hyväksyttävää vastausta. Tämän takia on tarkoituksenmukaista mallintaa todennäköisyysjakaumaa kaikista mahdollisista vastauksista yksittäisen vastauksen sijaan. Tässä diplomityössä tutkitaan normalisoivien virtausten malliluokan soveltamista digitaalisiin kuviin liittyviin käänteisongelmiin. Normalisoiva virtaus muuntaa yksinkertaisen todennäköisyysjakauman neuroverkoilla parametrosoiduilla kääntyvillä funktioilla monimutkaisemmaksi jakaumaksi, siten että havaintojen uskottavuudesta saadaan kuitenkin tarkka numeerinen arvo. Normalisoivat virtaukset mahdollistavat myös tehokkaan näytteiden ottamisen niiden mallintamasta monimutkaisesta todennäköisyysjakaumasta. Työssä määritetään, kuinka hyvin virtausmallit onnistuvat tehtävässään ja kuinka ne muodostavat uskottavia kuvia kohinasta. Työssä todetaan, että ehdollisten normalisoivien virtausten avulla voidaan tuottaa korkealaatuisia näytteitä useissa kuviin liittyvissä käänteisongelmissa

Random matrix theory and the loss surfaces of neural networks

Neural network models are one of the most successful approaches to machine learning, enjoying an enormous amount of development and research over recent years and finding concrete real-world applications in almost any conceivable area of science, engineering and modern life in general. The theoretical understanding of neural networks trails significantly behind their practical success and the engineering heuristics that have grown up around them. Random matrix theory provides a rich framework of tools with which aspects of neural network phenomenology can be explored theoretically. In this thesis, we establish significant extensions of prior work using random matrix theory to understand and describe the loss surfaces of large neural networks, particularly generalising to different architectures. Informed by the historical applications of random matrix theory in physics and elsewhere, we establish the presence of local random matrix universality in real neural networks and then utilise this as a modeling assumption to derive powerful and novel results about the Hessians of neural network loss surfaces and their spectra. In addition to these major contributions, we make use of random matrix models for neural network loss surfaces to shed light on modern neural network training approaches and even to derive a novel and effective variant of a popular optimisation algorithm. Overall, this thesis provides important contributions to cement the place of random matrix theory in the theoretical study of modern neural networks, reveals some of the limits of existing approaches and begins the study of an entirely new role for random matrix theory in the theory of deep learning with important experimental discoveries and novel theoretical results based on local random matrix universality.Comment: 320 pages, PhD thesi

Pattern Recognition

Pattern recognition is a very wide research field. It involves factors as diverse as sensors, feature extraction, pattern classification, decision fusion, applications and others. The signals processed are commonly one, two or three dimensional, the processing is done in real- time or takes hours and days, some systems look for one narrow object class, others search huge databases for entries with at least a small amount of similarity. No single person can claim expertise across the whole field, which develops rapidly, updates its paradigms and comprehends several philosophical approaches. This book reflects this diversity by presenting a selection of recent developments within the area of pattern recognition and related fields. It covers theoretical advances in classification and feature extraction as well as application-oriented works. Authors of these 25 works present and advocate recent achievements of their research related to the field of pattern recognition

Multiple cue integration for robust tracking in dynamic environments: application to video relighting

L'anàlisi de moviment i seguiment d'objectes ha estat un dels pricipals focus d'atenció en la comunitat de visió per computador durant les dues darreres dècades. L'interès per aquesta àrea de recerca resideix en el seu ample ventall d'aplicabilitat, que s'extén des de tasques de navegació de vehicles autònoms i robots, fins a aplications en la indústria de l'entreteniment i realitat virtual.Tot i que s'han aconseguit resultats espectaculars en problemes específics, el seguiment d'objectes continua essent un problema obert, ja que els mètodes disponibles són propensos a ser sensibles a diversos factors i condicions no estacionàries de l'entorn, com ara moviments impredictibles de l'objecte a seguir, canvis suaus o abruptes de la il·luminació, proximitat d'objectes similars o fons confusos. Enfront aquests factors de confusió la integració de múltiples característiques ha demostrat que permet millorar la robustesa dels algoritmes de seguiment. En els darrers anys, degut a la creixent capacitat de càlcul dels ordinadors, hi ha hagut un significatiu increment en el disseny de complexes sistemes de seguiment que consideren simultàniament múltiples característiques de l'objecte. No obstant, la majoria d'aquests algoritmes estan basats enheurístiques i regles ad-hoc formulades per aplications específiques, fent-ne impossible l'extrapolació a noves condicions de l'entorn.En aquesta tesi proposem un marc probabilístic general per integrar el nombre de característiques de l'objecte que siguin necessàries, permetent que interactuin mútuament per tal d'estimar-ne el seu estat amb precisió, i per tant, estimar amb precisió la posició de l'objecte que s'està seguint. Aquest marc, s'utilitza posteriorment per dissenyar un algoritme de seguiment, que es valida en diverses seqüències de vídeo que contenen canvis abruptes de posició i il·luminació, camuflament de l'objecte i deformacions no rígides. Entre les característiques que s'han utilitzat per representar l'objecte, cal destacar la paramatrització robusta del color en un espai de color dependent de l'objecte, que permet distingir-lo del fons més clarament que altres espais de color típicament ulitzats al llarg de la literatura.En la darrera part de la tesi dissenyem una tècnica per re-il·luminar tant escenes estàtiques com en moviment, de les que s'en desconeix la geometria. La re-il·luminació es realitza amb un mètode 'basat en imatges', on la generació de les images de l'escena sota noves condicions d'il·luminació s'aconsegueix a partir de combinacions lineals d'un conjunt d'imatges de referència pre-capturades, i que han estat generades il·luminant l'escena amb patrons de llum coneguts. Com que la posició i intensitat de les fonts d'il.luminació que formen aquests patrons de llum es pot controlar, és natural preguntar-nos: quina és la manera més òptima d'il·luminar una escena per tal de reduir el nombre d'imatges de referència? Demostrem que la millor manera d'il·luminar l'escena (és a dir, la que minimitza el nombre d'imatges de referència) no és utilitzant una seqüència de fonts d'il·luminació puntuals, com es fa generalment, sinó a través d'una seqüència de patrons de llum d'una base d'il·luminació depenent de l'objecte. És important destacar que quan es re-il·luminen seqüències de vídeo, les imatges successives s'han d'alinear respecte a un sistema de coordenades comú. Com que cada imatge ha estat generada per un patró de llum diferent il·uminant l'escena, es produiran canvis d'il·luminació bruscos entre imatges de referència consecutives. Sota aquestes circumstàncies, el mètode de seguiment proposat en aquesta tesi juga un paper fonamental. Finalment, presentem diversos resultats on re-il·luminem seqüències de vídeo reals d'objectes i cares d'actors en moviment. En cada cas, tot i que s'adquireix un únic vídeo, som capaços de re-il·luminar una i altra vegada, controlant la direcció de la llum, la seva intensitat, i el color.Motion analysis and object tracking has been one of the principal focus of attention over the past two decades within the computer vision community. The interest of this research area lies in its wide range of applicability, extending from autonomous vehicle and robot navigation tasks, to entertainment and virtual reality applications.Even though impressive results have been obtained in specific problems, object tracking is still an open problem, since available methods are prone to be sensitive to several artifacts and non-stationary environment conditions, such as unpredictable target movements, gradual or abrupt changes of illumination, proximity of similar objects or cluttered backgrounds. Multiple cue integration has been proved to enhance the robustness of the tracking algorithms in front of such disturbances. In recent years, due to the increasing power of the computers, there has been a significant interest in building complex tracking systems which simultaneously consider multiple cues. However, most of these algorithms are based on heuristics and ad-hoc rules formulated for specific applications, making impossible to extrapolate them to new environment conditions.In this dissertation we propose a general probabilistic framework to integrate as many object features as necessary, permitting them to mutually interact in order to obtain a precise estimation of its state, and thus, a precise estimate of the target position. This framework is utilized to design a tracking algorithm, which is validated on several video sequences involving abrupt position and illumination changes, target camouflaging and non-rigid deformations. Among the utilized features to represent the target, it is important to point out the use of a robust parameterization of the target color in an object dependent colorspace which allows to distinguish the object from the background more clearly than other colorspaces commonly used in the literature.In the last part of the dissertation, we design an approach for relighting static and moving scenes with unknown geometry. The relighting is performed through an -image-based' methodology, where the rendering under new lighting conditions is achieved by linear combinations of a set of pre-acquired reference images of the scene illuminated by known light patterns. Since the placement and brightness of the light sources composing such light patterns can be controlled, it is natural to ask: what is the optimal way to illuminate the scene to reduce the number of reference images that are needed? We show that the best way to light the scene (i.e., the way that minimizes the number of reference images) is not using a sequence of single, compact light sources as is most commonly done, but rather to use a sequence of lighting patterns as given by an object-dependent lighting basis. It is important to note that when relighting video sequences, consecutive images need to be aligned with respect to a common coordinate frame. However, since each frame is generated by a different light pattern illuminating the scene, abrupt illumination changes between consecutive reference images are produced. Under these circumstances, the tracking framework designed in this dissertation plays a central role. Finally, we present several relighting results on real video sequences of moving objects, moving faces, and scenes containing both. In each case, although a single video clip was captured, we are able to relight again and again, controlling the lighting direction, extent, and color.Postprint (published version

Processing and analysis of transient data from permanent down-hole gauges (PDG)

The Permanent Downhole Gauge (PDG) can monitor the reservoir in real time over a long period of time. This produces a huge amount of real time data which can potentially provide more information about wells and reservoirs. However, processing large numbers of data and extracting useful information from these data brings new challenges for industry and engineers. A new workflow for processing the PDG data is proposed in this study. The new approach processes PDG data from the view of gauge, well and reservoir. The gauge information is first filtered with data preprocessing and outlier removal. Then, the well event is identified using an improved wavelet approach. The further processing step of data denoise and data reduction is carried out before analyzing the reservoir information. The accurate production history is very essential for data analysis. However, the accurate production rate is hard to be acquired. Therefore, a new approach is created to recover flow rate history from the accumulated production and PDG pressure data. This new approach is based on the theory that the relation between production rate and the amplitude of detail coefficient are in direct proportion after wavelet transform. With accurate pressure and rate data, traditional well testing is applied to analyze the PDG pressure data to get dynamic reservoir parameters. The numerical well testing approach is also carried out to analyze more complex reservoir model with a new toolbox. However, these two approaches all suffer from the nonlinear problem of PDG pressure. So, a dynamic forward modelling approach is proposed to analyze PDG pressure data. The new approach uses the deconvolution method to diagnose the linear region in the nonlinear system. The nonlinear system can be divided into different linear systems which can be analyzed with the numerical well testing approach. Finally, a toolbox which includes a PDG data processing module and PDG data analysis module is designed with Matlab

Mathematical Models and Monte-Carlo Algorithms for Improved Detection of Targets in the Commercial Maritime Domain

Commercial Vessel Traffic Monitoring Services (VTMSs) are widely used by port authorities and the military to improve the safety and efficiency of navigation, as well as to ensure the security of ports and marine life as a whole. Technology based on the Kalman Filtering framework is in widespread use in modern operational VTMS systems. At a research level, there has also been a significant interest in Particle Filters, which are widely researched but far less widely applied to deliver an operational advantage. The Monte-Carlo nature of Particle Filters places them as the ideal candidate for solving the highly non-linear, non-Gaussian problems encountered by modern VTMS systems. However, somewhat counter-intuitively, while Particle Filters are best suited to exploit such non-linear, non-Gaussian problems, they are most frequently used within a context that is mostly linear and Gaussian. The engineering challenge tackled by the PhD project reported in this thesis was to study and experiment with models that are well placed to capitalise on the abilities of Particle Filters and to develop solutions that make use of such models to deliver a direct operational advantage in real applications within the commercial maritime domain