19 research outputs found

    Visualizing and Understanding Contrastive Learning

    Full text link
    Contrastive learning has revolutionized the field of computer vision, learning rich representations from unlabeled data, which generalize well to diverse vision tasks. Consequently, it has become increasingly important to explain these approaches and understand their inner workings mechanisms. Given that contrastive models are trained with interdependent and interacting inputs and aim to learn invariance through data augmentation, the existing methods for explaining single-image systems (e.g., image classification models) are inadequate as they fail to account for these factors. Additionally, there is a lack of evaluation metrics designed to assess pairs of explanations, and no analytical studies have been conducted to investigate the effectiveness of different techniques used to explaining contrastive learning. In this work, we design visual explanation methods that contribute towards understanding similarity learning tasks from pairs of images. We further adapt existing metrics, used to evaluate visual explanations of image classification systems, to suit pairs of explanations and evaluate our proposed methods with these metrics. Finally, we present a thorough analysis of visual explainability methods for contrastive learning, establish their correlation with downstream tasks and demonstrate the potential of our approaches to investigate their merits and drawbacks

    Learning and inverse problems: from theory to solar physics applications

    Get PDF
    The problem of approximating a function from a set of discrete measurements has been extensively studied since the seventies. Our theoretical analysis proposes a formalization of the function approximation problem which allows dealing with inverse problems and supervised kernel learning as two sides of the same coin. The proposed formalization takes into account arbitrary noisy data (deterministically or statistically defined), arbitrary loss functions (possibly seen as a log-likelihood), handling both direct and indirect measurements. The core idea of this part relies on the analogy between statistical learning and inverse problems. One of the main evidences of the connection occurring across these two areas is that regularization methods, usually developed for ill-posed inverse problems, can be used for solving learning problems. Furthermore, spectral regularization convergence rate analyses provided in these two areas, share the same source conditions but are carried out with either increasing number of samples in learning theory or decreasing noise level in inverse problems. Even more in general, regularization via sparsity-enhancing methods is widely used in both areas and it is possible to apply well-known ell1ell_1-penalized methods for solving both learning and inverse problems. In the first part of the Thesis, we analyze such a connection at three levels: (1) at an infinite dimensional level, we define an abstract function approximation problem from which the two problems can be derived; (2) at a discrete level, we provide a unified formulation according to a suitable definition of sampling; and (3) at a convergence rates level, we provide a comparison between convergence rates given in the two areas, by quantifying the relation between the noise level and the number of samples. In the second part of the Thesis, we focus on a specific class of problems where measurements are distributed according to a Poisson law. We provide a data-driven, asymptotically unbiased, and globally quadratic approximation of the Kullback-Leibler divergence and we propose Lasso-type methods for solving sparse Poisson regression problems, named PRiL for Poisson Reweighed Lasso and an adaptive version of this method, named APRiL for Adaptive Poisson Reweighted Lasso, proving consistency properties in estimation and variable selection, respectively. Finally we consider two problems in solar physics: 1) the problem of forecasting solar flares (learning application) and 2) the desaturation problem of solar flare images (inverse problem application). The first application concerns the prediction of solar storms using images of the magnetic field on the sun, in particular physics-based features extracted from active regions from data provided by Helioseismic and Magnetic Imager (HMI) on board the Solar Dynamics Observatory (SDO). The second application concerns the reconstruction problem of Extreme Ultra-Violet (EUV) solar flare images recorded by a second instrument on board SDO, the Atmospheric Imaging Assembly (AIA). We propose a novel sparsity-enhancing method SE-DESAT to reconstruct images affected by saturation and diffraction, without using any a priori estimate of the background solar activity

    NEW TECHNIQUES IN DERIVATIVE DOMAIN IMAGE FUSION AND THEIR APPLICATIONS

    Get PDF
    There are many applications where multiple images are fused to form a single summary greyscale or colour output, including computational photography (e.g. RGB-NIR), diffusion tensor imaging (medical), and remote sensing. Often, and intuitively, image fusion is carried out in the derivative domain (based on image gradients). In this thesis, we propose new derivative domain image fusion methods and metrics, and carry out experiments on a range of image fusion applications. After reviewing previous relevant methods in derivative domain image fusion, we make several new contributions. We present new applications for the Spectral Edge image fusion method, in thermal image fusion (using a FLIR smartphone accessory) and near-infrared image fusion (using an integrated visible and near-infrared sensor). We propose extensions of standard objective image fusion quality metrics for M to N channel image fusion measuring image fusion performance is an unsolved problem. Finally, and most importantly, we propose new methods in image fusion, which give improved results compared to previous methods (based on metric and subjective comparisons): we propose an iterative extension to the Spectral Edge image fusion method, producing improved detail transfer and colour vividness, and we propose a new derivative domain image fusion method, based on finding a local linear combination of input images to produce an output image with optimum gradient detail, without artefacts - this mapping can be calculated by finding the principal characteristic vector of the outer product of the Jacobian matrix of image derivatives, or by solving a least-squares regression (with regularization) to the target gradients calculated by the Spectral Edge theorem. We then use our new image fusion method on a range of image fusion applications, producing state of the art image fusion results with the potential for real-time performance

    Human-centered display design : balancing technology & perception

    Get PDF

    The geometry of colour

    Get PDF
    This thesis explores the geometric description of animal colour vision. It examines the relationship of colour spaces to behavior and to physiology. I provide a derivation of, and explore the limits of, geometric spaces derived from the notion of risk and uncertainty aversion as well as the geometric objects that enumerate the variety of achievable colours. Using these principles I go on to explore evolutionary questions concerning colourfulness, such as aposematism, mimicry and the idea of aesthetic preference

    Multispectral photography for earth resources

    Get PDF
    A guide for producing accurate multispectral results for earth resource applications is presented along with theoretical and analytical concepts of color and multispectral photography. Topics discussed include: capabilities and limitations of color and color infrared films; image color measurements; methods of relating ground phenomena to film density and color measurement; sensitometry; considerations in the selection of multispectral cameras and components; and mission planning

    Automated color correction for colorimetric applications using barcodes

    Get PDF
    [eng] Color-based sensor devices often offer qualitative solutions, where a material change its color from one color to another, and this is change is observed by a user who performs a manual reading. These materials change their color in response to changes in a certain physical or chemical magnitude. Nowadays, we can find colorimetric indicators with several sensing targets, such as: temperature, humidity, environmental gases, etc. The common approach to quantize these sensors is to place ad hoc electronic components, e.g., a reader device. With the rise of smartphone technology, the possibility to automatically acquire a digital image of those sensors and then compute a quantitative measure is near. By leveraging this measuring process to the smartphones, we avoid the use of ad hoc electronic components, thus reducing colorimetric application cost. However, there exists a challenge on how-to acquire the images of the colorimetric applications and how-to do it consistently, with the disparity of external factors affecting the measure, such as ambient light conditions or different camera modules. In this thesis, we tackle the challenges to digitize and quantize colorimetric applications, such as colorimetric indicators. We make a statement to use 2D barcodes, well-known computer vision patterns, as the base technology to overcome those challenges. We focus on four main challenges: (I) to capture barcodes on top of real-world challenging surfaces (bottles, food packages, etc.), which are the usual surface where colorimetric indicators are placed; (II) to define a new 2D barcode to embed colorimetric features in a back-compatible fashion; (III) to achieve image consistency when capturing images with smartphones by reviewing existent methods and proposing a new color correction method, based upon thin-plate splines mappings; and (IV) to demonstrate a specific application use case applied to a colorimetric indicator for sensing CO2 in the range of modified atmosphere packaging, MAP, one of the common food-packaging standards.[cat] Els dispositius de sensat basats en color, normalment ofereixen solucions qualitatives, en aquestes solucions un material canvia el seu color a un altre color, i aquest canvi de color és observat per un usuari que fa una mesura manual. Aquests materials canvien de color en resposta a un canvi en una magnitud física o química. Avui en dia, podem trobar indicadors colorimètrics que amb diferents objectius, per exemple: temperatura, humitat, gasos ambientals, etc. L'opció més comuna per quantitzar aquests sensors és l'ús d'electrònica addicional, és a dir, un lector. Amb l'augment de la tecnologia dels telèfons intel·ligents, la possibilitat d'automatitzar l'adquisició d'imatges digitals d'aquests sensors i després computar una mesura quantitativa és a prop. Desplaçant aquest procés de mesura als telèfons mòbils, evitem l'ús d'aquesta electrònica addicional, i així, es redueix el cost de l'aplicació colorimètrica. Tanmateix, existeixen reptes sobre com adquirir les imatges de les aplicacions colorimètriques i de com fer-ho de forma consistent, a causa de la disparitat de factors externs que afecten la mesura, com per exemple la llum ambient or les diferents càmeres utilitzades. En aquesta tesi, encarem els reptes de digitalitzar i quantitzar aplicacions colorimètriques, com els indicadors colorimètrics. Fem una proposició per utilitzar codis de barres en dues dimensions, que són coneguts patrons de visió per computador, com a base de la nostra tecnologia per superar aquests reptes. Ens focalitzem en quatre reptes principals: (I) capturar codis de barres sobre de superfícies del món real (ampolles, safates de menjar, etc.), que són les superfícies on usualment aquests indicadors colorimètrics estan situats; (II) definir un nou codi de barres en dues dimensions per encastar elements colorimètrics de forma retro-compatible; (III) aconseguir consistència en la captura d'imatges quan es capturen amb telèfons mòbils, revisant mètodes de correcció de color existents i proposant un nou mètode basat en transformacions geomètriques que utilitzen splines; i (IV) demostrar l'ús de la tecnologia en un cas específic aplicat a un indicador colorimètric per detectar CO2 en el rang per envasos amb atmosfera modificada, MAP, un dels estàndards en envasos de menjar.

    Investigating the genetic and immunological aetiology of myalgic encephalomyelitis/chronic fatigue syndrome

    Get PDF
    This thesis describes two investigations into the disease Myalgic Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS), specifically its genetic aetiology and immune system alterations. The first study investigated the genetic basis of ME/CFS using Genome-wide Association Studies (GWAS) by attempting to replicate and extend results previously found using UK Biobank cohort data. GWAS attempt to identify associations between DNA variants and phenotypes. This GWAS was novel, conducted on new phenotypes constructed by combining those in the most up-to-date UK Biobank data release. A new, previously unseen, genome-wide significant association was found on chromosome 6 for males with ME/CFS within the gene PDE10A. Further results were not genome-wide significant, but many were suggestive and hence independent replication may justify further research. A previous analysis on the UK Biobank cohort had identified an indicative association in females between variants around the SLC25A15 gene at genome-wide significance. I adopted a hypothesis that the dietary protein intake of people with the CFS risk variants would be lower than those with the alternative alleles, due to potentially reduced production of mitochondrial ornithine transporter 1 (ORNT1). However, this association with dietary protein intake was not supported by UK Biobank data. Additionally, I investigated associations between the human leukocyte antigen (HLA) alleles and the ME/CFS phenotype using UK Biobank data. Associations between alleles within the HLA-C and -DQB1 genes had previously been found in a cohort of Norwegian people with ME/CFS, and my goal was to seek replication of these results in a larger dataset. None of the associations found in the UK Biobank proved to be genome-wide significant. In my second study I investigated the use of T-cell clonal diversity as a potential biomarker for ME/CFS. This project used cells from CureME Biobank samples in collaboration with Systems Biology Laboratory (SBL). I developed a data analysis pipeline to analyse T-cell receptor (TCR) genomic DNA data based on the best practices currently used in the fields of immunology and mathematical biology. This approach used a mathematical notion of entropy as a measure for the diversity of TCR repertoires, in this way combining all of the most commonly used metrics in mathematical biology. When combined, these measures form a profile for each repertoire, which can be sorted using a machine learning algorithm to partition the repertoires into subgroups. My hypothesis was that the T-cell clonal expansion of people with ME/CFS would be greater than for healthy controls, and comparable to disease (multiple sclerosis) controls. Although this method was able to effectively classify TCR chains using simulated data, results from experimentally-derived data did not support the hypothesis, with the most effective classifications for both CD4+ and CD8+ cells failing to pass corrections for multiple hypothesis significance testing

    Representation of Object-Centered Space by Neurons of the Supplementary Eye Field

    Get PDF
    The supplementary eye field (SEF) is a region of cortex located on the dorsomedial shoulder of the frontal lobe, considered to be involved in the control of eye movements. SEF neurons show spatially selective activity during visually- and memory-guided saccades. The selectivity exhibited by SEF neurons has been described as being related to an eye- or head-centered reference frame. We have previously shown that SEF neurons exhibit selectivity in an object-centered reference frame: neurons will fire selectively when saccades are directed to one end of a bar or another, irrespective of the absolute location of the bar in space.It is not well known how SEF neurons display selectivity for object-centered locations. In order to better understand the mechanism of this phenomenon, we performed three studies. In the first study, we asked how SEF neurons encode locations in both egocentric and object-centered reference frames. We recorded from single SEF neurons while monkeys performed tasks requiring spatial representation in either eye-centered or object-centered reference frames. Different SEF neurons encoded locations in eye-centered coordinates only, object-centered coordinates only, or in complex combinations of the two.In the second study, we tested whether object-centered selectivity is an innate property of SEF neurons or whether it is acquired through learning. We recorded the activity of SEF neurons before and after training monkeys to perform an object-centered task. Some SEF neurons exhibited object-centered selectivity before training. Following training, this number was increased, as was the intensity of object-centered spatial selectivity.In the third study, we investigated whether the object-centered selectivity seen in SEF neurons during performance of an object-centered task is reduced during performance of a non-object-centered task. We recorded from SEF neurons while monkeys performed either an object-centered task or a color matching task with an object as a target. An equivalent number of neurons showed object-centered selectivity in both tasks, but the strength of selectivity was slightly higher during performance of the object-centered task. We conclude from the results of these studies that neurons in the SEF are critically involved in the dynamic representation of locations using multiple spatial reference frames

    Machine Learning As Tool And Theory For Computational Neuroscience

    Get PDF
    Computational neuroscience is in the midst of constructing a new framework for understanding the brain based on the ideas and methods of machine learning. This is effort has been encouraged, in part, by recent advances in neural network models. It is also driven by a recognition of the complexity of neural computation and the challenges that this poses for neuroscience’s methods. In this dissertation, I first work to describe these problems of complexity that have prompted a shift in focus. In particular, I develop machine learning tools for neurophysiology that help test whether tuning curves and other statistical models in fact capture the meaning of neural activity. Then, taking up a machine learning framework for understanding, I consider theories about how neural computation emerges from experience. Specifically, I develop hypotheses about the potential learning objectives of sensory plasticity, the potential learning algorithms in the brain, and finally the consequences for sensory representations of learning with such algorithms. These hypotheses pull from advances in several areas of machine learning, including optimization, representation learning, and deep learning theory. Each of these subfields has insights for neuroscience, offering up links for a chain of knowledge about how we learn and think. Together, this dissertation helps to further an understanding of the brain in the lens of machine learning
    corecore