Super-resolving multiresolution images with band-independant geometry of multispectral pixels

A new resolution enhancement method is presented for multispectral and multi-resolution images, such as these provided by the Sentinel-2 satellites. Starting from the highest resolution bands, band-dependent information (reflectance) is separated from information that is common to all bands (geometry of scene elements). This model is then applied to unmix low-resolution bands, preserving their reflectance, while propagating band-independent information to preserve the sub-pixel details. A reference implementation is provided, with an application example for super-resolving Sentinel-2 data.Comment: Source code with a ready-to-use script for super-resolving Sentinel-2 data is available at http://nicolas.brodu.net/recherche/superres

Extending minkowski norm illuminant estimation

The ability to obtain colour images invariant to changes of illumination is called colour constancy. An algorithm for colour constancy takes sensor responses - digital images - as input, estimates the ambient light and returns a corrected image in which the illuminant influence over the colours has been removed. In this thesis we investigate the step of illuminant estimation for colour constancy and aim to extend the state of the art in this field. We first revisit the Minkowski Family Norm framework for illuminant estimation. Because, of all the simple statistical approaches, it is the most general formulation and, crucially, delivers the best results. This thesis makes four technical contributions. First, we reformulate the Minkowski approach to provide better estimation when a constraint on illumination is employed. Second, we show how the method can (by orders of magnitude) be implemented to run much faster than previous algorithms. Third, we show how a simple edge based variant delivers improved estimation compared with the state of the art across many datasets. In contradistinction to the prior state of the art our definition of edges is fixed (a simple combination of first and second derivatives) i.e. we do not tune our algorithm to particular image datasets. This performance is further improved by incorporating a gamut constraint on surface colour -our 4th contribution. The thesis finishes by considering our approach in the context of a recent OSA competition run to benchmark computational algorithms operating on physiologically relevant cone based input data. Here we find that Constrained Minkowski Norms operi ii ating on spectrally sharpened cone sensors (linear combinations of the cones that behave more like camera sensors) supports competition leading illuminant estimation

Coding Strategies Underlying Visual Processing

Acquiring and representing knowledge about our environment involves a variety of core neural computations. The coding strategies underlying visual perception highlight many of these processes, and thus reveal general design principles in perception and cognition. I will review three studies where I have used different computational frameworks and analyses to address open questions in visual coding. The first project uses factor analyses of individual differences in perception to demonstrate fundamentally different representational structures for the stimulus features of color and motion. In the second project, I have explored visual adaptation in the context of population coding to address controversies regarding which coding schemes are implicated by different patterns of adaptation aftereffects. In the third, I have explored these adaptation effects in the context of Bayesian inference. This approach accounts for the full gamut of known aftereffects within the context of physiologically plausible models and provides principled quantitative predictions for why and how much the system should adapt. Together, these projects draw on the power of formal computational approaches both for analyzing neural representations and for revealing the computations and coding principles on which they are based

Evaluation and improvement of the workflow of digital imaging of fine art reproduction in museums

Fine arts refer to a broad spectrum of art formats, ie~painting, calligraphy, photography, architecture, and so forth. Fine art reproductions are to create surrogates of the original artwork that are able to faithfully deliver the aesthetics and feelings of the original. Traditionally, reproductions of fine art are made in the form of catalogs, postcards or books by museums, libraries, archives, and so on (hereafter called museums for simplicity). With the widespread adoption of digital archiving in museums, more and more artwork is reproduced to be viewed on a display. For example, artwork collections are made available through museum websites and Google Art Project for art lovers to view on their own displays. In the thesis, we study the fine art reproduction of paintings in the form of soft copy viewed on displays by answering four questions: (1) what is the impact of the viewing condition and original on image quality evaluation? (2) can image quality be improved by avoiding visual editing in current workflows of fine art reproduction? (3) can lightweight spectral imaging be used for fine art reproduction? and (4) what is the performance of spectral reproductions compared with reproductions by current workflows? We started with evaluating the perceived image quality of fine art reproduction created by representative museums in the United States under controlled and uncontrolled environments with and without the presence of the original artwork. The experimental results suggest that the image quality is highly correlated with the color accuracy of the reproduction only when the original is present and the reproduction is evaluated on a characterized display. We then examined the workflows to create these reproductions, and found that current workflows rely heavily on visual editing and retouching (global and local color adjustments on the digital reproduction) to improve the color accuracy of the reproduction. Visual editing and retouching can be both time-consuming and subjective in nature (depending on experts\u27 own experience and understanding of the artwork) lowering the efficiency of artwork digitization considerably. We therefore propose to improve the workflow of fine art reproduction by (1) automating the process of visual editing and retouching in current workflows based on RGB acquisition systems and by (2) recovering the spectral reflectance of the painting with off-the-shelf equipment under commonly available lighting conditions. Finally, we studied the perceived image quality of reproductions created by current three-channel (RGB) workflows with those by spectral imaging and those based on an exemplar-based method

Modelling Colour Appearance: Applications in Skin Image Perception

Humans are trichromatic, and yet their perception of colours is rich and complex. The research presented in this thesis explores the process of colour appearance of uniform patches and natural polychromatic stimuli. This is done through the measurement and analysis of the achromatic locus (Chapter 2), modelling of chromatic adaptation in a large dataset of unique hues settings (Chapter 3), and measurement of thresholds for uniform and polychromatic stimuli derived from simulated skin images (Chapter 4). Chapter 2 proposes a novel navigation scheme based on unique hues for traversing colour space. The results show that when colour adjustments are made using this novel scheme, the variability of achromatic settings made by observers is reduced compared to the classical method of making colour adjustments along the cardinal axes of the CIELUV colour space. This result holds across the tested luminance levels (5,20,50 cd/m^2) in each of the three tested ambient illumination conditions – dark, simulated daylight and cool white fluorescent lighting. The analysis also shows that the direction of maximum variance of the achromatic settings lies along the daylight locus. Chapter 3 evaluates models of chromatic adaptation by using unique hues settings measured under different ambient illumination conditions. It is shown that a simple diagonal model in cone excitation space is the most efficient in terms of the trade-off between accuracy and degrees of freedom. It is also found that diagonal and linear models show similar performances, reiterating their theoretical equivalence. Performances of these diagonalisable models are found to be worse for UR and UG unique hue planes compared to UY and UB planes. Chapter 4 presents a set of three experiments reporting estimations of perceptual thresholds for polychromatic and uniform stimuli in a 3-D chromaticity-luminance colour space. The first experiment reports thresholds for simulated skin images and uniform stimuli of the corresponding mean CIELAB colour. The second and third experiments investigate the effect of ambient illumination and the location of the stimuli in colour space. The thresholds for the polychromatic stimuli are found to be consistently higher than those for the uniform patches, for both the chromatic, and the luminance projections. The area of the chromaticity ellipses shows a gradual increase with distance from the illuminant chromaticity. The orientations of these ellipses for simulated skin are found to align with the vector joining the mean patch chromaticity and the illuminant chromaticity

Image Quality Evaluation in Lossy Compressed Images

This research focuses on the quantification of image quality in lossy compressed images, exploring the impact of digital artefacts and scene characteristics upon image quality evaluation. A subjective paired comparison test was implemented to assess perceived quality of JPEG 2000 against baseline JPEG over a range of different scene types. Interval scales were generated for both algorithms, which indicated a subjective preference for JPEG 2000, particularly at low bit rates, and these were confirmed by an objective distortion measure. The subjective results did not follow this trend for some scenes however, and both algorithms were found to be scene dependent as a result of the artefacts produced at high compression rates. The scene dependencies were explored from the interval scale results, which allowed scenes to be grouped according to their susceptibilities to each of the algorithms. Groupings were correlated with scene measures applied in a linked study. A pilot study was undertaken to explore perceptibility thresholds of JPEG 2000 of the same set of images. This work was developed with a further experiment to investigate the thresholds of perceptibility and acceptability of higher resolution JPEG 2000 compressed images. A set of images was captured using a professional level full-frame Digital Single Lens Reflex camera, using a raw workflow and carefully controlled image-processing pipeline. The scenes were quantified using a set of simple scene metrics to classify them according to whether they were average, higher than, or lower than average, for a number of scene properties known to affect image compression and perceived image quality; these were used to make a final selection of test images. Image fidelity was investigated using the method of constant stimuli to quantify perceptibility thresholds and just noticeable differences (JNDs) of perceptibility. Thresholds and JNDs of acceptability were also quantified to explore suprathreshold quality evaluation. The relationships between the two thresholds were examined and correlated with the results from the scene measures, to identify more or less susceptible scenes. It was found that the level and differences between the two thresholds was an indicator of scene dependency and could be predicted by certain types of scene characteristics. A third study implemented the soft copy quality ruler as an alternative psychophysical method, by matching the quality of compressed images to a set of images varying in a single attribute, separated by known JND increments of quality. The imaging chain and image processing workflow were evaluated using objective measures of tone reproduction and spatial frequency response. An alternative approach to the creation of ruler images was implemented and tested, and the resulting quality rulers were used to evaluate a subset of the images from the previous study. The quality ruler was found to be successful in identifying scene susceptibilities and observer sensitivity. The fourth investigation explored the implementation of four different image quality metrics. These were the Modular Image Difference Metric, the Structural Similarity Metric, The Multi-scale Structural Similarity Metric and the Weighted Structural Similarity Metric. The metrics were tested against the subjective results and all were found to have linear correlation in terms of predictability of image quality

NEURAL MECHANISMS UNDERLYING OBJECT SELECTIVITY IN MACAQUE INFEROTEMPORAL CORTEX

The inferotemporal cortex of the macaque monkey mediates the recognition of objects in the visual world. The purpose of the research presented in this dissertation was to investigate the neural mechanisms underlying two poorly understood aspects of object recognition. The first experiment addressed the question of how visual features are integrated in IT. In this study, we sought to determine whether feature selectivity for shape and color is integrated by IT neurons via a conjunction-coding mechanism, or via linear summation. We demonstrate that visual responses of most IT neurons encode shape and color information in a linear manner. Our results shed light on the computational strategy that the brain employs to construct a versatile representation of the visual world.The purpose of the second experiment was to investigate the neural mechanisms underlying repetition priming. Repetition priming is a form of rapid visual learning, whereby previous experience with an object allows for faster, more efficient perceptual processing of that object upon subsequent encounters. This behavioral process is believed to be dependent on activity reductions in single IT neurons, but this hypothesis has never been tested. Indeed, repetition priming has never been demonstrated before in monkeys. To address this issue, we adapted the experimental paradigm of repetition priming for use in primate physiology. We demonstrate that repetition priming at the level of behavior is accompanied by repetition suppression at the level of single neurons in IT. We further demonstrate that repetition suppression in IT results in a proportional scaling reduction of visual responses, and not in a sharpening of the stimulus selectivity. These findings constrain the possible mechanisms whereby visual response plasticity in IT could contribute to behavioral priming

Mapping the natural visual world of the zebrafish (Danio rerio): from sensory input to behavioural output

Vision is one of the most crucial senses for animals to catch prey, find mates and stay alive. The tetrachromatic zebrafish (Danio rerio) is a widely used model animal in visual neuroscience with four cone photoreceptors sensitive to UV, blue, green and red light. However, a detailed understanding of how their visual system is adapted to the natural environment, and what is important for the fish to see in their shallow freshwater habitats of the Indian subcontinent, has been missing. Therefore, it also has not been possible to carefully assess the importance of different parts of the light spectrum for their natural behaviours. In this thesis I introduce a new method for natural imaging, characterise the spectral composition of zebrafish’s natural visual world and demonstrate the role of UV light in their prey capture behaviours. To characterise the light conditions in natural environments, I developed and built two hyperspectral scanners to take spectrally detailed light measurements in shallow ponds and slowly moving streams in North-East India. As expected, the spectral profile becomes increasingly monochromatic and red shifted when moving from surface to the bottom. However, the short wavelength dominated surface and long wavelength dominated bottom are separated with colour-rich horizon. These spectral statistics match rather perfectly with the cone densities and colour processing abilities of the bipolar cells in the larval zebrafish retina. Previous work has demonstrated how prey capture behaviours on larval zebrafish can be triggered by small, bright spots. The short wavelength dominated upper part of the visual field projects light from UV bright prey items perfectly to the ventro-temporal part of the retina (“strike zone”) with high density of UV cones. Finally, with my behaviour experiments I demonstrate how prey capture behaviours are strongly driven by UV bright paramecia detected with the strike zone

Perceptual models for high-refresh-rate rendering

Rendering realistic images requires substantial computational power. With new high-refresh-rate displays as well as the renaissance of virtual reality (VR) and augmented reality (AR), one cannot expect that GPU performance will scale fast enough to meet the requirements of immersive photo-realistic rendering with current rendering techniques. In this dissertation, I follow the dual of the well-known computer vision approach: vision is inverse graphics: to improve graphical algorithms, I consider the operation of the human visual system. I propose to model and exploit the limitations of the visual system in the context of novel high-refresh-rate displays; specifically, I focus on spatio-temporal perception, a topic that has received remarkably less attention than spatial-only perception so far. I present three main contributions. First, I demonstrate the validity of the perceptual approach by presenting a conceptually simple rendering technique motivated by our eyes' limited sensitivity to high spatio-temporal change which reduces the rendering load and transmission requirement of current-generation VR headsets without introducing perceivable visual artefacts. Second, I present two visual models related to motion perception: (a) a metric for detecting flicker; and (b) a comprehensive visual model to predict perceived motion quality on monitors with arbitrary refresh rates and monitor resolutions. Third, I propose an adaptive rendering algorithm that utilises the proposed models. All algorithms operate on physical colorimetric units (instead of display-referenced pixel values), for which I provide the appropriate display measurements and models. All proposed algorithms and visual models are calibrated and validated with psychophysical experiments