Computational principles for an autonomous active vision system

Vision research has uncovered computational principles that generalize across species and brain area. However, these biological mechanisms are not frequently implemented in computer vision algorithms. In this thesis, models suitable for application in computer vision were developed to address the benefits of two biologically-inspired computational principles: multi-scale sampling and active, space-variant, vision. The first model investigated the role of multi-scale sampling in motion integration. It is known that receptive fields of different spatial and temporal scales exist in the visual cortex; however, models addressing how this basic principle is exploited by species are sparse and do not adequately explain the data. The developed model showed that the solution to a classical problem in motion integration, the aperture problem, can be reframed as an emergent property of multi-scale sampling facilitated by fast, parallel, bi-directional connections at different spatial resolutions. Humans and most other mammals actively move their eyes to sample a scene (active vision); moreover, the resolution of detail in this sampling process is not uniform across spatial locations (space-variant). It is known that these eye-movements are not simply guided by image saliency, but are also influenced by factors such as spatial attention, scene layout, and task-relevance. However, it is seldom questioned how previous eye movements shape how one learns and recognizes an object in a continuously-learning system. To explore this question, a model (CogEye) was developed that integrates active, space-variant sampling with eye-movement selection (the where visual stream), and object recognition (the what visual stream). The model hypothesizes that a signal from the recognition system helps the where stream select fixation locations that best disambiguate object identity between competing alternatives. The third study used eye-tracking coupled with an object disambiguation psychophysics experiment to validate the second model, CogEye. While humans outperformed the model in recognition accuracy, when the model used information from the recognition pathway to help select future fixations, it was more similar to human eye movement patterns than when the model relied on image saliency alone. Taken together these results show that computational principles in the mammalian visual system can be used to improve computer vision models

Multi-channel data acquisition using multiplexed imaging with spatial encoding

This paper was published in Optics Express and is made available as an electronic reprint with the permission of OSA. The paper can be found at the following URL on the OSA website: http://dx.doi.org/10.1364/OE.18.023041 Systematic or multiple reproduction or distribution to multiple locations via electronic or other means is prohibited and is subject to penalties under law

Shift-encoded optically multiplexed imaging

In a multiplexed image, multiple fields-of-view (FoVs) are superimposed onto a common focal plane. The attendant gain in sensor FoV provides a new degree of freedom in the design of an imaging system, allowing for performance tradeoffs not available in traditional optical designs. We explore design choices relating to a shift-encoded optically multiplexed imaging system and discuss their performance implications. Unlike in a traditional imaging system, a single multiplexed image has a fundamental ambiguity regarding the location of objects in the image. We present a system that can shift each FoV independently to break this ambiguity and compare it to other potential disambiguation techniques. We then discuss the optical, mechanical, and encoding design choices of a shift-encoding midwave infrared imaging system that multiplexes six 15×15  deg FoVs onto a single one megapixel focal plane. Using this sensor, we demonstrate a computationally demultiplexed wide FoV video.United States. Air Force Office of Scientific Research (FA8721-05-C-0002

Autonomous real-time surveillance system with distributed IP cameras

An autonomous Internet Protocol (IP) camera based object tracking and behaviour identification system, capable of running in real-time on an embedded system with limited memory and processing power is presented in this paper. The main contribution of this work is the integration of processor intensive image processing algorithms on an embedded platform capable of running at real-time for monitoring the behaviour of pedestrians. The Algorithm Based Object Recognition and Tracking (ABORAT) system architecture presented here was developed on an Intel PXA270-based development board clocked at 520 MHz. The platform was connected to a commercial stationary IP-based camera in a remote monitoring station for intelligent image processing. The system is capable of detecting moving objects and their shadows in a complex environment with varying lighting intensity and moving foliage. Objects moving close to each other are also detected to extract their trajectories which are then fed into an unsupervised neural network for autonomous classification. The novel intelligent video system presented is also capable of performing simple analytic functions such as tracking and generating alerts when objects enter/leave regions or cross tripwires superimposed on live video by the operator

Small-scale dynamics of photospheric magnetic activities and their chromospheric responses

The evolution of photospheric magnetic fields is considered as the fundamental source of forming atmospheric structures and triggering most solar activities, including flares and mass ejections on various scales (CMEs, jets, etc.). With the implementation of high-resolution observational instruments, small-scale details of magnetic features are recognized that can provide important information regarding the evolution in active regions and the connection between photospheric magnetic reconnection and jet-like ejections in the quiet Sun. This research takes advantage of the exceptionally high-resolution measurements of vector magnetic field and imaging observations by the Goode Solar Telescope, and UV/EUV imaging observations from space-based instruments. The studied topics include structural evolution of penumbra and shear flows in response to a flare eruption, flux emergence in the formation of an active region, and small-scale magnetic reconnections in the photosphere in a coronal boundary. The main findings in this dissertation work are listed as follows: Using high-resolution imaging observation in the TiO band by GST and photospheric vector magnetic field observation by the Helioseismic and Magnetic Imager, strong shear flow expansion in a precursor kernel near the polarity inversion line (PIL) is found to be closely associated with the M6.5 flare in AR 12371, which indicates a linkage between small-scale magnetic changes and the flare eruption. The flow field of the penumbra in the flaring core region is calculated using the differential affine velocity estimator. An enhancement of penumbral flows and an extension of penumbral fibrils after the first peak of the flare hard X-ray (HXR) emission are observed to be accompanied by a rapid increase of the horizontal field. These results provide evidence of the back reaction theory of coronal restructuring on the photosphere as a result of flare energy release. Taking advantage of high-resolution magnetic field measurements, TiO continuum, and H-alpha observations by GST, small-scale magnetic flux emergences in the AR 12665 are observed in two types of topology: magnetic flux sheet emergence associated with the newly forming granules, and the traditional magnetic flux loop emergence. Both types of flux emergence are associated with the darkening of granular boundaries, while only flux sheets elongate granules along the direction of emerging magnetic fields and expand laterally. H-alpha observations reveal transient brightenings in the events of magnetic loop emergence, which are most probably the signatures of Ellerman bombs. Statistical study of small-scale magnetic reconnections around the coronal hole boundary is achieved by tracking magnetic features with the Southwest Automatic Magnetic Identification Suite (SWAMIS). The tracking results show that the magnetic energy release rate by small-scale cancellation inside the coronal hole is 3 times higher than that outside the coronal hole. Spicules are likely associated with these kinds of reconnections, while only a small portion of the release magnetic energy is directly deposited to hot plasma

Natural landmark detection for visually-guided robot navigation

The main difficulty to attain fully autonomous robot navigation outdoors is the fast detection of reliable visual references, and their subsequent characterization as landmarks for immediate and unambiguous recognition. Aimed at speed, our strategy has been to track salient regions along image streams by just performing on-line pixel sampling. Persistent regions are considered good candidates for landmarks, which are then characterized by a set of subregions with given color and normalized shape. They are stored in a database for posterior recognition during the navigation process. Some experimental results showing landmark-based navigation of the legged robot Lauron III in an outdoor setting are provided.Peer Reviewe

Multimodal Imaging of Photoreceptor Structure in Choroideremia

Purpose Choroideremia is a progressive X-linked recessive dystrophy, characterized by degeneration of the retinal pigment epithelium (RPE), choroid, choriocapillaris, and photoreceptors. We examined photoreceptor structure in a series of subjects with choroideremia with particular attention to areas bordering atrophic lesions. Methods Twelve males with clinically-diagnosed choroideremia and confirmed hemizygous mutations in the CHM gene were examined. High-resolution images of the retina were obtained using spectral domain optical coherence tomography (SD-OCT) and both confocal and non-confocal split-detector adaptive optics scanning light ophthalmoscope (AOSLO) techniques. Results Eleven CHM gene mutations (3 novel) were identified; three subjects had the same mutation and one subject had two mutations. SD-OCT findings included interdigitation zone (IZ) attenuation or loss in 10/12 subjects, often in areas with intact ellipsoid zones; RPE thinning in all subjects; interlaminar bridges in the imaged areas of 10/12 subjects; and outer retinal tubulations (ORTs) in 10/12 subjects. Only split-detector AOSLO could reliably resolve cones near lesion borders, and such cones were abnormally heterogeneous in morphology, diameter and density. On split-detector imaging, the cone mosaic terminated sharply at lesion borders in 5/5 cases examined. Split-detector imaging detected remnant cone inner segments within ORTs, which were generally contiguous with a central patch of preserved retina. Conclusions Early IZ dropout and RPE thinning on SD-OCT are consistent with previously published results. Evidence of remnant cone inner segments within ORTs and the continuity of the ORTs with preserved retina suggests that these may represent an intermediate state of retinal degeneration prior to complete atrophy. Taken together, these results supports a model of choroideremia in which the RPE degenerates before photoreceptors

Mixed Selectivity via Unsupervised Learning in Neural Networks

Mixed selectivity characterises neurons that simultaneously respond to different input stimuli. Neurons with mixed selectivity have been observed in multiple brain regions, and are hypothesised to play important roles in neural computation. Recently, both experimental and theoretical work demonstrated the importance of mixed selectivity in context-dependent decision tasks. This thesis extends existing theoretical work on mixed selectivity, arguing for a general and statistical role of mixed selectivity in learning complex dependencies of input stimuli. This role can be motivated from unsupervised learning of generative models, and is exhibited in increased mutual information between the stimuli and their neural representation. This argument is supported empirically using simulation of a sequence disambiguation task that incorporated key aspects of related behaviour experiments. Mixed selectivity neurons that resembled hippocampal place cells were emerged from models optimised only for behaviour. To understand these results, as well as to generalise the findings to a wider range of computations, I provided a formal connection between learning robust models and mixed selectivity

Modelling the human perception of shape-from-shading

Shading conveys information on 3-D shape and the process of recovering this information is called shape-from-shading (SFS). This thesis divides the process of human SFS into two functional sub-units (luminance disambiguation and shape computation) and studies them individually. Based on results of a series of psychophysical experiments it is proposed that the interaction between first- and second-order channels plays an important role in disambiguating luminance. Based on this idea, two versions of a biologically plausible model are developed to explain the human performances observed here and elsewhere. An algorithm sharing the same idea is also developed as a solution to the problem of intrinsic image decomposition in the field of image processing. With regard to the shape computation unit, a link between luminance variations and estimated surface norms is identified by testing participants on simple gratings with several different luminance profiles. This methodology is unconventional but can be justified in the light of past studies of human SFS. Finally a computational algorithm for SFS containing two distinct operating modes is proposed. This algorithm is broadly consistent with the known psychophysics on human SFS