Event-based Vision: A Survey

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world

Invariance of visual operations at the level of receptive fields

Receptive field profiles registered by cell recordings have shown that mammalian vision has developed receptive fields tuned to different sizes and orientations in the image domain as well as to different image velocities in space-time. This article presents a theoretical model by which families of idealized receptive field profiles can be derived mathematically from a small set of basic assumptions that correspond to structural properties of the environment. The article also presents a theory for how basic invariance properties to variations in scale, viewing direction and relative motion can be obtained from the output of such receptive fields, using complementary selection mechanisms that operate over the output of families of receptive fields tuned to different parameters. Thereby, the theory shows how basic invariance properties of a visual system can be obtained already at the level of receptive fields, and we can explain the different shapes of receptive field profiles found in biological vision from a requirement that the visual system should be invariant to the natural types of image transformations that occur in its environment.Comment: 40 pages, 17 figure

Bioinspired symmetry detection on resource limited embedded platforms

This work is inspired by the vision of flying insects which enables them to detect and locate a set of relevant objects with remarkable effectiveness despite very limited brainpower. The bioinspired approach worked out here focuses on detection of symmetric objects to be performed by resource-limited embedded platforms such as micro air vehicles. Symmetry detection is posed as a pattern matching problem which is solved by an approach based on the use of composite correlation filters. Two variants of the approach are proposed, analysed and tested in which symmetry detection is cast as 1) static and 2) dynamic pattern matching problems. In the static variant, images of objects are input to two dimentional spatial composite correlation filters. In the dynamic variant, a video (resulting from platform motion) is input to a composite correlation filter of which its peak response is used to define symmetry. In both cases, a novel method is used for designing the composite filter templates for symmetry detection. This method significantly reduces the level of detail which needs to be matched to achieve good detection performance. The resulting performance is systematically quantified using the ROC analysis; it is demonstrated that the bioinspired detection approach is better and with a lower computational cost compared to the best state-of-the-art solution hitherto available

Computational Modeling of Human Dorsal Pathway for Motion Processing

Reliable motion estimation in videos is of crucial importance for background iden- tification, object tracking, action recognition, event analysis, self-navigation, etc. Re- constructing the motion field in the 2D image plane is very challenging, due to variations in image quality, scene geometry, lighting condition, and most importantly, camera jit- tering. Traditional optical flow models assume consistent image brightness and smooth motion field, which are violated by unstable illumination and motion discontinuities that are common in real world videos. To recognize observer (or camera) motion robustly in complex, realistic scenarios, we propose a biologically-inspired motion estimation system to overcome issues posed by real world videos. The bottom-up model is inspired from the infrastructure as well as functionalities of human dorsal pathway, and the hierarchical processing stream can be divided into three stages: 1) spatio-temporal processing for local motion, 2) recogni- tion for global motion patterns (camera motion), and 3) preemptive estimation of object motion. To extract effective and meaningful motion features, we apply a series of steer- able, spatio-temporal filters to detect local motion at different speeds and directions, in a way that\u27s selective of motion velocity. The intermediate response maps are cal- ibrated and combined to estimate dense motion fields in local regions, and then, local motions along two orthogonal axes are aggregated for recognizing planar, radial and circular patterns of global motion. We evaluate the model with an extensive, realistic video database that collected by hand with a mobile device (iPad) and the video content varies in scene geometry, lighting condition, view perspective and depth. We achieved high quality result and demonstrated that this bottom-up model is capable of extracting high-level semantic knowledge regarding self motion in realistic scenes. Once the global motion is known, we segment objects from moving backgrounds by compensating for camera motion. For videos captured with non-stationary cam- eras, we consider global motion as a combination of camera motion (background) and object motion (foreground). To estimate foreground motion, we exploit corollary dis- charge mechanism of biological systems and estimate motion preemptively. Since back- ground motions for each pixel are collectively introduced by camera movements, we apply spatial-temporal averaging to estimate the background motion at pixel level, and the initial estimation of foreground motion is derived by comparing global motion and background motion at multiple spatial levels. The real frame signals are compared with those derived by forward predictions, refining estimations for object motion. This mo- tion detection system is applied to detect objects with cluttered, moving backgrounds and is proved to be efficient in locating independently moving, non-rigid regions. The core contribution of this thesis is the invention of a robust motion estimation system for complicated real world videos, with challenges by real sensor noise, complex natural scenes, variations in illumination and depth, and motion discontinuities. The overall system demonstrates biological plausibility and holds great potential for other applications, such as camera motion removal, heading estimation, obstacle avoidance, route planning, and vision-based navigational assistance, etc

Advanced signal processing tools for ballistic missile defence and space situational awareness

The research presented in this Thesis deals with signal processing algorithms for the classification of sensitive targets for defence applications and with novel solutions for the detection of space objects. These novel tools include classification algorithms for Ballistic Targets (BTs) from both micro-Doppler (mD) and High Resolution Range Profiles (HRRPs) of a target, and a space-borne Passive Bistatic Radar (PBR) designed for exploiting the advantages guaranteed by the Forward Scattering (FS) configuration for the detection and identification of targets orbiting around the Earth.;Nowadays the challenge of the identification of Ballistic Missile (BM) warheads in a cloud of decoys and debris is essential in order to optimize the use of ammunition resources. In this Thesis, two different and efficient robust frameworks are presented. Both the frameworks exploit in different fashions the effect in the radar return of micro-motions exhibited by the target during its flight.;The first algorithm analyses the radar echo from the target in the time-frequency domain, with the aim to extract the mD information. Specifically, the Cadence Velocity Diagram (CVD) from the received signal is evaluated as mD profile of the target, where the mD components composing the radar echo and their repetition rates are shown.;Different feature extraction approaches are proposed based on the estimation of statistical indices from the 1-Dimensional (1D) Averaged CVD (ACVD), on the evaluation of pseudo-Zerike (pZ) and Krawtchouk (Kr) image moments and on the use of 2-Dimensional (2D) Gabor filter, considering the CVD as 2D image. The reliability of the proposed feature extraction approaches is tested on both simulated and real data, demonstrating the adaptivity of the framework to different radar scenarios and to different amount of available resources.;The real data are realized in laboratory, conducting an experiment for simulating the mD signature of a BT by using scaled replicas of the targets, a robotic manipulator for the micro-motions simulation and a Continuous Waveform (CW) radar for the radar measurements.;The second algorithm is based on the computation of the Inverse Radon Transform (IRT) of the target signature, represented by a HRRP frame acquired within an entire period of the main rotating motion of the target, which are precession for warheads and tumbling for decoys. Following, pZ moments of the resulting transformation are evaluated as final feature vector for the classifier. The features guarantee robustness against the target dimensions and the initial phase and the angular velocity of its motion.;The classification results on simulated data are shown for different polarization of the ElectroMagnetic (EM) radar waveform and for various operational conditions, confirming the the validity of the algorithm.The knowledge of space debris population is of fundamental importance for the safety of both the existing and new space missions. In this Thesis, a low budget solution to detect and possibly track space debris and satellites in Low Earth Orbit (LEO) is proposed.;The concept consists in a space-borne PBR installed on a CubeSaT flying at low altitude and detecting the occultations of radio signals coming from existing satellites flying at higher altitudes. The feasibility of such a PBR system is conducted, with key performance such as metrics the minimumsize of detectable objects, taking into account visibility and frequency constraints on existing radio sources, the receiver size and the compatibility with current CubeSaT's technology.;Different illuminator types and receiver altitudes are considered under the assumption that all illuminators and receivers are on circular orbits. Finally, the designed system can represent a possible solution to the the demand for Ballistic Missile Defence (BMD) systems able to provide early warning and classification and its potential has been assessed also for this purpose.The research presented in this Thesis deals with signal processing algorithms for the classification of sensitive targets for defence applications and with novel solutions for the detection of space objects. These novel tools include classification algorithms for Ballistic Targets (BTs) from both micro-Doppler (mD) and High Resolution Range Profiles (HRRPs) of a target, and a space-borne Passive Bistatic Radar (PBR) designed for exploiting the advantages guaranteed by the Forward Scattering (FS) configuration for the detection and identification of targets orbiting around the Earth.;Nowadays the challenge of the identification of Ballistic Missile (BM) warheads in a cloud of decoys and debris is essential in order to optimize the use of ammunition resources. In this Thesis, two different and efficient robust frameworks are presented. Both the frameworks exploit in different fashions the effect in the radar return of micro-motions exhibited by the target during its flight.;The first algorithm analyses the radar echo from the target in the time-frequency domain, with the aim to extract the mD information. Specifically, the Cadence Velocity Diagram (CVD) from the received signal is evaluated as mD profile of the target, where the mD components composing the radar echo and their repetition rates are shown.;Different feature extraction approaches are proposed based on the estimation of statistical indices from the 1-Dimensional (1D) Averaged CVD (ACVD), on the evaluation of pseudo-Zerike (pZ) and Krawtchouk (Kr) image moments and on the use of 2-Dimensional (2D) Gabor filter, considering the CVD as 2D image. The reliability of the proposed feature extraction approaches is tested on both simulated and real data, demonstrating the adaptivity of the framework to different radar scenarios and to different amount of available resources.;The real data are realized in laboratory, conducting an experiment for simulating the mD signature of a BT by using scaled replicas of the targets, a robotic manipulator for the micro-motions simulation and a Continuous Waveform (CW) radar for the radar measurements.;The second algorithm is based on the computation of the Inverse Radon Transform (IRT) of the target signature, represented by a HRRP frame acquired within an entire period of the main rotating motion of the target, which are precession for warheads and tumbling for decoys. Following, pZ moments of the resulting transformation are evaluated as final feature vector for the classifier. The features guarantee robustness against the target dimensions and the initial phase and the angular velocity of its motion.;The classification results on simulated data are shown for different polarization of the ElectroMagnetic (EM) radar waveform and for various operational conditions, confirming the the validity of the algorithm.The knowledge of space debris population is of fundamental importance for the safety of both the existing and new space missions. In this Thesis, a low budget solution to detect and possibly track space debris and satellites in Low Earth Orbit (LEO) is proposed.;The concept consists in a space-borne PBR installed on a CubeSaT flying at low altitude and detecting the occultations of radio signals coming from existing satellites flying at higher altitudes. The feasibility of such a PBR system is conducted, with key performance such as metrics the minimumsize of detectable objects, taking into account visibility and frequency constraints on existing radio sources, the receiver size and the compatibility with current CubeSaT's technology.;Different illuminator types and receiver altitudes are considered under the assumption that all illuminators and receivers are on circular orbits. Finally, the designed system can represent a possible solution to the the demand for Ballistic Missile Defence (BMD) systems able to provide early warning and classification and its potential has been assessed also for this purpose

Models for Motion Perception

As observers move through the environment or shift their direction of gaze, the world moves past them. In addition, there may be objects that are moving differently from the static background, either rigid-body motions or nonrigid (e.g., turbulent) ones. This dissertation discusses several models for motion perception. The models rely on first measuring motion energy, a multi-resolution representation of motion information extracted from image sequences. The image flow model combines the outputs of a set of spatiotemporal motion-energy filters to estimate image velocity, consonant with current views regarding the neurophysiology and psychophysics of motion perception. A parallel implementation computes a distributed representation of image velocity that encodes both a velocity estimate and the uncertainty in that estimate. In addition, a numerical measure of image-flow uncertainty is derived. The egomotion model poses the detection of moving objects and the recovery of depth from motion as sensor fusion problems that necessitate combining information from different sensors in the presence of noise and uncertainty. Image sequences are segmented by finding image regions corresponding to entire objects that are moving differently from the stationary background. The turbulent flow model utilizes a fractal-based model of turbulence, and estimates the fractal scaling parameter of fractal image sequences from the outputs of motion-energy filters. Some preliminary results demonstrate the model\u27s potential for discriminating image regions based on fractal scaling

Data mining based learning algorithms for semi-supervised object identification and tracking

Sensor exploitation (SE) is the crucial step in surveillance applications such as airport security and search and rescue operations. It allows localization and identification of movement in urban settings and can significantly boost knowledge gathering, interpretation and action. Data mining techniques offer the promise of precise and accurate knowledge acquisition techniques in high-dimensional data domains (and diminishing the “curse of dimensionality” prevalent in such datasets), coupled by algorithmic design in feature extraction, discriminative ranking, feature fusion and supervised learning (classification). Consequently, data mining techniques and algorithms can be used to refine and process captured data and to detect, recognize, classify, and track objects with predictable high degrees of specificity and sensitivity. Automatic object detection and tracking algorithms face several obstacles, such as large and incomplete datasets, ill-defined regions of interest (ROIs), variable scalability, lack of compactness, angular regions, partial occlusions, environmental variables, and unknown potential object classes, which work against their ability to achieve accurate real-time results. Methods must produce fast and accurate results by streamlining image processing, data compression and reduction, feature extraction, classification, and tracking algorithms. Data mining techniques can sufficiently address these challenges by implementing efficient and accurate dimensionality reduction with feature extraction to refine incomplete (ill-partitioning) data-space and addressing challenges related to object classification, intra-class variability, and inter-class dependencies. A series of methods have been developed to combat many of the challenges for the purpose of creating a sensor exploitation and tracking framework for real time image sensor inputs. The framework has been broken down into a series of sub-routines, which work in both series and parallel to accomplish tasks such as image pre-processing, data reduction, segmentation, object detection, tracking, and classification. These methods can be implemented either independently or together to form a synergistic solution to object detection and tracking. The main contributions to the SE field include novel feature extraction methods for highly discriminative object detection, classification, and tracking. Also, a new supervised classification scheme is presented for detecting objects in urban environments. This scheme incorporates both novel features and non-maximal suppression to reduce false alarms, which can be abundant in cluttered environments such as cities. Lastly, a performance evaluation of Graphical Processing Unit (GPU) implementations of the subtask algorithms is presented, which provides insight into speed-up gains throughout the SE framework to improve design for real time applications. The overall framework provides a comprehensive SE system, which can be tailored for integration into a layered sensing scheme to provide the war fighter with automated assistance and support. As more sensor technology and integration continues to advance, this SE framework can provide faster and more accurate decision support for both intelligence and civilian applications

Use of multi-scale phase-based methods to determine optical flow in dynamic scene analysis

Estimates of optical flow in images can be made by applying a complex periodic transform to the images and tracking the movement of points of constant phase in the complex output. This approach however suffers from the problem that filters of large width give information only about broad scale image features, whilst those of small spatial extent (high resolution) cannot track fast motion, which causes a feature to move a distance that is large compared to the filter-size. A method is presented in which the flow is measured at different scales, using a series of complex filters of decreasing width. The largest filter is used to give a large scale flow estimate at each image point. Estimates at smaller scales are then carried out by using the previous result as an a priori estimate. Rather than comparing the same region in different images in order to estimate flow, the regions to be compared are displaced from one another by an amount given by the most recent previous flow estimate. This results in an estimate of flow relative to the earlier estimate. The two estimates are then added together to give a new estimate of the absolute displacement. The process is repeated at successively smaller scales. The method can therefore detect small local velocity variations superimposed on the broad scale flow, even where the magnitude of the absolute displacement is larger than the scope of the smaller filters. Without the assistance of the earlier estimates in ‘tuning\u27 the smaller filters in this manner, a smaller filter could fail to capture these velocity variations, because the absolute displacement carry the feature out of range of-the filter during successive frames. The output of the method is a series of scale-dependent flow fields corresponding to different scales, reflecting the fact that motion in the real world is a scale-dependent quantity. Application of the method to some 1 dimensional test images gives good results, with realistic flow values that could be used as an aid to segmentation. Some synthetic 2-dimentional images containing only a small number of well defined features aIso yield good-results but the method performs poorly on a random-dot stereogram and on a real-world test image pair selected from the Hamburg Taxi sequence