Dynamic Vision Shape from Motion

Dynamic vision is a class of problems in computer vision that calculates properties of the3D world through the dynamics of the system observing by cameras. In this thesis we introduce the issue of motion and shape estimation with the aid of a single camera and for particular objects as points, planar surfaces and polyhedrons. This problem is well known in literature as Shape from Motion. We are interested in building a framework that, starting from a sequence of images that has been recorded from a camera, achieves the geometrical and dynamics properties of the 3D scene. In the first part we inspect the motion estimation through the 2D frames recorded by the camera. In particular we present a possible recursive approach for Optical Flow estimation. The second part introduces a new class of problems in system theory that are suitable for applications of computer vision. This subject has been called perspective system theory. We consider typical issue as observability, identifiability and realization theory for this branch of problems. Results has been shown through use of algebraic and recursive estimation algorithms

The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM

New vision sensors, such as the Dynamic and Active-pixel Vision sensor (DAVIS), incorporate a conventional global-shutter camera and an event-based sensor in the same pixel array. These sensors have great potential for high-speed robotics and computer vision because they allow us to combine the benefits of conventional cameras with those of event-based sensors: low latency, high temporal resolution, and very high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called "events") and synchronous grayscale frames. For this purpose, we present and release a collection of datasets captured with a DAVIS in a variety of synthetic and real environments, which we hope will motivate research on new algorithms for high-speed and high-dynamic-range robotics and computer-vision applications. In addition to global-shutter intensity images and asynchronous events, we provide inertial measurements and ground-truth camera poses from a motion-capture system. The latter allows comparing the pose accuracy of ego-motion estimation algorithms quantitatively. All the data are released both as standard text files and binary files (i.e., rosbag). This paper provides an overview of the available data and describes a simulator that we release open-source to create synthetic event-camera data.Comment: 7 pages, 4 figures, 3 table

Dynamic Adaptation in Fly Motion Vision

Sensory neurons process and convey information about our surroundings, providing the physiological basis for how we interact with the external world. In order to understand neuronal responses we must identify the rules governing how sensory information is encoded. It was proposed more than fifty years ago that neural codes constitute efficient representations of the natural world (Attneave, 1954; Barlow, 1961). In an information maximization paradigm, an efficient coding strategy will match the encoded neural response to the statistics of the input signals. Adaptation of the stimulus-response function to the statistics of the stimulus is one way to efficiently encode a stimulus when the response range and resolution are limited compared to the entire range of stimulus probabilities (Laughlin, 1981). Recent work has indeed shown that adaptation to the input statistics can occur in real time (Smirnakis et al., 1997) and that this form of adaptation can be used to efficiently encode the stimulus and maximize information transmission (Brenner et al., 2000). In this work I examined the mechanisms of dynamic adaptation in fly motion vision. The H1-cell is a large field tangential cell of the blowfly visual system that responds to motion in a directionally selective way. It also adapts its response properties to the second order statistics of an apparent motion stimulus (Fairhall et al., 2001). I measured the adaptation of the H1-cell to the variance and temporal correlations of a Gaussian low-pass filtered velocity signal that directed a sine wave visual grating. I found that the H1-cell adapted the slope, or gain, and range of its input-output function to the variance of the velocity signal over two orders of magnitude. The H1-cell also adapted its response properties to the low-pass filter time constant of the velocity signal over one order of magnitude. I compared the adaptation between flies by normalizing the gain of the stimulus-response function by the gain of the stimulus-response function during steady-state firing properties. This “dynamic gain” decreased as the velocity variance increased and broadened to cover the larger range of velocities. In contrast, as the time constant of the velocity fluctuations increased, the dynamic gain increased. The results of these experiments were then compared with simulations of the correlation-type or Reichardt motion detector model. The Reichardt detector is an algorithmic model for motion detection that explains the behavior of directionally selective large-field tangential cells in flies including the H1-cell, as well as directionally selective motion vision in humans (Zanker, 1996; Borst and Egelhaaf, 1989). The Reichardt detector model showed the same adaptive properties as the H1-cell in response to the same stimuli. Reichardt detector adaptation occurred without changing any of the model parameters; it was an automatic function of the dynamics of the model. This suggested that the mathematical properties of the Reichardt detector provide a mechanism for adaptation in the H1-cell of the blowfly. This adaptation was further characterized in both the Reichardt detector model and the H1-cell. The time course of this form of velocity adaptation in the H1-cell was examined by switching between two different variances and two different low-pass filter time constants of the velocity signal. The H1-cell adapted to the statistics or the time course of the new velocity signal within two seconds after the switch. The Reichardt detector showed a similar time course for adaptation as in the experiments. The effect of the visual pattern on adaptation was also examined, using a square wave pattern in addition to the sine wave used previously. The visual pattern affects the output of an array of Reichardt motion detectors and may therefore affect adaptation in the system. The overall shape of the adaptation function with respect to the stimulus variance was not different between the two stimulus patterns. In the experiments, the H1-cell showed a consistently higher dynamic gain with a square wave pattern. The Reichardt detector model, however, had a lower dynamic gain when the square wave pattern was presented. After careful investigation of the potential causes of this discrepancy I found that the steady-state firing rate of the H1-cell saturated when a square wave pattern was used, thereby altering the normalization under experimental conditions that was not accounted for in the simulations. These results suggest that contrast saturation is an important feature of fly motion vision that has not been explained by the Reichardt detector model. The Reichardt detector provides an automatic mechanism and mathematical explanation for adaptation in the fly visual system involving the nature of the incoming visual signals and the non-linearity in the motion detector model. Interestingly, the gradient detector model, although it is also non-linear, does not display automatic adaptation. It remains to be seen whether this type of adaptation is prominent in other sensory systems and whether it leads to and efficient and accurate representation of the natural world

Focus Is All You Need: Loss Functions For Event-based Vision

Event cameras are novel vision sensors that output pixel-level brightness changes ("events") instead of traditional video frames. These asynchronous sensors offer several advantages over traditional cameras, such as, high temporal resolution, very high dynamic range, and no motion blur. To unlock the potential of such sensors, motion compensation methods have been recently proposed. We present a collection and taxonomy of twenty two objective functions to analyze event alignment in motion compensation approaches (Fig. 1). We call them Focus Loss Functions since they have strong connections with functions used in traditional shape-from-focus applications. The proposed loss functions allow bringing mature computer vision tools to the realm of event cameras. We compare the accuracy and runtime performance of all loss functions on a publicly available dataset, and conclude that the variance, the gradient and the Laplacian magnitudes are among the best loss functions. The applicability of the loss functions is shown on multiple tasks: rotational motion, depth and optical flow estimation. The proposed focus loss functions allow to unlock the outstanding properties of event cameras.Comment: 29 pages, 19 figures, 4 table