An Event-Driven Multi-Kernel Convolution Processor Module for Event-Driven Vision Sensors

Event-Driven vision sensing is a new way of sensing visual reality in a frame-free manner. This is, the vision sensor (camera) is not capturing a sequence of still frames, as in conventional video and computer vision systems. In Event-Driven sensors each pixel autonomously and asynchronously decides when to send its address out. This way, the sensor output is a continuous stream of address events representing reality dynamically continuously and without constraining to frames. In this paper we present an Event-Driven Convolution Module for computing 2D convolutions on such event streams. The Convolution Module has been designed to assemble many of them for building modular and hierarchical Convolutional Neural Networks for robust shape and pose invariant object recognition. The Convolution Module has multi-kernel capability. This is, it will select the convolution kernel depending on the origin of the event. A proof-of-concept test prototype has been fabricated in a 0.35 m CMOS process and extensive experimental results are provided. The Convolution Processor has also been combined with an Event-Driven Dynamic Vision Sensor (DVS) for high-speed recognition examples. The chip can discriminate propellers rotating at 2 k revolutions per second, detect symbols on a 52 card deck when browsing all cards in 410 ms, or detect and follow the center of a phosphor oscilloscope trace rotating at 5 KHz.Unión Europea 216777 (NABAB)Ministerio de Ciencia e Innovación TEC2009-10639-C04-0

CMOS Architectures and circuits for high-speed decision-making from image flows

We present architectures, CMOS circuits and CMOS chips to process image flows at very high speed. This is achieved by exploiting bio-inspiration and performing processing tasks in parallel manner and concurrently with image acquisition. A vision system is presented which makes decisions within sub-msec range. This is very well suited for defense and security applications requiring segmentation and tracking of rapidly moving objects

Egocentric Perception using a Biologically Inspired Software Retina Integrated with a Deep CNN

We presented the concept of of a software retina, capable of significant visual data reduction in combination with scale and rotation invariance, for applications in egocentric and robot vision at the first EPIC workshop in Amsterdam [9]. Our method is based on the mammalian retino-cortical transform: a mapping between a pseudo-randomly tessellated retina model (used to sample an input image) and a CNN. The aim of this first pilot study is to demonstrate a functional retina-integrated CNN implementation and this produced the following results: a network using the full retino-cortical transform yielded an F1 score of 0.80 on a test set during a 4-way classification task, while an identical network not using the proposed method yielded an F1 score of 0.86 on the same task. On a 40K node retina the method reduced the visual data bye×7, the input data to the CNN by 40% and the number of CNN training epochs by 36%. These results demonstrate the viability of our method and hint at the potential of exploiting functional traits of natural vision systems in CNNs. In addition, to the above study, we present further recent developments in porting the retina to an Apple iPhone, an implementation in CUDA C for NVIDIA GPU platforms and extensions of the retina model we have adopted

Bioinspired engineering of exploration systems for NASA and DoD

A new approach called bioinspired engineering of exploration systems (BEES) and its value for solving pressing NASA and DoD needs are described. Insects (for example honeybees and dragonflies) cope remarkably well with their world, despite possessing a brain containing less than 0.01% as many neurons as the human brain. Although most insects have immobile eyes with fixed focus optics and lack stereo vision, they use a number of ingenious, computationally simple strategies for perceiving their world in three dimensions and navigating successfully within it. We are distilling selected insect-inspired strategies to obtain novel solutions for navigation, hazard avoidance, altitude hold, stable flight, terrain following, and gentle deployment of payload. Such functionality provides potential solutions for future autonomous robotic space and planetary explorers. A BEES approach to developing lightweight low-power autonomous flight systems should be useful for flight control of such biomorphic flyers for both NASA and DoD needs. Recent biological studies of mammalian retinas confirm that representations of multiple features of the visual world are systematically parsed and processed in parallel. Features are mapped to a stack of cellular strata within the retina. Each of these representations can be efficiently modeled in semiconductor cellular nonlinear network (CNN) chips. We describe recent breakthroughs in exploring the feasibility of the unique blending of insect strategies of navigation with mammalian visual search, pattern recognition, and image understanding into hybrid biomorphic flyers for future planetary and terrestrial applications. We describe a few future mission scenarios for Mars exploration, uniquely enabled by these newly developed biomorphic flyers

Temporal Dynamics of Binocular Display Processing with Corticogeniculate Interactions

A neural model of binocular vision is developed to simulate psychophysical and neurobiological data concerning the dynamics of binocular disparity processing. The model shows how feedforward and feedback interactions among LGN ON and OFF cells and cortical simple, complex, and hypercomplex cells can simulate binocular summation, the Pulfrich effect, and the fusion of delayed anticorrelated stereograms. Model retinal ON and OFF cells are linked by an opponent process capable of generating antagonistic rebounds from OFF cells after offset of an ON cell input. Spatially displaced ON and OFF cells excite simple cells. Opposite polarity simple cells compete before their half-wave rectified outputs excite complex cells. Complex cells binocularly match like-polarity simple cell outputs before pooling half-wave rectified signals frorn opposite polarities. Competitive feedback among complex cells leads to sharpening of disparity selectivity and normalizes cell activity. Slow inhibitory interneurons help to reset complex cells after input offset. The Pulfrich effect occurs because the delayed input from the one eye fuses with the present input from the other eye to create a disparity. Binocular summation occurs for stimuli of brief duration or of low contrast because competitive normalization takes time, and cannot occur for very brief or weak stimuli. At brief SOAs, anticorrelatecd stereograms can be fused because the rebound mechanism ensures that the present image to one eye can fuse with the afterimage from a previous image to the other eye. Corticogeniculate feedback embodies a matching process that enhances the speed and temporal accuracy of complex cell disparity tuning. Model mechanisms interact to control the stable development of sharp disparity tuning.Air Force Office of Scientific Research (F19620-92-J-0499, F49620-92-J-0334, F49620-92-J-0225); Office of Naval Research (N00014-95-1-0409, N00014-95-l-0657, N00014-92-J-1015, N00014-91-J-4100

A spatial contrast retina with on-chip calibration for neuromorphic spike-based AER vision systems

We present a 32 32 pixels contrast retina microchip that provides its output as an address event representation (AER) stream. Spatial contrast is computed as the ratio between pixel photocurrent and a local average between neighboring pixels obtained with a diffuser network. This current-based computation produces an important amount of mismatch between neighboring pixels, because the currents can be as low as a few pico-amperes. Consequently, a compact calibration circuitry has been included to trimm each pixel. Measurements show a reduction in mismatch standard deviation from 57% to 6.6% (indoor light). The paper describes the design of the pixel with its spatial contrast computation and calibration sections. About one third of pixel area is used for a 5-bit calibration circuit. Area of pixel is 58 m 56 m, while its current consumption is about 20 nA at 1-kHz event rate. Extensive experimental results are provided for a prototype fabricated in a standard 0.35- m CMOS process.Gobierno de España TIC2003-08164-C03-01, TEC2006-11730-C03-01European Union IST-2001-3412

Spike-based VITE control with Dynamic Vision Sensor applied to an Arm Robot.

Spike-based motor control is very important in the field of robotics and also for the neuromorphic engineering community to bridge the gap between sensing / processing devices and motor control without losing the spike philosophy that enhances speed response and reduces power consumption. This paper shows an accurate neuro-inspired spike-based system composed of a DVS retina, a visual processing system that detects and tracks objects, and a SVITE motor control, where everything follows the spike-based philosophy. The control system is a spike version of the neuroinspired open loop VITE control algorithm implemented in a couple of FPGA boards: the first one runs the algorithm and the second one drives the motors with spikes. The robotic platform is a low cost arm with four degrees of freedom.Ministerio de Ciencia e Innovación TEC2009-10639-C04-02/01Ministerio de Economía y Competitividad TEC2012-37868-C04-02/0

Bcl-xL-mediated remodeling of rod and cone synaptic mitochondria after postnatal lead exposure: electron microscopy, tomography and oxygen consumption.

PurposePostnatal lead exposure produces rod-selective and Bax-mediated apoptosis, decreased scotopic electroretinograms (ERGs), and scotopic and mesopic vision deficits in humans and/or experimental animals. Rod, but not cone, inner segment mitochondria were considered the primary site of action. However, photoreceptor synaptic mitochondria were not examined. Thus, our experiments investigated the structural and functional effects of environmentally relevant postnatal lead exposure on rod spherule and cone pedicle mitochondria and whether Bcl-xL overexpression provided neuroprotection.MethodsC57BL/6N mice pups were exposed to lead only during lactation via dams drinking water containing lead acetate. The blood [Pb] at weaning was 20.6±4.7 µg/dl, which decreased to the control value by 2 months. To assess synaptic mitochondrial structural differences and vulnerability to lead exposure, wild-type and transgenic mice overexpressing Bcl-xL in photoreceptors were used. Electron microscopy, three-dimensional electron tomography, and retinal and photoreceptor synaptic terminal oxygen consumption (QO(2)) studies were conducted in adult control, Bcl-xL, lead, and Bcl-xL/lead mice.ResultsThe spherule and pedicle mitochondria in lead-treated mice were swollen, and the cristae structure was markedly changed. In the lead-treated mice, the mitochondrial cristae surface area and volume (abundance: measure correlated with ATP (ATP) synthesis) were decreased in the spherules and increased in the pedicles. Pedicles also had an increased number of crista segments per volume. In the lead-treated mice, the number of segments/crista and fraction of cristae with multiple segments (branching) similarly increased in spherule and pedicle mitochondria. Lead-induced remodeling of spherule mitochondria produced smaller cristae with more branching, whereas pedicle mitochondria had larger cristae with more branching and increased crista junction (CJ) diameter. Lead decreased dark- and light-adapted photoreceptor and dark-adapted photoreceptor synaptic terminal QO(2). Bcl-xL partially blocked many of the lead-induced alterations relative to controls. However, spherules still had partially decreased abundance, whereas pedicles still had increased branching, increased crista segments per volume, and increased crista junction diameter. Moreover, photoreceptor and synaptic QO(2) were only partially recovered.ConclusionsThese findings reveal cellular and compartmental specific differences in the structure and vulnerability of rod and cone inner segment and synaptic mitochondria to postnatal lead exposure. Spherule and pedicle mitochondria in lead-exposed mice displayed complex and distinguishing patterns of cristae and matrix damage and remodeling consistent with studies showing that synaptic mitochondria are more sensitive to Ca(2+) overload, oxidative stress, and ATP loss than non-synaptic mitochondria. The lead-induced decreases in QO(2) likely resulted from the decreased spherule cristae abundance and smaller cristae, perhaps due to Bax-mediated effects as they occurred in apoptotic rod inner segments. The increase in pedicle cristae abundance and CJ diameter could have resulted from increased Drp1-mediated fission, as small mitochondrial fragments were observed. The mechanisms of Bcl-xL-mediated remodeling might occur via interaction with formation of CJ protein 1 (Fcj1), whereas the partial protection of synaptic QO(2) might result from the enhanced efficiency of energy metabolism via Bcl-xL's direct interaction with the F1F0 ATP synthase and/or regulation of cellular redox status. These lead-induced alterations in photoreceptor synaptic terminal mitochondria likely underlie the persistent scotopic and mesopic deficits in lead-exposed children, workers, and experimental animals. Our findings stress the clinical and scientific importance of examining synaptic dysfunction following injury or disease during development, and developing therapeutic treatments that prevent synaptic degeneration in retinal and neurodegenerative disorders even when apoptosis is blocked