FPGA-Based Multimodal Embedded Sensor System Integrating Low- and Mid-Level Vision

Motion estimation is a low-level vision task that is especially relevant due to its wide range of applications in the real world. Many of the best motion estimation algorithms include some of the features that are found in mammalians, which would demand huge computational resources and therefore are not usually available in real-time. In this paper we present a novel bioinspired sensor based on the synergy between optical flow and orthogonal variant moments. The bioinspired sensor has been designed for Very Large Scale Integration (VLSI) using properties of the mammalian cortical motion pathway. This sensor combines low-level primitives (optical flow and image moments) in order to produce a mid-level vision abstraction layer. The results are described trough experiments showing the validity of the proposed system and an analysis of the computational resources and performance of the applied algorithms

Computational intelligence approaches to robotics, automation, and control [Volume guest editors]

No abstract available

Robust Models for Optic Flow Coding in Natural Scenes Inspired by Insect Biology

The extraction of accurate self-motion information from the visual world is a difficult problem that has been solved very efficiently by biological organisms utilizing non-linear processing. Previous bio-inspired models for motion detection based on a correlation mechanism have been dogged by issues that arise from their sensitivity to undesired properties of the image, such as contrast, which vary widely between images. Here we present a model with multiple levels of non-linear dynamic adaptive components based directly on the known or suspected responses of neurons within the visual motion pathway of the fly brain. By testing the model under realistic high-dynamic range conditions we show that the addition of these elements makes the motion detection model robust across a large variety of images, velocities and accelerations. Furthermore the performance of the entire system is more than the incremental improvements offered by the individual components, indicating beneficial non-linear interactions between processing stages. The algorithms underlying the model can be implemented in either digital or analog hardware, including neuromorphic analog VLSI, but defy an analytical solution due to their dynamic non-linear operation. The successful application of this algorithm has applications in the development of miniature autonomous systems in defense and civilian roles, including robotics, miniature unmanned aerial vehicles and collision avoidance sensors

Passive Energy Recapture in Jellyfish Contributes to Propulsive Advantage over other Metazoans

Gelatinous zooplankton populations are well known for their ability to take over perturbed ecosystems. The ability of these animals to outcompete and functionally replace fish that exhibit an effective visual predatory mode is counterintuitive because jellyfish are described as inefficient swimmers that must rely on direct contact with prey to feed. We show that jellyfish exhibit a unique mechanism of passive energy recapture, which is exploited to allow them to travel 30% further each swimming cycle, thereby reducing metabolic energy demand by swimming muscles. By accounting for large interspecific differences in net metabolic rates, we demonstrate, contrary to prevailing views, that the jellyfish (Aurelia aurita) is one of the most energetically efficient propulsors on the planet, exhibiting a cost of transport (joules per kilogram per meter) lower than other metazoans. We estimate that reduced metabolic demand by passive energy recapture improves the cost of transport by 48%, allowing jellyfish to achieve the large sizes required for sufficient prey encounters. Pressure calculations, using both computational fluid dynamics and a newly developed method from empirical velocity field measurements, demonstrate that this extra thrust results from positive pressure created by a vortex ring underneath the bell during the refilling phase of swimming. These results demonstrate a physical basis for the ecological success of medusan swimmers despite their simple body plan. Results from this study also have implications for bioinspired design, where low-energy propulsion is required

Event transformer FlowNet for optical flow estimation

Event cameras are bioinspired sensors that produce asynchronous and sparse streams of events at image locations where intensity change is detected. They can detect fast motion with low latency, high dynamic range, and low power consumption. Over the past decade, efforts have been conducted in developing solutions with event cameras for robotics applications. In this work, we address their use for fast and robust computation of optical flow. We present ET-FlowNet, a hybrid RNN-ViT architecture for optical flow estimation. Visual transformers (ViTs) are ideal candidates for the learning of global context in visual tasks, and we argue that rigid body motion is a prime case for the use of ViTs since long-range dependencies in the image hold during rigid body motion. We perform end-to-end training with self-supervised learning method. Our results show comparable and in some cases exceeding performance with state-of-the-art coarse-to-fine event-based optical flow estimation.This work was supported by projects EBSLAM DPI2017-89564-P and EBCON PID2020-119244GB-I00 funded by CIN/AEI/10.13039/501100011033 and by an FI AGAUR PhD grant to Yi Tian.Postprint (published version