Event-based Vision: A Survey

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world

Rask Policy-Læring Gjennom Imitation Learning og Reinforcement Learning i Løfteoperasjoner

I denne forskningen ble implementering og evaluering av en ny læringsmetode for autonom kranoperasjon kalt LOKI-G (Locally Optimal search after K-step Imitation - Generalized) ved hjelp av Closed-form Continous-time (CfC) Artificial Neural Network (ANN) utforsket. Studien dreide seg om å takle Sim-to-real gapet ved å tillate modellen å lære "on edge" med minimale eksempler, noe som reduserer behovet for simulatorer. Det ble lagt vekt på å skape en effektiv, robust, pålitelig og forklarlig modell som kunne trenes for anvendelser i den virkelige verden. Forskningen involverte fem eksperimenter hvor modellens ytelse under varierende forhold ble gransket. Modellens reaksjon under basisforhold, sensorisk deprivasjon, endret målposisjon og objektgeneralisering ga betydelige innsikter i modellens evner og potensielle områder for forbedring. Resultatene demonstrerte CfC ANN's evne til å lære den grunnleggende oppgaven med høy nøyaktighet, og viste pålitelig oppførsel og utmerket ytelse under Zero-Shot Learning. Modellen viste imidlertid begrensninger med hensyn til å forstå dybde. Disse funnene har betydelige konsekvenser for å akselerere utviklingen av autonomi i kraner, noe som øker industriell effektivitet og sikkerhet, reduserer karbonutslipp og baner vei for bred adopsjon av autonome løfteoperasjoner. Fremtidige forskningsretninger antyder potensialet for å forbedre modellen ved å optimalisere hyperparametre, utvide modellen til multimodal operasjon, forbedre sikkerhet gjennom bruk av BarrierNet, og adoptere nye læringsmetoder for raskere konvergens. Refleksjoner om viktigheten av å vente under oppgaver og mengden og kvaliteten på data for trening dukket også opp i studien. Som konklusjon har dette arbeidet gitt et eksperimentelt bevis på konsept og et springbrett for fremtidig forskning i utviklingen av tilpasningsdyktige, robuste og pålitelige AI-modeller for autonome industrioperasjoner.In this research, the implementation and evaluation of a novel learning approach for an autonomous crane operation called LOKI-G (Locally Optimal search after K-step Imitation - Generalized) using Closed-form Continous-time (CfC) Artificial Neural Network (ANN) was explored. The study revolved around addressing the Sim-to-real gap by allowing the model to learn on edge with minimal examples, mitigating the need for simulators. An emphasis was placed on creating a sparse, robust, reliable, and explainable model that could be trained for real-world applications. The research involved five experiments where the model's performance under varying conditions was scrutinized. The model's response under baseline conditions, sensory deprivation, altered environment, and object generalization provided significant insights into the model's capabilities and potential areas for improvement. The results demonstrated the CfC ANN's ability to learn the fundamental task with high accuracy, exhibiting reliable behaviour and excellent performance during Zero-Shot Learning. The model, however, showed limitations in regard to understanding depth. These findings have significant implications for accelerating the development of autonomy in cranes, thus increasing industrial efficiency and safety, reducing carbon emissions and paving the way for the wide-scale adoption of autonomous lifting operations. Future research directions suggest the potential of improving the model by optimizing hyperparameters, extending the model to multimodal operation, ensuring safety through the application of BarrierNet, and adopting new learning methods for faster convergence. Reflections on the importance of waiting during tasks and the quantity and quality of data for training also surfaced during the study. In conclusion, this work has provided an experimental proof of concept and a springboard for future research into the development of adaptable, robust, and trustworthy AI models for autonomous industrial operations