Event-based Vision: A Survey

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world

Nonrigid reconstruction of 3D breast surfaces with a low-cost RGBD camera for surgical planning and aesthetic evaluation

Accounting for 26% of all new cancer cases worldwide, breast cancer remains the most common form of cancer in women. Although early breast cancer has a favourable long-term prognosis, roughly a third of patients suffer from a suboptimal aesthetic outcome despite breast conserving cancer treatment. Clinical-quality 3D modelling of the breast surface therefore assumes an increasingly important role in advancing treatment planning, prediction and evaluation of breast cosmesis. Yet, existing 3D torso scanners are expensive and either infrastructure-heavy or subject to motion artefacts. In this paper we employ a single consumer-grade RGBD camera with an ICP-based registration approach to jointly align all points from a sequence of depth images non-rigidly. Subtle body deformation due to postural sway and respiration is successfully mitigated leading to a higher geometric accuracy through regularised locally affine transformations. We present results from 6 clinical cases where our method compares well with the gold standard and outperforms a previous approach. We show that our method produces better reconstructions qualitatively by visual assessment and quantitatively by consistently obtaining lower landmark error scores and yielding more accurate breast volume estimates

A Survey on Deep Learning in Medical Image Registration: New Technologies, Uncertainty, Evaluation Metrics, and Beyond

Over the past decade, deep learning technologies have greatly advanced the field of medical image registration. The initial developments, such as ResNet-based and U-Net-based networks, laid the groundwork for deep learning-driven image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, and uncertainty estimation. These advancements have not only enriched the field of deformable image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration

Preoperative Systems for Computer Aided Diagnosis based on Image Registration: Applications to Breast Cancer and Atherosclerosis

Computer Aided Diagnosis (CAD) systems assist clinicians including radiologists and cardiologists to detect abnormalities and highlight conspicuous possible disease. Implementing a pre-operative CAD system contains a framework that accepts related technical as well as clinical parameters as input by analyzing the predefined method and demonstrates the prospective output. In this work we developed the Computer Aided Diagnostic System for biomedical imaging analysis of two applications on Breast Cancer and Atherosclerosis. The aim of the first CAD application is to optimize the registration strategy specifically for Breast Dynamic Infrared Imaging and to make it user-independent. Base on the fact that automated motion reduction in dynamic infrared imaging is on demand in clinical applications, since movement disarranges time-temperature series of each pixel, thus originating thermal artifacts that might bias the clinical decision. All previously proposed registration methods are feature based algorithms requiring manual intervention. We implemented and evaluated 3 different 3D time-series registration methods: 1. Linear affine, 2. Non-linear Bspline, 3. Demons applied to 12 datasets of healthy breast thermal images. The results are evaluated through normalized mutual information with average values of 0.70±0.03, 0.74±0.03 and 0.81±0.09 (out of 1) for Affine, BSpline and Demons registration, respectively, as well as breast boundary overlap and Jacobian determinant of the deformation field. The statistical analysis of the results showed that symmetric diffeomorphic Demons registration method outperforms also with the best breast alignment and non-negative Jacobian values which guarantee image similarity and anatomical consistency of the transformation, due to homologous forces enforcing the pixel geometric disparities to be shortened on all the frames. We propose Demons registration as an effective technique for time-series dynamic infrared registration, to stabilize the local temperature oscillation. The aim of the second implemented CAD application is to assess contribution of calcification in plaque vulnerability and wall rupture and to find its maximum resistance before break in image-based models of carotid artery stenting. The role of calcification inside fibroatheroma during carotid artery stenting operation is controversial in which cardiologists face two major problems during the placement: (i) “plaque protrusion” (i.e. elastic fibrous caps containing early calcifications that penetrate inside the stent); (ii) “plaque vulnerability” (i.e. stiff plaques with advanced calcifications that break the arterial wall or stent). Finite Element Analysis was used to simulate the balloon and stent expansion as a preoperative patient-specific virtual framework. A nonlinear static structural analysis was performed on 20 patients acquired using in vivo MDCT angiography. The Agatston Calcium score was obtained for each patient and subject-specific local Elastic Modulus (EM) was calculated. The in silico results showed that by imposing average ultimate external load of 1.1MPa and 2.3MPa on balloon and stent respectively, average ultimate stress of 55.7±41.2kPa and 171±41.2kPa are obtained on calcifications. The study reveals that a significant positive correlation (R=0.85, p<0.0001) exists on stent expansion between EM of calcification and ultimate stress as well as Plaque Wall Stress (PWS) (R=0.92, p<0.0001), comparing to Ca score that showed insignificant associations with ultimate stress (R=0.44, p=0.057) and PWS (R=0.38, p=0.103), suggesting minor impact of Ca score in plaque rupture. These average data are in good agreement with results obtained by other research groups and we believe this approach enriches the arsenal of tools available for pre-operative prediction of carotid artery stenting procedure in the presence of calcified plaques