Focus Is All You Need: Loss Functions For Event-based Vision

Event cameras are novel vision sensors that output pixel-level brightness changes ("events") instead of traditional video frames. These asynchronous sensors offer several advantages over traditional cameras, such as, high temporal resolution, very high dynamic range, and no motion blur. To unlock the potential of such sensors, motion compensation methods have been recently proposed. We present a collection and taxonomy of twenty two objective functions to analyze event alignment in motion compensation approaches (Fig. 1). We call them Focus Loss Functions since they have strong connections with functions used in traditional shape-from-focus applications. The proposed loss functions allow bringing mature computer vision tools to the realm of event cameras. We compare the accuracy and runtime performance of all loss functions on a publicly available dataset, and conclude that the variance, the gradient and the Laplacian magnitudes are among the best loss functions. The applicability of the loss functions is shown on multiple tasks: rotational motion, depth and optical flow estimation. The proposed focus loss functions allow to unlock the outstanding properties of event cameras.Comment: 29 pages, 19 figures, 4 table

Modeling and applications of the focus cue in conventional digital cameras

El enfoque en cámaras digitales juega un papel fundamental tanto en la calidad de la imagen como en la percepción del entorno. Esta tesis estudia el enfoque en cámaras digitales convencionales, tales como cámaras de móviles, fotográficas, webcams y similares. Una revisión rigurosa de los conceptos teóricos detras del enfoque en cámaras convencionales muestra que, a pasar de su utilidad, el modelo clásico del thin lens presenta muchas limitaciones para aplicación en diferentes problemas relacionados con el foco. En esta tesis, el focus profile es propuesto como una alternativa a conceptos clásicos como la profundidad de campo. Los nuevos conceptos introducidos en esta tesis son aplicados a diferentes problemas relacionados con el foco, tales como la adquisición eficiente de imágenes, estimación de profundidad, integración de elementos perceptuales y fusión de imágenes. Los resultados experimentales muestran la aplicación exitosa de los modelos propuestos.The focus of digital cameras plays a fundamental role in both the quality of the acquired images and the perception of the imaged scene. This thesis studies the focus cue in conventional cameras with focus control, such as cellphone cameras, photography cameras, webcams and the like. A deep review of the theoretical concepts behind focus in conventional cameras reveals that, despite its usefulness, the widely known thin lens model has several limitations for solving different focus-related problems in computer vision. In order to overcome these limitations, the focus profile model is introduced as an alternative to classic concepts, such as the near and far limits of the depth-of-field. The new concepts introduced in this dissertation are exploited for solving diverse focus-related problems, such as efficient image capture, depth estimation, visual cue integration and image fusion. The results obtained through an exhaustive experimental validation demonstrate the applicability of the proposed models

3D image acquisition system based on shape from focus technique

agent Agrosup Dijon de l'UMREcolDurGEAPSIThis paper describes the design of a 3D image acquisition system dedicated to natural complex scenes composed of randomly distributed objects with spatial discontinuities. In agronomic sciences, the 3D acquisition of natural scene is difficult due to the complex nature of the scenes. Our system is based on the Shape from Focus technique initially used in the microscopic domain. We propose to adapt this technique to the macroscopic domain and we detail the system as well as the image processing used to perform such technique. The Shape from Focus technique is a monocular and passive 3D acquisition method that resolves the occlusion problem affecting the multi-cameras systems. Indeed, this problem occurs frequently in natural complex scenes like agronomic scenes. The depth information is obtained by acting on optical parameters and mainly the depth of field. A focus measure is applied on a 2D image stack previously acquired by the system. When this focus measure is performed, we can create the depth map of the scene

The development of a predictive autofocus algorithm using a general image formation model

Bibliography: pages 97-99.This were outlines the development of a general imaging model for use in autofocus, astigmatism correction, and resolution analysis. The model is based on the modulation transfer function of the imaging system in the presence of aberrations, in particular defocus. The extension of the model to include astigmatism is also included. The signals used are related to the ratios of the Fourier transforms of images captured under different operating conditions. Methods are developed for working with these signals in a consistent manner. The model described is then applied to the problem of autofocus. A general autofocus algorithm is presented and results given which reflect the predictive properties of this model. The imaging system used for the generation of results was a scanning electron microscope, although the conclusions should be valid across a far wider range of instruments. It is however the specific requirements of the SEM that make the generalisation presented here particularly useful

Learning Lens Blur Fields

Optical blur is an inherent property of any lens system and is challenging to model in modern cameras because of their complex optical elements. To tackle this challenge, we introduce a high-dimensional neural representation of blur$-$$\textit{the lens blur field}$$-$and a practical method for acquiring it. The lens blur field is a multilayer perceptron (MLP) designed to (1) accurately capture variations of the lens 2D point spread function over image plane location, focus setting and, optionally, depth and (2) represent these variations parametrically as a single, sensor-specific function. The representation models the combined effects of defocus, diffraction, aberration, and accounts for sensor features such as pixel color filters and pixel-specific micro-lenses. To learn the real-world blur field of a given device, we formulate a generalized non-blind deconvolution problem that directly optimizes the MLP weights using a small set of focal stacks as the only input. We also provide a first-of-its-kind dataset of 5D blur fields$-$for smartphone cameras, camera bodies equipped with a variety of lenses, etc. Lastly, we show that acquired 5D blur fields are expressive and accurate enough to reveal, for the first time, differences in optical behavior of smartphone devices of the same make and model

탈초점 흐림 정도의 예측 및 그 신뢰도를 이용한 깊이 맵 작성 기법

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2016. 2. 김태정.깊이 맵이란 영상 내에서 촬영 장치로부터 가깝고 먼 정도를 수치적으로 나타낸 것으로서 영상의 3차원 구조를 나타내기 위해 널리 쓰이는 표현 방식이다. 2차원 영상으로부터 깊이 맵을 예측하기 위해서는 탈초점 흐림, 장면의 기하학적 구조, 객체의 주목도 및 움직임 등 다양한 종류의 깊이 정보가 활용된다. 그 중에서도 탈초점 흐림은 널리 이용되는 강력한 정보로서 탈초점 흐림으로부터 깊이를 예측하는 문제는 깊이를 예측하는 데 있어서 매우 중요한 역할을 한다. 본 연구는 2차원 영상만을 이용하여 깊이 맵을 예측하는 것을 목표로 하며 이 때, 촬영 장치로부터 영상 내 각 영역의 거리를 알아내기 위해 탈초점 거리 예측을 이용한다. 먼저 영상을 촬영할 때 영상 내 가장 가까운 곳에 초점이 맞춰져 있다고 가정하면 촬영 장치로부터 멀어짐에 따라 탈초점 흐림의 정도가 증가하게 된다. 탈초점 거리 기반 깊이 맵 예측 방법은 이를 이용하여 탈초점 흐림의 정도를 측정함으로써 거리를 예측하는 방식이다. 본 연구에서는 탈초점 거리로부터 깊이 맵을 구하는 새로운 방법을 제안한다. 먼저 인간의 깊이 지각 방식을 고려한 지각 깊이를 정의하고 이를 이용하여 탈초점 거리 예측의 (실제) 신뢰도를 정의하였다. 다음으로 그래디언트 및 2차 미분 값에 기반한 탈초점 거리 예측 결과에 대하여 신뢰도를 예측하는 방법을 설계하였다. 이렇게 예측한 신뢰도 값은 기존의 신뢰도 예측 방법으로 예측한 것에 비하여 더 정확하였다. 제안하는 깊이 맵 작성 방법은 조각 단위 평면 모델에 기반하였으며, 비용 함수는 데이터 항과 평활도 항으로 구성되었다. 깊이 맵의 전체 비용 함수를 최적화하는 과정에서는 반복적 지역 최적화 방식을 사용하였다. 제안하는 방법을 검증하기 위한 실험에는 인공 영상 및 실제 영상들을 사용하여 제안하는 방법과 기존의 탈초점 거리 기반 깊이 맵 예측 방법들을 비교하였다. 그 결과, 제안하는 방법은 기존의 방법들보다 더 나은 결과를 보여주었다.The depth map is an absolute or relative expression of how far from a capturing device each region of an image is, and a popular representation of the 3D (three-dimensional) structure of an image. There are many depth cues for depth map estimation using only a 2D (two-dimensional) image, such as the defocus blur, the geometric structure of a scene, the saliency of an object, and motion parallax. Among them, the defocus blur is a popular and powerful depth cue, and as such, the DFD (depth from defocus) problem is important for depth estimation. This paper aims to estimate the depth map of a 2D image using defocus blur estimation. It assumes that the focus region of an image is nearest, and therefore, the blur radius of the defocus blur increases with the distance from the capturing device so that the distance can be estimated using the amount of defocus blur. In this paper, a new solution for the DFD problem is proposed. First, the perceptual depth, which is based on human depth perception, is defined, and then the (true) confidence values of defocus blur estimation are defined using the perceptual depth. Estimation methods of confidence values were designed for the gradient- and second-derivative-based focus measures. These estimated confidence values are more correct than those of the existing methods. The proposed focus depth map estimation method is based on the segment-wise planar model, and the total cost function consists of the data term and the smoothness term. The data term is the sum of the fitting error costs of each segment at the fitting process, and the confidence values are used as fitting weights. The smoothness term means the amount of decrease of total cost function by merging two adjacent segments. It consists of the boundary cost and the similarity term. To solve the cost optimization problem of the total cost function, iterative local optimization based on the greedy algorithm is used. In experiments to evaluate the proposed method and the existing DFD methods, the synthetic and real images are used for qualitative evaluation. Based on the results, the proposed method showed better performances than the existing approaches for depth map estimation.Chapter 1 Introduction 1 1.1 Focus Depth Map 1 1.1.1 Depth from Defocus Blur 2 1.1.2 Absolute Depth vs. Relative Depth 3 1.2 Focus Measure 4 1.3 Approaches of the Paper 5 Chapter 2 Blur Estimation Methods Using Focus Measures 6 2.1 Various Blur Estimation Methods 6 2.1.1 Gradient-based Methods 6 2.1.2 Laplacian-based Methods 8 2.1.3 Gaussian-filtering-based Methods 12 2.1.4 Focus Measure Based on Adaptive Derivative Filters 12 2.2 Comparison of the Blur Estimators 15 Chapter 3 Confidence Values of Focus Measures 21 3.1 True Confidence Value 21 3.1.1 Perceptual Depth by the Parallactic Angle 21 3.1.2 True Confidence Value Using the Perceptual Depth and Blur Radius 23 3.1.3 Examples of True Confidence Values 26 3.2 Confidence Value Estimation Methods for Various Focus Measures 27 3.2.1 Blur Estimator Based on the Gradient Focus Measure 27 3.2.2 Blur Estimator Based on the Second Derivative Focus Measure 29 Chapter 4 Focus Depth Map Estimation 31 4.1 Piecewise Planar Model 31 4.2 The Proposed Focus Depth Map Estimation Method 34 4.2.1 Cost Function 34 4.2.2 Depth Map Generation Algorithm 38 Chapter 5 Experimental Results 40 5.1 Comparison of the Confidences Value Estimation Methods of Focus Measures 40 5.2 Performances of the Proposed Depth Map Generation Method 70 5.2.1 Experiments on Synthetic Images 70 5.2.2 The Experiments on Real Images 73 5.2.3 Execution Time 81 Chapter 6 Conclusion 84 Bibliography 86 국문 초록 91Docto

Coded aperture and coded exposure photography : an investigation into applications and methods

This dissertation presents an introduction to the field of computational photography, and provides a survey of recent research. Specific attention is given to coded aperture and coded exposure theory and methods, as these form the basis for the experiments performed

Microcantilever biosensors

The cross-sensitivity of microcantilever sensors presents a major obstacle in the development of a commercially viable microcantilever biosensor for point of care testing. This thesis concerns electrothermally actuated bi-material microcantilevers with piezoresistive read out, developed for use as a blood coagulometer. Thermal properties of the sensor environment including the heat capacity and thermal conductivity affect the ‘thermal profile’ onto which the higher frequency mechanical signal is superimposed. In addition, polymer microcantilevers are known to have cross-sensitivity to relative humidity due to moisture absorption in the beam. However it is not known whether any of these cross sensitivities have a significant impact on performance of the sensor during pulsed mode operation or following immersion into liquid. When analysing patient blood samples, any change in signal that is not caused by the change in blood viscosity during clotting could lead to a false result and consequently an incorrect dose of anticoagulants may be taken by the patient. In order to address these issues three aspects of the operation of polymer bi-material strip cantilevers has been researched and investigated: relative humidity; viscosity/density, and thermal conductivity of a liquid environment. The relative humidity was not found to affect the resonant frequency of a microcantilever operated in air, or to affect the ability of the cantilever to measure clot times. However, a decrease in deflection with increasing relative humidity of the SmartStrip microcantilever beams is observed at 1.1 ± 0.4 μm per 1% RH, and is constant with temperature over the range 10 – 37 °C, which is an issue that should be considered in quality control. In this study, the SmartStrip was shown to have viscosity sensitivity of 2 cP within the range 0.7 – 15.2 cP, and it was also shown that the influence of inertial effects is negligible in comparison to the viscosity. To investigate cross-sensitivity to the thermal properties of the environment, the first demonstration of a cantilever designed specifically to observe the thermal background is presented. Characterisation experiments showed that the piezoresistive component of the signal was minimised to -0.8% ± 0.2% of the total signal by repositioning the read out tracks onto the neutral axis of the beam. Characterisations of the signal in a range of silicone oils with different thermal conductivities gave a resolution to thermal conductivity of 0.3 Wm-1K-1 and resulted in a suggestion for design improvements in the sensor: the time taken for the thermal background signal to reach a maximum can be increased by increasing the distance between the heater and sensor, thus lessening the impact of the thermal crosstalk within the cantilever beam. A preliminary investigation into thermal properties of clotting blood plasma showed that the sensor can distinguish the change between fresh and clotted plasma