Motion blur in digital images - analys, detection and correction of motion blur in photogrammetry

Unmanned aerial vehicles (UAV) have become an interesting and active research topic for photogrammetry. Current research is based on images acquired by an UAV, which have a high ground resolution and good spectral and radiometrical resolution, due to the low flight altitudes combined with a high resolution camera. UAV image flights are also cost effective and have become attractive for many applications including, change detection in small scale areas. One of the main problems preventing full automation of data processing of UAV imagery is the degradation effect of blur caused by camera movement during image acquisition. This can be caused by the normal flight movement of the UAV as well as strong winds, turbulence or sudden operator inputs. This blur disturbs the visual analysis and interpretation of the data, causes errors and can degrade the accuracy in automatic photogrammetric processing algorithms. The detection and removal of these images is currently achieved manually, which is both time consuming and prone to error, particularly for large image-sets. To increase the quality of data processing an automated process is necessary, which must be both reliable and quick. This thesis proves the negative affect that blurred images have on photogrammetric processing. It shows that small amounts of blur do have serious impacts on target detection and that it slows down processing speed due to the requirement of human intervention. Larger blur can make an image completely unusable and needs to be excluded from processing. To exclude images out of large image datasets an algorithm was developed. The newly developed method makes it possible to detect blur caused by linear camera displacement. The method is based on human detection of blur. Humans detect blurred images best by comparing it to other images in order to establish whether an image is blurred or not. The developed algorithm simulates this procedure by creating an image for comparison using image processing. Creating internally a comparable image makes the method independent of additional images. However, the calculated blur value named SIEDS (saturation image edge difference standard-deviation) on its own does not provide an absolute number to judge if an image is blurred or not. To achieve a reliable judgement of image sharpness the SIEDS value has to be compared to other SIEDS values of the same dataset. This algorithm enables the exclusion of blurred images and subsequently allows photogrammetric processing without them. However, it is also possible to use deblurring techniques to restor blurred images. Deblurring of images is a widely researched topic and often based on the Wiener or Richardson-Lucy deconvolution, which require precise knowledge of both the blur path and extent. Even with knowledge about the blur kernel, the correction causes errors such as ringing, and the deblurred image appears muddy and not completely sharp. In the study reported in this paper, overlapping images are used to support the deblurring process. An algorithm based on the Fourier transformation is presented. This works well in flat areas, but the need for geometrically correct sharp images for deblurring may limit the application. Another method to enhance the image is the unsharp mask method, which improves images significantly and makes photogrammetric processing more successful. However, deblurring of images needs to focus on geometric correct deblurring to assure geometric correct measurements. Furthermore, a novel edge shifting approach was developed which aims to do geometrically correct deblurring. The idea of edge shifting appears to be promising but requires more advanced programming

Towards High-Frequency Tracking and Fast Edge-Aware Optimization

This dissertation advances the state of the art for AR/VR tracking systems by increasing the tracking frequency by orders of magnitude and proposes an efficient algorithm for the problem of edge-aware optimization. AR/VR is a natural way of interacting with computers, where the physical and digital worlds coexist. We are on the cusp of a radical change in how humans perform and interact with computing. Humans are sensitive to small misalignments between the real and the virtual world, and tracking at kilo-Hertz frequencies becomes essential. Current vision-based systems fall short, as their tracking frequency is implicitly limited by the frame-rate of the camera. This thesis presents a prototype system which can track at orders of magnitude higher than the state-of-the-art methods using multiple commodity cameras. The proposed system exploits characteristics of the camera traditionally considered as flaws, namely rolling shutter and radial distortion. The experimental evaluation shows the effectiveness of the method for various degrees of motion. Furthermore, edge-aware optimization is an indispensable tool in the computer vision arsenal for accurate filtering of depth-data and image-based rendering, which is increasingly being used for content creation and geometry processing for AR/VR. As applications increasingly demand higher resolution and speed, there exists a need to develop methods that scale accordingly. This dissertation proposes such an edge-aware optimization framework which is efficient, accurate, and algorithmically scales well, all of which are much desirable traits not found jointly in the state of the art. The experiments show the effectiveness of the framework in a multitude of computer vision tasks such as computational photography and stereo.Comment: PhD thesi

Machine Learning Approaches to Image Deconvolution

Image blur is a fundamental problem in both photography and scientific imaging. Even the most well-engineered optics are imperfect, and finite exposure times cause motion blur. To reconstruct the original sharp image, the field of image deconvolution tries to recover recorded photographs algorithmically. When the blur is known, this problem is called non-blind deconvolution. When the blur is unknown and has to be inferred from the observed image, it is called blind deconvolution. The key to reconstructing information lost due to blur and noise is to use prior knowledge. To this end, this thesis develops approaches inspired by machine learning that include more available information and advance the current state of the art for both non-blind and blind image deconvolution. Optical aberrations of a lens are encoded in an initial calibration step as a spatially-varying point spread function. With prior information about the distribution of gradients in natural images, the original image is reconstructed in a maximum a posteriori (MAP) estimation, with results comparing favorably to previous methods. By including the camera’s color filter array in the forward model, the estimation procedure can perform demosaicing and deconvolution jointly and thereby surpass the quality of the results yielded by a separate demosaicing step. The applicability of removing optical aberrations is broadened further by estimating the point spread function from the image itself. We extend an existing MAP-based blind deconvolution approach to the first algorithm that is able to remove spatially-varying lens blur blindly, including chromatic aberrations. The properties of lenses restrict the class of possible point spread functions and reduce the space of parameters to be inferred, enabling results on par with the best non-blind approaches for the lenses tested in our experiments. To capture more information about the distribution of natural images and capitalize on the abundance of training data, neural networks prove to be a useful tool. As other successful non-blind deconvolution methods, a regularized inversion of the blur is performed in the Fourier domain as an initial step. Next, a large neural network learns the mapping from the preprocessed image back to the uncorrupted original. The trained network surpasses results of state-of-the-art algorithms on both artificial and real-world examples. For the first time, a learning approach also succeeds in blind image deconvolution. A deep neural network “unrolls” the estimation procedure of existing methods for this task. After training end-to-end on artificially generated example images, the network achieves performance competitive with state-of-the-art methods in the generic case, and even goes beyond when trained for a specific image category.Unscharfe Bilder sind ein häufiges Problem, sowohl in der Fotografie als auch in der wissenschaftlichen Bildgebung. Auch die leistungsfähigsten optischen Systeme sind nicht perfekt, und endliche Belichtungszeiten verursachen Bewegungsunschärfe. Dekonvolution hat das Ziel das ursprünglich scharfe Bild aus der Aufnahme mit Hilfe von algorithmischen Verfahren wiederherzustellen. Kennt man die exakte Form der Unschärfe, so wird dieses Rekonstruktions-Problem als nicht-blinde Dekonvolution bezeichnet. Wenn die Unschärfe aus dem Bild selbst inferiert werden muss, so spricht man von blinder Dekonvolution. Der Schlüssel zum Wiederherstellen von verlorengegangener Bildinformation liegt im Verwenden von verfügbarem Vorwissen über Bilder und die Entstehung der Unschärfe. Hierzu entwickelt diese Arbeit verschiedene Ansätze um dieses Vorwissen besser verwenden zu können, basierend auf Methoden des maschinellen Lernens, und verbessert damit den Stand der Technik, sowohl für nicht-blinde als auch für blinde Dekonvolution. Optische Abbildungsfehler lassen sich in einem einmal ausgeführten Kalibrierungsschritt vermessen und als eine ortsabhängige Punktverteilungsfunktion des einfallenden Lichtes beschreiben. Mit dem Vorwissen über die Verteilung von Gradienten in Bildern kann das ursprüngliche Bild durch eine Maximum-a-posteriori (MAP) Schätzung wiederhergestellt werden, wobei die resultierenden Ergebnisse vergleichbare Methoden übertreffen. Wenn man des Weiteren im Vorwärtsmodell die Farbfilter des Sensors berücksichtigt, so kann das Schätzverfahren Demosaicking und Dekonvolution simultan ausführen, in einer Qualität die den Ergebnissen durch Demosaicking in einem separaten Schritt überlegen ist. Die Korrektur von Linsenfehlern wird breiter anwendbar indem man die Punktverteilungsfunktion vom Bild selbst inferiert. Wir erweitern einen existierenden MAP-basierenden Ansatz für blinde Dekonvolution zum ersten Algorithmus, der in der Lage ist auch ortsabhängige optische Unschärfen blind zu entfernen, einschließlich chromatischer Aberration. Die spezifischen Eigenschaften von Kamera-Objektiven schränken den Raum der zu schätzenden Punktverteilungsfunktionen weit genug ein, so dass für die in unseren Experimenten untersuchten Objektive die erreichte Bildrekonstruktion ähnlich erfolgreich ist wie bei nicht-blinden Verfahren. Es zeigt sich, dass neuronale Netze von im Überfluss vorhandenen Bilddatenbanken profitieren können um mehr über die Bildern zugrundeliegende Wahrscheinlichkeitsverteilung zu lernen. Ähnlich wie in anderen erfolgreichen nicht-blinden Dekonvolutions-Ansätzen wird die Unschärfe zuerst durch eine regularisierte Inversion im Fourier-Raum vermindert. Danach ist es einem neuronalen Netz mit großer Kapazität möglich zu lernen, wie aus einem derart vorverarbeiteten Bild das fehlerfreie Original geschätzt werden kann. Das trainierte Netz produziert anderen Methoden überlegene Ergebnisse, sowohl auf künstlich generierten Beispielen als auch auf tatsächlichen unscharfen Fotos. Zum ersten Mal ist ein lernendes Verfahren auch hinsichtlich der blinden Bild-Dekonvolution erfolgreich. Ein tiefes neuronales Netz modelliert die Herangehensweise von bisherigen Schätzverfahren und wird auf künstlich generierten Beispielen trainiert die Unschärfe vorherzusagen. Nach Abschluss des Trainings ist es in der Lage, mit anderen aktuellen Methoden vergleichbare Ergebnisse zu erzielen, und geht über deren Ergebnisse hinaus, wenn man speziell für eine bestimmten Subtyp von Bildern trainiert

Foundations, Inference, and Deconvolution in Image Restoration

Image restoration is a critical preprocessing step in computer vision, producing images with reduced noise, blur, and pixel defects. This enables precise higher-level reasoning as to the scene content in later stages of the vision pipeline (e.g., object segmentation, detection, recognition, and tracking). Restoration techniques have found extensive usage in a broad range of applications from industry, medicine, astronomy, biology, and photography. The recovery of high-grade results requires models of the image degradation process, giving rise to a class of often heavily underconstrained, inverse problems. A further challenge specific to the problem of blur removal is noise amplification, which may cause strong distortion by ringing artifacts. This dissertation presents new insights and problem solving procedures for three areas of image restoration, namely (1) model foundations, (2) Bayesian inference for high-order Markov random fields (MRFs), and (3) blind image deblurring (deconvolution). As basic research on model foundations, we contribute to reconciling the perceived differences between probabilistic MRFs on the one hand, and deterministic variational models on the other. To do so, we restrict the variational functional to locally supported finite elements (FE) and integrate over the domain. This yields a sum of terms depending locally on FE basis coefficients, and by identifying the latter with pixels, the terms resolve to MRF potential functions. In contrast with previous literature, we place special emphasis on robust regularizers used commonly in contemporary computer vision. Moreover, we draw samples from the derived models to further demonstrate the probabilistic connection. Another focal issue is a class of high-order Field of Experts MRFs which are learned generatively from natural image data and yield best quantitative results under Bayesian estimation. This involves minimizing an integral expression, which has no closed form solution in general. However, the MRF class under study has Gaussian mixture potentials, permitting expansion by indicator variables as a technical measure. As approximate inference method, we study Gibbs sampling in the context of non-blind deblurring and obtain excellent results, yet at the cost of high computing effort. In reaction to this, we turn to the mean field algorithm, and show that it scales quadratically in the clique size for a standard restoration setting with linear degradation model. An empirical study of mean field over several restoration scenarios confirms advantageous properties with regard to both image quality and computational runtime. This dissertation further examines the problem of blind deconvolution, beginning with localized blur from fast moving objects in the scene, or from camera defocus. Forgoing dedicated hardware or user labels, we rely only on the image as input and introduce a latent variable model to explain the non-uniform blur. The inference procedure estimates freely varying kernels and we demonstrate its generality by extensive experiments. We further present a discriminative method for blind removal of camera shake. In particular, we interleave discriminative non-blind deconvolution steps with kernel estimation and leverage the error cancellation effects of the Regression Tree Field model to attain a deblurring process with tightly linked sequential stages

Human gait modelling with step estimation and phase classification utilising a single thigh mounted IMU for vision impaired indoor navigation

This research is focused on human gait modelling for infrastructure free inertial navigation for vision impaired. A pedometer based on a single thigh mounted gyroscope, an efficient algorithm to estimate thigh flexion and extension, gait models for level walking, a model to estimate step length and a technique to detect gait phases based on a single thigh mounted Inertial Measurement Unit (IMU) were developed and confirmed higher accuracies