Depth and IMU aided image deblurring based on deep learning

Abstract. With the wide usage and spread of camera phones, it becomes necessary to tackle the problem of the image blur. Embedding a camera in those small devices implies obviously small sensor size compared to sensors in professional cameras such as full-frame Digital Single-Lens Reflex (DSLR) cameras. As a result, this can dramatically affect the collected amount of photons on the image sensor. To overcome this, a long exposure time is needed, but with slight motions that often happen in handheld devices, experiencing image blur is inevitable. Our interest in this thesis is the motion blur that can be caused by the camera motion, scene (objects in the scene) motion, or generally the relative motion between the camera and scene. We use deep neural network (DNN) models in contrary to conventional (non DNN-based) methods which are computationally expensive and time-consuming. The process of deblurring an image is guided by utilizing the scene depth and camera’s inertial measurement unit (IMU) records. One of the challenges of adopting DNN solutions is that a relatively huge amount of data is needed to train the neural network. Moreover, several hyperparameters need to be tuned including the network architecture itself. To train our network, a novel and promising method of synthesizing spatially-variant motion blur is proposed that considers the depth variations in the scene, which showed improvement of results against other methods. In addition to the synthetic dataset generation algorithm, a real blurry and sharp dataset collection setup is designed. This setup can provide thousands of real blurry and sharp images which can be of paramount benefit in DNN training or fine-tuning

Towards High-Frequency Tracking and Fast Edge-Aware Optimization

This dissertation advances the state of the art for AR/VR tracking systems by increasing the tracking frequency by orders of magnitude and proposes an efficient algorithm for the problem of edge-aware optimization. AR/VR is a natural way of interacting with computers, where the physical and digital worlds coexist. We are on the cusp of a radical change in how humans perform and interact with computing. Humans are sensitive to small misalignments between the real and the virtual world, and tracking at kilo-Hertz frequencies becomes essential. Current vision-based systems fall short, as their tracking frequency is implicitly limited by the frame-rate of the camera. This thesis presents a prototype system which can track at orders of magnitude higher than the state-of-the-art methods using multiple commodity cameras. The proposed system exploits characteristics of the camera traditionally considered as flaws, namely rolling shutter and radial distortion. The experimental evaluation shows the effectiveness of the method for various degrees of motion. Furthermore, edge-aware optimization is an indispensable tool in the computer vision arsenal for accurate filtering of depth-data and image-based rendering, which is increasingly being used for content creation and geometry processing for AR/VR. As applications increasingly demand higher resolution and speed, there exists a need to develop methods that scale accordingly. This dissertation proposes such an edge-aware optimization framework which is efficient, accurate, and algorithmically scales well, all of which are much desirable traits not found jointly in the state of the art. The experiments show the effectiveness of the framework in a multitude of computer vision tasks such as computational photography and stereo.Comment: PhD thesi

UAV Based Agricultural Planning and Landslide Monitoring

Unmanned Aerial Vehicle (UAV) is finding a wide application field in areas such as map production, land survey, landslide, erosion, agricultural activities, and forest fires monitoring. In this study, an UAV equipped with SONY 6000 camera was used. The flight plan was prepared from 100 m height, and having 80% overlap and 60% sidelap rates. GNSS geodetic receivers and Ground Control Points (GCPs) were observed. GNSS signals were processed with LGO V.8.4 software to receive precise location information. 291 photographs for 50 hectares of landslide area were taken by UAV. All photos were processed by PIX4D software. In the field of the landslide area, 8 GCPs were included in the evaluation. 3D model were produced with pixel matching algorithms. Six period flights in different months were made for the landslide area and ground movements between the periods were observed. During this time interval , the volume of moving soil was determined. At the end of the study, RMSE for soil movement was obtained ±1.79 cm for landslide area. This study demonstrates that UAV-based high resolution orthophoto, 3D terrain model and point cloud data sets can be used to monitor the landslide, especially in micro small areas. It also was revealed that this method has some advantages over other traditional geomatics methods

Remote sensing of subglacial bedforms from the British Ice Sheet using an Unmanned Aerial System (UAS): Problems and Potential.

Photogrammetry can be applied to the results of UAS (Unmanned Aerial Systems) based photographic surveys to produce high resolution DEMs (Digital Elevation Models) of small areas (c. 1 km2). However, this method has not been widely used in academia due to photogrammetric programmes working poorly with the ill constrained intrinsic and extrinsic properties that often accompany UAS based photographs. In this study a PAMS (Personal Aerial Mapping System) SmartOne B UAS was used to provide image sets for testing a number of different photogrammetry packages; LPS, Bundler, PhotoSynth and PhotoScan, with the aim of producing sub-metric accuracy DEMs with a low complexity methodology and without significant financial investment. To demonstrate the potential use of a UAS photogrammetric survey methodology it was applied here to an investigation into scale dependant remote sensing of glacial geomorphology. Subglacial bedforms, landforms produced by the flow of ice over land, are thought to ‘seed’ with a minimum horizontal dimension of 100 m. This hypothesis is based on surveys of bedforms across the UK and Ireland using NEXTMap DEMs with 1 m accuracy and 5 m resolution. Here we test that hypothesis using sub-metric accuracy DEMs produced via photogrammetry of an area in the Eden Valley drumlin field, NW England. The UAS was found to be suitable for this type of survey, but only one of the four photogrammetry programmes provided an effective and low complexity methodology. This programme, PhotoScan, was shown to require minimal user training and could produce DEMs from the survey imagery on the day of flying with a standard high performance computer at a resolution of 0.12 m2. The DEM produced was down sampled and validated against pre-existing 1 m LiDAR (Light Detection And Ranging) data of the same area. It showed poor absolute accuracy due to a systematic parabolic error introduced during processing that made quantification of the DEM error problematic. However, estimates of the error additional to this systematic error put it at around 0.5 m which makes the DEM suitable for mapping low amplitude bedforms. Use of the DEM for mapping subglacial bedforms yielded ambiguous results. 17 additional linear ridges were identified that were not visible on the NEXTMap DEM. Their dimensions were not remarkably shorter than the 100 m limit, with only 6 measuring <100 m, but their width was much narrower than those mapped previously. However, whilst these dimensions could suggest that bedforms do not ‘seed’ at a certain size and may fine into smaller features such as flutes, there was no way to demonstrate that they were in fact glacial in origin. This highlighted that whilst sub-metric resolution DEMs are undoubtedly highly useful tools in the survey of glacial bedforms, they may require additional data from field investigations in order for robust conclusions to be drawn due to the numerous processes capable of produce geomorphic features at a sub-metric vertical scale

Contribution to ground-based and UAV SAR systems for Earth observation

Mankind's way of life is the main driver of a planetary-scale change that is marked by the growing of human population's demand of energy, food, goods, services and information. As a result, it have emerged new ecological, economical, social and geopolitical concerns. In this scenario, SAR remote sensing is a potential tool that provides unique information about the Earth's properties and processes that can be used to solve societal challenges of local and global dimension. SARs, which are coherent systems that are able to provide high resolution images with weather independence, represent a suitable alternative for EO with diverse applications. Some examples of SAR application areas are topography (DEM generation with interferometry), agriculture (crop classification or soil moisture), or geology (monitoring surface deformation). In this framework, the encompassing objective of the present doctoral work has been part of the implementation and the subsequent evaluation of capabilities of two X-band SAR sensors. On the one hand, the RISKSAR-X radar designed to be operated at ground to monitor small-scale areas of observation and, on the other, the ARBRES-X sensor designed to be integrated into small UAVs. Despite its inherently dissimilar conception, the concurrence of both sensors has been evidenced along this manuscript. By taking advantage of the similarities between them, it has been possible to analogously assess both sensors to obtain conclusions. In this context, the common link has been the development of the polarimetric OtF operation mode of the RISKSAR-X, allowing this sensor to be operated equivalently to the ARBRES-X. Regarding the RISKSAR-X SAR sensor, several hardware contributions have been developed during part of this Ph.D. with the aim of improving the system performance. By endowing the system with the capability to operate in the fully polarimetric OtF acquisition mode, the relative long scanning time has been reduced. It is of great interest since the measured scatterers that present a short term variable reflectivity during the scanning time, such as moving vegetation, may degrade the extracted parameters from the retrieved data and the SAR image reconstruction. During this doctoral activity, it has been studied the image blurring, the decorrelation and the coherence degradation introduced by this effect. Furthermore, a new term in the differential interferometric coherence that takes into account the image blurring has been introduced. Concerning the ARBRES-X SAR system, one of the main objectives pursued during this Ph.D. has been the integration of the sensor into a small UAV MP overcoming restrictions of weight, size and aerodynamics of the platform. The use of this type of platforms is expected to open up new possibilities in airborne SAR remote sensing, since it offers much more versatility than the commonly used fixed wings UAVs. Different innovative flight strategies with this type of platforms have been assessed and some preliminary results have been obtained with the use of the ARBRES-X SAR system. During the course of the present doctoral work, much effort has been devoted to achieve the first experimental repeat-pass interfereometric results obtained with the UAV MP together with the ARBRES-X. Moreover, the sensor has been endowed with fully polarimetric capabilities by applying the improvements developed to the RISKSAR-X radar, which is another example of the duality between both systems. Finally, a vertical and a semicircular aperture have been successfully performed obtaining SLC images of the scenario, which envisages the capability of the UAV MP to perform tomographic images and complete circular apertures in the future. In conclusion, the UAV MP is a promising platform that opens new potentials for several applications, such as repeat-pass interferometry or differential tomography imaging with the realization of almost arbitrary trajectories.El mode de viure de la humanitat és el principal motor d'un canvi a escala planetària que està marcat per la creixent demanda d'energia, d'aliment, de béns, de serveis i d'informació de les poblacions humanes. Com a resultat, han sorgit noves inquietuds ecològiques, econòmiques, socials i geopolítiques. En aquest escenari, la detecció remota SAR és una eina potencial que proporciona informació única sobre les propietats i processos de la Terra que es pot utilitzar per resoldre reptes socials de dimensió local i global. Els SARs, que són sistemes coherents que poden proporcionar imatges d'alta resolució amb independència del temps, representen una alternativa adequada per a l'observació de la Terra. Alguns exemples d'àrees d'aplicació SAR són la topografia (generació de DEM amb interferometria), l'agricultura (classificació de cultius o humitat del sòl) o la geologia (monitoratge de deformació superficial). En aquest context, l'objectiu general del present doctorat ha estat part de la implementació i posterior avaluació de les capacitats de dos sensors SAR de banda X. D'una banda, el radar RISKSAR-X dissenyat per funcionar a terra i monitoritzar àrees d'observació a petita escala i, d'altra, el sensor ARBRES-X dissenyat per ser integrat en petits UAVs. Malgrat la seva concepció inherentment diferent, la concurrència d'ambdós sensors s'ha evidenciat al llarg d'aquest manuscrit. Aprofitant les similituds entre ells, s'han pogut avaluar de forma anàloga els dos sensors per obtenir conclusions. En aquest sentit, el vincle comú ha estat el desenvolupament del mode de funcionament polimètric OtF del RISKSAR-X, permetent que aquest sensor operi de forma equivalent a l'ARBRES-X. Pel que fa al sensor RISKSAR-X, s'han desenvolupat diverses contribucions hardware durant part d'aquest doctorat amb l'objectiu de millorar el rendiment del sistema. En dotar el sistema de la possibilitat d'operar en el mode d'adquisició totalment polarimètric OtF, s'ha reduït el relatiu llarg temps d'escaneig. Això és de gran interès ja que els blancs mesurats que presenten una reflectivitat variable a curt termini, com ara la vegetació en moviment, poden degradar els paràmetres extrets de les dades recuperades i la reconstrucció d'imatges SAR. Durant aquesta activitat doctoral s'ha estudiat el desenfocat de la imatge, la decorrelació i la degradació de la coherència introduïts per aquest efecte. A més, s'ha introduït un nou terme en la coherència interferomètrica diferencial que té en compte el desenfocat de la imatge. Pel que fa al sistema ARBRES-X, un dels principals objectius perseguits durant aquest doctorat ha estat la integració del sensor en un petit UAV MP superant les restriccions de pes, grandària i aerodinàmica de la plataforma. S'espera que l'ús d'aquest tipus de plataformes obri noves possibilitats en la detecció remota SAR aerotransportada, ja que ofereix molta més versatilitat que els UAV d'ales fixes habituals. S'han avaluat diferents estratègies de vol innovadores amb aquest tipus de plataformes i s'han obtingut resultats preliminars amb l'ús del sistema ARBRES-X. Durant el transcurs del present treball, s'ha dedicat molt esforç a assolir els primers resultats experimentals d'interferometria de múltiple passada obtinguts amb l'UAV MP conjuntament amb l'ARBRES-X. A més, el sensor ha estat dotat de capacitats totalment polarimètriques aplicant les millores desenvolupades al radar RISKSAR-X, el qual constitueix un altre exemple de la dualitat entre ambdós sistemes. Finalment, s'han realitzat amb èxit una apertura vertical i semicircular obtenint imatges SLC de l'escenari, el qual permet preveure la capacitat de l'UAV MP per a realitzar imatges tomogràfiques i apertures circulars completes en el futur. En conclusió, l'UAV MP és una plataforma prometedora que obre nous potencials per a diverses aplicacions, com ara la interferometria de múltiple passada o la tomografia diferencial amb la realització de trajectòries gairebé arbitràries

Contribution to ground-based and UAV SAR systems for Earth observation

Mankind's way of life is the main driver of a planetary-scale change that is marked by the growing of human population's demand of energy, food, goods, services and information. As a result, it have emerged new ecological, economical, social and geopolitical concerns. In this scenario, SAR remote sensing is a potential tool that provides unique information about the Earth's properties and processes that can be used to solve societal challenges of local and global dimension. SARs, which are coherent systems that are able to provide high resolution images with weather independence, represent a suitable alternative for EO with diverse applications. Some examples of SAR application areas are topography (DEM generation with interferometry), agriculture (crop classification or soil moisture), or geology (monitoring surface deformation). In this framework, the encompassing objective of the present doctoral work has been part of the implementation and the subsequent evaluation of capabilities of two X-band SAR sensors. On the one hand, the RISKSAR-X radar designed to be operated at ground to monitor small-scale areas of observation and, on the other, the ARBRES-X sensor designed to be integrated into small UAVs. Despite its inherently dissimilar conception, the concurrence of both sensors has been evidenced along this manuscript. By taking advantage of the similarities between them, it has been possible to analogously assess both sensors to obtain conclusions. In this context, the common link has been the development of the polarimetric OtF operation mode of the RISKSAR-X, allowing this sensor to be operated equivalently to the ARBRES-X. Regarding the RISKSAR-X SAR sensor, several hardware contributions have been developed during part of this Ph.D. with the aim of improving the system performance. By endowing the system with the capability to operate in the fully polarimetric OtF acquisition mode, the relative long scanning time has been reduced. It is of great interest since the measured scatterers that present a short term variable reflectivity during the scanning time, such as moving vegetation, may degrade the extracted parameters from the retrieved data and the SAR image reconstruction. During this doctoral activity, it has been studied the image blurring, the decorrelation and the coherence degradation introduced by this effect. Furthermore, a new term in the differential interferometric coherence that takes into account the image blurring has been introduced. Concerning the ARBRES-X SAR system, one of the main objectives pursued during this Ph.D. has been the integration of the sensor into a small UAV MP overcoming restrictions of weight, size and aerodynamics of the platform. The use of this type of platforms is expected to open up new possibilities in airborne SAR remote sensing, since it offers much more versatility than the commonly used fixed wings UAVs. Different innovative flight strategies with this type of platforms have been assessed and some preliminary results have been obtained with the use of the ARBRES-X SAR system. During the course of the present doctoral work, much effort has been devoted to achieve the first experimental repeat-pass interfereometric results obtained with the UAV MP together with the ARBRES-X. Moreover, the sensor has been endowed with fully polarimetric capabilities by applying the improvements developed to the RISKSAR-X radar, which is another example of the duality between both systems. Finally, a vertical and a semicircular aperture have been successfully performed obtaining SLC images of the scenario, which envisages the capability of the UAV MP to perform tomographic images and complete circular apertures in the future. In conclusion, the UAV MP is a promising platform that opens new potentials for several applications, such as repeat-pass interferometry or differential tomography imaging with the realization of almost arbitrary trajectories.El mode de viure de la humanitat és el principal motor d'un canvi a escala planetària que està marcat per la creixent demanda d'energia, d'aliment, de béns, de serveis i d'informació de les poblacions humanes. Com a resultat, han sorgit noves inquietuds ecològiques, econòmiques, socials i geopolítiques. En aquest escenari, la detecció remota SAR és una eina potencial que proporciona informació única sobre les propietats i processos de la Terra que es pot utilitzar per resoldre reptes socials de dimensió local i global. Els SARs, que són sistemes coherents que poden proporcionar imatges d'alta resolució amb independència del temps, representen una alternativa adequada per a l'observació de la Terra. Alguns exemples d'àrees d'aplicació SAR són la topografia (generació de DEM amb interferometria), l'agricultura (classificació de cultius o humitat del sòl) o la geologia (monitoratge de deformació superficial). En aquest context, l'objectiu general del present doctorat ha estat part de la implementació i posterior avaluació de les capacitats de dos sensors SAR de banda X. D'una banda, el radar RISKSAR-X dissenyat per funcionar a terra i monitoritzar àrees d'observació a petita escala i, d'altra, el sensor ARBRES-X dissenyat per ser integrat en petits UAVs. Malgrat la seva concepció inherentment diferent, la concurrència d'ambdós sensors s'ha evidenciat al llarg d'aquest manuscrit. Aprofitant les similituds entre ells, s'han pogut avaluar de forma anàloga els dos sensors per obtenir conclusions. En aquest sentit, el vincle comú ha estat el desenvolupament del mode de funcionament polimètric OtF del RISKSAR-X, permetent que aquest sensor operi de forma equivalent a l'ARBRES-X. Pel que fa al sensor RISKSAR-X, s'han desenvolupat diverses contribucions hardware durant part d'aquest doctorat amb l'objectiu de millorar el rendiment del sistema. En dotar el sistema de la possibilitat d'operar en el mode d'adquisició totalment polarimètric OtF, s'ha reduït el relatiu llarg temps d'escaneig. Això és de gran interès ja que els blancs mesurats que presenten una reflectivitat variable a curt termini, com ara la vegetació en moviment, poden degradar els paràmetres extrets de les dades recuperades i la reconstrucció d'imatges SAR. Durant aquesta activitat doctoral s'ha estudiat el desenfocat de la imatge, la decorrelació i la degradació de la coherència introduïts per aquest efecte. A més, s'ha introduït un nou terme en la coherència interferomètrica diferencial que té en compte el desenfocat de la imatge. Pel que fa al sistema ARBRES-X, un dels principals objectius perseguits durant aquest doctorat ha estat la integració del sensor en un petit UAV MP superant les restriccions de pes, grandària i aerodinàmica de la plataforma. S'espera que l'ús d'aquest tipus de plataformes obri noves possibilitats en la detecció remota SAR aerotransportada, ja que ofereix molta més versatilitat que els UAV d'ales fixes habituals. S'han avaluat diferents estratègies de vol innovadores amb aquest tipus de plataformes i s'han obtingut resultats preliminars amb l'ús del sistema ARBRES-X. Durant el transcurs del present treball, s'ha dedicat molt esforç a assolir els primers resultats experimentals d'interferometria de múltiple passada obtinguts amb l'UAV MP conjuntament amb l'ARBRES-X. A més, el sensor ha estat dotat de capacitats totalment polarimètriques aplicant les millores desenvolupades al radar RISKSAR-X, el qual constitueix un altre exemple de la dualitat entre ambdós sistemes. Finalment, s'han realitzat amb èxit una apertura vertical i semicircular obtenint imatges SLC de l'escenari, el qual permet preveure la capacitat de l'UAV MP per a realitzar imatges tomogràfiques i apertures circulars completes en el futur. En conclusió, l'UAV MP és una plataforma prometedora que obre nous potencials per a diverses aplicacions, com ara la interferometria de múltiple passada o la tomografia diferencial amb la realització de trajectòries gairebé arbitràries.Postprint (published version

Motion blur in digital images - analys, detection and correction of motion blur in photogrammetry

Unmanned aerial vehicles (UAV) have become an interesting and active research topic for photogrammetry. Current research is based on images acquired by an UAV, which have a high ground resolution and good spectral and radiometrical resolution, due to the low flight altitudes combined with a high resolution camera. UAV image flights are also cost effective and have become attractive for many applications including, change detection in small scale areas. One of the main problems preventing full automation of data processing of UAV imagery is the degradation effect of blur caused by camera movement during image acquisition. This can be caused by the normal flight movement of the UAV as well as strong winds, turbulence or sudden operator inputs. This blur disturbs the visual analysis and interpretation of the data, causes errors and can degrade the accuracy in automatic photogrammetric processing algorithms. The detection and removal of these images is currently achieved manually, which is both time consuming and prone to error, particularly for large image-sets. To increase the quality of data processing an automated process is necessary, which must be both reliable and quick. This thesis proves the negative affect that blurred images have on photogrammetric processing. It shows that small amounts of blur do have serious impacts on target detection and that it slows down processing speed due to the requirement of human intervention. Larger blur can make an image completely unusable and needs to be excluded from processing. To exclude images out of large image datasets an algorithm was developed. The newly developed method makes it possible to detect blur caused by linear camera displacement. The method is based on human detection of blur. Humans detect blurred images best by comparing it to other images in order to establish whether an image is blurred or not. The developed algorithm simulates this procedure by creating an image for comparison using image processing. Creating internally a comparable image makes the method independent of additional images. However, the calculated blur value named SIEDS (saturation image edge difference standard-deviation) on its own does not provide an absolute number to judge if an image is blurred or not. To achieve a reliable judgement of image sharpness the SIEDS value has to be compared to other SIEDS values of the same dataset. This algorithm enables the exclusion of blurred images and subsequently allows photogrammetric processing without them. However, it is also possible to use deblurring techniques to restor blurred images. Deblurring of images is a widely researched topic and often based on the Wiener or Richardson-Lucy deconvolution, which require precise knowledge of both the blur path and extent. Even with knowledge about the blur kernel, the correction causes errors such as ringing, and the deblurred image appears muddy and not completely sharp. In the study reported in this paper, overlapping images are used to support the deblurring process. An algorithm based on the Fourier transformation is presented. This works well in flat areas, but the need for geometrically correct sharp images for deblurring may limit the application. Another method to enhance the image is the unsharp mask method, which improves images significantly and makes photogrammetric processing more successful. However, deblurring of images needs to focus on geometric correct deblurring to assure geometric correct measurements. Furthermore, a novel edge shifting approach was developed which aims to do geometrically correct deblurring. The idea of edge shifting appears to be promising but requires more advanced programming