Resolving Measurement Errors Inherent with Time-of-Flight Range Imaging Cameras

Range imaging cameras measure the distance to objects in the field-of-view (FoV) of the camera, these cameras enable new machine vision applications in robotics, manufacturing, and human computer interaction. Time-of-flight (ToF) range cameras operate by illuminating the scene with amplitude modulated continuous wave (AMCW) light and measuring the phase difference between the emitted and reflected modulation envelope. Currently ToF range cameras suffer from measurement errors that are highly scene dependent, and these errors limit the accuracy of the depth measurement. The major cause of measurement errors is multiple propagation paths from the light source to pixel, known as multi path interference. Multi-path interference typically arises from: inter reflections, lens flare, subsurface scattering, volumetric scattering, and translucent objects. This thesis contributes three novel methods for resolving multi-path interference: coding in time, coding in frequency, and coding in space. Time coding is implemented by replacing the single frequency amplitude modulation with a binary sequence. Fundamental to ToF range cameras is the cross-correlation between the reflected light and a reference signal. The measured cross-correlation depends on the selection of the binary sequence. With selection of an appropriate binary sequence and using sparse deconvolution on the measured cross-correlation the multiple return path lengths and their amplitudes can be recovered. However, the minimal resolvable path length is dependent on the highest frequency in the binary sequence. Frequency coding is implemented by taking multiple measurements at different modulation frequencies. A subset of frequency coding is operating the camera in a mode analogous to stepped frequency continuous wave (SFCW). Frequency coding uses techniques from radar to resolve multiple propagation paths. The minimal resolvable path length is dependent on the camera's modulation bandwidth and the spectrum estimation technique used to recover distance, and it is shown that SFCW can be used to measure depth of objects behind a translucent sheet, while AMCW measurements can not. Path lengths below quarter a wavelength of the highest modulation frequency are difficult to resolve. The use of spatial coding is used to resolve diffuse multi-path interference. The original technique comes from direct and global separation in computer graphics, and it is modified to operate on the complex data produced by a ToF range camera. By illuminating the scene with a pattern the illuminated areas contain the direct return and the scattering (global return). The non-illuminated regions contain the scattering return, assuming the global component is spatially smooth. The direct and global separation with sinusoidal patterns is combining with the sinusoidal modulation signal of ToF range cameras for a closed form solution to multi-path interference in nine frames. With nine raw frames it is possible to implement direct and global separation at video frame rates. The RMSE of a corner is reduced from 0.0952 m to 0.0112 m. Direct and global separation correctly measures the depth of a diffuse corner, and resolves subsurface scattering however fails to resolve specular reflections. Finally the direct and global separation is combined with replacing the illumination and reference signals with a binary sequence. The combination allows for resolving diffuse multi-path interference present in a corner, with the sparse multi-path interference caused mixed pixels between the foreground and background. The corner is correctly measured and the number of mixed pixels is reduced by 90%. With the development of new methods to resolve multi-path interference ToF range cameras can measure scenes with more confidence. ToF range cameras can be built into small form factors as they require a small number of parts: a pixel array, a light source and a lens. The small form factor coupled with accurate range measurements allows ToF range cameras to be embedded in cellphones and consumer electronic devices, enabling wider adoption and advantages over competing range imaging technologies

An Imaging Fourier Transform Spectrometer (IFTS) for Climate Observations

Climate change is an ongoing global phenomenon having a greater impact at higher latitudes. The instrument development reported herein is aimed at demonstrating the feasibility of using an Imaging Fourier Transform Spectrometer (IFTS) to measure carbon dioxide (CO2), and methane (CH4) mixing ratios at high latitudes using oxygen A-band measurements as a surface pressure reference. This thesis details the optical design, instrumental setup, and development criteria for the IFTS. The development of a software package to control and acquire data is also discussed. The instrument is developed to achieve the Technology Readiness Level 4 standard which covers the breadboard validation of a space system in a laboratory environment. Hardware specifications and software algorithms of the instrument are presented. Results from an external Helium-Neon (HeNe) laser and a broadband light source limited by spectral bandpass filters are presented. Finally, recommendations and future improvements to this research and development program are listed

Deep Learning Techniques for Backscattering Vector Estimation in ToF Data

In this work we propose a new approach that is not already mentioned in the literature to correct the multi-path interference phenomenon which occurs in time-of-flight cameras. We introduce a deep learning approach to learn the typical reflection structure of the light in a real environment and use it as strong prior to estimate the shape of the time-dependent scene impulse response, called backscattering vector

A Comprehensive Overview on 5G-and-Beyond Networks with UAVs: From Communications to Sensing and Intelligence

Due to the advancements in cellular technologies and the dense deployment of cellular infrastructure, integrating unmanned aerial vehicles (UAVs) into the fifth-generation (5G) and beyond cellular networks is a promising solution to achieve safe UAV operation as well as enabling diversified applications with mission-specific payload data delivery. In particular, 5G networks need to support three typical usage scenarios, namely, enhanced mobile broadband (eMBB), ultra-reliable low-latency communications (URLLC), and massive machine-type communications (mMTC). On the one hand, UAVs can be leveraged as cost-effective aerial platforms to provide ground users with enhanced communication services by exploiting their high cruising altitude and controllable maneuverability in three-dimensional (3D) space. On the other hand, providing such communication services simultaneously for both UAV and ground users poses new challenges due to the need for ubiquitous 3D signal coverage as well as the strong air-ground network interference. Besides the requirement of high-performance wireless communications, the ability to support effective and efficient sensing as well as network intelligence is also essential for 5G-and-beyond 3D heterogeneous wireless networks with coexisting aerial and ground users. In this paper, we provide a comprehensive overview of the latest research efforts on integrating UAVs into cellular networks, with an emphasis on how to exploit advanced techniques (e.g., intelligent reflecting surface, short packet transmission, energy harvesting, joint communication and radar sensing, and edge intelligence) to meet the diversified service requirements of next-generation wireless systems. Moreover, we highlight important directions for further investigation in future work.Comment: Accepted by IEEE JSA

Compressive Sensing and Its Applications in Automotive Radar Systems

Die Entwicklung in Richtung zu autonomem Fahren verspricht, künftig einen sicheren Verkehr ohne tödliche Unfälle zu ermöglichen, indem menschliche Fahrer vollständig ersetzt werden. Dadurch entfällt der Faktor des menschlichen Fehlers, der aus Müdigkeit, Unachtsamkeit oder Alkoholeinfluss resultiert. Um jedoch eine breite Akzeptanz für autonome Fahrzeuge zu erreichen und es somit eines Tages vollständig umzusetzen, sind noch eine Vielzahl von Herausforderungen zu lösen. Da in einem autonomen Fahrzeug kein menschlicher Fahrer mehr in Notfällen eingreifen kann, müssen sich autonome Fahrzeuge auf leistungsfähige und robuste Sensorsysteme verlassen können, um in kritischen Situationen auch unter widrigen Bedingungen angemessen reagieren zu können. Daher ist die Entwicklung von Sensorsystemen erforderlich, die für Funktionalitäten jenseits der aktuellen advanced driver assistance systems eingesetzt werden können. Dies resultiert in neuen Anforderungen, die erfüllt werden müssen, um sichere und zuverlässige autonome Fahrzeuge zu realisieren, die weder Fahrzeuginsassen noch Passanten gefährden. Radarsysteme gehören zu den Schlüsselkomponenten unter der Vielzahl der verfügbaren Sensorsysteme, da sie im Gegensatz zu visuellen Sensoren von widrigen Wetter- und Umgebungsbedingungen kaum beeinträchtigt werden. Darüber hinaus liefern Radarsysteme zusätzliche Umgebungsinformationen wie Abstand, Winkel und relative Geschwindigkeit zwischen Sensor und reflektierenden Zielen. Die vorliegende Dissertation deckt im Wesentlichen zwei Hauptaspekte der Forschung und Entwicklung auf dem Gebiet der Radarsysteme im Automobilbereich ab. Ein Aspekt ist die Steigerung der Effizienz und Robustheit der Signalerfassung und -verarbeitung für die Radarperzeption. Der andere Aspekt ist die Beschleunigung der Validierung und Verifizierung von automated cyber-physical systems, die parallel zum Automatisierungsgrad auch eine höhere Komplexität aufweisen. Nach der Analyse zahlreicher möglicher Compressive Sensing Methoden, die im Bereich Fahrzeugradarsysteme angewendet werden können, wird ein rauschmoduliertes gepulstes Radarsystem vorgestellt, das kommerzielle Fahrzeugradarsysteme in seiner Robustheit gegenüber Rauschen übertrifft. Die Nachteile anderer gepulster Radarsysteme hinsichtlich des Signalerfassungsaufwands und der Laufzeit werden durch die Verwendung eines Compressive Sensing-Signalerfassungs- und Rekonstruktionsverfahrens in Kombination mit einer Rauschmodulation deutlich verringert. Mit Compressive Sensing konnte der Aufwand für die Signalerfassung um 70% reduziert werden, während gleichzeitig die Robustheit der Radarwahrnehmung auch für signal-to-noise-ratio-Pegel nahe oder unter Null erreicht wird. Mit einem validierten Radarsensormodell wurde das Rauschradarsystem emuliert und mit einem kommerziellen Fahrzeugradarsystem verglichen. Datengetriebene Wettermodelle wurden entwickelt und während der Simulation angewendet, um die Radarleistung unter widrigen Bedingungen zu bewerten. Während eine Besprühung mit Wasser die Radomdämpfung um 10 dB erhöht und Spritzwasser sogar um 20 dB, ergibt sich die eigentliche Begrenzung aus der Rauschzahl und Empfindlichkeit des Empfängers. Es konnte bewiesen werden, dass das vorgeschlagene Compressive Sensing Rauschradarsystem mit einer zusätzlichen Signaldämpfung von bis zu 60 dB umgehen kann und damit eine hohe Robustheit in ungünstigen Umwelt- und Wetterbedingungen aufweist. Neben der Robustheit wird auch die Interferenz berücksichtigt. Zum einen wird die erhöhte Störfestigkeit des Störradarsystems nachgewiesen. Auf der anderen Seite werden die Auswirkungen auf bestehende Fahrzeugradarsysteme bewertet und Strategien zur Minderung der Auswirkungen vorgestellt. Die Struktur der Arbeit ist folgende. Nach der Einführung der Grundlagen und Methoden für Fahrzeugradarsysteme werden die Theorie und Metriken hinter Compressive Sensing gezeigt. Darüber hinaus werden weitere Aspekte wie Umgebungsbedingungen, unterschiedliche Radararchitekturen und Interferenz erläutert. Der Stand der Technik gibt einen Überblick über Compressive Sensing-Ansätze und Implementierungen mit einem Fokus auf Radar. Darüber hinaus werden Aspekte von Fahrzeug- und Rauschradarsystemen behandelt. Der Hauptteil beginnt mit der Vorstellung verschiedener Ansätze zur Nutzung von Compressive Sensing für Fahrzeugradarsysteme, die in der Lage sind, die Erfassung und Wahrnehmung von Radarsignalen zu verbessern oder zu erweitern. Anschließend wird der Fokus auf ein Rauschradarsystem gelegt, das mit Compressive Sensing eine effiziente Signalerfassung und -rekonstruktion ermöglicht. Es wurde mit verschiedenen Compressive Sensing-Metriken analysiert und in einer Proof-of-Concept-Simulation bewertet. Mit einer Emulation des Rauschradarsystems wurde das Potential der Compressive Sensing Signalerfassung und -verarbeitung in einem realistischeren Szenario demonstriert. Die Entwicklung und Validierung des zugrunde liegenden Sensormodells wird ebenso dokumentiert wie die Entwicklung der datengetriebenen Wettermodelle. Nach der Betrachtung von Interferenz und der Koexistenz des Rauschradars mit kommerziellen Radarsystemen schließt ein letztes Kapitel mit Schlussfolgerungen und einem Ausblick die Arbeit ab.Developments towards autonomous driving promise to lead to safer traffic, where fatal accidents can be avoided after making human drivers obsolete and hence removing the factor of human error. However, to ensure the acceptance of automated driving and make it a reality one day, still a huge amount of challenges need to be solved. With having no human supervisors, automated vehicles have to rely on capable and robust sensor systems to ensure adequate reactions in critical situations, even during adverse conditions. Therefore, the development of sensor systems is required that can be applied for functionalities beyond current advanced driver assistance systems. New requirements need to be met in order to realize safe and reliable automated vehicles that do not harm passersby. Radar systems belong to the key components among the variety of sensor systems. Other than visual sensors, radar is less vulnerable towards adverse weather and environment conditions. In addition, radar provides complementary environment information such as target distance, angular position or relative velocity, too. The thesis ad hand covers basically two main aspects of research and development in the field of automotive radar systems. One aspect is to increase efficiency and robustness in signal acquisition and processing for radar perception. The other aspect is to accelerate validation and verification of automated cyber-physical systems that feature more complexity along with the level of automation. After analyzing a variety of possible Compressive Sensing methods for automotive radar systems, a noise modulated pulsed radar system is suggested in the thesis at hand, which outperforms commercial automotive radar systems in its robustness towards noise. Compared to other pulsed radar systems, their drawbacks regarding signal acquisition effort and computation run time are resolved by using noise modulation for implementing a Compressive Sensing signal acquisition and reconstruction method. Using Compressive Sensing, the effort in signal acquisition was reduced by 70%, while obtaining a radar perception robustness even for signal-to-noise-ratio levels close to or below zero. With a validated radar sensor model the noise radar was emulated and compared to a commercial automotive radar system. Data-driven weather models were developed and applied during simulation to evaluate radar performance in adverse conditions. While water sprinkles increase radome attenuation by 10 dB and splash water even by 20 dB, the actual limitation comes from noise figure and sensitivity of the receiver. The additional signal attenuation that can be handled by the proposed compressive sensing noise radar system proved to be even up to 60 dB, which ensures a high robustness of the receiver during adverse weather and environment conditions. Besides robustness, interference is also considered. On the one hand the increased robustness towards interference of the noise radar system is demonstrated. On the other hand, the impact on existing automotive radar systems is evaluated and strategies to mitigate the impact are presented. The structure of the thesis is the following. After introducing basic principles and methods for automotive radar systems, the theory and metrics of Compressive Sensing is presented. Furthermore some particular aspects are highlighted such as environmental conditions, different radar architectures and interference. The state of the art provides an overview on Compressive Sensing approaches and implementations with focus on radar. In addition, it covers automotive radar and noise radar related aspects. The main part starts with presenting different approaches on making use of Compressive Sensing for automotive radar systems, that are capable of either improving or extending radar signal acquisition and perception. Afterwards the focus is put on a noise radar system that uses Compressive Sensing for an efficient signal acquisition and reconstruction. It was analyzed using different Compressive Sensing metrics and evaluated in a proof-of-concept simulation. With an emulation of the noise radar system the feasibility of the Compressive Sensing signal acquisition and processing was demonstrated in a more realistic scenario. The development and validation of the underlying sensor model is documented as well as the development of the data-driven weather models. After considering interference and co-existence with commercial radar systems, a final chapter with conclusions and an outlook completes the work

System level risk analysis of electromagnetic environmental effects and lightning effects in aircraft -- steady state and transient

2017 Summer.Includes bibliographical references.This dissertation is an investigation of the system level risk of electromagnetic and lightning effects in aircraft. It begins with an analysis to define a system, and a discussion of emergence as a characteristic of a system. Against this backdrop, risk is defined as an undesirable emergent property of a system. A procedure to translate the system level non-functional attributes to lower level functional requirements is developed. With this foundation, a model for risk analysis, resolution and management is developed by employing the standard risk model. The developed risk model is applied to evaluation of electromagnetic environmental effects and lightning effects in aircraft. Examples are shown to demonstrate the validity of the model. Object Process Methodology and systems thinking principles are used extensively throughout this work. The dissertation concludes with a summary and suggestions for additional work

Digital Image Processing

This book presents several recent advances that are related or fall under the umbrella of 'digital image processing', with the purpose of providing an insight into the possibilities offered by digital image processing algorithms in various fields. The presented mathematical algorithms are accompanied by graphical representations and illustrative examples for an enhanced readability. The chapters are written in a manner that allows even a reader with basic experience and knowledge in the digital image processing field to properly understand the presented algorithms. Concurrently, the structure of the information in this book is such that fellow scientists will be able to use it to push the development of the presented subjects even further

Ultrafast Electron Diffraction: Visualizing Dynamic States of Matter

Since the discovery of electron-wave duality, electron scattering instrumentation has developed into a powerful array of techniques for revealing the atomic structure of matter. Beyond detecting local lattice variations in equilibrium structures with the highest possible spatial resolution, recent research efforts have been directed towards the long sought-after dream of visualizing the dynamic evolution of matter in real-time. The atomic behavior at ultrafast timescales carries critical information on phase transition and chemical reaction dynamics, the coupling of electronic and nuclear degrees of freedom in materials and molecules, the correlation between structure, function and previously hidden metastable or nonequilibrium states of matter. Ultrafast electron pulses play an essential role in this scientific endeavor, and their generation has been facilitated by rapid technical advances in both ultrafast laser and particle accelerator technologies. This review presents a summary of the remarkable developments in this field over the last few decades. The physics and technology of ultrafast electron beams is presented with an emphasis on the figures of merit most relevant for ultrafast electron diffraction (UED) experiments. We discuss recent developments in the generation, manipulation and characterization of ultrashort electron beams aimed at improving the combined spatio-temporal resolution of these measurements. The fundamentals of electron scattering from atomic matter and the theoretical frameworks for retrieving dynamic structural information from solid-state and gas-phase samples is described. Essential experimental techniques and several landmark works that have applied these approaches are also highlighted to demonstrate the widening applicability of these methods. Ultrafast electron probes with ever improving capabilities, combined with other complementary photon-based or spectroscopic approaches, hold tremendous potential for revolutionizing our ability to observe and understand energy and matter at atomic scales

Development of a Nano-Illumination Microscope

[eng] This doctoral thesis proposes and explores a new approach to lensless microscopy, focusing on making high resolution imaging ubiquitous and low cost. A short introduction to microscopy frames the state of current techniques: Abbe’s law limits the resolving power for visible light microscopes with lenses, techniques using UV, X-rays, or electrons are incompatible with live samples and all of them, including super-resolution microscopy methods, are complex devices not suitable for being used in the field as mobile devices. Some lensless microscopy methods try to solve these issues. The microscopy method is named Nano Illumination Microscopy (NIM) because it relies on using nanometric light sources in an ordered array to illuminate a sample placed in close proximity to them, and a photodetector at the other side to measure the amount of light arriving from each LED. In a setup like this, the resolving power is provided by the nano-LEDs and their distribution instead of the sensing devices, as is the case in the other methods. Since the resolving power depends on the pitch of the LED array, this method also opens a path to obtain super-resolution images, depending only on obtaining LED arrays with pitches smaller than Abbe’s limit for the wavelength. After the introduction to microscopy setting the context of the thesis, the thesis continues explaining the main components used to build the microscope: a SPAD camera, designed within the context of this work, and the electronics to control the nano-LED array. The third chapter of this thesis provides an overview of the microscopy method and its fundaments, exploring the requirements and capabilities. Image formation is first introduced with simulations, and this information is then used to build the very first prototype, a microscope capable of forming 8x8 pixel images -since that is the form factor of the LED array used, with LEDs of 5 μm in size (and 10 μm in pitch). The first results from this technique are presented and compared with the simulations, showing the agreement between both, validating the method, and offering insight on building the next prototypes, which will use smaller LEDs in an attempt to study the technological limits. The thesis continues with the work done in search of the limits of the technique, building and testing new improved versions of the microscope and confronting the limitations which arise. Some of those came from the structure of the LED arrays themselves: while nano-LEDs well below the sizes used have been reported, those have been isolated structures or non individually addressable. Selecting exactly which LED will emit is one of the main problems to solve since with increasingly large arrays, the connections required increase exponentially until routing is impossible. The thesis also studies this problem, as the LED arrays were changed in search of the proper solution. This implied moving from a direct addressing strategy, in which each LED was selected individually, towards a matrix-addressing format, in which the LEDs are selected by polarising the appropriate row and columns. The microscopy technique is validated and the more advanced prototypes presented. Images with a maximum resolving power of 800 nm are shown, and the results discussed, since the physical limitations on fabricating the chips limit the maximum resolving power below what was theoretically expected. The thesis also offers a short overview into the future of the Nano Illumination Microscopy technique.[cat] Aquesta tesi doctoral proposa i explora una nova aproximació a la microscopia sense lents, amb la intenció de facilitar l’obtenció d’imatges d’alta resolució amb baix cost i disponible arreu. S’ha batejat aquest mètode de microscòpia com a Microscopia de Nano-Il·luminació (MNI) perquè la imatge es construeix a partir de fonts de llum de mida nanomètrica distribuïdes en una matriu que il·luminen la mostra de forma consecutiva i ordenada. Un sensor a l’altre costat recull la intensitat de llum que arriba de cada LED, creant un mapa de l’objecte observat. Aquest mètode fa que la resolució de les imatges depengui de la mida i distribució dels LEDs, en comptes de la del sensor com és el cas convencionalment, obrint la porta a noves integracions. En la tesi s’ofereix una introducció general a la microscòpia abans d’entrar a detallar els components del microscopi i com s’integren per muntar-lo. A continuació es presenta i s’estudia el funcionament del mètode, començant amb simulacions i seguint amb la construcció del primer prototip de microscopi amb el que s’obtenen les primeres imatges. La tesi procedeix a continuació a investigar els límits actuals de la tècnica de microscòpia, utilitzant noves versions de la matriu de LEDs i estratègies alternatives per intentar superar-ne les complicacions tècniques. Així, s’obtenen imatges amb una resolució de 800 nm i es discuteix la problemàtica d’implementar dispositius que s’aproximin a les expectatives teòriques per la tècnica