Visual localization in challenging environments

Visual localization, the method of self-localization based on camera images, has established as an additional, GNSS-free technology that is investigated in increasingly real and challenging applications. Particularly demanding is the self-localization of first responders in unstructured and unknown environments, for which visual localization can substantially contribute to increase the situational awareness and safety of first responders. Challenges arise from the operation under adverse conditions on computationally restricted platforms in the presence of dynamic objects. Current solutions are quickly pushed to their limits and the development of more robust approaches is of high demand. This thesis investigates the application of visual localization in dynamic, adverse environments to identify challenges and accordingly to increase the robustness, on the example of a dedicated visual-inertial navigation system. The methodical contributions of this work relate to the introduction of semantic understanding, improvements in error propagation and the development of a digital twin. The geometric visual odometry component is extended to a hybrid approach that includes a deep neural network for semantic segmentation to ignore distracting image areas of certain object classes. A Sensor-AI approach complements this method by directly training the network to segment image areas that are critical for the considered visual odometry system. Another improvement results from analyses and modifications of the existing error propagation in visual odometry. Furthermore, a digital twin is presented that closely replicates geometric and radiometric properties of the real sensor system in simulation in order to multiply experimental possibilities. The experiments are based on datasets from inspections that are used to motivate three first responder scenarios, namely indoor rescue, flood disaster and wildfire. The datasets were recorded in corridor, mall, coast, river and fumarole environments and aim to analyze the influence of the dynamic elements person, water and smoke. Each investigation starts with extensive in-depth analyses in simulation based on created synthetic video clones of the respective dynamic environments. Specifically, a combined sensitivity analysis allows to jointly consider environment, system design, sensor property and calibration error parameters to account for adverse conditions. All investigations are verified with experiments based on the real system. The results show the susceptibility of geometric approaches to dynamic objects in challenging scenarios. The introduction of the segmentation aid within the hybrid system contributes well in terms of robustness by preventing significant failures, but understandably it cannot compensate for a lack of visible static backgrounds. As a consequence, future visual localization systems require both the ability of semantic understanding and its integration into a complementary multi-sensor system

Orthogonal enzyme-driven timers for DNA strand displacement reactions

Here, we demonstrate a strategy to rationally program a delayed onset of toehold-mediated DNA strand displacement reactions (SDRs). The approach is based on blocker strands that efficiently inhibit the strand displacement by binding to the toehold domain of the target DNA. Specific enzymatic degradation of the blocker strand subsequently enables SDR. The kinetics of the blocker enzymatic degradation thus controls the time at which the SDR starts. By varying the concentration of the blocker strand and the concentration of the enzyme, we show that we can finely tune and modulate the delayed onset of SDR. Additionally, we show that the strategy is versatile and can be orthogonally controlled by different enzymes each specifically targeting a different blocker strand. We designed and established three different delayed SDRs using RNase H and two DNA repair enzymes (formamidopyrimidine DNA glycosylase and uracil-DNA glycosylase) and corresponding blockers. The achieved temporal delay can be programed with high flexibility without undesired leak and can be conveniently predicted using kinetic modeling. Finally, we show three possible applications of the delayed SDRs to temporally control the ligand release from a DNA nanodevice, the inhibition of a target protein by a DNA aptamer, and the output signal generated by a DNA logic circuit

Dissipative control over the toehold-mediated DNA strand displacement reaction

Here we show a general approach to achieve dissipative control over toehold-mediated strand-displacement, the most widely employed reaction in the field of DNA nanotechnology. The approach relies on rationally re-engineering the classic strand displacement reaction such that the high-energy invader strand (fuel) is converted into a low-energy waste product through an energy-dissipating reaction allowing the spontaneous return to the original state over time. We show that such dissipative control over the toehold-mediated strand displacement process is reversible (up to 10 cycles), highly controllable and enables unique temporal activation of DNA systems. We show here two possible applications of this strategy: the transient labelling of DNA structures and the additional temporal control of cascade reactions

Finally! Insights into the ARCHES Lunar Planetary Exploration Analogue Campaign on Etna in summer 2022

This paper summarises the first outcomes of the space demonstration mission of the ARCHES project which could have been performed this year from 13 june until 10 july on Italy’s Mt. Etna in Sicily. After the second postponement related to COVID from the initially for 2020 planed campaign, we are now very happy to report, that the whole campaign with more than 65 participants for four weeks has been successfully conduced. In this short overview paper, we will refer to all other publication here on IAC22. This paper includes an overview of the performed 4-week campaign and the achieved mission goals and first results but also share our findings on the organisational and planning aspects

Simulation-based Sensitivity Analyses of Visual Localization in Challenging Environments

Digital twins of sensor systems enable in-depth analysis of visual localization methods based on synthetic video clones. This is necessary to develop and prepare them for reliable operation in challenging and hazardous real-world environments. Exemplary for a visual-inertial navigation system, this poster presents experiments from Monte-Carlo-based multiparameter sensitivity analyses to assess significant error sources from different environmental, system design, sensor property and calibration error parameters. Three different synthetic video clones are used that closely replicate datasets from corridor, volcanic coast and fumarole environments, and contain the dynamic elements person, water, and smoke

Generalizing mechanisms of secondary structure dynamics in biopolymers

Secondary structure dynamics of biopolymers play a vital role in many of the complex processes within a cell. However, due to the substantial number of atoms in the involved biopolymers along with the multitude of interactions that occur between the molecules, understanding these processes in detail is challenging and often involves computationally demanding simulations. In this thesis, the secondary structure dynamics of three different biopolymer systems were modeled using a single approach, which is based on intuitive principles that facilitate the interpretation. To this end, the kinetic behavior of each system was experimentally determined, and described by simplified reaction schemes, which were then connected to Markov chain models encompassing all principal secondary structural conformations. Firstly, we investigated the toehold-mediated strand displacement reaction, which is widely applied in nanotechnology to create DNA-based nano-devices and biochemical reaction networks. Our model correctly described the impact of base pair mismatches on the kinetics of these reactions, as measured by bulk fluorescence experiments. Additionally, it revealed that incumbent dissociation, base pair fraying, and internal loop formation are important processes during strand displacement. Furthermore, we established two dissipative elements to enhance temporal control over toehold-mediated strand displacement reactions. The first element allowed a reversible and repeatable incumbent strand release, whereas the second element provided the possibility to start the displacement reaction after a programmable temporal delay. Secondly, we studied the target recognition by the CRISPR-Cas effector complex Cascade, a highly promising protein for applications in genome engineering. Our model successfully reproduced all aspects of the torque- and mismatch-dependent R-loop formation time by Cascade obtained by single-molecule torque and bulk fluorescence measurements. Furthermore, we demonstrated that the seed effect observed for Cascade results from DNA supercoiling, rather than a structural property of the protein complex. Lastly, we explored the folding/unfolding of α-helices, which plays a critical role in the folding and function of proteins. Our model accurately described α-helix unfolding kinetics obtained by fast triplet-triplet energy transfer. Moreover, we showed that the complex α-helix unfolding does not follow a simple Einstein-type diffusion but is a combination of the sub-diffusive boundary diffusion and the rather peptide-length-independent coil nucleation. The presented models enabled access to the diverse timescales of the characterized processes, which are generally difficult to access experimentally, despite utilizing just a single approach. In particular, we obtained: tens of microseconds for the branch migration step time of the toehold-mediated strand displacement, hundreds of microseconds for the R-loop formation steps by Cascade, and tens of nanoseconds for folding or unfolding of an α-helix by a single residue. Given the simplicity and accessibility of the established models, we are confident that they will become useful tools for researchers to analyze the dynamics of biomolecules, and anticipate that similar modeling approaches can be applied to other biopolymer systems, being well-described by probabilistic models.Die Sekundärstrukturdynamik von Biopolymeren spielt eine entscheidende Rolle bei vielen komplexen Prozessen innerhalb einer Zelle. Aufgrund der beträchtlichen Anzahl von Atomen in den beteiligten Biopolymeren und der Vielzahl an Wechselwirkungen zwischen den Molekülen ist es jedoch eine Herausforderung diese Prozesse im Detail zu verstehen, und erfordert oft rechenintensive Simulationen. In dieser Arbeit wurde die Sekundärstrukturdynamik von drei verschiedenen Biopolymersystemen mit einem einzigen Ansatz modelliert, welcher auf intuitiven Prinzipien beruht und somit eine erleichterte Interpretation der Ergebnisse ermöglicht. Hierzu wurde das kinetische Verhalten jedes Systems experimentell bestimmt und durch vereinfachte Reaktionsschemata beschrieben. Diese wurden anschließend mit Markov-Kettenmodellen verknüpft, welche alle wichtigen Konformationen der Sekundärstruktur abbilden. Als erstes System untersuchten wir die DNA Strangaustauschreaktion, welche in der Nanotechnologie häufig zur Herstellung von DNA-basierten Nanomaschinen und biochemischen Reaktionsnetzwerken eingesetzt wird. Unser Modell beschrieb die durch Ensemble-Fluoreszenz-Experimente gemessenen Auswirkungen von Basenfehlpaarungen auf die Kinetik dieser Reaktionen korrekt. Des Weiteren zeigte sich, dass die vorzeitige Strangablösung, das Ausfransen von Basenpaaren und die Bildung interner Schleifen wichtige Prozesse während des Strangaustausches sind. Darüber hinaus konnten wir zwei dissipative Elemente etablieren, um die zeitliche Kontrolle über die Strangaustauschreaktionen zu verbessern. Das erste Element ermöglicht eine reversible und wiederholbare Strangablösung, während das zweite Element die Möglichkeit bietet die Strangaustauschreaktionen nach einer programmierbaren zeitlichen Verzögerung zu starten. Zweitens untersuchten wir den Zielerkennungsprozess durch den CRISPR-Cas Komplex Cascade, ein vielversprechendes Protein für Anwendungen in der Genomtechnologie. Unser Modell reproduzierte erfolgreich alle Aspekte der torsions- und fehlpaarungs-abhängigen R-Schleifenbildung durch Cascade, welche durch Einzelmolekül-Torsions- und Ensemble-Fluoreszenz-Messungen ermittelt wurden. Zusätzlich konnten wir nachweisen, dass der für Cascade beobachtete „seed“-Effekt auf DNA-Verdrehung und nicht auf eine strukturelle Eigenschaft des Proteinkomplexes zurückzuführen ist. Schließlich untersuchten wir die Faltung/Entfaltung von α-Helices, welche eine entscheidende Rolle bei der Faltung und Funktion von Proteinen spielen. Unser Modell beschrieb die durch schnelle Triplett-Triplett-Energietransfer Experimente ermittelte α-Helix-Entfaltungskinetik exakt. Darüber hinaus konnten wir zeigen, dass die komplexe α-Helix-Entfaltung nicht einer einfachen Diffusion vom Einstein-Typ folgt, sondern eine Kombination aus subdiffusiver Grenzdiffusion und der eher peptidlängenunabhängigen Coil-Nukleation ist. Obwohl nur ein einziger Ansatz verwendet wurde, ermöglichten die vorgestellten Modelle den Zugang zu den vielschichtigen Zeitskalen der charakterisierten Prozesse, welche im Allgemeinen experimentell schwer zugänglich sind. Insbesondere konnten die folgenden zeitlichen Bereiche bestimmt werden: Dutzende von Mikrosekunden für die Schrittzeit der Strangaustauschreaktion, Hunderte von Mikrosekunden für die Schritte der R-Schleifenbildung durch Cascade, und Dutzende von Nanosekunden für die Faltung oder Entfaltung einer α-Helix um ein einzelnes Segment. Angesichts der Simplizität und Zugänglichkeit der etablierten Modelle sind wir zuversichtlich, dass sie zu nützlichen Werkzeugen für Forscher werden, um die Dynamik von Biomolekülen zu analysieren. Zusätzlich gehen wir davon aus, dass ähnliche Modellierungsansätze auf andere Biopolymersysteme angewendet werden können, sofern sie gut durch probabilistische Modelle beschrieben werden

Camera-based distance estimation for autonomous vehicles

The aim of this work is the investigation of camera-based techniques for distance estimation between two autonomous vehicles. While both monocular- and stereo-camera methods are explored, this study focuses on the usage of fiducial markers. Therefore, existing fiducial markers are discussed and selected. Based on this selection, three configurations of markers are proposed and applied to different distance estimation methods. The chosen markers are AprilTag and WhyCon. Their distances are estimated by means of Perspective-n-Point, 3D position calculation of a circle and stereo-based triangulation. Within this study the presented methods are evaluated based on their distance estimation accuracy and applicable range. They are compared with each other and with the common stereo method Semi-Global-Matching. Moreover, the influence of uncertainties is explored with reference to geometrical calibration. A setup is presented to evaluate the techniques based on real-world and simulated data. In order to gain insights on the methods properties, a simulation is used that facilitates variation of the image data. In addition, a Monte-Carlo-Simulation allows to model calibration uncertainty. The obtained observations are substantiated based on two real-world experiments. The results demonstrate the potential of fiducial markers for relative distance estimation of vehicles in terms of high accuracy and low uncertainty. The lower sensitivity to uncertainties in camera calibration makes fiducial markers preferable to stereo methods

A Sensor-AI approach to improve visual odometry in dynamic environments

A key component of the interdisciplinary research field Sensor-AI is the close interaction and combination of physical models, data based models and classical approaches in one sensor system, primary for applications that are defined by strict energy requirements. This concept finds practical use cases in many domains. I propose a Sensor-AI based approach that targets to improve visual odometry specifically in uncommon high dynamic environments. It combines classical feature-based visual odometry, which relies on physical models and analytical error propagation, with a data-based model in form of a deep neural network for semantic segmentation. The former is used to automatically generate semantic labels to train the neural network offline, simultaneously for multiple environments, and use its inference output to generate a mask for feature selection online. The proposed method is evaluated on datasets that contain the two dynamic environment elements steam and water, recorded at a volcanic fumarole field, a coast line and a river