Modellierung von Unsicherheit und Wahrnehmung für Lernende Systeme

Durch die enormen Fortschritte im Bereich der künstlichen Intelligenz (KI) erreicht eine Vielzahl lernender Systeme den Alltag von Menschen. Aktuell agiert zwar der Großteil dieser lernenden, KI-basierten Systeme noch für den Endnutzer meist unsichtbar, rein online, (beispielsweise als Recommender-System in Online-Shops), doch erste Systeme wie selbstfahrende Autos und autonome Lieferroboter sind bereits in die physische Welt vorgedrungen. Ebenso wird mit Hochdruck an Visionen wie der vollautomatisierten Fabrik (Smart Factory) oder an adaptiven Stromnetzen (Smart Grids) gearbeitet. Spätestens mit der Präsenz der lernenden Systeme in der physischen Welt gewinnen Fragen der Zuverlässigkeit und Sicherheit höchste Relevanz. Aktuell existieren jedoch kaum Verfahren, die die Grenzen und stets vorhandenen Unsicherheiten der lernenden Systeme zuverlässig erfassen können. Ebenso wäre die Modellierung eines gemeinsamen Wahrnehmungsverständnisses zwischen Mensch und Maschine von großer Wichtigkeit, um Missverständnisse und Fehler in der Interaktion zu minimieren. In der vorliegenden Arbeit werden Ansätze vorgestellt, die eine solche Modellierung von Unsicherheit und Wahrnehmung, insbesondere in Kombination mit aktuellen Deep Learning Verfahren ermöglichen. Im ersten Teil dieser Arbeit werden Ansätze zur Modellierung eines räumlichen Wahrnehmungsverständnisses für lernende Systeme behandelt. Hierfür wird eine Isovisten-basierte Quantifizierung der Wahrnehmung mit Machine Learning (ML) basierter Modellierung kombiniert. Evaluationsergebnisse zeigen, dass die so entwickelte Modellierung in der Lage ist, semantische Strukturen abzubilden. Eine in einem zweiten Schritt mit Bayesschen Verfahren erweiterte, probabilistische Wahrnehmungsmodellierung wird anschließend unter anderem zur Charakterisierung von Routen eingesetzt. Da besonders im Forschungsfeld autonom handelnder Systeme aktuell Reinforcement Learning (RL) Ansätze dominieren, stehen im zweiten Teil der Arbeit Methoden zur Unsicherheitsmodellierung in Value-basiertem Deep RL im Fokus. Es werden sowohl Methoden der approximativen Bayesschen Inferenz, als auch ensemble-basierte Verfahren dahingehend evaluiert, ob sie zuverlässig in der Lage sind, epistemische Unsicherheit zu modellieren. Da die zuvor vorgestellten Ansätze nicht direkt mit Policy-basiertem RL kombinierbar sind, wird im dritten Teil der Arbeit ein neu entwickelter, auf der Entropie der Policy basierender Ansatz zur Erkennung von Out-of-Distribution (OOD) Situationen vorgestellt. Zusammengenommen schaffen die im Rahmen der vorliegenden Arbeit vorgestellten Modellierungsansätze von Unsicherheit und Wahrnehmung eine Grundlage zur Entwicklung zuverlässiger, sicherer, lernender Systeme.With the enormous progress in the field of artificial intelligence (AI), a plethora of learning systems is becoming an integral part of people's everyday lives. Currently, most of these learning, AI-based systems are restricted to virtual spaces (for example as recommender systems in online shops). But the first ones like self-driving cars or autonomous delivery robots have already entered the physical world. Work is also progressing to realize visions like fully automated industrial plants (Smart Factory) or adaptive electrical grids (Smart Grid). With this increasing presence of learning systems in the physical world, questions of reliability and safety are now of utmost importance. Currently, however, hardly any methods exist that are able to reliably capture the limitations and ever-present uncertainties of learning systems. Likewise, modeling a common understanding of perception between humans and machines is of great importance, to minimize misunderstandings and errors in their interaction. The work at hand presents approaches which allow this kind of modeling of uncertainty and perception, especially when combined with current Deep Learning methods. The first part of this thesis addresses approaches for modeling a spatial perception for learning systems. For this purpose, an Isovist-based quantification of perception is combined with Machine Learning (ML) based modeling. Evaluation results show that the developed modeling approaches are capable of representing semantic structures. A probabilistic modeling approach, extended using Bayesian methods, is subsequently presented. It is used, among other things, for an uncertainty based characterization of trajectories. For the development of autonomous systems, Reinforcement Learning (RL) approaches currently dominate. Consequently, the second part of this thesis focuses on methods for uncertainty modeling in Value-based Deep RL. Methods of approximate Bayesian Inference as well as Ensemble-based approaches are evaluated, regarding their ability to reliability model epistemic uncertainty. As the previous approaches are not directly applicable to Policy-based RL, the third part of this thesis presents a newly developed policy entropy based approach for detecting out-of-distribution (OOD) situations. Together, the approaches to modeling uncertainty and perception, presented in the context of the work at hand, provide a foundation for the future development of reliable, safe, learning systems

Surface modification and scan imaging of upconverting nanoparticles

The surface modification of upconverting nanoparticles was a main focus of this thesis and a wide variety of different functionalities were introduced on the nanoparticle surface in the process. The growth of a silica shell with a preferably simultaneous silanization proved to be a convenient way to create both a hydrophilic surface crucial for bioanalytical applications and an accessible functionality for subsequent modification. The utilization of silanol, amine, carboxyl, or phosphonate groups yielded UCNPs with strongly differing stabilities in dispersion and surface charges. A strong aggregation was observed for both an amine modification and the non-modified silanol groups of a pure silica shell with a tendency enhancement for a reduced shell thickness. Furthermore, surface phosphonate groups led to the formation of aggregates with high uniformity in both size and shape. In contrast, an optimized modification process to introduce carboxylic acids on the surface yielded monodisperse UCNPs with a diameter of 13 nm that are efficiently stabilized in aqueous dispersion. The presence of these predominantly single nanoparticles in high abundance was confirmed both by dynamic light scattering and transmission electron microscopy studies. This high monodispersity was also realizable by a ligand exchange utilizing the polymer polyacrylic acid. The method applied relied on a prior exchange of the hydrophobic oleic acid with BF4- ions allowing a subsequent modification with the polymer with simplified process conditions and reduced expenditure of time. Furthermore, the copper-catalyzed azide-alkyne cycloaddition (CuAAC) was studied as a potential beneficial coupling reaction since it merged the advantages of the "Click chemistry" and "bioorthogonal reactions" concept. The associated alkyne and azide groups were bound to the nanoparticle surface by silanization of silica-coated UCNPs and the nanoparticles differed drastically in their dispersion stability. While an azide-functionalized surface exerted a certain degree of stabilization, alkyne functionalities led to a strong nanoparticle aggregation and precipitation in a very short time. However, both functional groups allowed an efficient and highly specific binding of their respective counterpart shown by an exemplary Click reaction with a fluorophore and also the important role of the copper(I) catalyst was demonstrated. The attachment of more complex molecules including biochemical functionalities or additionally stabilizing moieties revealed a good feasibility of a modification by the copper-catalyzed azide-alkyne cycloaddition. A simultaneous modification of the UCNPs with azide and carboxyl groups led to an enhanced nanoparticle stability compared to a pure azide functionalization. Studies about the creation of a biomimetic nanoparticle surface comprised the encapsulation of UCNPs by virus capsid leading to the formation of virus-like particles (VLPs). After the process feasibility was confirmed to both dissemble and reassemble the capsid of a brome mosaic virus (BMV), small nanoparticles with an optimized surface were utilized to act as an artificial nucleation grain for self-assembly. These modified UCNPs met the three requirements for encapsulation: a hydrophilic surface to be dispersible in aqueous systems, an absolute diameter below 16 nm to fit in the capsid cavity of the BMV, and a negative surface charge to initiate the self-assembly of the capsomers similar to the viral RNA. In addition to the options described, citric acid was investigated as a potential surface ligand. While silanol and phosphonate groups were inappropriate to induce any kind of self-assembly, a form of unspecific interaction of the viral proteins with carboxylic acids was found both for covalent and non-covalent methods. An exception was citric acid, since it was prone to an irreversible removal from the UCNP surface during the VLP preparation process. However, for all nanoparticles with a modified surface no clear evidence of encapsulation was found as confirmed by both TEM imaging and immunogold staining. The second research focus of this thesis was the development of a new imaging method of UCNPs. At first, different imaging parameters of the scan mode of a Hidex Plate Chameleon Multilabel Detection Platform with a 980 nm excitation source were studied and optimized for UCNPs of the type NaYF4: Yb3+, Er3+ or NaY4:Yb3+, Tm3+. A collecting time from 250 to 500 ms was sufficient to obtain an upconversion emission signal distinguishable from the background and to minimize laser-induced damages to the sample due to heating effects. Furthermore, both the overall emission of the UCNPs and the emission in a narrow wavelength domain were suitable as the detection signal for the image acquisition. The respective filter or filter combination additionally influenced the signal-to-noise ratio and thus the detection sensitivity. While the green emission of erbium-doped UCNPs was favorable in this regard, limitations of the detector sensitivity in the near infrared range hampered the utilization of the strong near infrared emission of thulium-doped UCNPs for scans with sensitive detection. The lateral resolution of the resulting scan images was reduced to 200 µm providing both a good resolution of luminescent structures and a good discrimination of signal and background. Since a lower scan point distance resulted in longer scan times without additional structural information, a lateral resolution of 200 µm was defined as the lower limit of the lateral resolution of the scan modus. Finally, a limit of detection of 1 ng was determined for both erbium- and thulium-doped UCNPs with high accuracy regardless of the emission utilized as the detection signal. The applicability of this optimized process to real samples was demonstrated for both electrophoresis gels and lateral flow assays. The imaging of these gels and assays showed a high accuracy and reproducibility and allowed a good discrimination of the UCNP signal from the background and an illustration of differences in the UCNP concentrations or sample materials. Furthermore, the downconversion emission of fluorescein doped in the silica shell of the UCNPs allowed a comparison between the imaging methods based either on up- and downconversion. In addition to a high conformity of the images regarding the position and intensity of the gel bands the absence of any signal from the fluorophore in the upconversion scan images of the gels confirmed a high discriminability of both signal types. This enables a bimodal readout. Moreover, the evaluation of the lateral flow assays yielded a limit of detection of the exemplary analyte "Schistosoma circulating anodic antigen (CAA)" of 44 or 64 pg/mL for the wet and dry condition of the array, respectively. For these measurements the the overall emission of the erbium-doped UCNPs was utilized as the detection signal. Consequently, a high sensitivity of this new imaging method was evident compared to other instrument options. Finally, the scan mode of the Chameleon reader was also utilized for studies about a potential enhancement of the upconversion emission by surface plasmon resonance of a gold surface. Both NaYF4: Yb3+, Er3+ or NaY4:Yb3+, Tm3+ were applied on a gold or silica surface of a commercial wafer with gold electrodes whose complete surfce was modified before with the same functionality to ensure the same chemical properties. The comparison of the emission intensities of UCNP on the different surface materials indicated a strong dependency of the enhancement effect on the emission wavelength. While the intensity of the green or near infrared emission of erbium- or thulium-doped UCNPs was increased by the gold surface, the overall emission of both UCNP types was reduced indicating a simultaneous quenching of the upconversion emission at other wavelengths

Surface modification and characterization of photon-upconverting nanoparticles for bioanalytical applications

Photon-upconverting nanoparticles (UCNPs) can be excited by near-infrared light and emit visible light (anti-Stokes emission) which prevents autofluorescence and light scattering of biological samples. The potential for background-free imaging has attracted wide interest in UCNPs in recent years. Small and homogeneous lanthanide-doped UCNPs that display high upconversion efficiency have typically been synthesized in organic solvents. Bioanalytical applications, however, require a subsequent phase transfer to aqueous solutions. Hence, the surface properties of UCNPs must be well designed and characterized to grant both a stable aqueous colloidal dispersion and the ability to conjugate biomolecules and other ligands on the nanoparticle surface. In this review, we introduce various routes for the surface modification of UCNPs and critically discuss their advantages and disadvantages. The last part covers various analytical methods that enable a thorough examination of the progress and success of the surface functionalization

Uncertainty-Based Out-of-Distribution Classification in Deep Reinforcement Learning

Robustness to out-of-distribution (OOD) data is an important goal in building reliable machine learning systems. Especially in autonomous systems, wrong predictions for OOD inputs can cause safety critical situations. As a first step towards a solution, we consider the problem of detecting such data in a value-based deep reinforcement learning (RL) setting. Modelling this problem as a one-class classification problem, we propose a framework for uncertainty-based OOD classification: UBOOD. It is based on the effect that an agent's epistemic uncertainty is reduced for situations encountered during training (in-distribution), and thus lower than for unencountered (OOD) situations. Being agnostic towards the approach used for estimating epistemic uncertainty, combinations with different uncertainty estimation methods, e.g. approximate Bayesian inference methods or ensembling techniques are possible. We further present a first viable solution for calculating a dynamic classification threshold, based on the uncertainty distribution of the training data. Evaluation shows that the framework produces reliable classification results when combined with ensemble-based estimators, while the combination with concrete dropout-based estimators fails to reliably detect OOD situations. In summary, UBOOD presents a viable approach for OOD classification in deep RL settings by leveraging the epistemic uncertainty of the agent's value function.Comment: arXiv admin note: text overlap with arXiv:1901.0221

Bayesian Surprise in Indoor Environments

This paper proposes a novel method to identify unexpected structures in 2D floor plans using the concept of Bayesian Surprise. Taking into account that a person's expectation is an important aspect of the perception of space, we exploit the theory of Bayesian Surprise to robustly model expectation and thus surprise in the context of building structures. We use Isovist Analysis, which is a popular space syntax technique, to turn qualitative object attributes into quantitative environmental information. Since isovists are location-specific patterns of visibility, a sequence of isovists describes the spatial perception during a movement along multiple points in space. We then use Bayesian Surprise in a feature space consisting of these isovist readings. To demonstrate the suitability of our approach, we take "snapshots" of an agent's local environment to provide a short list of images that characterize a traversed trajectory through a 2D indoor environment. Those fingerprints represent surprising regions of a tour, characterize the traversed map and enable indoor LBS to focus more on important regions. Given this idea, we propose to use "surprise" as a new dimension of context in indoor location-based services (LBS). Agents of LBS, such as mobile robots or non-player characters in computer games, may use the context surprise to focus more on important regions of a map for a better use or understanding of the floor plan.Comment: 10 pages, 16 figure

The scenario coevolution paradigm: adaptive quality assurance for adaptive systems

Systems are becoming increasingly more adaptive, using techniques like machine learning to enhance their behavior on their own rather than only through human developers programming them. We analyze the impact the advent of these new techniques has on the discipline of rigorous software engineering, especially on the issue of quality assurance. To this end, we provide a general description of the processes related to machine learning and embed them into a formal framework for the analysis of adaptivity, recognizing that to test an adaptive system a new approach to adaptive testing is necessary. We introduce scenario coevolution as a design pattern describing how system and test can work as antagonists in the process of software evolution. While the general pattern applies to large-scale processes (including human developers further augmenting the system), we show all techniques on a smaller-scale example of an agent navigating a simple smart factory. We point out new aspects in software engineering for adaptive systems that may be tackled naturally using scenario coevolution. This work is a substantially extended take on Gabor et al. (International symposium on leveraging applications of formal methods, Springer, pp 137–154, 2018)

Association between physical activity, grip strength and sedentary behaviour with incidence of malignant melanoma: results from the UK Biobank

Background Physical activity has been positively related to malignant melanoma. However, that association may be confounded by ultraviolet radiation (UV), a variable closely related to both outdoor physical activity and malignant melanoma. We examined physical activity, grip strength and sedentary behaviour in relation to risk of malignant melanoma, accounting for relevant confounders using data from a prospective cohort study. Methods In 350,512 UK Biobank participants aged 38–73 years at baseline, physical activity was assessed with a modified version of the International Physical Activity Questionnaire Short Form, grip strength was measured with a hand dynamometer, and sedentary behaviour was recorded with three specific questions. Multivariable hazard ratios (HR) and corresponding 95% confidence intervals (CI) were estimated using Cox proportional hazards regression. Results During 7 years of follow-up, 1239 incident malignant melanoma diagnoses were recorded. Physical activity and sedentary behaviour were unrelated to malignant melanoma (HRs 1.01 (95% CI 0.95–1.07) and 1.04 (95% CI 0.97–1.12), respectively), and the initially positive association with grip strength in the basic model (HR 1.23, 95% CI 1.08–1.40) was attenuated after full adjustment (HR 1.10, 95% CI 0.96–1.26). Conclusion Physical activity, grip strength and sedentary behaviour are not associated with malignant melanoma risk

Glutaminyl cyclase-mediated toxicity of pyroglutamate-beta amyloid induces striatal neurodegeneration

Background Posttranslational modifications of beta amyloid (Aβ) have been shown to affect its biophysical and neurophysiological properties. One of these modifications is N-terminal pyroglutamate (pE) formation. Enzymatic glutaminyl cyclase (QC) activity catalyzes cyclization of truncated Aβ(3-x), generating pE3-Aβ. Compared to unmodified Aβ, pE3-Aβ is more hydrophobic and neurotoxic. In addition, it accelerates aggregation of other Aβ species. To directly investigate pE3-Aβ formation and toxicity in vivo, transgenic (tg) ETNA (E at the truncated N-terminus of Aβ) mice expressing truncated human Aβ(3–42) were generated and comprehensively characterized. To further investigate the role of QC in pE3-Aβ formation in vivo, ETNA mice were intercrossed with tg mice overexpressing human QC (hQC) to generate double tg ETNA-hQC mice. Results Expression of truncated Aβ(3–42) was detected mainly in the lateral striatum of ETNA mice, leading to progressive accumulation of pE3-Aβ. This ultimately resulted in astrocytosis, loss of DARPP-32 immunoreactivity, and neuronal loss at the sites of pE3-Aβ formation. Neuropathology in ETNA mice was associated with behavioral alterations. In particular, hyperactivity and impaired acoustic sensorimotor gating were detected. Double tg ETNA-hQC mice showed similar Aβ levels and expression sites, while pE3-Aβ were significantly increased, entailing increased astrocytosis and neuronal loss. Conclusions ETNA and ETNA-hQC mice represent novel mouse models for QC-mediated toxicity of truncated and pE-modified Aβ. Due to their significant striatal neurodegeneration these mice can also be used for analysis of striatal regulation of basal locomotor activity and sensorimotor gating, and possibly for DARPP-32-dependent neurophysiology and neuropathology. The spatio-temporal correlation of pE3-Aβ and neuropathology strongly argues for an important role of this Aβ species in neurodegenerative processes in these models