Robotic Manipulation under Transparency and Translucency from Light-field Sensing

From frosted windows to plastic containers to refractive fluids, transparency and translucency are prevalent in human environments. The material properties of translucent objects challenge many of our assumptions in robotic perception. For example, the most common RGB-D sensors require the sensing of an infrared structured pattern from a Lambertian reflectance of surfaces. As such, transparent and translucent objects often remain invisible to robot perception. Thus, introducing methods that would enable robots to correctly perceive and then interact with the environment would be highly beneficial. Light-field (or plenoptic) cameras, for instance, which carry light direction and intensity, make it possible to perceive visual clues on transparent and translucent objects. In this dissertation, we explore the inference of transparent and translucent objects from plenoptic observations for robotic perception and manipulation. We propose a novel plenoptic descriptor, Depth Likelihood Volume (DLV), that incorporates plenoptic observations to represent depth of a pixel as a distribution rather than a single value. Building on the DLV, we present the Plenoptic Monte Carlo Localization algorithm, PMCL, as a generative method to infer 6-DoF poses of objects in settings with translucency. PMCL is able to localize both isolated transparent objects and opaque objects behind translucent objects using a DLV computed from a single view plenoptic observation. The uncertainty induced by transparency and translucency for pose estimation increases greatly as scenes become more cluttered. Under this scenario, we propose GlassLoc to localize feasible grasp poses directly from local DLV features. In GlassLoc, a convolutional neural network is introduced to learn DLV features for classifying grasp poses with grasping confidence. GlassLoc also suppresses the reflectance over multi-view plenoptic observations, which leads to more stable DLV representation. We evaluate GlassLoc in the context of a pick-and-place task for transparent tableware in a cluttered tabletop environment. We further observe that the transparent and translucent objects will generate distinguishable features in the light-field epipolar image plane. With this insight, we propose Light-field Inference of Transparency, LIT, as a two-stage generative-discriminative refractive object localization approach. In the discriminative stage, LIT uses convolutional neural networks to learn reflection and distortion features from photorealistic-rendered light-field images. The learned features guide generative object location inference through local depth estimation and particle optimization. We compare LIT with four state-of-the-art pose estimators to show our efficacy in the transparent object localization task. We perform a robot demonstration by building a champagne tower using the LIT pipeline.PHDRoboticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/169707/1/zhezhou_1.pd

Semantic mapping for service robots: building and using maps for mobile manipulators in semi-structured environments

Although much progress has been made in the field of robotic mapping, many challenges remain including: efficient semantic segmentation using RGB-D sensors, map representations that include complex features (structures and objects), and interfaces for interactive annotation of maps. This thesis addresses how prior knowledge of semi-structured human environments can be leveraged to improve segmentation, mapping, and semantic annotation of maps. We present an organized connected component approach for segmenting RGB-D data into planes and clusters. These segments serve as input to our mapping approach that utilizes them as planar landmarks and object landmarks for Simultaneous Localization and Mapping (SLAM), providing necessary information for service robot tasks and improving data association and loop closure. These features are meaningful to humans, enabling annotation of mapped features to establish common ground and simplifying tasking. A modular, open-source software framework, the OmniMapper, is also presented that allows a number of different sensors and features to be combined to generate a combined map representation, and enabling easy addition of new feature types.Ph.D

Proceedings of the 2011 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Laboratory

This book is a collection of 15 reviewed technical reports summarizing the presentations at the 2011 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Laboratory. The covered topics include image processing, optical signal processing, visual inspection, pattern recognition and classification, human-machine interaction, world and situation modeling, autonomous system localization and mapping, information fusion, and trust propagation in sensor networks

The role of allostery in nNOS and SynGAP activity

Understanding the molecular basis of neuroplasticity is an ambition of neuroscience. SynGAP and nNOS are two neuronal proteins which are central regulators of neuroplasticity. nNOS synthesizes nitric oxide, a second messenger important for retrograde communication between synapses, while SynGAP stimulates the GTPase activity of Ras and Rap. Regulation of these enzymes is vital for synaptic homeostasis, but there is much to learn about the details of nNOS and SynGAP activity. In this dissertation I address two questions about nNOS and SynGAP structural biology: what is the structural basis for nNOS activation by calmodulin? How do the auxiliary domains of SynGAP enable RapGAP activity? Both nNOS and SynGAP are large proteins with modular domain organization and flexible linkers. These features make structural analysis by mainstay techniques such as NMR or X-Ray crystallography excessively difficult. Hydrogen-deuterium exchange mass spectrometry (HDX-MS) is well-suited for investigating the structure-activity relationships in proteins such as nNOS and SynGAP. Hydrogen-deuterium exchange of amide protons with the solvent, as measured in HDX-MS, provides a wealth of information on the structural dynamics of proteins. Mass spectrometry in combination with proteolytic digestion of samples enables assignment of those perturbation to discrete peptides within the protein. Using this approach, I demonstrate a role for allosteric effects in both activation of nNOS by calmodulin and the RapGAP function of SynGAP

Context-driven Object Detection and Segmentation with Auxiliary Information

One fundamental problem in computer vision and robotics is to localize objects of interest in an image. The task can either be formulated as an object detection problem if the objects are described by a set of pose parameters, or an object segmentation one if we recover object boundary precisely. A key issue in object detection and segmentation concerns exploiting the spatial context, as local evidence is often insufficient to determine object pose in the presence of heavy occlusions or large object appearance variations. This thesis addresses the object detection and segmentation problem in such adverse conditions with auxiliary depth data provided by RGBD cameras. We focus on four main issues in context-aware object detection and segmentation: 1) what are the effective context representations? 2) how can we work with limited and imperfect depth data? 3) how to design depth-aware features and integrate depth cues into conventional visual inference tasks? 4) how to make use of unlabeled data to relax the labeling requirements for training data? We discuss three object detection and segmentation scenarios based on varying amounts of available auxiliary information. In the first case, depth data are available for model training but not available for testing. We propose a structured Hough voting method for detecting objects with heavy occlusion in indoor environments, in which we extend the Hough hypothesis space to include both the object's location, and its visibility pattern. We design a new score function that accumulates votes for object detection and occlusion prediction. In addition, we explore the correlation between objects and their environment, building a depth-encoded object-context model based on RGBD data. In the second case, we address the problem of localizing glass objects with noisy and incomplete depth data. Our method integrates the intensity and depth information from a single view point, and builds a Markov Random Field that predicts glass boundary and region jointly. In addition, we propose a nonparametric, data-driven label transfer scheme for local glass boundary estimation. A weighted voting scheme based on a joint feature manifold is adopted to integrate depth and appearance cues, and we learn a distance metric on the depth-encoded feature manifold. In the third case, we make use of unlabeled data to relax the annotation requirements for object detection and segmentation, and propose a novel data-dependent margin distribution learning criterion for boosting, which utilizes the intrinsic geometric structure of datasets. One key aspect of this method is that it can seamlessly incorporate unlabeled data by including a graph Laplacian regularizer. We demonstrate the performance of our models and compare with baseline methods on several real-world object detection and segmentation tasks, including indoor object detection, glass object segmentation and foreground segmentation in video

Senseable Spaces: from a theoretical perspective to the application in augmented environments

Grazie all’ enorme diffusione di dispositivi senzienti nella vita di tutti i giorni, nell’ ultimo decennio abbiamo assistito ad un cambio definitivo nel modo in cui gli utenti interagiscono con lo spazio circostante. Viene coniato il termine Spazio Sensibile, per descrivere quegli spazi in grado di fornire servizi contestuali agli utenti, misurando e analizzando le dinamiche che in esso avvengono, e di reagire conseguentemente a questo continuo flusso di dati bidirezionale. La ricerca è stata condotta abbracciando diversi domini di applicazione, le cui singole esigenze hanno reso necessario testare il concetto di Spazi Sensibili in diverse declinazioni, mantenendo al centro della ricerca l’utente, con la duplice accezione di end-user e manager. Molteplici sono i contributi rispetto allo stato dell’ arte. Il concetto di Spazio Sensibile è stato calato nel settore dei Beni Culturali, degli Spazi Pubblici, delle Geosciences e del Retail. I casi studio nei musei e nella archeologia dimostrano come l’ utilizzo della Realtà Aumentata possa essere sfruttata di fronte a un dipinto o in outdoor per la visualizzazione di modelli complessi, In ambito urbano, il monitoraggio di dati generati dagli utenti ha consentito di capire le dinamiche di un evento di massa, durante il quale le stesse persone fruivano di servizi contestuali. Una innovativa applicazione di Realtà Aumentata è stata come servizio per facilitare l’ ispezione di fasce tampone lungo i fiumi, standardizzando flussi di dati e modelli provenienti da un Sistema Informativo Territoriale. Infine, un robusto sistema di indoor localization è stato istallato in ambiente retail, per scopi classificazione dei percorsi e per determinare le potenzialità di un punto vendita. La tesi è inoltre una dimostrazione di come Space Sensing e Geomatica siano discipline complementari: la geomatica consente di acquisire e misurare dati geo spaziali e spazio temporali a diversa scala, lo Space Sensing utilizza questi dati per fornire servizi all’ utente precisi e contestuali.Given the tremendous growth of ubiquitous services in our daily lives, during the last few decades we have witnessed a definitive change in the way users' experience their surroundings. At the current state of art, devices are able to sense the environment and users’ location, enabling them to experience improved digital services, creating synergistic loop between the use of the technology, and the use of the space itself. We coined the term Senseable Space, to define the kinds of spaces able to provide users with contextual services, to measure and analyse their dynamics and to react accordingly, in a seamless exchange of information. Following the paradigm of Senseable Spaces as the main thread, we selected a set of experiences carried out in different fields; central to this investigation there is of course the user, placed in the dual roles of end-user and manager. The main contribution of this thesis lies in the definition of this new paradigm, realized in the following domains: Cultural Heritage, Public Open Spaces, Geosciences and Retail. For the Cultural Heritage panorama, different pilot projects have been constructed from creating museum based installations to developing mobile applications for archaeological settings. Dealing with urban areas, app-based services are designed to facilitate the route finding in a urban park and to provide contextual information in a city festival. We also outlined a novel application to facilitate the on-site inspection by risk managers thanks to the use of Augmented Reality services. Finally, a robust indoor localization system has been developed, designed to ease customer profiling in the retail sector. The thesis also demonstrates how Space Sensing and Geomatics are complementary to one another, given the assumption that the branches of Geomatics cover all the different scales of data collection, whilst Space Sensing gives one the possibility to provide the services at the correct location, at the correct time

Senseable Spaces: from a theoretical perspective to the application in augmented environments

openGrazie all’ enorme diffusione di dispositivi senzienti nella vita di tutti i giorni, nell’ ultimo decennio abbiamo assistito ad un cambio definitivo nel modo in cui gli utenti interagiscono con lo spazio circostante. Viene coniato il termine Spazio Sensibile, per descrivere quegli spazi in grado di fornire servizi contestuali agli utenti, misurando e analizzando le dinamiche che in esso avvengono, e di reagire conseguentemente a questo continuo flusso di dati bidirezionale. La ricerca è stata condotta abbracciando diversi domini di applicazione, le cui singole esigenze hanno reso necessario testare il concetto di Spazi Sensibili in diverse declinazioni, mantenendo al centro della ricerca l’utente, con la duplice accezione di end-user e manager. Molteplici sono i contributi rispetto allo stato dell’ arte. Il concetto di Spazio Sensibile è stato calato nel settore dei Beni Culturali, degli Spazi Pubblici, delle Geosciences e del Retail. I casi studio nei musei e nella archeologia dimostrano come l’ utilizzo della Realtà Aumentata possa essere sfruttata di fronte a un dipinto o in outdoor per la visualizzazione di modelli complessi, In ambito urbano, il monitoraggio di dati generati dagli utenti ha consentito di capire le dinamiche di un evento di massa, durante il quale le stesse persone fruivano di servizi contestuali. Una innovativa applicazione di Realtà Aumentata è stata come servizio per facilitare l’ ispezione di fasce tampone lungo i fiumi, standardizzando flussi di dati e modelli provenienti da un Sistema Informativo Territoriale. Infine, un robusto sistema di indoor localization è stato istallato in ambiente retail, per scopi classificazione dei percorsi e per determinare le potenzialità di un punto vendita. La tesi è inoltre una dimostrazione di come Space Sensing e Geomatica siano discipline complementari: la geomatica consente di acquisire e misurare dati geo spaziali e spazio temporali a diversa scala, lo Space Sensing utilizza questi dati per fornire servizi all’ utente precisi e contestuali.Given the tremendous growth of ubiquitous services in our daily lives, during the last few decades we have witnessed a definitive change in the way users' experience their surroundings. At the current state of art, devices are able to sense the environment and users’ location, enabling them to experience improved digital services, creating synergistic loop between the use of the technology, and the use of the space itself. We coined the term Senseable Space, to define the kinds of spaces able to provide users with contextual services, to measure and analyse their dynamics and to react accordingly, in a seamless exchange of information. Following the paradigm of Senseable Spaces as the main thread, we selected a set of experiences carried out in different fields; central to this investigation there is of course the user, placed in the dual roles of end-user and manager. The main contribution of this thesis lies in the definition of this new paradigm, realized in the following domains: Cultural Heritage, Public Open Spaces, Geosciences and Retail. For the Cultural Heritage panorama, different pilot projects have been constructed from creating museum based installations to developing mobile applications for archaeological settings. Dealing with urban areas, app-based services are designed to facilitate the route finding in a urban park and to provide contextual information in a city festival. We also outlined a novel application to facilitate the on-site inspection by risk managers thanks to the use of Augmented Reality services. Finally, a robust indoor localization system has been developed, designed to ease customer profiling in the retail sector. The thesis also demonstrates how Space Sensing and Geomatics are complementary to one another, given the assumption that the branches of Geomatics cover all the different scales of data collection, whilst Space Sensing gives one the possibility to provide the services at the correct location, at the correct time.INGEGNERIA DELL'INFORMAZIONEembargoed_20181001Pierdicca, RobertoPierdicca, Robert

MECHANISMS AND CYTOTOXIC EFFECTS OF NQO1-DIRECTED LAVENDAMYCIN DERIVATIVES AND MITOCHONDRIA-TARGETED ANTHRACENYL ISOXAZOLE AMIDES AS NOVEL ANTITUMOR AGENTS

Cancer is a common, complex, and oftentimes fatal disease. Despite extensive research in the field of cancer drug discovery, there are still improvements to be made in the design of effective anticancer agents. This project involved three separate but related studies that fall under the category of anticancer drug discovery as a whole. The overall goal of this project was to design and investigate the mechanisms of action of new antitumor agents to be used against solid tumors. First, we developed a series of anthracenyl isoxazole amides (AIMs) designed to bind to G-quadruplex DNA and inhibit telomerase. Results from this study demonstrated an alternative mitochondrial mechanism of action of the AIMs not yet fully described in the literature. Investigation of lead compound AIM 1 showed localization of the AIM in mitochondria with resulting induction of apoptosis, generation of mitochondrial superoxide, disruption of mitochondrial membrane potential, and activation of caspase-9. The second goal of this project was to assay a series of 4-isoxazolyl-1,4-dihydropyridines (IDHPs) that function as P-gp inhibitors to determine their contribution to enhanced cytotoxicity of the AIMs when co-dosed together in vitro. Because so many anticancer agents are substrates for P-gp and are therefore limited in their ability to reach intended targets, the development of P-gp inhibitors is an important area of research. Results from this study indicate that IDHPs are a viable class of P-gp inhibitors that can be co-dosed with P-gp substrates to increase substrate cytotoxicity. The third goal of this study was to determine the NQO1 substrate potential of a series of lavendamycin derivative quinolinequinones and assess their corresponding antitumor potential. Surprisingly, few of the quinolinequinones tested showed preferential specificity for NQO1-expressing cells compared to NQO1-null cells. However, in our series of aryl-substituted quinolinequinones, the active molecules appear to be the quinone derivatives and not derivatives of hydroquinones or semiquinones. These data suggest a mode of action that differs from that of previously studied lavendamycin analogues that are activated by NQO1 reduction. While this project focused on the general targeting of solid tumors, the type of tumor explicitly studied varied from brain cancer to breast cancer and encompassed multiple drug targets. Collectively the results of this study are expansive and offer much to the field of anticancer drug discovery