Reasoning about Geometric Object Interactions in 3D for Manipulation Action Understanding

In order to efficiently interact with human users, intelligent agents and autonomous systems need the ability of interpreting human actions. We focus our attention on manipulation actions, wherein an agent typically grasps an object and moves it, possibly altering its physical state. Agent-object and object-object interactions during a manipulation are a defining part of the performed action itself. In this thesis, we focus on extracting semantic cues, derived from geometric object interactions in 3D space during a manipulation, that are useful for action understanding at the cognitive level. First, we introduce a simple grounding model for the most common pairwise spatial relations between objects and investigate the descriptive power of their temporal evolution for action characterization. We propose a compact, abstract action descriptor that encodes the geometric object interactions during action execution, as captured by the spatial relation dynamics. Our experiments on a diverse dataset confirm both the validity and effectiveness of our spatial relation models and the discriminative power of our representation with respect to the underlying action semantics. Second, we model and detect lower level interactions, namely object contacts and separations, viewing them as topological scene changes within a dense motion estimation setting. In addition to improving motion estimation accuracy in the challenging case of motion boundaries induced by these events, our approach shows promising performance in the explicit detection and classification of the latter. Building upon dense motion estimation and using detected contact events as an attention mechanism, we propose a bottom-up pipeline for the guided segmentation and rigid motion extraction of manipulated objects. Finally, in addition to our methodological contributions, we introduce a new open-source software library for point cloud data processing, developed for the needs of this thesis, which aims at providing an easy to use, flexible, and efficient framework for the rapid development of performant software for a range of 3D perception tasks

Mobile Wound Assessment and 3D Modeling from a Single Image

The prevalence of camera-enabled mobile phones have made mobile wound assessment a viable treatment option for millions of previously difficult to reach patients. We have designed a complete mobile wound assessment platform to ameliorate the many challenges related to chronic wound care. Chronic wounds and infections are the most severe, costly and fatal types of wounds, placing them at the center of mobile wound assessment. Wound physicians assess thousands of single-view wound images from all over the world, and it may be difficult to determine the location of the wound on the body, for example, if the wound is taken at close range. In our solution, end-users capture an image of the wound by taking a picture with their mobile camera. The wound image is segmented and classified using modern convolution neural networks, and is stored securely in the cloud for remote tracking. We use an interactive semi-automated approach to allow users to specify the location of the wound on the body. To accomplish this we have created, to the best our knowledge, the first 3D human surface anatomy labeling system, based off the current NYU and Anatomy Mapper labeling systems. To interactively view wounds in 3D, we have presented an efficient projective texture mapping algorithm for texturing wounds onto a 3D human anatomy model. In so doing, we have demonstrated an approach to 3D wound reconstruction that works even for a single wound image

Modeling and Simulation in Engineering

This book provides an open platform to establish and share knowledge developed by scholars, scientists, and engineers from all over the world, about various applications of the modeling and simulation in the design process of products, in various engineering fields. The book consists of 12 chapters arranged in two sections (3D Modeling and Virtual Prototyping), reflecting the multidimensionality of applications related to modeling and simulation. Some of the most recent modeling and simulation techniques, as well as some of the most accurate and sophisticated software in treating complex systems, are applied. All the original contributions in this book are jointed by the basic principle of a successful modeling and simulation process: as complex as necessary, and as simple as possible. The idea is to manipulate the simplifying assumptions in a way that reduces the complexity of the model (in order to make a real-time simulation), but without altering the precision of the results

Contributions to the cornerstones of interaction in visualization: strengthening the interaction of visualization

Visualization has become an accepted means for data exploration and analysis. Although interaction is an important component of visualization approaches, current visualization research pays less attention to interaction than to aspects of the graphical representation. Therefore, the goal of this work is to strengthen the interaction side of visualization. To this end, we establish a unified view on interaction in visualization. This unified view covers four cornerstones: the data, the tasks, the technology, and the human.Visualisierung hat sich zu einem unverzichtbaren Werkzeug für die Exploration und Analyse von Daten entwickelt. Obwohl Interaktion ein wichtiger Bestandteil solcher Werkzeuge ist, wird der Interaktion in der aktuellen Visualisierungsforschung weniger Aufmerksamkeit gewidmet als Aspekten der graphischen Repräsentation. Daher ist es das Ziel dieser Arbeit, die Interaktion im Bereich der Visualisierung zu stärken. Hierzu wird eine einheitliche Sicht auf Interaktion in der Visualisierung entwickelt

An Agile Roadmap for Live, Virtual and Constructive-Integrating Training Architecture (LVC-ITA): A Case Study Using a Component based Integrated Simulation Engine

Conducting seamless Live Virtual Constructive (LVC) simulation remains the most challenging issue of Modeling and Simulation (M&S). There is a lack of interoperability, limited reuse and loose integration between the Live, Virtual and/or Constructive assets across multiple Standard Simulation Architectures (SSAs). There have been various theoretical research endeavors about solving these problems but their solutions resulted in complex and inflexible integration, long user-usage time and high cost for LVC simulation. The goal of this research is to provide an Agile Roadmap for the Live Virtual Constructive-Integrating Training Architecture (LVC-ITA) that will address the above problems and introduce interoperable LVC simulation. Therefore, this research describes how the newest M&S technologies can be utilized for LVC simulation interoperability and integration. Then, we will examine the optimal procedure to develop an agile roadmap for the LVC-ITA. In addition, this research illustrated a case study using an Adaptive distributed parallel Simulation environment for Interoperable and reusable Model (AddSIM) that is a component based integrated simulation engine. The agile roadmap of the LVC-ITA that reflects the lessons learned from the case study will contribute to guide M&S communities to an efficient path to increase interaction of M&S simulation across systems

Design and implementation of an indoor modeling method through crowdsensing

While automatic modeling and mapping of outdoor environments is well-established, the indoor equivalent of automated generation of building floor plans poses a challenge. In fact, outdoor localization is commonly available and inexpensive through the existing satellite positioning systems, such as GPS and Galileo. However, these technologies are not applicable in indoor environments, since a direct line of sight to the satellites, orbiting the globes, is required. As a substitution, the technical literature comprises several proposals for the development of simultaneous indoor localization and mapping (SLAM). In these approaches, the authors mostly exploit indoor resources such as the WiFi access points and the mobile smart devices carried by individuals in the indoor environment. Collecting data from several mobile devices is referred to as crowdsensing. To enable the generation of two-dimensional (2D) as well as three-dimensional (3D) maps, we propose crowdsensing of point clouds, which are 3D data structures of points in space. For localization, we integrate two features of a recently developed mobile device, called Project Tango. Specifically, the Tango platform provides two main technologies for reliable localization, namely motion tracking and area learning. Moreover, Tango-powered devices provide us with the ability to collect point clouds though a third technology, called depth perception. In the past few years, spatial data obtained from range imaging was used to generate indoor maps. Nevertheless, range images are expensive and not always available. The required equipment, e.g. laser range scanners, are both expensive in procurement and require trained personnel for proper setup and operation. In this thesis, we aim for obtaining spatial point clouds via crowdsensing. The main idea is to use sensor data which can be scanned by volunteering individuals using easy to handle mobile devices. Specifically, we depend on depth perception capabilities as provided by Google Tango-powered tablet computers. A crowdsensing infrastructure assigns scanning tasks to individuals carrying a Tango device. Execution of such a task consists of taking scans of e.g. offices in a public building. The scanning results contain both spatial information about the room layout and its position. Energy consumption on the mobile device is reduced by applying Octree compression to the scanned point clouds, which results in a significant reduction of the amount of data, which has to be transferred to a back-end server. Afterwards, the back-end is responsible for assembling the received scans and the extraction of an indoors model. The modeling process - developed in this thesis - comprises two-phases. First, we extract a basic model from the obtained point clouds, which may contain outliers, inaccuracies and gaps. In the second phase, we refine the model by exploiting formal grammars. It is worth to mention here that we are the first to exploit formal grammars as a model fitting tool. We feed the information obtained in the first phase to an indoors grammar, which has been developed in the ComNSense project, University of Stuttgart. The resultant model both contains much less deviations from the ground truth and provides improved robustness against aberrations with respect to localization during the scanning process. Thus, instead of scanning multiple point clouds per room, we need only one scan to be able to construct an indoor map. During evaluation of this process, using scans of offices of our department, we were able to reproduce a model which is very close to the ground truth

Explorative Graph Visualization

Netzwerkstrukturen (Graphen) sind heutzutage weit verbreitet. Ihre Untersuchung dient dazu, ein besseres Verständnis ihrer Struktur und der durch sie modellierten realen Aspekte zu gewinnen. Die Exploration solcher Netzwerke wird zumeist mit Visualisierungstechniken unterstützt. Ziel dieser Arbeit ist es, einen Überblick über die Probleme dieser Visualisierungen zu geben und konkrete Lösungsansätze aufzuzeigen. Dabei werden neue Visualisierungstechniken eingeführt, um den Nutzen der geführten Diskussion für die explorative Graphvisualisierung am konkreten Beispiel zu belegen.Network structures (graphs) have become a natural part of everyday life and their analysis helps to gain an understanding of their inherent structure and the real-world aspects thereby expressed. The exploration of graphs is largely supported and driven by visual means. The aim of this thesis is to give a comprehensive view on the problems associated with these visual means and to detail concrete solution approaches for them. Concrete visualization techniques are introduced to underline the value of this comprehensive discussion for supporting explorative graph visualization