Envisioning the qualitative effects of robot manipulation actions using simulation-based projections

Autonomous robots that are to perform complex everyday tasks such as making pancakes have to understand how the effects of an action depend on the way the action is executed. Within Artificial Intelligence, classical planning reasons about whether actions are executable, but makes the assumption that the actions will succeed (with some probability). In this work, we have designed, implemented, and analyzed a framework that allows us to envision the physical effects of robot manipulation actions. We consider envisioning to be a qualitative reasoning method that reasons about actions and their effects based on simulation-based projections. Thereby it allows a robot to infer what could happen when it performs a task in a certain way. This is achieved by translating a qualitative physics problem into a parameterized simulation problem; performing a detailed physics-based simulation of a robot plan; logging the state evolution into appropriate data structures; and then translating these sub-symbolic data structures into interval-based first-order symbolic, qualitative representations, called timelines. The result of the envisioning is a set of detailed narratives represented by timelines which are then used to infer answers to qualitative reasoning problems. By envisioning the outcome of actions before committing to them, a robot is able to reason about physical phenomena and can therefore prevent itself from ending up in unwanted situations. Using this approach, robots can perform manipulation tasks more efficiently, robustly, and flexibly, and they can even successfully accomplish previously unknown variations of tasks

Model-Based Environmental Visual Perception for Humanoid Robots

The visual perception of a robot should answer two fundamental questions: What? and Where? In order to properly and efficiently reply to these questions, it is essential to establish a bidirectional coupling between the external stimuli and the internal representations. This coupling links the physical world with the inner abstraction models by sensor transformation, recognition, matching and optimization algorithms. The objective of this PhD is to establish this sensor-model coupling

Investigating moral judgements in autistic children: integrating the observer’s and the speaker’s mind

This study investigates the moral judgements that autistic children make in everyday situations. Moral Transgressions (MT) and Faux Pas (FP) stories were compared, in which stories the agent’s morality and intention varied (MT: bad, FP: good), and were divided by the mediator (personal / material). Thirty autistic and 32 neurotypical children answered forced-choice questions. The two groups did not differ significantly when responding to either the MT or the FP questions. In between-group comparisons, the autistic group found difficulties in understanding the MT stories when the action directly affected another person (personal mediator). Comparisons between agent’s morality and intention variables revealed that autistic children judged the morality of the agent in FP stories as severe as in the MT task, even when the agent’s intention was understood. These subtle problems could shed some light on how autistic individuals would judge social situations, from the lack of a robust ToM to difficulties being socially flexible

Categorization of Affordances and Prediction of Future Object Interactions using Qualitative Spatial Relations

The application of deep neural networks on robotic platforms has successfully advanced robot perception in tasks related to human-robot collaboration scenarios. Tasks such as scene understanding, object categorization, affordance detection, interaction anticipation, are facilitated by the acquisition of knowledge about the object interactions taking place in the scene. The contributions of this thesis are two-fold: 1) it shows how representations of object interactions learned in an unsupervised way can be used to predict categories of objects depending on the affordances; 2) it shows how future frame-independent interaction can be learned in a self-supervised way by exploiting high-level graph representations of the object interactions. The aim of this research is to create representations and perform predictions of interactions which abstract from the image space and attain generalization across various scenes and objects. Interactions can be static, eg. holding a bottle, as well as dynamic, eg. playing with a ball, where the temporal aspect of the sequence of several static interactions is of importance to make the dynamic interaction distinguishable. Moreover, occlusion of objects in the 2D domain should be handled to avoid false positive interaction detections. Thus, RGB-D video data is exploited for these tasks. As humans tend to use objects in many different ways depending on the scene and the objects' availability, learning object affordances in everyday-life scenarios is a challenging task, particularly in the presence of an open-set of interactions and class-agnostic objects. In order to abstract from the continuous representation of spatio-temporal interactions in video data, a novel set of high-level qualitative depth-informed spatial relations is presented. Learning similarities via an unsupervised method exploiting graph representations of object interactions induces a hierarchy of clusters of objects with similar affordances. The proposed method handles object occlusions by capturing effectively possible interactions and without imposing any object or scene constraints. Moreover, interaction and action anticipation remains a challenging problem, especially considering the generalizability constraints of trained models from visual data or exploiting visual video embeddings. State of the art methods allow predictions approximately up to three seconds of time in the future. Hence, most everyday-life activities, which consist of actions of more than five seconds in duration, are not predictable. This thesis presents a novel approach for solving the task of interaction anticipation between objects in a video scene by utilizing high-level qualitative frame-number-independent spatial graphs to represent object interactions. A deep recurrent neural network learns in a self-supervised way to predict graph structures of future object interactions, whilst being decoupled from the visual information, the underlying activity, and the duration of each interaction taking place. Finally, the proposed methods are evaluated on RGB-D video datasets capturing everyday-life activities of human agents, and are compared against closely-related and state-of-the-art methods

User producer interaction in context: a classification

Science, Technology and Innovation Studies show that intensified user producer interaction (UPI) increases chances for successful innovations, especially in the case of emerging technology. It is not always clear, however, what type of interaction is necessary in a particular context. This paper proposes a conceptualization of contexts in terms of three dimensions – the phase of technology development, the flexibility of the technology, and the heterogeneity of user populations – resulting in a classification scheme with eight different contextual situations. The paper identifies and classifies types of interaction, like demand articulation, interactive learning, learning by using and domestication. It appears that each contextual situation demands a different set of UPI types. To illustrate the potential value of the classification scheme, four examples of innovations with varying technological and user characteristics are explored: the refrigerator, clinical anaesthesia, video cassette recording, and the bicycle. For each example the relevant UPI types are discussed and it is shown how these types highlight certain activities and interactions during key events of innovation processes. Finally, some directions for further research are suggested alongside a number of comments on the utility of the classification

User producer interaction in context: A classification

Science, Technology and Innovation Studies show that intensified user producer interaction (UPI) increases chances for successful innovations, especially in the case of emerging technology. It is not always clear, however, what type of interaction is necessary in a particular context. This paper proposes a conceptualization of contexts in terms of three dimensions – the phase of technology development, the flexibility of the technology, and the heterogeneity of user populations – resulting in a classification scheme with eight different contextual situations. The paper identifies and classifies types of interaction, like demand articulation, interactive learning, learning by using and domestication. It appears that each contextual situation demands a different set of UPI types. To illustrate the potential value of the classification scheme, four examples of innovations with varying technological and user characteristics are explored: the refrigerator, clinical anaesthesia, video cassette recording, and the bicycle. For each example the relevant UPI types are discussed and it is shown how these types highlight certain activities and interactions during key events of innovation processes. Finally, some directions for further research are suggested alongside a number of comments on the utility of the classification.Innovation, users, interaction, learning, typology of UPI

Taxonomy of Human Actions for Action-based Learning Assessment in Virtual Training Environments

This design research project developed and validated a taxonomy of human actions to be used in action-based learning assessment. The taxonomy, titled ‘BEHAVE,’ was shown to have both internal and external validity and allows actions performed by learners, for example in digital performance spaces, to be formally represented with consistency and to be compared with expert reference actions, to generate automated post-performance formative feedback

The robot's vista space : a computational 3D scene analysis

Swadzba A. The robot's vista space : a computational 3D scene analysis. Bielefeld (Germany): Bielefeld University; 2011.The space that can be explored quickly from a fixed view point without locomotion is known as the vista space. In indoor environments single rooms and room parts follow this definition. The vista space plays an important role in situations with agent-agent interaction as it is the directly surrounding environment in which the interaction takes place. A collaborative interaction of the partners in and with the environment requires that both partners know where they are, what spatial structures they are talking about, and what scene elements they are going to manipulate. This thesis focuses on the analysis of a robot's vista space. Mechanisms for extracting relevant spatial information are developed which enable the robot to recognize in which place it is, to detect the scene elements the human partner is talking about, and to segment scene structures the human is changing. These abilities are addressed by the proposed holistic, aligned, and articulated modeling approach. For a smooth human-robot interaction, the computed models should be aligned to the partner's representations. Therefore, the design of the computational models is based on the combination of psychological results from studies on human scene perception with basic physical properties of the perceived scene and the perception itself. The holistic modeling realizes a categorization of room percepts based on the observed 3D spatial layout. Room layouts have room type specific features and fMRI studies have shown that some of the human brain areas being active in scene recognition are sensitive to the 3D geometry of a room. With the aligned modeling, the robot is able to extract the hierarchical scene representation underlying a scene description given by a human tutor. Furthermore, it is able to ground the inferred scene elements in its own visual perception of the scene. This modeling follows the assumption that cognition and language schematize the world in the same way. This is visible in the fact that a scene depiction mainly consists of relations between an object and its supporting structure or between objects located on the same supporting structure. Last, the articulated modeling equips the robot with a methodology for articulated scene part extraction and fast background learning under short and disturbed observation conditions typical for human-robot interaction scenarios. Articulated scene parts are detected model-less by observing scene changes caused by their manipulation. Change detection and background learning are closely coupled because change is defined phenomenologically as variation of structure. This means that change detection involves a comparison of currently visible structures with a representation in memory. In range sensing this comparison can be nicely implement as subtraction of these two representations. The three modeling approaches enable the robot to enrich its visual perceptions of the surrounding environment, the vista space, with semantic information about meaningful spatial structures useful for further interaction with the environment and the human partner

Human factors of ubiquitous computing: ambient cueing in the digital kitchen?

This thesis is concerned with the uses of Ubiquitous Computing (UbiComp) in everyday domestic environments. The concept of UbiComp promises to shift computing away from the desktop into everyday objects and settings. It has the twin goals of providing ‘transparent’ technologies where the information has been thoroughly embedded into everyday activities and objects (thus making the computer invisible to the user) and also (and more importantly) of seamless integration of these technologies into the activities of their users. However, this raises the challenge of how best to support interaction with a ‘transparent’ or ‘invisible’ technology; if the technology is made visible, it will attract the user's attention to it and away from the task at hand, but if it is hidden, then how can the user cope with malfunctions or other problems in the technology? We approach the design of Human-Computer Interaction in the ubiquitous environment through the use of ambient displays, i.e. the use of subtle cueing, embedded in the environment which is intended to guide human activity. This thesis draws on the concept of stimulus-response compatibility and applies this to the design ambient display. This thesis emphasizes the need to understand the users’ perspectives and responses in any particular approach that has been proposed. Therefore, the main contributions of this thesis focus on approaches to improve human performance in the ubiquitous environment through ambient display