Multimodal Shared-Control Interaction for Mobile Robots in AAL Environments

This dissertation investigates the design, development and implementation of cognitively adequate, safe and robust, spatially-related, multimodal interaction between human operators and mobile robots in Ambient Assisted Living environments both from the theoretical and practical perspectives. By focusing on different aspects of the concept Interaction, the essential contribution of this dissertation is divided into three main research packages; namely, Formal Interaction, Spatial Interaction and Multimodal Interaction in AAL. As the principle package, in Formal Interaction, research effort is dedicated to developing a formal language based interaction modelling and management solution process and a unified dialogue modelling approach. This package aims to enable a robust, flexible, and context-sensitive, yet formally controllable and tractable interaction. This type of interaction can be used to support the interaction management of any complex interactive systems, including the ones covered in the other two research packages. In the second research package, Spatial Interaction, a general qualitative spatial knowledge based multi-level conceptual model is developed and proposed. The goal is to support a spatially-related interaction in human-robot collaborative navigation. With a model-based computational framework, the proposed conceptual model has been implemented and integrated into a practical interactive system which has been evaluated by empirical studies. It has been particularly tested with respect to a set of high-level and model-based conceptual strategies for resolving the frequent spatially-related communication problems in human-robot interaction. Last but not least, in Multimodal Interaction in AAL, attention is drawn to design, development and implementation of multimodal interaction for elderly persons. In this elderly-friendly scenario, ageing-related characteristics are carefully considered for an effective and efficient interaction. Moreover, a standard model based empirical framework for evaluating multimodal interaction is provided. This framework was especially applied to evaluate a minutely developed and systematically improved elderly-friendly multimodal interactive system through a series of empirical studies with groups of elderly persons

Towards gestural understanding for intelligent robots

Fritsch JN. Towards gestural understanding for intelligent robots. Bielefeld: Universität Bielefeld; 2012.A strong driving force of scientific progress in the technical sciences is the quest for systems that assist humans in their daily life and make their life easier and more enjoyable. Nowadays smartphones are probably the most typical instances of such systems. Another class of systems that is getting increasing attention are intelligent robots. Instead of offering a smartphone touch screen to select actions, these systems are intended to offer a more natural human-machine interface to their users. Out of the large range of actions performed by humans, gestures performed with the hands play a very important role especially when humans interact with their direct surrounding like, e.g., pointing to an object or manipulating it. Consequently, a robot has to understand such gestures to offer an intuitive interface. Gestural understanding is, therefore, a key capability on the way to intelligent robots. This book deals with vision-based approaches for gestural understanding. Over the past two decades, this has been an intensive field of research which has resulted in a variety of algorithms to analyze human hand motions. Following a categorization of different gesture types and a review of other sensing techniques, the design of vision systems that achieve hand gesture understanding for intelligent robots is analyzed. For each of the individual algorithmic steps – hand detection, hand tracking, and trajectory-based gesture recognition – a separate Chapter introduces common techniques and algorithms and provides example methods. The resulting recognition algorithms are considering gestures in isolation and are often not sufficient for interacting with a robot who can only understand such gestures when incorporating the context like, e.g., what object was pointed at or manipulated. Going beyond a purely trajectory-based gesture recognition by incorporating context is an important prerequisite to achieve gesture understanding and is addressed explicitly in a separate Chapter of this book. Two types of context, user-provided context and situational context, are reviewed and existing approaches to incorporate context for gestural understanding are reviewed. Example approaches for both context types provide a deeper algorithmic insight into this field of research. An overview of recent robots capable of gesture recognition and understanding summarizes the currently realized human-robot interaction quality. The approaches for gesture understanding covered in this book are manually designed while humans learn to recognize gestures automatically during growing up. Promising research targeted at analyzing developmental learning in children in order to mimic this capability in technical systems is highlighted in the last Chapter completing this book as this research direction may be highly influential for creating future gesture understanding systems

Observing Environments

> Context • Society is faced with “wicked” problems of environmental sustainability, which are inherently multiperspectival, and there is a need for explicitly constructivist and perspectivist theories to address them. > Problem • However, different constructivist theories construe the environment in different ways. The aim of this paper is to clarify the conceptions of environment in constructivist approaches, and thereby to assist the sciences of complex systems and complex environmental problems. > Method • We describe the terms used for “the environment” in von Uexküll, Maturana & Varela, and Luhmann, and analyse how their conceptions of environment are connected to differences of perspective and observation. > Results • We show the need to distinguish between inside and outside perspectives on the environment, and identify two very different and complementary logics of observation, the logic of distinction and the logic of representation, in the three constructivist theories. > Implications • Luhmann’s theory of social systems can be a helpful perspective on the wicked environmental problems of society if we consider carefully the theory’s own blind spots: that it confines itself to systems of communication, and that it is based fully on the conception of observation as indication by means of distinction

The attentive robot companion: learning spatial information from observation and verbal interaction

Ziegler L. The attentive robot companion: learning spatial information from observation and verbal interaction. Bielefeld: Universität Bielefeld; 2015.This doctoral thesis investigates how a robot companion can gain a certain degree of situational awareness through observation and interaction with its surroundings. The focus lies on the representation of the spatial knowledge gathered constantly over time in an indoor environment. However, from the background of research on an interactive service robot, methods for deployment in inference and verbal communication tasks are presented. The design and application of the models are guided by the requirements of referential communication. The approach here involves the analysis of the dynamic properties of structures in the robot’s field of view allowing it to distinguish objects of interest from other agents and background structures. The use of multiple persistent models representing these dynamic properties enables the robot to track changes in multiple scenes over time to establish spatial and temporal references. This work includes building a coherent representation considering allocentric and egocentric aspects of spatial knowledge for these models. Spatial analysis is extended with a semantic interpretation of objects and regions. This top-down approach for generating additional context information enhances the grounding process in communication. A holistic, boosting-based classification approach using a wide range of 2D and 3D visual features anchored in the spatial representation allows the system to identify room types. The process of grounding referential descriptions from a human interlocutor in the spatial representation is evaluated through referencing furniture. This method uses a probabilistic network for handling ambiguities in the descriptions and employs a strategy for resolving conflicts. In order to approve the real-world applicability of these approaches, this system was deployed on the mobile robot BIRON in a realistic apartment scenario involving observation and verbal interaction with an interlocutor

Factors shaping the evolution of electronic documentation systems

The main goal is to prepare the space station technical and managerial structure for likely changes in the creation, capture, transfer, and utilization of knowledge. By anticipating advances, the design of Space Station Project (SSP) information systems can be tailored to facilitate a progression of increasingly sophisticated strategies as the space station evolves. Future generations of advanced information systems will use increases in power to deliver environmentally meaningful, contextually targeted, interconnected data (knowledge). The concept of a Knowledge Base Management System is emerging when the problem is focused on how information systems can perform such a conversion of raw data. Such a system would include traditional management functions for large space databases. Added artificial intelligence features might encompass co-existing knowledge representation schemes; effective control structures for deductive, plausible, and inductive reasoning; means for knowledge acquisition, refinement, and validation; explanation facilities; and dynamic human intervention. The major areas covered include: alternative knowledge representation approaches; advanced user interface capabilities; computer-supported cooperative work; the evolution of information system hardware; standardization, compatibility, and connectivity; and organizational impacts of information intensive environments

The robot's vista space : a computational 3D scene analysis

Swadzba A. The robot's vista space : a computational 3D scene analysis. Bielefeld (Germany): Bielefeld University; 2011.The space that can be explored quickly from a fixed view point without locomotion is known as the vista space. In indoor environments single rooms and room parts follow this definition. The vista space plays an important role in situations with agent-agent interaction as it is the directly surrounding environment in which the interaction takes place. A collaborative interaction of the partners in and with the environment requires that both partners know where they are, what spatial structures they are talking about, and what scene elements they are going to manipulate. This thesis focuses on the analysis of a robot's vista space. Mechanisms for extracting relevant spatial information are developed which enable the robot to recognize in which place it is, to detect the scene elements the human partner is talking about, and to segment scene structures the human is changing. These abilities are addressed by the proposed holistic, aligned, and articulated modeling approach. For a smooth human-robot interaction, the computed models should be aligned to the partner's representations. Therefore, the design of the computational models is based on the combination of psychological results from studies on human scene perception with basic physical properties of the perceived scene and the perception itself. The holistic modeling realizes a categorization of room percepts based on the observed 3D spatial layout. Room layouts have room type specific features and fMRI studies have shown that some of the human brain areas being active in scene recognition are sensitive to the 3D geometry of a room. With the aligned modeling, the robot is able to extract the hierarchical scene representation underlying a scene description given by a human tutor. Furthermore, it is able to ground the inferred scene elements in its own visual perception of the scene. This modeling follows the assumption that cognition and language schematize the world in the same way. This is visible in the fact that a scene depiction mainly consists of relations between an object and its supporting structure or between objects located on the same supporting structure. Last, the articulated modeling equips the robot with a methodology for articulated scene part extraction and fast background learning under short and disturbed observation conditions typical for human-robot interaction scenarios. Articulated scene parts are detected model-less by observing scene changes caused by their manipulation. Change detection and background learning are closely coupled because change is defined phenomenologically as variation of structure. This means that change detection involves a comparison of currently visible structures with a representation in memory. In range sensing this comparison can be nicely implement as subtraction of these two representations. The three modeling approaches enable the robot to enrich its visual perceptions of the surrounding environment, the vista space, with semantic information about meaningful spatial structures useful for further interaction with the environment and the human partner

Toward Understanding Human Expression in Human-Robot Interaction

Intelligent devices are quickly becoming necessities to support our activities during both work and play. We are already bound in a symbiotic relationship with these devices. An unfortunate effect of the pervasiveness of intelligent devices is the substantial investment of our time and effort to communicate intent. Even though our increasing reliance on these intelligent devices is inevitable, the limits of conventional methods for devices to perceive human expression hinders communication efficiency. These constraints restrict the usefulness of intelligent devices to support our activities. Our communication time and effort must be minimized to leverage the benefits of intelligent devices and seamlessly integrate them into society. Minimizing the time and effort needed to communicate our intent will allow us to concentrate on tasks in which we excel, including creative thought and problem solving. An intuitive method to minimize human communication effort with intelligent devices is to take advantage of our existing interpersonal communication experience. Recent advances in speech, hand gesture, and facial expression recognition provide alternate viable modes of communication that are more natural than conventional tactile interfaces. Use of natural human communication eliminates the need to adapt and invest time and effort using less intuitive techniques required for traditional keyboard and mouse based interfaces. Although the state of the art in natural but isolated modes of communication achieves impressive results, significant hurdles must be conquered before communication with devices in our daily lives will feel natural and effortless. Research has shown that combining information between multiple noise-prone modalities improves accuracy. Leveraging this complementary and redundant content will improve communication robustness and relax current unimodal limitations. This research presents and evaluates a novel multimodal framework to help reduce the total human effort and time required to communicate with intelligent devices. This reduction is realized by determining human intent using a knowledge-based architecture that combines and leverages conflicting information available across multiple natural communication modes and modalities. The effectiveness of this approach is demonstrated using dynamic hand gestures and simple facial expressions characterizing basic emotions. It is important to note that the framework is not restricted to these two forms of communication. The framework presented in this research provides the flexibility necessary to include additional or alternate modalities and channels of information in future research, including improving the robustness of speech understanding. The primary contributions of this research include the leveraging of conflicts in a closed-loop multimodal framework, explicit use of uncertainty in knowledge representation and reasoning across multiple modalities, and a flexible approach for leveraging domain specific knowledge to help understand multimodal human expression. Experiments using a manually defined knowledge base demonstrate an improved average accuracy of individual concepts and an improved average accuracy of overall intents when leveraging conflicts as compared to an open-loop approach

Referring Expression Generation in Situated Interaction

While most current frameworks for reference handling are based on binary truth-theoretic knowledge representation, in this thesis I argue for a perspective on reference which emphasises the collaborative nature of reference. I present the Probabilistic Reference And GRounding mechanism (PRAGR) which uses flexible concept assignment based on vague property models and situational context in order to maximise the chance of communicative success. I demonstrate that PRAGR is capable of dealing with several property domains with different internal structures, such as graded adjectives, colour, shape, projective terms, and projective regions. Further, I show that PRAGR is fit to handle in an integrated fashion the most relevant challenges of Referring Expression Generation, in particular graded properties, spatial relations, and salience effects. In three empirical evaluation studies, I demonstrate the usefulness of PRAGR for situated referential communication

인공지능 기반 교육 플랫폼 사용에 대한 중국 교사의 인식

학위논문 (석사) -- 서울대학교 대학원 : 사범대학 교육학과, 2021. 2. 조영환.최근 교육 분야에서 인공지능(AI)의 도입이 큰 관심을 끌고 있다. 특히 AI 기술과 학습 분석이 결합한 인공지능 기반 교육 플랫폼은 지금껏 실현되기 어려웠던 맞춤형 학습(personalized learning)과 적응적 학습(adaptive learning)에 도움이 될 수 있도록 발전하고 있다. 인공지능 기반 교육 플랫폼(AI-based education platform)은 학습자의 행동 추적 등을 통해 이들의 특성을 분석하고 진단을 제공한 뒤 분석 결과를 토대로 학습자에게 인지 수준에 맞는 맞춤형 학습자원과 피드백을 제공한다. 인공지능 기반 교육 플랫폼은 교사와 학생에게 실시간 학습 데이터와 분석 결과, 그리고 피드백을 제공할 수 있어 인공지능 기반 교육 플랫폼이 맞춤형 학습에 긍정적인 의미가 있다는 선행 연구도 있었다. 그럼에도 불구하고, 기존 연구는 모델 개발의 차원에서나 엄밀한 실험실 환경에서 인공지능 기반 교육 플랫폼의 효과를 연구해왔으며, 인공지능 기반 교육 플랫폼에 대한 교사의 인식과 관련된 연구는 드물었다. 교사는 인공지능 교육 기술의 사용자이기 때문에 인공지능 교육 기술의 교육 도입에 있어 교사들의 인식과 의견은 중요하다. 본 연구는 인공지능 기반 교육 플랫폼을 활용하는 것에 대한 교사들의 인식을 탐구하였다. 아래 연구 문제를 다루기 위해 질적 연구를 시행하였다. 첫째, 중국 교사들은 인공지능 기반 교육 플랫폼이 중학교 교육에 활용 있어 어떠한 장점이 있다고 인식하는가? 둘째, 중국 교사들은 인공지능 기반 교육 플랫폼과 중학교 교수 활동 요소 간 어떠한 모순이 있다고 인식하는가? 셋째, 중국 교사들은 인공지능 기반 교육 플랫폼을 중학교 교육에 도입할 때 무엇이 필요하다고 인식하는가? 본 연구는 중국 교사들을 연구대상으로 온라인 심층 면담을 하였다. 문헌 리뷰를 통해 면담 질문지를 설계하되 눈덩이표집법 (snowball sampling)을 통해 중국 중학교 교사 14명을 연구참여자로 선정하였다. 선정된 교사들은 모두 인공지능 기반 교육 플랫폼 사용 경험이 있으며 각 교사를 대상으로 약 1시간 정도 면담을 진행하고 녹음하였다. 면담이 끝난 후 녹음 내용을 전사하였으며, 주제분석을 사용하여 면담 내용을 초기 코드 생성하고 면담 자료 속에서 주제를 도출하였다. 특히 연구 문제 2번의 경우, 인공지능 기반 교육 플랫폼 활용과 교수 학습활동 내 여러 요소 간의 모순을 분석하기 위해 활동이론을 연구의 틀로 이용하였다. 최종적으로 연구문제 1에 대한 주제 4개, 연구문제 2에 대한 주제 6개, 연구문제 3에 대한 주제 4개를 도출하였다. 연구 결과로 교사들은 인공지능 기반 교육 플랫폼의 장점에 대해 즉각적인 피드백 제공, 교수학습 지원, 교사의 업무량 감소 등으로 인식하였고, 인공지능 기반 교육 플랫폼이 다양한 교수학습 자원을 통합할 수 있다고 인식하였다. 아울러 교사들은 인공지능 기반 교육 플랫폼의 사용에 있어 기존의 교수학습 활동과 상충된 부분이 있다는 점을 인식하였다. 교사들은 기존 인공지능 기반 교육 플랫폼의 추천 모델이 차별화된 학생들에게 잘 적용되지 못한다는 것을 인식하였다. 그리고 기존 인공지능 기반 교육 플랫폼이 다양한 학습 자원을 잘 분류되지 못하기 때문에 교사들이 사용하기 불편하다. 인공지능 기반 교육 플랫폼을 이용할 때 교사의 지적재산권을 보호하기 위한 명확한 규제가 부족하다고 인식하였다. 이와 함께 학부모들은 인공지능 기반 교육 플랫폼을 사용함으로써 발생할 수 있는 학습자의 인터넷 남용과 시력 저하 문제를 우려하였다. 또 중국의 사회문화적 배경과 교육 특성으로 인해 인공지능 기반 교육 플랫폼을 활용하는 데 학생들의 글씨 쓰기 능력에 영향을 미칠 수 있으며, 학교 내 전자기기 사용 제한도 데이터 수집의 지속성과 효율성에 영향을 미칠 수 있다고 인식하였다. 교사들은 위의 문제들이 인공지능 교육 플랫폼 사용에 대한 규칙 마련과 인공지능 기술을 개선함으로써 완화될 수 있다고 인식하였다. 또한 교사의 실제 요구에 맞게 개발될 수 있도록 인공지능 기반 교육 플랫폼 개발 과정에 교육 전문가와 교사가 참여할 필요가 있다. 본 연구는 중국 교사들이 인공지능 기반 교육 플랫폼에 대한 인식을 탐색하였으며, 인공지능 기반 교육 플랫폼이 교수학습에서의 장점과 문제점을 밝혔다. 아울러 본 연구는 인공지능 기반 교육 플랫폼이 교육 분야에 대규모로 도입될 수 있도록 규칙, 인공지능 기술, 그리고 교육 공학의 차원에서 사용 규범과 기술 개선을 제안하였다. 본 연구를 통해 탐색한 내용이 향후 교육 분야의 인공지능 기반 교육 플랫폼 도입에 활용된다면 인공지능 교육 기술에 관한 연구의 발전에도 기여할 수 있을 것으로 기대된다.In recent years, the introduction of artificial intelligence (AI) in education has attracted widespread attention. In particular, the AI-based education platform based on the combination of AI technology and learning analysis brings new light to the long-standing difficulties in personalized learning and adaptive learning. The AI-based education platform analyzes learners' characteristics by collecting their data and tracking their learning behavior. It then generates cognitive diagnosis for learners and provides them with personalized learning resources and adaptive feedback that match their cognitive level based on systematic analysis. With the help of the AI-based education platform, teachers and students can get real-time educational data and analysis result，as well as the feedback and treatment corresponding to the results. Previous studies have already demonstrated and proved its positive significance to personalized learning. However, these studies mostly start from a model development perspective or in a rigorous laboratory environment. There has been little research on teachers' perceptions of AI-based education platform. As a direct user of AI educational technologies, teachers' perceptions and suggestions are vital for introducing AIEd in education. In this study, the researcher explored teachers' perceptions of using AI-based education platform in teaching. The study conducted qualitative research to address the following research questions: 1) How do Chinese teachers perceive the advantages of AI-based education platforms for teaching and learning in secondary school? 2) How do Chinese teachers perceive the contradictions between AI-based education platforms and the secondary school system? 3）How do Chinese teachers suggest applying AI-based education platforms in secondary school? And it referred to the in-depth online interview with Chinese teachers who had experience with AI-based education platform. Interview questions were constructed through the literature review, and 14 secondary school teachers were selected by the snowball sampling method. The interviews lasted for an average of one hour per teacher and were transcribed from the audio recordings to text documents when finished. Afterward, the data were analyzed using thematic analysis, including generating initial codes, searching and reviewing the categories, and deriving the themes finally. Notably, for research question two, the researcher used the activity theory framework to analyze the contradictions among the use of the AI-based education platform and the various elements of the teaching and learning activities. Finally, four themes for research question 1, six themes for research question 2, and four themes for research question 3 were derived. As for the advantages, teachers believe that AI-based education platforms can provide instant feedback, targeted and systematic teaching support, and reduce teachers' workload. At the same time, AI-based education platforms can also integrate teaching resources in different areas. Teachers also recognized that the AI-based education platforms might trigger contradictions in existing teaching activities. They are aware of the situation that the recommended model of the AI-based education platform is not suitable for all levels of students; that a large number of learning resources are not classified properly enough to meet the needs of teachers, and that there lack clear rules and regulations to protect teachers' intellectual property rights when using the platform. Besides, parents are also concerned about the potential risk of internet addiction and vision problems using AI-based education platforms. Moreover, the use of the AI-based education platform may also affect students' ability to write Chinese characters due to the socio-historical background and educational characteristics in China. Furthermore, the restricted use of electronic devices on campus may also impact the consistent and effective education data collection. Teachers believe that these problems can be solved by improving rules and AI technology. Moreover, to make the platform more in line with the actual teaching requirements, teachers and education experts can also be involved in the development process of AI-based education platform. This study explored how Chinese teachers perceive the AI-based education platform and found that the AI-based education platform was conducive to personalized teaching and learning. At the same time, this study put forward some suggestions from the perspective of rules, AI technology, and educational technology, hoping to provide a good value for the future large-scale introduction of AI-based education platforms in education.CHAPTER 1. INTRODUCTION 1 1.1. Problem Statement 1 1.2. Purpose of Research 7 1.3. Definition of Terms 8 CHAPTER 2. LITERATURE REVIEW 10 2.1. AI in Education 10 2.1.1 AI for Learning and Teaching 10 2.1.2 AI-based Education Platform 14 2.1.3 Teachers' Perception on AI-based Education Platform 18 2.2. Activity Theory 20 CHAPTER 3. RESEARCH METHOD 23 3.1. Research Design 23 3.2. Participants 25 3.3. Instrumentation 26 3.3.1 Potential Value of AI System in Education 26 3.4. Data Collection 33 3.5. Data Analysis 34 CHAPTER 4. FINDINGS 36 4.1. Advantages of Using AI-based Education Platform 36 4.1.1 Instant Feedback 37 4.1.2 Targeted and Systematic Teaching Support 42 4.1.3 Educational Resources Sharing 46 4.1.4 Reducing Workload 49 4.2. Tensions of Using AI-based Education Platform 51 4.2.1 Inadequately Meet the Needs of Teachers 52 4.2.2 Failure to Satisfy Low and High Achievers 54 4.2.3 Intellectual Property Violation 56 4.2.4 Guardian's Concern 57 4.2.5 School Rules about the Use of Electronic Devices 58 4.2.6 Implication for Chinese Character Education 59 4.3. Suggestion of Using AI-based Education Platform 61 4.3.1 Improving Rules of Using the AI-based Education Platform 61 4.3.2 Improving Rules of Protecting Teachers Right 62 4.3.3 Improving AI Technology 64 4.3.4 Participatory Design 66 CHAPTER 5. DISCUSSION AND CONCLUSION 68 5.1. Discussion 68 5.2. Conclusion 72 REFERENCE 75 APPENDIX 1 98 APPENDIX 2 100 국문초록 112Maste