MoKey - A motion based keyboard interpreter

In this paper the authors present a new middleware for mapping gestures, obtained by a motion sensing camera device, to key events which are consumed by a standard off-the-shelf application. The aim is twofold: accessibility for users which are not able to use the keyboard because of physical impairments and the use of standard games for doing physical exercises. Hereby, special attention is laid on the adaptability to user requirements and easiness of configuration for the user himself and non-expert assistants. The actual state of our system is compared to similar proposals in terms of usability and performance, finally future working directions are outlined

TouchEditor: Interaction design and evaluation of a flexible touchpad for text editing of head-mounted displays in speech-unfriendly environments

A text editing solution that adapts to speech-unfriendly (inconvenient to speak or difficult to recognize speech) environments is essential for head-mounted displays (HMDs) to work universally. For existing schemes, e.g., touch bar, virtual keyboard and physical keyboard, there are shortcomings such as insufficient speed, uncomfortable experience or restrictions on user location and posture. To mitigate these restrictions, we propose TouchEditor, a novel text editing system for HMDs based on a flexible piezoresistive film sensor, supporting cursor positioning, text selection, text retyping and editing commands (i.e., Copy, Paste, Delete, etc.). Through literature overview and heuristic study, we design a pressure-controlled menu and a shortcut gesture set for entering editing commands, and propose an area-and-pressure-based method for cursor positioning and text selection that skillfully maps gestures in different areas and with different strengths to cursor movements with different directions and granularities. The evaluation results show that TouchEditor i) adapts to various contents and scenes well with a stable correction speed of 0.075 corrections per second; ii) achieves 95.4% gesture recognition accuracy; iii) reaches a considerable level with a mobile phone in text selection tasks. The comparison results with the speech-dependent EYEditor and the built-in touch bar further prove the flexibility and robustness of TouchEditor in speech-unfriendly environments

Interactive Spaces: Model for Motion-based Music Applications

With the extensive utilization of touch screens, smartphones and various reactive surfaces, reality- based and intuitive interaction styles have now become customary. The employment of larger interactive areas, like floors or peripersonal three-dimensional spaces, further increase the reality- based interaction affordances, allowing full-body involvement and the development of a co- located, shared user experience. Embodied and spatial cognition play a fundamental role for the interaction in this kind of spaces, where users act in the reality with no device in the hands and obtain an audio and graphical output depending on their movements. Starting from the early experiments of Myron Krueger in 1971, responsive floors have been developed through various technologies including sensorized tiles and computer vision systems, to be employed in learn- ing environments, entertainment, games and rehabilitation. Responsive floors allow the spatial representation of concepts and for this reason are suitable for immediate communication and engagement. As many musical features have meaningful spatial representations, they can easily be reproduced in the physical space through a conceptual blending approach and be made available to a great number of users. This is the key idea for the design of the original music applications presented in this thesis. The applications, devoted to music learning, production and active listening, introduce a novel creative approach to music, which can be further assumed as a general paradigm for the design of motion-based learning environments. Application assessment with upper elementary and high school students has proved that users engagement and bodily inter- action have a high learning power, which can be a valid resource for deeper music knowledge and more creative learning processes. Although further interface tests showed that touch screen interaction performs better than full-body interaction, some important guidelines for the design of reactive floors applications have been obtained on the basis of these test results. Moreover, the conceptual framework developed for the design of music applications can represent a valid paradigm also in the general field of human-computer interaction

Development of a virtual audience concept for public speaking training in immersive virtual environments

Public speaking is common in today's work and study environment, yet many people dread giving presentations and sometimes even avoid them altogether. A frequent solution is to take up public speaking courses where speakers can learn presentation skills from speech trainers and rehearse in front of other trainees. By combining the opportunities of virtual reality with the characteristics and challenges of training sessions, the dissertation proposes a design concept for a virtual audience that can serve as speech training tool in immersive virtual environments. Through this approach the research addresses public speaking from a training angle - a novel view that wishes to complement the existing speech anxiety treatment studies in virtual reality. The work rests on a multidisciplinary ground and includes perspectives from interpersonal communication, virtual reality, human-computer interaction, and design of virtual learning environments.Die vorliegende Dissertation schlägt einen neuen Ansatz für Kommunikationstrainings und Sprechübungen vor, und führt ein Konzept für ein virtuelles Publikum ein. Das Publikum soll hierbei als Trainingsinstrument dienen und richtet sich an diejenigen, welche ihre rhetorischen Fähigkeiten und Präsentationstechniken verbessern möchten. Die Schlüssel-Komponente des Publikums bilden virtuelle Menschen, wobei diese möglichst realistisch aussehen und agieren sollen. Dies wirft eine zentrale Frage auf: Was sind die Merkmale eines realen Publikums und was machen die Menschen überhaupt, wenn sie vor einem Sprecher sitzen? Um diese Frage zu beantworten, wurden zwei Studien durchgeführt. Die erste explorative Studie beschäftigt sich mit Experten im Kommunikationstraining und VR Bereich. Sieben Experten wurden sowohl über die Rolle von Publikumsanpassung, Interaktion, und Gruppendynamik als auch über technische Anwendungsmöglichkeiten in der virtuellen Realität befragt. Die Ergebnisse zeigen, dass die Kommunikationstrainingsprogramme hohe Publikumsanpassungen und Interaktivitätsmöglichkeiten erfordern. Um die Ergebnisse zu komplementieren, wurde ein Trainingsseminar beobachtet. Die Resultate der Beobachtung verdeutlichen die Bedeutung von Sprechübungen im Rahmen eines Kommunikationsprogramms sowie die bevorzugten Feedbackmethoden. Die zweite Studie umfasst die Beobachtung eines studentischen Publikums im Rahmen einer Vorlesung an der Technischen Universität Ilmenau. Die nonverbalen Verhaltensweisen von 14 Studierenden wurden kodiert und in aufmerksame oder unaufmerksame Verhaltensweisen aufgeteilt. Weiterhin wurde die Frequenz, Komplexität und Dauer der ausgewiesenen Reaktionen analysiert. Die Ergebnisse dieser zwei Studien bilden das Fundament für die Entwicklung eines Designkonzepts für ein virtuelles Publikum. Dieses bezieht unterschiedliche Merkmale (z.B. demographische Charakteristika, Vorschläge für virtuelle Räumlichkeiten, etc.) ein. Zusätzlich wird eine Liste mit Indikatoren für Aufmerksamkeit und Unaufmerksamkeit eingebunden und ein fünfminütiges Szenario mit fünf virtuellen Menschen vorgestellt.The present dissertation proposes a novel approach to public speaking training and introduces a concept for a virtual audience in immersive virtual environments. The key component of such an audience are the virtual humans (VHs) whose role is to look and behave as closely as possible to a real public. Virtual audiences can then be used as training tools for people who wish to improve their public speaking and presentation skills. Two empirical studies were conducted to help identify relevant audience features and behaviors that occur in real life and that can be modeled for a virtual public. The first one is an exploratory study with experts in communication training and in fields related to virtual reality (VR). Seven experts were interviewed on the role and importance of audience customization, interaction, and group dynamic during training, as well as on technical possibilities to design virtual audiences with such features. The results show that audiences for communication training require extensive customization and interactivity options. To complement the findings, a speech practice session in a communication training seminar was observed and helped reveal the role of speech practice in the economy of the whole training seminar as well as the preferred feedback methods. In the second study, a video observation of a student audience during a lecture at Technische Universität Ilmenau was conducted. The nonverbal behaviors of 14 students were coded and divided into attentive and inattentive manifestations. The identified behaviors are further described and analyzed in terms of their frequency, complexity, and duration. The findings of both these studies helped create a design concept for a virtual audience with various characteristics (e.g., demographic features and virtual spaces they could inhabit) and a list with attentive and inattentive nonverbal behaviors the virtual humans could display. A five-minute scenario with virtual listeners is suggested as well

Psychophysiological analysis of a pedagogical agent and robotic peer for individuals with autism spectrum disorders.

Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by ongoing problems in social interaction and communication, and engagement in repetitive behaviors. According to Centers for Disease Control and Prevention, an estimated 1 in 68 children in the United States has ASD. Mounting evidence shows that many of these individuals display an interest in social interaction with computers and robots and, in general, feel comfortable spending time in such environments. It is known that the subtlety and unpredictability of people’s social behavior are intimidating and confusing for many individuals with ASD. Computerized learning environments and robots, however, prepare a predictable, dependable, and less complicated environment, where the interaction complexity can be adjusted so as to account for these individuals’ needs. The first phase of this dissertation presents an artificial-intelligence-based tutoring system which uses an interactive computer character as a pedagogical agent (PA) that simulates a human tutor teaching sight word reading to individuals with ASD. This phase examines the efficacy of an instructional package comprised of an autonomous pedagogical agent, automatic speech recognition, and an evidence-based instructional procedure referred to as constant time delay (CTD). A concurrent multiple-baseline across-participants design is used to evaluate the efficacy of intervention. Additionally, post-treatment probes are conducted to assess maintenance and generalization. The results suggest that all three participants acquired and maintained new sight words and demonstrated generalized responding. The second phase of this dissertation describes the augmentation of the tutoring system developed in the first phase with an autonomous humanoid robot which serves the instructional role of a peer for the student. In this tutoring paradigm, the robot adopts a peer metaphor, where its function is to act as a peer. With the introduction of the robotic peer (RP), the traditional dyadic interaction in tutoring systems is augmented to a novel triadic interaction in order to enhance the social richness of the tutoring system, and to facilitate learning through peer observation. This phase evaluates the feasibility and effects of using PA-delivered sight word instruction, based on a CTD procedure, within a small-group arrangement including a student with ASD and the robotic peer. A multiple-probe design across word sets, replicated across three participants, is used to evaluate the efficacy of intervention. The findings illustrate that all three participants acquired, maintained, and generalized all the words targeted for instruction. Furthermore, they learned a high percentage (94.44% on average) of the non-target words exclusively instructed to the RP. The data show that not only did the participants learn nontargeted words by observing the instruction to the RP but they also acquired their target words more efficiently and with less errors by the addition of an observational component to the direct instruction. The third and fourth phases of this dissertation focus on physiology-based modeling of the participants’ affective experiences during naturalistic interaction with the developed tutoring system. While computers and robots have begun to co-exist with humans and cooperatively share various tasks; they are still deficient in interpreting and responding to humans as emotional beings. Wearable biosensors that can be used for computerized emotion recognition offer great potential for addressing this issue. The third phase presents a Bluetooth-enabled eyewear – EmotiGO – for unobtrusive acquisition of a set of physiological signals, i.e., skin conductivity, photoplethysmography, and skin temperature, which can be used as autonomic readouts of emotions. EmotiGO is unobtrusive and sufficiently lightweight to be worn comfortably without interfering with the users’ usual activities. This phase presents the architecture of the device and results from testing that verify its effectiveness against an FDA-approved system for physiological measurement. The fourth and final phase attempts to model the students’ engagement levels using their physiological signals collected with EmotiGO during naturalistic interaction with the tutoring system developed in the second phase. Several physiological indices are extracted from each of the signals. The students’ engagement levels during the interaction with the tutoring system are rated by two trained coders using the video recordings of the instructional sessions. Supervised pattern recognition algorithms are subsequently used to map the physiological indices to the engagement scores. The results indicate that the trained models are successful at classifying participants’ engagement levels with the mean classification accuracy of 86.50%. These models are an important step toward an intelligent tutoring system that can dynamically adapt its pedagogical strategies to the affective needs of learners with ASD

Report on the 2015 NSF Workshop on Unified Annotation Tooling

On March 30 & 31, 2015, an international group of twenty-three researchers with expertise in linguistic annotation convened in Sunny Isles Beach, Florida to discuss problems with and potential solutions for the state of linguistic annotation tooling. The participants comprised 14 researchers from the U.S. and 9 from outside the U.S., with 7 countries and 4 continents represented, and hailed from fields and specialties including computational linguistics, artificial intelligence, speech processing, multi-modal data processing, clinical & medical natural language processing, linguistics, documentary linguistics, sign-language linguistics, corpus linguistics, and the digital humanities. The motivating problem of the workshop was the balkanization of annotation tooling, namely, that even though linguistic annotation requires sophisticated tool support to efficiently generate high-quality data, the landscape of tools for the field is fractured, incompatible, inconsistent, and lacks key capabilities. The overall goal of the workshop was to chart the way forward, centering on five key questions: (1) What are the problems with current tool landscape? (2) What are the possible benefits of solving some or all of these problems? (3) What capabilities are most needed? (4) How should we go about implementing these capabilities? And, (5) How should we ensure longevity and sustainability of the solution? I surveyed the participants before their arrival, which provided significant raw material for ideas, and the workshop discussion itself resulted in identification of ten specific classes of problems, five sets of most-needed capabilities. Importantly, we identified annotation project managers in computational linguistics as the key recipients and users of any solution, thereby succinctly addressing questions about the scope and audience of potential solutions. We discussed management and sustainability of potential solutions at length. The participants agreed on sixteen recommendations for future work. This technical report contains a detailed discussion of all these topics, a point-by-point review of the discussion in the workshop as it unfolded, detailed information on the participants and their expertise, and the summarized data from the surveys

Mixed Reality Interiors: Exploring Augmented Reality Immersive Space Planning Design Archetypes for the Creation of Interior Spatial Volume 3D User Interfaces

Augmented reality is an increasingly relevant medium of interaction and media reception with the advances in user worn or hand-held input/output technologies endowing perception of the digital nested within and reactive to the native physical. Our interior spaces are becoming the media interface and this emergence affords designers the opportunity to delve further into crafting an aesthetics for the medium. Beyond having the virtual assets and applications in correct registration with the real-world environment, critical topics are addressed such as the compositional roles of virtual and physical design features including their purpose, modulation, interaction potentials and implementation into varying indoor settings. Examining and formulating methodologies for mixed reality interior 3D UI schemes derived from the convergence of digital media and interior design disciplines comprise the scope of this design research endeavor. A holistic approach is investigated to produce a framework for augmented reality 3D user interface interiors through research and development of pattern language systems for the balanced blending of complimentary digital and physical design elements. These foundational attributes serve in the creation, organization and exploration of interactive possibilities and implications of these hybrid futuristic spatial interface layouts.M.S., Digital Media -- Drexel University, 201

Some problems of designing for augmentative and alternative communication users: an enquiry through practical design activity

The submission is concerned with, and addresses, problems of designing for people with disabilities, with specific reference to people who are illiterate and cannot speak. People with such disabilities often depend on electronic AAC (Augmentative and Alternative Communication) devices for interpersonal communication. A central theme of the thesis, however, is that such products, and products intended for people with disabilities more generally, have characteristics that inadequately attend to users' needs. Through a combination of practical product development and literature reviews, the thesis demonstrates how improvements to AAC devices 'can be made through user-participatory, usercentred and more sensitive and perceptive design. Literature reviews in the following subjects are reported: AAC; the operational knowledge base for design and disability; user participatory design; and wearable computing. At the core of the thesis is the presentation and discussion of an empirical case study, carried out by the researcher, to design and develop the Portland Communication Aid (PCA). The PCA was conceived as an AAC product that would attempt to redress the inadequacies of predecessor devices. The design activity for the PCA is traced in the thesis, from initial concepts and development models through to a working prototype. Key ideas and essential principles of the design are illustrated. Throughout the work on the PCA, many problems associated with designing for people with severe communication disabilities were encountered. These problems, as with their resolutions, comprised matters of both designing (as an activity) and design (as product specification). The thesis contains comprehensive exposure and analysis of these problems and resolutions. In particular, the value of shaping meaning, metaphor, and other product semantics into devices intended for use by people with disabilities is explored. The study provides two substantive conclusions. First, that both the activity and the outcomes of Industrial Design have a valuable role in the empowerment and rehabilitation of AAC users. And second, that key principles have been identified that will enable designers to better identify, articulate and respond to the needs of people with communication disabilities (and the needs of people with disabilities more generally

Computational approaches to explainable artificial intelligence: Advances in theory, applications and trends

Deep Learning (DL), a groundbreaking branch of Machine Learning (ML), has emerged as a driving force in both theoretical and applied Artificial Intelligence (AI). DL algorithms, rooted in complex and non-linear artificial neural systems, excel at extracting high-level features from data. DL has demonstrated human-level performance in real-world tasks, including clinical diagnostics, and has unlocked solutions to previously intractable problems in virtual agent design, robotics, genomics, neuroimaging, computer vision, and industrial automation. In this paper, the most relevant advances from the last few years in Artificial Intelligence (AI) and several applications to neuroscience, neuroimaging, computer vision, and robotics are presented, reviewed and discussed. In this way, we summarize the state-of-the-art in AI methods, models and applications within a collection of works presented at the 9th International Conference on the Interplay between Natural and Artificial Computation (IWINAC). The works presented in this paper are excellent examples of new scientific discoveries made in laboratories that have successfully transitioned to real-life applications.MCIU - Nvidia(UMA18-FEDERJA-084

Computational Approaches to Explainable Artificial Intelligence:Advances in Theory, Applications and Trends

Deep Learning (DL), a groundbreaking branch of Machine Learning (ML), has emerged as a driving force in both theoretical and applied Artificial Intelligence (AI). DL algorithms, rooted in complex and non-linear artificial neural systems, excel at extracting high-level features from data. DL has demonstrated human-level performance in real-world tasks, including clinical diagnostics, and has unlocked solutions to previously intractable problems in virtual agent design, robotics, genomics, neuroimaging, computer vision, and industrial automation. In this paper, the most relevant advances from the last few years in Artificial Intelligence (AI) and several applications to neuroscience, neuroimaging, computer vision, and robotics are presented, reviewed and discussed. In this way, we summarize the state-of-the-art in AI methods, models and applications within a collection of works presented at the 9 International Conference on the Interplay between Natural and Artificial Computation (IWINAC). The works presented in this paper are excellent examples of new scientific discoveries made in laboratories that have successfully transitioned to real-life applications