Designing Embodied Interactive Software Agents for E-Learning: Principles, Components, and Roles

Embodied interactive software agents are complex autonomous, adaptive, and social software systems with a digital embodiment that enables them to act on and react to other entities (users, objects, and other agents) in their environment through bodily actions, which include the use of verbal and non-verbal communicative behaviors in face-to-face interactions with the user. These agents have been developed for various roles in different application domains, in which they perform tasks that have been assigned to them by their developers or delegated to them by their users or by other agents. In computer-assisted learning, embodied interactive pedagogical software agents have the general task to promote human learning by working with students (and other agents) in computer-based learning environments, among them e-learning platforms based on Internet technologies, such as the Virtual Linguistics Campus (www.linguistics-online.com). In these environments, pedagogical agents provide contextualized, qualified, personalized, and timely assistance, cooperation, instruction, motivation, and services for both individual learners and groups of learners. This thesis develops a comprehensive, multidisciplinary, and user-oriented view of the design of embodied interactive pedagogical software agents, which integrates theoretical and practical insights from various academic and other fields. The research intends to contribute to the scientific understanding of issues, methods, theories, and technologies that are involved in the design, implementation, and evaluation of embodied interactive software agents for different roles in e-learning and other areas. For developers, the thesis provides sixteen basic principles (Added Value, Perceptible Qualities, Balanced Design, Coherence, Consistency, Completeness, Comprehensibility, Individuality, Variability, Communicative Ability, Modularity, Teamwork, Participatory Design, Role Awareness, Cultural Awareness, and Relationship Building) plus a large number of specific guidelines for the design of embodied interactive software agents and their components. Furthermore, it offers critical reviews of theories, concepts, approaches, and technologies from different areas and disciplines that are relevant to agent design. Finally, it discusses three pedagogical agent roles (virtual native speaker, coach, and peer) in the scenario of the linguistic fieldwork classes on the Virtual Linguistics Campus and presents detailed considerations for the design of an agent for one of these roles (the virtual native speaker)

Dois peixes movendo-se em seus mares : como se mostra a linguagem corpórea de professores que ensinam equações matemáticas?

Background: “Culture hides much more than it reveals and, strangely enough, what it hides, it hides more effectively from its own participants” (Hall, 1959, p. 39). This quote corresponds well to a Persian proverb, also a well-known aphorism that has been widely cited in many ethnographic articles: “a fish will be the last to discover water.” Being immersed in water, surrounded by it, makes it invisible and almost impossible to perceive. In other words, we often do not know our interactional behaviour as mathematics teachers when we perform it in our usual and localised professional practice. Objective: To discuss mathematics teacher’s body language when teaching equations and thus perceive this language in terms of possible fruitful educational action when teaching equations in the classroom. Design: Qualitative methodology. Data collection and analysis: Based on theoretical references that deal with body language, corporeality, and perception, we analysed individually and comparatively the classes of two mathematics teachers who taught equations in Birmingham (United Kingdom) and Rolante (Brazil). Thus, particularly attentive to mathematical culture in the classroom and analysing the localised gestures in the teachers’ teaching of equations and the non-verbal behaviour, we can understand mathematics teaching through body movement, which often goes unnoticed. Results: We understand from the results of this research that perceiving the body language of mathematics teachers, which is produced with speech, gives us indications of the materialisation of the meanings attributed to the equation and how this will possibly affect the very constitution of the student’s mathematical knowledge, in terms of possible meanings attributed to each gesture. Conclusions: We consider that knowing the body language can favour the teacher’s teaching, i.e., metaphorically, knowing the sea can favour the fish to swim.Contexto: “A cultura esconde mais do que revela e, por mais estranho que pareça se esconder, esconde-se mais efetivamente de seus próprios participantes” (Hall, 1959, p. 39). Esta citação se enquadra muito bem com um provérbio persa, também, um aforismo bem conhecido que tem sido bastante citado em muitos artigos etnográficos, o qual se apresenta como “um peixe é a última criatura a descobrir a água”. Estar imerso na água, cercado por ela, torna invisível e quase impossível percebê-la. Em outras palavras, muitas vezes desconhecemos nosso comportamento interacional, enquanto professores de matemática, quando o realizamos em nossa prática profissional habitual e localizada. Objetivo: Discutir como se mostra a linguagem corpórea do professor de matemática ao ensinar equações e, assim, perceber essa linguagem em termos de ação educacional profícua ao se ensinar equações em sala de aula, por exemplo. Metodologia: Metodologia qualitativa. Coleta de dados e análise: A partir de referenciais teóricos que tratam de linguagem corpórea, corporeidade e percepção, analisamos as aulas de dois professores de matemática que ensinavam equações, em Birmingham (Reino Unido) e em Rolante (Brasil), individualmente e comparativamente. Assim, prestando atenção especial à cultura matemática em sala de aula e analisando os gestos localmente situados no ensino de equações e o comportamento não-verbal dos professores, podemos compreender o ensino de matemática por meio do movimento do corpo, o qual muitas vezes passa despercebido. Resultados: Compreendemos com os resultados dessa pesquisa que perceber a linguagem corpórea dos professores de matemática, a qual é produzida-coma-fala, nos dá indicativos da materialização do sentidos atribuídos à equação e como isso possivelmente afetará a própria constituição do conhecimento matemático do estudante, em termos de possíveis sentidos atribuídos a cada gesto. Conclusões: Consideramos que conhecer a linguagem corpórea pode favorecer o próprio ensinar do professor, ou seja, metaforicamente, conhecer o mar pode favorecer o peixe a nadar

Embodied cognition and executive functioning : the effect of whole body interaction on children's planning and inhibition

Modern user interfaces (UI) are becoming more ‘embodied’ as they facilitate bodily processes. Games consoles now often include body tracking hardware. Tenants of the theories of embodied cognition and executive function (EF) have stipulated that cognition is to some extent tied to the motor system, and so, that cognitive processing benefits from physical interaction. To date however, the research in this domain has focussed on adult populations. Ultimately, children are going to experience this UI revolution throughout the lifespan. So, in the following thesis I examined whether whole body interaction supported by a gaming floor mat improved children’s performance on a set of EF tasks. A set of new, gamified EF tasks were developed and completed using two interfaces (a floor mat and a keyboard) at separate sessions. The results revealed children were equally competent at each EF task using either device. Another notable finding was the effect of gamification on performance. The findings are discussed in the context of developmental psychology, experiment composition, and children’s interactions with technology

Assessment of Audio Interfaces for use in Smartphone Based Spatial Learning Systems for the Blind

Recent advancements in the field of indoor positioning and mobile computing promise development of smart phone based indoor navigation systems. Currently, the preliminary implementations of such systems only use visual interfaces—meaning that they are inaccessible to blind and low vision users. According to the World Health Organization, about 39 million people in the world are blind. This necessitates the need for development and evaluation of non-visual interfaces for indoor navigation systems that support safe and efficient spatial learning and navigation behavior. This thesis research has empirically evaluated several different approaches through which spatial information about the environment can be conveyed through audio. In the first experiment, blindfolded participants standing at an origin in a lab learned the distance and azimuth of target objects that were specified by four audio modes. The first three modes were perceptual interfaces and did not require cognitive mediation on the part of the user. The fourth mode was a non-perceptual mode where object descriptions were given via spatial language using clockface angles. After learning the targets through the four modes, the participants spatially updated the position of the targets and localized them by walking to each of them from two indirect waypoints. The results also indicate hand motion triggered mode to be better than the head motion triggered mode and comparable to auditory snapshot. In the second experiment, blindfolded participants learned target object arrays with two spatial audio modes and a visual mode. In the first mode, head tracking was enabled, whereas in the second mode hand tracking was enabled. In the third mode, serving as a control, the participants were allowed to learn the targets visually. We again compared spatial updating performance with these modes and found no significant performance differences between modes. These results indicate that we can develop 3D audio interfaces on sensor rich off the shelf smartphone devices, without the need of expensive head tracking hardware. Finally, a third study, evaluated room layout learning performance by blindfolded participants with an android smartphone. Three perceptual and one non-perceptual mode were tested for cognitive map development. As expected the perceptual interfaces performed significantly better than the non-perceptual language based mode in an allocentric pointing judgment and in overall subjective rating. In sum, the perceptual interfaces led to better spatial learning performance and higher user ratings. Also there is no significant difference in a cognitive map developed through spatial audio based on tracking user’s head or hand. These results have important implications as they support development of accessible perceptually driven interfaces for smartphones

Patau sündroomi mosaiikvariandiga subjekti kommunikatiivsed võimed: juhtumiuuring

Väitekirja elektrooniline versioon ei sisalda publikatsioone.Kõnevõime puudumine tekitab keerulisi olukordi ega lase suhtlusel sujuda. Kõnetul indiviidil tekib kergesti frustratsioon ning hädas on ka tema lähedased ja ametiisikud, kellega isik kokku puutub. Kuigi verbaalne suuline kommunikatsioon on kõige levinum mõtete edastamise viis, saame suhelda ka sõnadest erinevate märkide abil, multimodaalselt. Üks võimalus kõnetut inimest aidata on analüüsida tema suhtlust, leida sellest tema jaoks tähtsaimad ja kõige kasutatavamad suhtlusmodaalsused ning sel viisil tema mõistmiseni jõuda. Käesoleva väitekirja näol on tegemist Eesti esimese multimodaalse suhtluse uuringuga, kus vaadeldakse vaimupuude ja düspraksiaga indiviidi toimetulekut suhtlussituatsioonides. Ka ei ole eri trisoomiatega isikute suhtlusmodaalsusi (uurimistöö subjekti põhidiagnoos on Patau sündroomi e 13. kromosoomi trisoomia mosaiikvariant) siin varem uuritud ning kogu maailmas on sellise kommunikatsiooni kohta väga vähe andmeid. Samas on valdkond oluline ja suure praktilise väärtusega, kuna ka sellise puudega – ekspressiivse kõne häirega – indiviididel on vajadus ja õigus inimestevahelises suhtluses osaleda, informatsiooni vastu võtta ja ennast arusaadavaks teha. Temaatikat oleks kohe vaja Eesti ühiskonnas rohkem tutvustada. Ainuüksi multimodaalsel mikroanalüüsil põhineva lähenemise teadvustamisest võib olla abi selliste puuetega inimeste paremaks mõistmiseks nende igapäevases elus. Artiklipõhine väitekiri on juhtumiuuring, mille ühe tulemusena koostati uuritava indiviidi suhtlusmodaalsuste leksikon. Valminud loendi eesmärk on parandada subjekti elukvaliteeti – selle olemasolu võimaldab indiviidi suhtluspartneritel temast aru saada, temaga suhelda ja tagab nii sujuvama kommunikatsiooni. Töö autor näeb ka vajadust töötada välja sobiv suhtlusmetoodika ekspressiivse kõnepuudega inimeste ja nende suhtluspartnerite jaoks, dissertatsioon on esimene samm selles suunas.The absence of speech ability causes a lot of misunderstanding and makes communication unsmooth. Not only a person with partial or total loss of the ability to speak experiences a lot of frustrations, but the problem also influences his or her interlocutors, especially the closest family members. Although the most common way to communicate is to use words, we also have the opportunity to interact multimodally, applying other signs different from words. One possibility to help a speech impaired person is to analyze his or her communication practices in order to find the most important and most used communicative modalities, and through this process comprehend him or her better. Current thesis presents the first Estonian study of multimodal communication, which explores coping ability of an individual with mental retardation and dyspraxia in communicative situations. In addition, communication means of people with various trisomies ( the subject of the thesis has the mosaic version of Patau syndrome) have not been studied here before, and throughout the world there are very limited data on such communication. At the same time, the field is relevant and loaded with enormous practical value, because individuals with such disability – impairment of expressive speech – yet have the need and the right to participate in interpersonal communication, to receive information and to make themselves understandable. Raising the awareness of Estonian society on this topic is an example of one immediate necessity. Sole consciousness of the approach based on multimodal microanalysis might enhance comprehension of people with such disabilities in their everyday life. The article-based dissertation is a case study, which one outcome is the lexicon of the subject`s verbal and corporal communication modalities. The objective of compiling the lexicon was to improve life quality of the subject – existence of the lexicon enables the individual´s communication partners to comprehend her and to communicate with her, ensuring thus a smoother communication. The author of current thesis considers it important to develop a suitable communication methodology for individuals with expressive speech impairment and for their communication partners; this study constitutes the first step in the wishful direction

A Tangible User Interface for Interactive Data Visualisation

Information visualisation (infovis) tools are integral for the analysis of large abstract data, where interactive processes are adopted to explore data, investigate hypotheses and detect patterns. New technologies exist beyond post-windows, icons, menus and pointing (WIMP), such as tangible user interfaces (TUIs). TUIs expand on the affordance of physical objects and surfaces to better exploit motor and perceptual abilities and allow for the direct manipulation of data. TUIs have rarely been studied in the field of infovis. The overall aim of this thesis is to design, develop and evaluate a TUI for infovis, using expression quantitative trait loci (eQTL) as a case study. The research began with eliciting eQTL analysis requirements that identified high- level tasks and themes for quantitative genetic and eQTL that were explored in a graphical prototype. The main contributions of this thesis are as follows. First, a rich set of interface design options for touch and an interactive surface with exclusively tangible objects were explored for the infovis case study. This work includes characterising touch and tangible interactions to understand how best to use them at various levels of metaphoric representation and embodiment. These design were then compared to identify a set of options for a TUI that exploits the advantages of touch and tangible interaction. Existing research shows computer vision commonly utilised as the TUI technology of choice. This thesis contributes a rigorous technical evaluation of another promising technology, micro-controllers and sensors, as well as computer vision. However the findings showed that some sensors used with micro-controllers are lacking in capability, so computer vision was adopted for the development of the TUI. The majority of TUIs for infovis are presented as technical developments or design case studies, but lack formal evaluation. The last contribution of this thesis is a quantitative and qualitative comparison of the TUI and touch UI for the infovis case study. Participants adopted more effective strategies to explore patterns and performed fewer unnecessary analyses with the TUI, which led to significantly faster performance. Contrary to common belief bimanual interactions were infrequently used for both interfaces, while epistemic actions were strongly promoted for the TUI and contributed to participants’ efficient exploration strategies

From pixels to gestures: learning visual representations for human analysis in color and depth data sequences

[cat] L’anàlisi visual de persones a partir d'imatges és un tema de recerca molt important, atesa la rellevància que té a una gran quantitat d'aplicacions dins la visió per computador, com per exemple: detecció de vianants, monitorització i vigilància,interacció persona-màquina, “e-salut” o sistemes de recuperació d’matges a partir de contingut, entre d'altres. En aquesta tesi volem aprendre diferents representacions visuals del cos humà, que siguin útils per a la anàlisi visual de persones en imatges i vídeos. Per a tal efecte, analitzem diferents modalitats d'imatge com són les imatges de color RGB i les imatges de profunditat, i adrecem el problema a diferents nivells d'abstracció, des dels píxels fins als gestos: segmentació de persones, estimació de la pose humana i reconeixement de gestos. Primer, mostrem com la segmentació binària (objecte vs. fons) del cos humà en seqüències d'imatges ajuda a eliminar soroll pertanyent al fons de l'escena en qüestió. El mètode presentat, basat en optimització “Graph cuts”, imposa consistència espai-temporal a Ies màscares de segmentació obtingudes en “frames” consecutius. En segon lloc, presentem un marc metodològic per a la segmentació multi-classe, amb la qual podem obtenir una descripció més detallada del cos humà, en comptes d'obtenir una simple representació binària separant el cos humà del fons, podem obtenir màscares de segmentació més detallades, separant i categoritzant les diferents parts del cos. A un nivell d'abstraccíó més alt, tenim com a objectiu obtenir representacions del cos humà més simples, tot i ésser suficientment descriptives. Els mètodes d'estimació de la pose humana sovint es basen en models esqueletals del cos humà, formats per segments (o rectangles) que representen les extremitats del cos, connectades unes amb altres seguint les restriccions cinemàtiques del cos humà. A la pràctica, aquests models esqueletals han de complir certes restriccions per tal de poder aplicar mètodes d'inferència que permeten trobar la solució òptima de forma eficient, però a la vegada aquestes restriccions suposen una gran limitació en l'expressivitat que aques.ts models son capaços de capturar. Per tal de fer front a aquest problema, proposem un enfoc “top-down” per a predir la posició de les parts del cos del model esqueletal, introduïnt una representació de parts de mig nivell basada en “Poselets”. Finalment. proposem un marc metodològic per al reconeixement de gestos, basat en els “bag of visual words”. Aprofitem els avantatges de les imatges RGB i les imatges; de profunditat combinant vocabularis visuals específiques per a cada modalitat, emprant late fusion. Proposem un nou descriptor per a imatges de profunditat invariant a rotació, que millora l'estat de l'art, i fem servir piràmides espai-temporals per capturar certa estructura espaial i temporal dels gestos. Addicionalment, presentem una reformulació probabilística del mètode “Dynamic Time Warping” per al reconeixement de gestos en seqüències d'imatges. Més específicament, modelem els gestos amb un model probabilistic gaussià que implícitament codifica possibles deformacions tant en el domini espaial com en el temporal.[eng] The visual analysis of humans from images is an important topic of interest due to its relevance to many computer vision applications like pedestrian detection, monitoring and surveillance, human-computer interaction, e-health or content-based image retrieval, among others. In this dissertation in learning different visual representations of the human body that are helpful for the visual analysis of humans in images and video sequences. To that end, we analyze both RCB and depth image modalities and address the problem from three different research lines, at different levels of abstraction; from pixels to gestures: human segmentation, human pose estimation and gesture recognition. First, we show how binary segmentation (object vs. background) of the human body in image sequences is helpful to remove all the background clutter present in the scene. The presented method, based on “Graph cuts” optimization, enforces spatio-temporal consistency of the produced segmentation masks among consecutive frames. Secondly, we present a framework for multi-label segmentation for obtaining much more detailed segmentation masks: instead of just obtaining a binary representation separating the human body from the background, finer segmentation masks can be obtained separating the different body parts. At a higher level of abstraction, we aim for a simpler yet descriptive representation of the human body. Human pose estimation methods usually rely on skeletal models of the human body, formed by segments (or rectangles) that represent the body limbs, appropriately connected following the kinematic constraints of the human body, In practice, such skeletal models must fulfill some constraints in order to allow for efficient inference, while actually Iimiting the expressiveness of the model. In order to cope with this, we introduce a top-down approach for predicting the position of the body parts in the model, using a mid-level part representation based on Poselets. Finally, we propose a framework for gesture recognition based on the bag of visual words framework. We leverage the benefits of RGB and depth image modalities by combining modality-specific visual vocabularies in a late fusion fashion. A new rotation-variant depth descriptor is presented, yielding better results than other state-of-the-art descriptors. Moreover, spatio-temporal pyramids are used to encode rough spatial and temporal structure. In addition, we present a probabilistic reformulation of Dynamic Time Warping for gesture segmentation in video sequences, A Gaussian-based probabilistic model of a gesture is learnt, implicitly encoding possible deformations in both spatial and time domains

Tele-proximity: The Experienced Educator Perspective on Human to Human Connection in Distance Education.

Distance education is an integral part of many universities worldwide, aiming to offer students opportunities for life-long learning and equitable access. Distant learners face many barriers and as a result, they drop out more frequently than on-campus students. Educators seem to be equally affected by the 'transactional distance' and the new digital skills needed for teaching online. The purpose of this study is to explore the educators' perspective on how synchronous video communication (SVC) could offer alternative educational forms for distance learning. Specifically the study aims to fill gaps in the existing research literature: to describe synchronous teaching approaches that are used worldwide to support e-learners, to investigate the role of the educator, to discover how learning and subject content are affected by SVC, to see if it enhances the sense of togetherness (immediacy and intimacy) to specify the contextual factors influencing teaching and learning synchronously, and to formulate a theory. The Informed Grounded Theory (Thornberg, 2012) and the Community of Inquiry model (Garrison, Anderson & Archer, 2000) provided a scaffolding framework for designing interview questions and analysing findings. The findings of the study illustrate the empirical evidence of the value of SVC, its potential forms (teaching approaches), possibilities (linking educational purposes to the medium) and contextual factors (University policy, time zones, technologies, learning objectives, educator and student identities). Specifically, audiovisual cues have an impact on educators, the learning process and immediacy. The theory of Tele-proximity formulated to explain how educators and students are connected in synchronous networked environment via tele-operations. The study aims at helping educators/instructional designers, and administrators to face the difficulties of transactional distance and make informed decisions about synchronous video enhanced communication. Researchers may use Tele-proximity as a heuristic tool to continue the scientific dialogue on the potential of synchronous video-enhanced technologies in distance education

DOKY: A Multi-Modal User Interface for Non-Visual Presentation, Navigation and Manipulation of Structured Documents on Mobile and Wearable Devices

There are a large number of highly structured documents available on the Internet. The logical document structure is very important for the reader in order to efficiently handling the document content. In graphical user interfaces, each logical structure element is presented by a specific visualisation, a graphical icon. This representation allows visual readers to recognise the structure at a glance. Another advantage is that it enables direct navigation and manipulation. Blind and visually impaired persons are unable to use graphical user interfaces and for the emerging category of mobile and wearable devices, where there are only small visual displays available or no visual display at all, a non-visual alternative is required too. A multi-modal user interface for non-visual presentation, navigation and manipulation of structured documents on mobile and wearable devices like smart phones, smart watches or smart tablets has been developed as a result of inductive research among 205 blind and visually impaired participants. It enables the user to get a fast overview over the document structure and to efficiently skim and scan over the document content by identifying the type, level, position, length, relationship and content text of each element as well as to focus, select, activate, move, remove and insert structure elements or text. These interactions are presented in a non-visual way using Earcons, Tactons and synthetic speech utterances, serving the auditory and tactile human sense. Navigation and manipulation is provided by using the multitouch, motion (linear acceleration and rotation) or speech recognition input modality. It is a complete solution for reading, creating and editing structured documents in a non-visual way. There is no special hardware required. The name DOKY is derived from a short form of the terms document, and accessibility. A flexible platform-independent and event-driven software architecture implementing the DOKY user interface as well as the automated structured observation research method employed for the investigation into the effectiveness of the proposed user interface has been presented. Because it is platform- and language-neutral, it can be used in a wide variety of platforms, environments and applications for mobile and wearable devices. Each component is defined by interfaces and abstract classes only, so that it can be easily changed or extended, and grouped in a semantically self-containing package. An investigation into the effectiveness of the proposed DOKY user interface has been carried out to see whether the proposed user interface design concepts and user interaction design concepts are effective means for non-visual presentation, navigation and manipulation of structured documents on mobile and wearable devices, by automated structured observations of 876 blind and visually impaired research subjects performing 19 exercises among a highly structured example document using the DOKY Structured Observation App on their own mobile or wearable device remotely over the Internet. The results showed that the proposed user interface design concepts for presentation and navigation and the user interaction design concepts for manipulation are effective and that their effectiveness depends on the input modality and hardware device employed as well as on the use of screen readers