Augmenting Situated Spoken Language Interaction with Listener Gaze

Collaborative task solving in a shared environment requires referential success. Human speakers follow the listener’s behavior in order to monitor language comprehension (Clark, 1996). Furthermore, a natural language generation (NLG) system can exploit listener gaze to realize an effective interaction strategy by responding to it with verbal feedback in virtual environments (Garoufi, Staudte, Koller, & Crocker, 2016). We augment situated spoken language interaction with listener gaze and investigate its role in human-human and human-machine interactions. Firstly, we evaluate its impact on prediction of reference resolution using a mulitimodal corpus collection from virtual environments. Secondly, we explore if and how a human speaker uses listener gaze in an indoor guidance task, while spontaneously referring to real-world objects in a real environment. Thirdly, we consider an object identification task for assembly under system instruction. We developed a multimodal interactive system and two NLG systems that integrate listener gaze in the generation mechanisms. The NLG system “Feedback” reacts to gaze with verbal feedback, either underspecified or contrastive. The NLG system “Installments” uses gaze to incrementally refer to an object in the form of installments. Our results showed that gaze features improved the accuracy of automatic prediction of reference resolution. Further, we found that human speakers are very good at producing referring expressions, and showing listener gaze did not improve performance, but elicited more negative feedback. In contrast, we showed that an NLG system that exploits listener gaze benefits the listener’s understanding. Specifically, combining a short, ambiguous instruction with con- trastive feedback resulted in faster interactions compared to underspecified feedback, and even outperformed following long, unambiguous instructions. Moreover, alternating the underspecified and contrastive responses in an interleaved manner led to better engagement with the system and an effcient information uptake, and resulted in equally good performance. Somewhat surprisingly, when gaze was incorporated more indirectly in the generation procedure and used to trigger installments, the non-interactive approach that outputs an instruction all at once was more effective. However, if the spatial expression was mentioned first, referring in gaze-driven installments was as efficient as following an exhaustive instruction. In sum, we provide a proof of concept that listener gaze can effectively be used in situated human-machine interaction. An assistance system using gaze cues is more attentive and adapts to listener behavior to ensure communicative success

Modifiability of the psychomotor domain

Bibliography: p. 194-225

Eyewear Computing \u2013 Augmenting the Human with Head-Mounted Wearable Assistants

The seminar was composed of workshops and tutorials on head-mounted eye tracking, egocentric vision, optics, and head-mounted displays. The seminar welcomed 30 academic and industry researchers from Europe, the US, and Asia with a diverse background, including wearable and ubiquitous computing, computer vision, developmental psychology, optics, and human-computer interaction. In contrast to several previous Dagstuhl seminars, we used an ignite talk format to reduce the time of talks to one half-day and to leave the rest of the week for hands-on sessions, group work, general discussions, and socialising. The key results of this seminar are 1) the identification of key research challenges and summaries of breakout groups on multimodal eyewear computing, egocentric vision, security and privacy issues, skill augmentation and task guidance, eyewear computing for gaming, as well as prototyping of VR applications, 2) a list of datasets and research tools for eyewear computing, 3) three small-scale datasets recorded during the seminar, 4) an article in ACM Interactions entitled \u201cEyewear Computers for Human-Computer Interaction\u201d, as well as 5) two follow-up workshops on \u201cEgocentric Perception, Interaction, and Computing\u201d at the European Conference on Computer Vision (ECCV) as well as \u201cEyewear Computing\u201d at the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)

Young children’s grounding of mathematical thinking in sensory-motor experiences

Dissertation articles have been removed from the digital version, due to copyright issues. They can be read in the printed edition.Hovedmålet med avhandling min er å utvikle kunnskap om små barns forankring av matematisk tenkning i sensoriske-motoriske erfaringer, mens et delmål er å øke forståelsen for hvordan utendørs design kan støtte slike erfaringer. To intervensjoner som engasjerte 27 barn i alderen 3 til 5 år ble gjennomført som et samarbeid mellom 4 barnehagelærere i to barnehager og to forskere. Intervensjonene la grunnlaget for tre fokusstudier der det empiriske materialet består av video av barna i individuelle etter-tester. Teorien kroppslig situert kognisjon (Embodied Cognition) er brukt i analysen der en detaljert vurdering av koherens mellom oppgave adferd og det matematiske målområdet i det respektive delstudiet dannet grunnlag for sammenlikning og en fler-kasus dybdeanalyse av karakteristikker innenfor og på tvers av identifiserte mønstre for kroppslig situering av matematisk tenking. Tre forskjellige aspekter ble fokusert på: Karteristiske trekk ved delmengde-kjenneres (dvs. barn som ikke viser ferdigheter i bruk av telling for å produsere små mengder) evner til å behandle små mengder som helheter gjennom tale og kroppslig interaksjon i en stor sirkel med 50 merker (fokus studie 1); barnas evner til å gjenskape symmetrisk strukturerte kroppslige erfaringer med tallmengder for å støtte additive resonnement, og barnas evner til kroppslig modellering av tellebasert addisjon (fokus studie 3). Resultatene viste karakteristiske og divergerende trekk ved delmengde-kjenneres kroppslig situering av kardinaltallbegrepet som også omfattet produksjon av små mengder over målt begrepsnivå, og videre hvordan sensoriske-motoriske erfaringer kan støtte telle-basert addisjon og resonnement rundt del-helhet relasjoner. Avhandlingen gir et bidrag til forskningsområdet knyttet til tidlig strukturbasert kroppslig læring i matematikk, og spesielt gjennom funn som viser karakteristikker i barns situering, avlastning og koherens av matematisk tenkning i bevegelse og motorisk interaksjon. Uventede funn var inkludering av estetiske, rytmiske og sammensatte bevegelsesmønstre i den kroppsbaserte matematiske tenkningen. I lys av det kroppslige situerte perspektivet bør resultatene oppmuntre til design av utendørs aktiviteter som involverer bevegelse og rytme i den tidlige læringen av matematikk. For å konkludere understreker avhandlingen rollen som kroppslig bevegelse og fysisk interaksjon med romlige strukturer kan utgjøre i små barns matematiske tenkning.ABSTRACT: The main objective of my dissertation is to develop knowledge about young children’s grounding of mathematical thinking in sensory-motor experiences while a sub-goal is to increase the understanding of how outdoor embodied designs can facilitate such experiences. Two embodied training programmes engaging 27 children aged 3 to 5 years were conducted as a collaboration between four kindergarten teachers in two kindergartens and two researchers. These interventions led to three focal studies, where the empirical material consists of video footage of the children in individual post-tests. The data are analysed through the framework of Embodied Cognition, involving detailed attention to each child’s cohering of task behaviour with the mathematical targeting domain addressed in the respective focal study, followed by a cross-case comparison and a multi-case analysis across and within the identified patterns of grounding of mathematical thinking in bodily action. Three different aspects were focused on: Characteristic features of subset-knowers’ (i.e., children unable to use counting for exact numbering) abilities in establishing congruence between the idea of cardinality and verbalised body-spatial mapping of small sets (focal study 1); children’s re-enactment of canonical structured experiences of numerosity in reasoning about additive compositions (focal study 2) and children’s congruency in the physical grounding of counting based addition (focal study 3). The results showed recurring and deviating patterns of subset-knowers’ grounding of the idea of cardinality in bodily production of small sets that also exceeded their knower-level, and the findings showed how sensory-motor action might concur with counting-based addition and support reasoning about additive compositions. Unexpected findings comprise the inclusion of expressive body movements (e.g., rotation, rhythm, force, and tempo) in the situating of mathematical thinking. The dissertation study contributes to the field of educational research on early structured-based bodily learning of mathematics as it revealed characteristics of young children’s situating, off-loading and cohering of mathematical thinking in full-body interaction. In light of the embodied perspective, this should encourage the design of activities outdoors that involve movement and rhythm in the early learning of mathematics. In conclusion, this dissertation underlines the role that bodily movement and physical interaction with spatial structures can play in young children’s mathematical thinking

Gender differences in navigation dialogues with computer systems

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Gender is among the most influential of the factors underlying differences in spatial abilities, human communication and interactions with and through computers. Past research has offered important insights into gender differences in navigation and language use. Yet, given the multidimensionality of these domains, many issues remain contentious while others unexplored. Moreover, having been derived from non-interactive, and often artificial, studies, the generalisability of this research to interactive contexts of use, particularly in the practical domain of Human-Computer Interaction (HCI), may be problematic. At the same time, little is known about how gender strategies, behaviours and preferences interact with the features of technology in various domains of HCI, including collaborative systems and systems with natural language interfaces. Targeting these knowledge gaps, the thesis aims to address the central question of how gender differences emerge and operate in spatial navigation dialogues with computer systems. To this end, an empirical study is undertaken, in which, mixed-gender and same-gender pairs communicate to complete an urban navigation task, with one of the participants being under the impression that he/she interacts with a robot. Performance and dialogue data were collected using a custom system that supported synchronous navigation and communication between the user and the robot. Based on this empirical data, the thesis describes the key role of the interaction of gender in navigation performance and communication processes, which outweighed the effect of individual gender, moderating gender differences and reversing predicted patterns of performance and language use. This thesis has produced several contributions; theoretical, methodological and practical. From a theoretical perspective, it offers novel findings in gender differences in navigation and communication. The methodological contribution concerns the successful application of dialogue as a naturalistic, and yet experimentally sound, research paradigm to study gender and spatial language. The practical contributions include concrete design guidelines for natural language systems and implications for the development of gender-neutral interfaces in specific domains of HCI

A third space pedagogy: embracing complexity in a super-diverse, early childhood education setting

In Britain, ‘super-diverse’ communities, where children navigate multiple cultural repertoires, are increasingly prevalent. However, Reception teachers are pressured to ensure children, aged four and five, conform to a narrow conception of ‘school-readiness’. Research demonstrates children in multicultural contexts construct a ‘third space’, bridging their home and school discourses. This research shows how opportunities for third space creation are inherently tied to the nature of physical space, and its concomitant social expectations. It is argued that complexity in super-diverse communities can be harnessed and embraced, rather than reduced. Data presented were drawn from a year-long collaborative ethnographic study of children in a Reception class in the north of England. Children co-created cartoons, collaborating with the researcher in interpreting the data. Significantly, findings indicate that teachers can incorporate the third space as an alternative lens through which to understand and meet the challenges of teaching a linguistically and culturally diverse student cohort

A sensorimotor account of visual attention in natural behaviour

The real-world sensorimotor paradigm is based on the premise that sufficient ecological complexity is a prerequisite for inducing naturally relevant sensorimotor relations in the experimental context. The aim of this thesis is to embed visual attention research within the real-world sensorimotor paradigm using an innovative mobile gaze-tracking system (EyeSeeCam, Schneider et al., 2009). Common laboratory set-ups in the field of attention research fail to create natural two-way interaction between observer and situation because they deliver pre-selected stimuli and human observer is essentially neutral or passive. EyeSeeCam, by contrast, permits an experimental design whereby the observer freely and spontaneously engages in real-world situations. By aligning a video camera in real time to the movements of the eyes, the system directly measures the observer’s perspective in a video recording and thus allows us to study vision in the context of authentic human behaviour, namely as resulting from past actions and as originating future actions. The results of this thesis demonstrate that (1) humans, when freely exploring natural environments, prefer directing their attention to local structural features of the world, (2) eyes, head and body perform distinct functions throughout this process, and (3) coordinated eye and head movements do not fully stabilize but rather continuously adjust the retinal image also during periods of quasi-stable “fixation”. These findings validate and extend the common laboratory concept of feature salience within whole-body sensorimotor actions outside the laboratory. Head and body movements roughly orient gaze, potentially driven by early stages of processing. The eyes then fine-tune the direction of gaze, potentially during higher-level stages of visual-spatial behaviour (Studies 1 and 2). Additional head-centred recordings reveal distinctive spatial biases both in the visual stimulation and the spatial allocation of gaze generated in a particular real-world situation. These spatial structures may result both from the environment and form the idiosyncrasies of the natural behaviour afforded by the situation. By contrast, when the head-centred videos are re-played as stimuli in the laboratory, gaze directions reveal a bias towards the centre of the screen. This “central bias” is likely a consequence of the laboratory set-up with its limitation to eye-in-head movements and its restricted screen (Study 3). Temporal analysis of natural visual behaviour reveals frequent synergistic interactions of eye and head that direct rather than stabilize gaze in the quasi-stable eye movement periods following saccades, leading to rich temporal dynamics of real-world retinal input (Study 4) typically not addressed in laboratory studies. Direct comparison to earlier data with respect to the visual system of cats (CatCam), frequently taken as proxy for human vision, shows that stabilizing eye movements play an even less dominant role in the natural behaviour of cats. This highlights the importance of realistic temporal dynamics of vision for models and experiments (Study 5). The approach and findings presented in this thesis demonstrate the need for and feasibility of real- world research on visual attention. Real-world paradigms permit the identification of relevant features triggered in the natural interplay between internal-physiological and external-situational sensorimotor factors. Realistic spatial and temporal characteristics of eye, head and body interactions are essential qualitative properties of reliable sensorimotor models of attention but difficult to obtain under laboratory conditions. Taken together, the data and theory presented in this thesis suggest that visual attention does not represent a pre-processing stage of object recognition but rather is an integral component of embodied action in the real world

A Question Of Language Disorder: Studies Of Assessment, Management And Parent Attitude.

Children labelled "language disordered" failed to make academic progress in schools promoting investigations into the nature of their problems, attendant attitudes and contextual influences. Studies find differences in haptic, auditory and visual processing between normal and language disordered children. Individual management does not acknowledge this range of modality input problems and selectively targets language form for systematic development. The approach does not result in better school attainments. Perhaps the label "language disorder" limits perception of the extent of difficulties in other areas. In contrast, an interactive method is described, taking account of "inside" and "outside" the child factors. Educational success is produced by inter-relating the language system with the learning context. Full implementation of the interactive model seems unlikely given the present styles of professional training and existing institutional constraints. However, consumer dissatisfaction with current language Learning provision suggests this procedure offers a promising alternative

Drawing as a Method for Accessing Young Children's Perspectives in Research

Researchers have taken a particular interest in children’s drawings as a means of representing and communicating knowledge and perspectives but a review of literature reveals that researchers routinely use drawings as a way of obtaining data without considering their function or value. This ESRC-funded research aims to explore drawing as a method of accessing children's perspectives and has three central research objectives which consider methodological and analytical factors relating to the use of children’s drawings as a research tool. These are: to develop a principled approach to analysing and interpreting children’s drawings, to create guidelines for the use of drawing as a research tool, and to gather children’s perspectives on play through the method of drawing. The research objectives were achieved by asking the following three questions: How can children’s drawings be analysed using a principled approach? What are the major factors to be considered when using drawing as a research tool? What can drawings reveal about children's perspectives on play? The study involved two visits to the homes of eight preschool children aged four. The sample included four girls and four boys from central and north-east Scotland with half of the families being categorised as being of low socioeconomic status. Visits were flexible and unstructured allowing the child autonomy regarding our level of interaction and the types of activities (such as free play and conversation) with which they wished to engage. The second visit included a prompted drawing activity in which I invited children to express their perspectives on play. The topic of play was chosen (i) to offer children a meaningful research activity to investigate the issues surrounding the method, (ii) to explore the task of representing an abstract, yet familiar, concept and how this may influence children’s drawings and representations of play, and (iii) as an extension of the ESRC project Young Children Learning with Toys and Technology at Home (Plowman et al., 2012) by giving greater emphasis to children's own perspectives on play and exploring the ways in which this can be achieved. My theoretical approach is not to consider drawings as reproductions of reality, but to value and attempt to understand children’s drawings as a semiotic vehicle in which messages are created and conveyed during the drawing process through representation and signification. Informed by social semiotics (Kress & van Leeuwen, 1996) the research presents an innovative four-step approach to analysing children's drawings (4-SASA). The protocol, a key contribution of the research, was developed to promote a more systematic analysis, involving (i) isolating signs within drawings through manual annotation, (ii) documenting the child’s understanding of signs and the significance attributed to them, (iii) organising signs using specific categories of social semiotic analysis (mode, size, colour, salience) and identifying the child’s motivation and interest for specific sign production, and (iv) synthesis of the child’s perspectives from steps 1-3. Post hoc methodological examinations elucidated the following four key factors to be considered when using young children’s drawings: (i) contextual sensitivity of the drawing process, (ii) children’s perceptions of the research task, (iii) the complex task of representing an abstract and elusive concept such as play, and (iv) whether there is a fundamental difference between drawing spontaneously (non-commissioned) and drawing on request. Evidence from the study supports previous literature in demonstrating the potential of drawing as a method of accessing children’s perspectives. However, findings suggest that rather than routinely selecting drawing as a method for representing children’s perspectives, researchers need to be more thoughtful about the ways in which factors such as the social and contextual framing of drawing and approaches to data collection can affect research outcomes. The thesis concludes by discussing how these emerging issues impact research outcomes, along with implications for future implementation and analysis of drawings