Quantifying the predictability of visual scanpaths using active information storage

Entropy-based measures are an important tool for studying human gaze behavior under various conditions. In particular, gaze transition entropy (GTE) is a popular method to quantify the predictability of fixation transitions. However, GTE does not account for temporal dependencies beyond two consecutive fixations and may thus underestimate a scanpath's actual predictability. Instead, we propose to quantify scanpath predictability by estimating the active information storage (AIS), which can account for dependencies spanning multiple fixations. AIS is calculated as the mutual information between a processes' multivariate past state and its next value. It is thus able to measure how much information a sequence of past fixations provides about the next fixation, hence covering a longer temporal horizon. Applying the proposed approach, we were able to distinguish between induced observer states based on estimated AIS, providing first evidence that AIS may be used in the inference of user states to improve human-machine interaction.Comment: 19 pages, 3 figure

A cloud-based robot system for long-term interaction: principles, implementation, lessons learned

Making the transition to long-term interaction with social-robot systems has been identified as one of the main challenges in human-robot interaction. This article identifies four design principles to address this challenge and applies them in a real-world implementation: cloud-based robot control, a modular design, one common knowledge base for all applications, and hybrid artificial intelligence for decision making and reasoning. The control architecture for this robot includes a common Knowledge-base (ontologies), Data-base, “Hybrid Artificial Brain” (dialogue manager, action selection and explainable AI), Activities Centre (Timeline, Quiz, Break and Sort, Memory, Tip of the Day, ), Embodied Conversational Agent (ECA, i.e., robot and avatar), and Dashboards (for authoring and monitoring the interaction). Further, the ECA is integrated with an expandable set of (mobile) health applications. The resulting system is a Personal Assistant for a healthy Lifestyle (PAL), which supports diabetic children with self-management and educates them on health-related issues (48 children, aged 6–14, recruited via hospitals in the Netherlands and in Italy). It is capable of autonomous interaction “in the wild” for prolonged periods of time without the need for a “Wizard-of-Oz” (up until 6 months online). PAL is an exemplary system that provides personalised, stable and diverse, long-term human-robot interaction

Annotations of maps in collaborative work at a distance

This thesis inquires how map annotations can be used to sustain remote collaboration. Maps condense the interplay of space and communication, solving linguistic references by linking conversational content to the actual places to which it refers. This is a mechanism people are accustomed to. When we are face-to-face, we can point to things around us. However, at a distance, we need to recreate a context that can help disambiguate what we mean. A map can help recreate this context. However other technological solutions are required to allow deictic gestures over a shared map when collaborators are not co-located. This mechanism is here termed Explicit Referencing. Several systems that allow sharing maps annotations are reviewed critically. A taxonomy is then proposed to compare their features. Two filed experiments were conducted to investigate the production of collaborative annotations of maps with mobile devices, looking for the reasons why people might want to produce these notes and how they might do so. Both studies led to very disappointing results. The reasons for this failure are attributed to the lack of a critical mass of users (social network), the lack of useful content, and limited social awareness. More importantly, the study identified a compelling effect of the way messages were organized in the tested application, which caused participants to refrain from engaging in content-driven explorations and synchronous discussions. This last qualitative observation was refined in a controlled experiment where remote participants had to solve a problem collaboratively, using chat tools that differed in the way a user could relate an utterance to a shared map. Results indicated that team performance is improved by the Explicit Referencing mechanisms. However, when this is implemented in a way that is detrimental to the linearity of the conversation, resulting in the visual dispersion or scattering of messages, its use has negative consequences for collaborative work at a distance. Additionally, an analysis of the eye movements of the participants over the map helped to ascertain the interplay of deixis and gaze in collaboration. A primary relation was found between the pair's recurrence of eye movements and their task performance. Finally, this thesis presents an algorithm that detects misunderstandings in collaborative work at a distance. It analyses the movements of collaborators' eyes over the shared map, their utterances containing references to this workspace, and the availability of "remote" deictic gestures. The algorithm associates the distance between the gazes of the emitter and gazes of the receiver of a message with the probability that the recipient did not understand the message

Speech production, dual-process theory, and the attentive addressee

This thesis outlines a model of Speaker-Addressee interaction that suggests some answers to two linked problems current in speech production. The first concerns an under-researched issue in psycholinguistics: how are decisions about speech content – conceptualization – carried out? The second, a pragmatics problem, asks how Speakers, working under the heavy time pressures of normal dialogue, achieve optimal relevance often enough for successful communication to take place. Links between these problems are discussed in Chapter 1; Chapter 2 reviews existing research on speech production and dialogue. Chapter 3 presents the central claim of my thesis: that the Addressee exerts a significant influence over the Speaker’s decision-making at a level below the latter’s consciousness. Using evidence drawn from psycholinguistics, developmental psychology and human-computer interaction, Chapter 4 presents evidence to support this claim, demonstrating that a Speaker’s performance can be decisively affected at a preconscious level by the degree of attentiveness shown by the Addressee. Lack of attentiveness, in particular, appears to damage speech production at the conceptualization level. I suggest, therefore, that Speaker and Addressee are linked in a feedback loop: unless a Speaker achieves and maintains relevance to an Addressee, the Addressee’s interest will be lost, and this will impair the Speaker’s production abilities and hence the communication process itself. Chapters 5 and 6 consider some automatic mechanisms that may help Speakers dovetail their productions to Addressee need. These include the neural mechanisms underlying face perception and social rejection; automatic aspects of theory of mind; intuitive memory and inference systems of the type being explored in dual-process theory; and connections between verbal performance and behavioural priming currently being investigated. Chapter 7 summarizes the complete argument, discusses its wider implications, and includes suggestions for further work

Social vulnerability in Williams syndrome

This thesis focused on the high levels of social vulnerability experienced by individuals with Williams Syndrome (WS). The investigation began with parent interviews about social approach behaviour, with parents emphasising the lack of awareness of social boundaries that many individuals with WS display. The qualitative analysis also highlighted the within-syndrome variability in the parental accounts, prompting discussion on the heterogeneity of the WS social profile. Based on the atypical social approach behaviour described by parents, the subsequent studies addressed issues of personal space and interpersonal distance. Using a parent report questionnaire, it was found that children with WS were more likely to violate the personal space of others. This was followed up with a stop-distance paradigm which showed that children with WS failed to regulate their distance based on familiarity, and stood the same distance from a stranger as they did their parent, which was not the case for typically developing individuals. Given these findings, the research progressed to explore the issue of trust in WS. It was found that children with WS displayed higher levels of trust behaviour, compared to their mental age matched typically developing peers and struggled to decipher trustworthiness from faces. Taken together, these findings seem to suggest that children with WS could be experiencing high levels of social vulnerability on a daily basis. It is widely accepted that this social vulnerability continues into adulthood, with increased levels of both independence and isolation posing a new set of challenges. The subsequent chapter probed the level of insight that adults with WS had about their own vulnerability. Using the Social Vulnerability Questionnaire, it was found that adults with WS consistently reported lower levels of vulnerability, compared to parent reports. This emphasised the need for multi-informant methods, and called for interventions which target self-awareness in order to increase intervention efficacy. The final chapters looked at how this social vulnerability manifests in the online environment. It was found that adults with WS frequently use the Internet and the majority visit social networking sites every day or almost every day, with little parental supervision or oversight. These individuals were more likely to agree to engage in socially risky behaviours (e.g. meeting an “online friend” in person) compared to risky behaviours that were not social in nature when online (e.g. giving out passwords). A case study interview with an adult with WS and their parent highlighted that this individual held a broad definition of what a friend was and found they used the Internet as a tool to expand their social network, which was of great concern to their parent. The findings included in this thesis provide in-depth information relating to social vulnerability in WS and offer the first insights into online social behaviour and online vulnerability in adults with WS. The theoretical and real-world implications of these findings are emphasised throughout and a number of suggestions are made to help the research progress towards intervention development

Contextual Factors Affecting Information Sharing Patterns in Technology Mediated Communication

In this thesis, we investigate how and what contextual factors affect user’s information sharing. We build our work on six individual research projects which cover a variety of systems (search engines, social network sites, teleconferencing systems, monitoring technology, and general purpose conversational agents) in a variety of communication scenarios with diverse relationships and dispositions of users. Alongside detailed findings for particular systems and communication scenarios from each individual project, we provide a consolidated analysis of these results across systems and scenarios, which allows us to identify patterns specific for different system types and aspects shared between systems. In particular, we show that depending on the system’s position between a user and an intended information receiving agent – whether communication happens through, around, or directly with the system – the system should have different patterns of operational adaptation to communication context. Specifically, when communication happens through the system, the system needs to gather communication context unavailable to the user and integrate it into information communication; when communication happens around the system, the system should adapt its operations to provide information in the most contextually suitable format; finally, when a user communicates with the system, the role of the system is to “match” this context in communication with the user. We then argue that despite the differences between system types in patterns of required context-based adaptation, there are contextual factors affecting user’s information sharing intent that should be acknowledged across systems. Grounded in our cumulative findings and analysis of related literature, we identify four such high-level contextual factors. We then present these four factors synthesized into an early design framework, which we call SART according to the included factors of space, addressee, reason, and time. Each factor in SART is presented as a continuum defined through a descriptive dichotomy: perceived breadth of communication space (public to private); perceived specificity of an information addressee (defined to undefined); intended reason for information sharing (instrumental to objective); and perceived time of information relevance and life-span (immediate to indefinite)

The Intercultural Communication Experiences of Arab Muslims Studying in New Zealand: Academic and Social Perspectives

The number of Arab Muslim students in New Zealand has increased significantly in recent years, yet there is a lack of New Zealand studies that investigate this phenomenon. Studies that have examined the experiences of these students in Western academic contexts suggest, however, that there is a need for further investigation to understand the extent to which these students (re)construct and (re)ne¬go¬tiate their identities as a result of their intercultural communication experiences. The purpose of this study is to examine how universities’ communication practices influence the negotiation process of these students’ cultural and religious identities. In addition, the study investigates which communication practices adopted by students facilitate, or inhibit, good communication with New Zealanders. Cultural identity theory and structuration theory were used as the theoretical framework to understand the reconstruction and renegotiation of students’ cultural and religious identities. In-depth phenomenological interviews were conducted with 45 male and female participants to elicit their personal stories. Eight university administrators were also interviewed, and university documents were analysed to explore the organisational perspectives in dealing with the presence of these students. Thematic, content, and structuration analyses were conducted with the assistance of NVivo software. Given that the researcher is also an Arab Muslim student, methodological and ethical challenges (e.g., recruiting and engaging with participants, and conducting semistructured face-to-face interviews) were explored reflexively. Analysis of the data for this study suggested three main findings. First, both Islamic and cultural values guided the direction of Arab Muslim students’ daily lives. Participants noted a number of issues that reflected their emphatic, forthright identification with their own cultural and religious heritage. These issues involved insisting on the role of social networks to protect feminine identity and the integrity of people’s social reputation; the importance of consuming halal food and securing a space to perform daily prayers; the avoidance of working with the opposite sex, and avoid any university and community activities that include practices contrary to their own values. Second, the negotiation of power between universities and participants was observed. As New Zealand universities used human and nonhuman resources, they were able to wield power over participants and influence them to negotiate and reflect on their own values and norms. Participants appealed to the concept of the purification of Islam to rationalise their motivation for reflecting on and questioning their own values and norms. This reflection resulted in the adoption by students of guidelines and strategies for interaction, avoidance, and normalisation. Third, the students’ length of residence appeared to be an indicator of disiden¬ti¬fi¬cation with their own values and identification with the dominant values in matters relating to the segregation of the sexes and modesty. A redefining of concepts of gender roles, being alone, freedom, and others was observed over time. The experience of negotiating cultural and religious identities affirmed that cultural identity is constructed in the intercultural communication context as participants worked out a sort of two identities which combines elements from the old and the new. The study contributes significantly to existing research on intercultural communication by hearing Arab Muslim students’ voices on issues that arise as they encounter new cultural values, seek to maintain their cultural and religious identities, and navigate between home and host values. Among significant contributions to theoretical knowledge, we can include conceptualising the gender roles, being alone in a Western country, gender relations, segregation of the sexes, modesty, freedom, hijab, and personal freedom. In particular, the study enriches the extant literature examining the negotiation process of individual identity from the structuration and cultural identity point of view. In addition to these contributions, implications were drawn for educational institutions, government policies, and future Arab Muslim students to help them obtain constructive intercultural communication experiences in a dominant culture. The suggestion for future studies was made to further explore the role of female gender and length of residency in the reconstruction and renegotiation of cultural identity