Thinking head: Towards human centred robotics

Thinking Head project is a multidisciplinary approach to building intelligent agents for human machine interaction. The Thinking Head Framework evolved out of the Thinking Head Project and it facilitates loose coupling between various components and forms the central nerve system in a multimodal perception-action system. The paper presents the overall architecture, components and the attention system. The paper then concludes with a preliminary behavioral experiment that studies the intelligibility of the audiovisual speech output produced by the Embodied Conversational Agent (ECA) that is part of the system. These results provide the baseline for future evaluations of the system as the project progresses through multiple evaluate and refine cycles

The Effect of Familiarity and Complexity of Environments, and Mode of Wayfinding on Gaze and Memory

Whether we travel the well-known route to work or unfamiliar streets in a new city, we use wayfinding in order to determine and follow a route to get to our destination. Wayfinding is dependent on cognitive processes, specifically attention and memory which has been reinforced in the observation of challenges in wayfinding among individuals who have suffered cognitive dysfunction as the result of neurologic injury or diseases (eg stroke or Alzheimer’s). The studies are focused on advancing the understanding of the role of visual attention, associated gaze behaviour, and memory on the control of wayfinding. The first study focused on determining how changes in the familiarity and complexity of an environment influence visual attention during wayfinding. The second study investigated how the method of learning an environment, either actively or passively, would influence gaze behaviour. The results from study 1 showed that both novel and visually complex environments were characterized by an increase in the number of fixations, and therefore the amount of directed attention towards landmarks when wayfinding. Study 2 revealed that when learning an environment actively there is an increase in fixations and directed attention when compared to learning an environment passively. However this increase in the number of fixations did not lead to better wayfinding performance when attempting to repeat the route, or an ability to recall landmarks from memory. By understanding what components of an environment, and how we learn an environment influence the allocation of attention and the ability to store our surroundings into memory in healthy individuals we may reveal potential tools for designing rehabilitation techniques for cognitively impaired populations

Semantic and action influences on visual perception: the role of action affordances and object functionality in visual selection, memory encoding and post-perceptual processes

The current thesis explores semantic and action effects on visual perception and specifically how higher-level knowledge can co-affect the process of visual perception, along with the well established effects of low level image characteristics, such as colour, image/object saliency and general gist of the scene. Recent evidence on object recognition supports perceptual grouping effects of familiar pairings of functionally interacting objects. This leads to an advantage for their perception as compared with objects positioned in a non-interacting configuration, in cases where there are attentional limitations in perception. Similar effects were previously reported in clinical cases of people diagnosed with neglect (Humphreys & Riddoch, 2001, 2007; Riddoch et al, 2003, 2006), but the fact that they are also present in normally functioning individuals (Green & Hummel, 2006) makes them a clear example of higher order effects on perception. Given the evidence about the abstract nature of the information stored in visual memory and the fact that orientation is part of the spatial information related to an object representation, our first series of experiments aimed at further exploring the nature of this perceptual grouping and whether objects separation would have an effect on it. By combining this paradigm with a paradigm used to explore linguistic factors of perceiving space (Carlson-Radvansky & Radvansky, 1996; Carlson-Radvansky, Covey & Lattanzi, 1999; Carlson-Radvansky & Tang, 2000), we additionally explored the effect of functional interactions at higher levels of post-perceptual processing. We manipulated the locations of various pairs of objects as well as the semantic and functional relationship between them to explore if spatial configurations affect the way people talk about the relationship of the objects in the same way as they affect the same objects‟ recognition. The results revealed a difference, with the same distance manipulation affecting linguistic descriptions of spatial relationships between pairs of objects but having no effect in their perceptual grouping. One of the aims of this thesis is the interpretation of such effects according to a recently growing body of evidence on the interaction between action and perception systems. These systems which were traditionally considered to be two separate disciplines seem to connect, with information from action systems feeding on perceptual systems. Through such an interaction, for example, information about the functionally related objects could lead to their perceptual grouping. A series of experiments have demonstrated effects of action affordances on object perception and their combined results seem to imply pre-attentive effects on object perception independent of the person‟s intention to act on an object (Riddoch, Humphreys, Edwards, Baker & Wilson, 2002; Tipper, Paul & Hayes, 2006; Symes, Ellis & Tucker, 2007).To further explore the role of functional relationships and action affordances in natural scene viewing, a second series of experiments was designed. These experiments also provided evidence to an old debate about the nature of visual memory and its organisation, adding further evidence for the role of semantic relationship and action affordances in the memory encoding of a scene. This series of experiments took advantage of the phenomenon of object prioritization during unexpected object onsets or feature changes while viewing real world scenes (Brockmole & Henderson, 2005a). Using a variation of classic change detection paradigms, eye-tracking data were recorded to measure at which point action affordance manipulations would have an effect and to reveal whether object functionality changes can still produce attention capture (quantified as fixation probability to the object of interest), similarly to previously tested semantic changes. Functionality manipulation was achieved by orientation changes of a critical object in the scene, but in a way which constitutes it non functional to the specific context. By comparing action affordance interference during object onsets against interference during object orientation changes we differentiated between pre-attentive and post-selection mechanisms. Our results indicate that although there is no evidence of pre-attentive modulation of object prioritization, action affordances do have an effect in post-selection mechanisms, with functionally inconsistent objects attracting attention faster and affecting the encoding of an object in the scene representation during memory guided prioritization but not during oculomotor capture. Our results also support the existence of two separate mechanisms for object prioritization. As a summary, this family of semantic relationships, action affordances and the interplay between action and perception systems has been tested during my PhD research from the very early stages of perception until post perceptual and linguistic accounts of the perceived image. Their role in attention capture and their mediating role to visual memory have also been explored using eye-tracking technology and realistic and rich in information real world scenes. Overall my thesis is oriented towards the aspects that tie all these effects together and further explores the role of action affordances in memory encoding