“You, Move There!”: Investigating the Impact of Feedback on Voice Control in Virtual Environments

Current virtual environment (VEs) input techniques often overlook speech as a useful control modality. Speech could improve interaction in multimodal VEs by enabling users to address objects, locations, and agents, yet research on how to design effective speech for VEs is limited. Our paper investigates the effect of agent feedback on speech VE experiences. Through a lab study, users commanded agents to navigate a VE, receiving either auditory, visual or behavioural feedback. Based on a post interaction semi-structured interview, we find that the type of feedback given by agents is critical to user experience. Specifically auditory mechanisms are preferred, allowing users to engage with other modalities seamlessly during interaction. Although command-like utterances were frequently used, it was perceived as contextually appropriate, ensuring users were understood. Many also found it difficult to discover speech-based functionality. Drawing on these, we discuss key challenges for designing speech input for VEs

LGBTQ-AI? Exploring Expressions of Gender and Sexual Orientation in Chatbots

Chatbots are popular machine partners for task-oriented and so- cial interactions. Human-human computer-mediated communica- tion research has explored how people express their gender and sexuality in online social interactions, but little is known about whether and in what way chatbots do the same. We conducted semi-structured interviews with 5 text-based conversational agents to explore this topic Through these interviews, we identified 6 com- mon themes around the expression of gender and sexual identity: identity description, identity formation, peer acceptance, positive reflection, uncomfortable feelings and off-topic responses. Chat- bots express gender and sexuality explicitly and through relation of experience and emotions, mimicking the human language on which they are trained. It is nevertheless evident that chatbots dif- fer from human dialogue partners as they lack the flexibility and understanding enabled by lived human experience. While chatbots are proficient in using language to express identity, they also dis- play a lack of authentic experiences of gender and sexuality

Building and Designing Expressive Speech Synthesis

We know there is something special about speech. Our voices are not just a means of communicating. They also give a deep impression of who we are and what we might know. They can betray our upbringing, our emotional state, our state of health. They can be used to persuade and convince, to calm and to excite. As speech systems enter the social domain they are required to interact, support and mediate our social relationships with 1) each other, 2) with digital information, and, increasingly, 3) with AI-based algorithms and processes. Socially Interactive Agents (SIAs) are at the fore- front of research and innovation in this area. There is an assumption that in the future “spoken language will provide a natural conversational interface between human beings and so-called intelligent systems.” [Moore 2017, p. 283]. A considerable amount of previous research work has tested this assumption with mixed results. However, as pointed out “voice interfaces have become notorious for fostering frustration and failure” [Nass and Brave 2005, p.6]. It is within this context, between our exceptional and intelligent human use of speech to communicate and interact with other humans, and our desire to leverage this means of communication for artificial systems, that the technology, often termed expressive speech synthesis uncomfortably falls. Uncomfortably, because it is often overshadowed by issues in interactivity and the underlying intelligence of the system which is something that emerges from the interaction of many of the components in a SIA. This is especially true of what we might term conversational speech, where decoupling how things are spoken, from when and to whom they are spoken, can seem an impossible task. This is an even greater challenge in evaluation and in characterising full systems which have made use of expressive speech. Furthermore when designing an interaction with a SIA, we must not only consider how SIAs should speak but how much, and whether they should even speak at all. These considerations cannot be ignored. Any speech synthesis that is used in the context of an artificial agent will have a perceived accent, a vocal style, an underlying emotion and an intonational model. Dimensions like accent and personality (cross speaker parameters) as well as vocal style, emotion and intonation during an interaction (within-speaker parameters) need to be built in the design of a synthetic voice. Even a default or neutral voice has to consider these same expressive speech synthesis components. Such design parameters have a strong influence on how effectively a system will interact, how it is perceived and its assumed ability to perform a task or function. To ignore these is to blindly accept a set of design decisions that ignores the complex effect speech has on the user’s successful interaction with a system. Thus expressive speech synthesis is a key design component in SIAs. This chapter explores the world of expressive speech synthesis, aiming to act as a starting point for those interested in the design, building and evaluation of such artificial speech. The debates and literature within this topic are vast and are fundamentally multidisciplinary in focus, covering a wide range of disciplines such as linguistics, pragmatics, psychology, speech and language technology, robotics and human-computer interaction (HCI), to name a few. It is not our aim to synthesise these areas but to give a scaffold and a starting point for the reader by exploring the critical dimensions and decisions they may need to consider when choosing to use expressive speech. To do this, the chapter explores the building of expressive synthesis, highlighting key decisions and parameters as well as emphasising future challenges in expressive speech research and development. Yet, before these are expanded upon we must first try and define what we actually mean by expressive speech

Finding a New Voice: Transitioning Designers from GUI to VUI Design

As Voice User Interfaces (VUIs) become widely popular, designers must handle new usability challenges. However, compared to other established domains such as Graphical User Interfaces (GUIs), VUI designers have fewer resources (training support, usability heuris- tics, design patterns) to guide them. On the other hand, GUI-trained designers may also be solicited upon to design VUIs given the in- creased demand for such interfaces. This raises the question: how can we best support such designers as they transition from GUI to VUI design? To answer this, we focus on usability heuristics as a key resource, and conduct several workshops with GUI design experts, exploring how they map their design experience onto VUI design. Based on this, we suggest that the “path of least resistance” to transitioning designers from GUI to VUI may be the adaptation of familiar resources and concepts (such as GUI heuristics) to the VUI design space, instead of the imposition of novel VUI-specific heuristics on GUI-trained designers. This finding can inform the development of design resources that can support the increase demand for VUIs

The journey of researching on to researching with : theoretical and methodological challenges within educational research

PhD Thesis (Published Articles have been removed due to copyright issues. The thesis can be viewed in full, on request via the print copy at the University Library)This thesis focuses on the relationship between participatory research and visual methods. Firstly, I explore how methodology can be participative, investigating the conceptual base, the possibilities, significance and usefulness. Secondly, I explore whether using visual research methods can contribute to participatory research and how we can do this better as researchers. As I gained more confidence as a researcher, I started to carve out such space within projects to design and use more creative, innovative and visual research tools as a way of engaging with the participants in my research. The thesis elaborates on three main themes: 1. Ideals vs Practice of participatory research: How I have come to understand the difference between the theoretical and conceptual underpinnings of participatory research within the everyday reality (and the affordances and constraints) of educational research when trying to do it well. My early discussion relates to the methodological, practical and ethical challenges faced when, as a researcher, I was keen to be evaluatively formative, inclusive and collaborative (Publications 1, 2, 3). I also relate this to the range of knowledge this can produce. In this theme I explore the underlying principles of participatory research – and how these fit well with my own values as a researcher - and the notion of participation, consider linear modelling and question the concept of voice. I explore the mismatch between what I intended (the ideal) and what happened in reality (practice). I consider whether and how participation can be conceptualised in the less-than-ideal situations of real world research. 2. Quality in practice using visual methods: How visual methods can help individuals think differently. With reference to the development of particular visual research tools, I explore what visual methods can add to the quality of participatory research particularly in terms of ethics, inclusivity and appropriateness (Publications 4 and 5). I argue that visual methods enable me to reject a deterministic framework for exploring human behaviour and experiences, but instead position visual methods as facilitative with the aim of creating ‘space’ – ‘visually-mediated encounters’ - for meaningful dialogue between the researcher and participant. I critically explore the affordances of using visual methods and the different pieces of knowledge that visual methods can facilitate. I argue that the use of visual methods in a participatory setting can evoke a variety of viewpoints, from a range of participants, leading to a more complete and better research process. 3. Making connections: Implications for policy and practice: Revisiting the early concepts in my work, Publication 6 develops my earlier ideas further and proposes a model for effective participatory research. Publication 6 is a result of this journey to date, - as I reflect, refine and further develop tools to improve the research process and the experiences of people within it. During this journey so far I have moved from the structural issues of conducting participatory research (section 3), through to 7 managing the research encounter (section 4) and bringing all that I have learnt through to a policy and practice context (section 5). This thesis draws upon twenty three years as a researcher at Newcastle University, and my experiences of conducting over 60 research projects in many diverse educational settings. These different environments include community-based settings, prisons, and primary and secondary schools. However, it is not the particular settings in which this research takes place that is important in this thesis. It is to some extent about the participants within the thesis, and these include young offenders in the community, prisoners and children and young people. These participants could be described as unheard, or the have nots in the research process (e.g. Munro et al., 2005; Liamputtong, 2007; Arnstein, 1969) and so this thesis will discuss the particular considerations and sensitivities of being a researcher faced with subject groups who are sometimes placed at the margins of society. It explores the ethical, practical and methodological implications of researching with such groups (or for) rather than as objects of research (see Griffiths, 1998). My research experiences and reflections are placed in the wider context of other researchers in the area who advocate an inclusive, and collaborative methodology alongside ‘user involvement’ and ‘participation’ (e.g. Cook, 2003; Crozier and Reay, 2004; Nind, 2014). However, such concepts are contested, often overlapping, used interchangeably and are therefore not unproblematic, as will become evident. Rather than have a single study focus, the thesis charts my journey as an academic across both a series of projects and a timeframe and focuses on the reflection, learning and the thinking which took place within this work over time. This thesis is based on 6 pieces of work published between 2006 and 2012 – five are published in independent, peer-reviewed journals - and the majority of these publications are joint-authored. This reflects the collaborative nature of my work – I have never worked as a lone researcher (nor have I had the desire to) and I have always enjoyed being part of larger research and writing teams. This thesis reflects my own perspectives and therefore my own contribution to this work. Moreover, the publications are not all academic journal articles, one is a report (Publication 6), which is soundly based on academic evidence and robust research (funded through the joint Research Councils UK Connected Communities programme), and has been written specifically for a wider audience. This report is primarily aimed at practitioners and policy-makers and reflects my gradual realisation that by broadening the dissemination from academic journals, such publications can be accessed and utilised by different audiences, academics and non-academics, and perhaps have different kinds of impact. For a full list of the submitted publications for this thesis, please see Table 1. Contextual publications (related to my thesis, but not submitted as part of my thesis) which help to provide the context for my work are referred to in the thesis as [a], [b] etc., and a full reference list is included in Appendix 1

Let’s Talk About CUIs: Putting Conversational User Interface Design Into Practice

As CUIs become more prevalent in both academic research and the commercial market, it becomes more essential to design usable and adoptable CUIs. Though research on the usability and design of CUIs has been growing greatly over the past decade, we see that many usability issues are still prevalent in current conversational voice interfaces, from issues in feedback and visibility, to learnability, to error correction, and more. These issues still exist in the most current conversational interfaces in the commercial market, like the Google Assistant, Amazon Alexa, and Siri. The aim of this workshop therefore is to bring both academics and industry practitioners together to bridge the gaps of knowledge in regards to the tools, practices, and methods used in the design of CUIs. This workshop will bring together both the research performed by academics in the field, and the practical experience and needs from industry practitioners, in order to have deeper discussions about the resources that require more research and development, in order to build better and more usable CUIs