Social VR design features and experiential outcomes:narrative review and relationship map for dyadic agent conversations

The application of virtual reality to the study of conversation and social interaction is a relatively new field of study. While the affordances of VR in the domain compared to traditional methods are promising, the current state of the field is plagued by a lack of methodological standards and shared understanding of how design features of the immersive experience impact participants. In order to address this, this paper develops a relationship map between design features and experiential outcomes, along with expectations for how those features interact with each other. Based on the results of a narrative review drawing from diverse fields, this relationship map focuses on dyadic conversations with agents. The experiential outcomes chosen include presence &amp; engagement, psychological discomfort, and simulator sickness. The relevant design features contained in the framework include scenario agency, visual fidelity, agent automation, environmental context, and audio features. We conclude by discussing the findings of the review and framework, such as the multimodal nature of social VR being highlighted, and the importance of environmental context, and lastly provide recommendations for future research in social VR.</p

Reconstructing the Dynamic Directivity of Unconstrained Speech

This article presents a method for estimating and reconstructing the spatial energy distribution pattern of natural speech, which is crucial for achieving realistic vocal presence in virtual communication settings. The method comprises two stages. First, recordings of speech captured by a real, static microphone array are used to create an egocentric virtual array that tracks the movement of the speaker over time. This virtual array is used to measure and encode the high-resolution directivity pattern of the speech signal as it evolves dynamically with natural speech and movement. In the second stage, the encoded directivity representation is utilized to train a machine learning model that can estimate the full, dynamic directivity pattern given a limited set of speech signals, such as those recorded using the microphones on a head-mounted display. Our results show that neural networks can accurately estimate the full directivity pattern of natural, unconstrained speech from limited information. The proposed method for estimating and reconstructing the spatial energy distribution pattern of natural speech, along with the evaluation of various machine learning models and training paradigms, provides an important contribution to the development of realistic vocal presence in virtual communication settings.Comment: In proceedings of I3DA 2023 - The 2023 International Conference on Immersive and 3D Audio. DOI coming soo

Perceptual Significance of Tone-Dependent Directivity Patterns of Musical Instruments

Musical instruments are complex sound sources that exhibit directivity patterns that not only vary depending on the frequency, but can also change as a function of the played tone. It is yet unclear whether the directivity variation as a function of the played tone leads to a perceptible difference compared to an auralization that uses an averaged directivity pattern. This paper examines the directivity of 38 musical instruments from a publicly available database and then selects three representative instruments among those with similar radiation characteristics (oboe, violin, and trumpet). To evaluate the listeners\u27 ability to perceive a difference between auralizations of virtual environments using tone-dependent and averaged directivities, a listening test was conducted using the directivity patterns of the three selected instruments in both anechoic and reverberant conditions. The results show that, in anechoic conditions, listeners can reliably detect differences between the tone-dependent and averaged directivities for the oboe but not for the violin or the trumpet. Nevertheless, in reverberant conditions, listeners can distinguish tone-dependent directivity from averaged directivity for all instruments under study

Explorations in the Deictic Field

This paper focuses on the ways in which speakers make reference to themselves, to one another, and to objects in the everyday settings of talk. Drawing on research in linguistic anthropology, sociology, and linguistics, it proposes an approach to language based on the concepts of communicative practice, deictic field, and socially constituted objects of reference. Found in all human languages, deictics are expressions like English "this," "that," "here," and "there" whose meanings depend strictly on the occasions of their use. This paper critically examines current approaches to deixis, proposes an alternative framework based on the sociological concept of field, and applies this framework to deictic practice in Yucatec Maya. Drawing on the work of Buhler, Goffman, and Bourdieu, it adapts the field concept to the semiotic structure of deixis. The result is an analysis of deictic practice as an emergent construal of socially embedded deictic fields involving practical equivalences, counterpart relations among objects, and rules of thumb

Designing Sound for Social Robots: Advancing Professional Practice through Design Principles

Sound is one of the core modalities social robots can use to communicate with the humans around them in rich, engaging, and effective ways. While a robot's auditory communication happens predominantly through speech, a growing body of work demonstrates the various ways non-verbal robot sound can affect humans, and researchers have begun to formulate design recommendations that encourage using the medium to its full potential. However, formal strategies for successful robot sound design have so far not emerged, current frameworks and principles are largely untested and no effort has been made to survey creative robot sound design practice. In this dissertation, I combine creative practice, expert interviews, and human-robot interaction studies to advance our understanding of how designers can best ideate, create, and implement robot sound. In a first step, I map out a design space that combines established sound design frameworks with insights from interviews with robot sound design experts. I then systematically traverse this space across three robot sound design explorations, investigating (i) the effect of artificial movement sound on how robots are perceived, (ii) the benefits of applying compositional theory to robot sound design, and (iii) the role and potential of spatially distributed robot sound. Finally, I implement the designs from prior chapters into humanoid robot Diamandini, and deploy it as a case study. Based on a synthesis of the data collection and design practice conducted across the thesis, I argue that the creation of robot sound is best guided by four design perspectives: fiction (sound as a means to convey a narrative), composition (sound as its own separate listening experience), plasticity (sound as something that can vary and adapt over time), and space (spatial distribution of sound as a separate communication channel). The conclusion of the thesis presents these four perspectives and proposes eleven design principles across them which are supported by detailed examples. This work contributes an extensive body of design principles, process models, and techniques providing researchers and designers with new tools to enrich the way robots communicate with humans

Directives and stance in Finnish conversation: An interactional-linguistic study of the conditional mood and the partitive case

Wydział NeofilologiiPrzedmiotem niniejszej pracy są konstrukcje językowe z trybem przypuszczającym i przypadkiem partitivusem jako wzorce konwersacyjne wyłaniające się z fińskich interakcji codziennych i instytucjonalnych. Konstrukcje te funkcjonują jako formaty dwóch typów działań społecznych: dyrektyw i postawy. Podstawa teoretyczna skonstruowana jest z trzech nurtów Lingwistyki Interakcyjnej: Gramatyki Konstrukcyjnej, Gramatyki Powstającej (Emergent Grammar) oraz Formatów Działań Społecznych (Social Action Formats). Materiał badawczy obejmuje 12 godzin fińskich rozmów codziennych i 12 godzin rozmów instytucjonalnych. Źródłem tych ostatnich są interakcje w fińskim urzędzie spraw socjalnych KELA. Praca wykazuje, iż różnice między tymi dwoma typami konwersacji wpływają na cechy dystrybucyjne i funkcjonalne badanych konstrukcji. Praca zidentyfikowała 22 konstrukcji z trybem warunkowym i 8 konstrukcji z partitivusem. Dyrektywy w rozmowach instytucjonalnych formułowane są głównie w stronę partnera interakcyjnego (są to głównie prośby), a w rozmowach codziennych dominują propozycje działań podejmowanych wspólnie. Konstrukcje z trybem warunkowym są powszechne dla przyjmowania postawy jako osiągnięcia intersubiektywnego, ale mają one też funkcje związane z organizacją interakcji. Dla partitivusa stwierdza się, że w kontekstach wyrażania postawy epistemicznej i afektywnej występuje on w swoim typowym środowisku niskiej tranzytywności. Praca koncentruje się ponadto na postawie w służbie innych czynności jako na poniekąd odrębnej funkcji konstrukcji językowych.The focus of the present study is on linguistic constructions with the conditional mood and the partitive case as patterns emergent from Finnish spoken interactions, functioning as patterns of directives and stance. Three main orientations in Interactional Linguistics are combined to form the theoretical foundation of the study: Construction Grammar, Emergent Grammar, and Social Action Formats. The research material comprises 12 hours of casual and 12 hours of institutional spoken Finnish. The institutional data come from the Finnish social welfare institution KELA. It is shown that the different characteristics of casual and institutional conversations influence the distributional and functional properties of the constructions under investigation. The study has identified 22 constructions with the conditional and 8 constructions containing partitive NPs. The directives at KELA interactions are mostly oriented at the other participant (they are primarily requests), whereas in casual conversations proposal for common activities are more ubiquitous. In addition to performing stance as an intersubjective achievement, constructions with the Finnish conditional also serve functions of discourse organisation. In terms of the partitive, it is demonstrated that this grammatical case inhibits its typical contexts of low transitivity when being a device for epistemic or affective stance taking. The study has also made a case for stance in the service of other actions, including directives, and identified a number of constructions dedicated to this function

The sound effect: a study in radical sound design

This research project combines a theoretical intervention into sound ontology, with an empirical investigation into listening experience, in parallel with two technologically focused, research-led creative practice projects. The design follows an iterative cycle of research and creative practice that integrates theory, practice and empirical approaches. The research makes an initial contribution to the field of sound studies by re-appraising the work of pioneers in the field—Pierre Schaeffer and R. Murray Schafer—in light of the concept of the sonic effect. This concept is developed as an effective tool for both sound studies and sound design. This theoretical work attempts to critically and creatively examine the ontology or mode of existence of sonic phenomena and is informed by the post-structural theory of the effect. The theory of the sonic effect is empirically investigated by examining verbal accounts of listening experience elicited by semi-structured interview. Finally, having deconstructed sonic phenomena in terms of their potential to be actualised in diverse contexts, sonic effects are interrogated as a creative strategy in the field of sound design for performance and installed sonic art. Two projects are documented. One is a hybrid live performance installation utilising a novel software design for sound composition and projection. The other is a sound installation work demonstrating a novel loudspeaker design for the creation of very dense sound fields. In this context, design occurs as an effect at the intersection of new technologies of sound production and the production of audible sense. This approach enacts a radical pragmatism that underlies the radical sound design strategy outlined in the thesis