Simulating Auditory Hallucinations in a Video Game: Three Prototype Mechanisms

In previous work the authors have proposed the concept of 'ASC Simulation1: including audio-visual installations and experiences, as well as interactive video game systems, which simulate altered states of consciousness (ASCs) such as dreams and hallucinations. Building on the discussion of the authors' previous paper, where a large-scale qualitative study explored the changes to auditory perception that users of various intoxicating substances report, here the authors present three prototype audio mechanisms for simulating hallucinations in a video game. These were designed in the Unity video game engine as an early proof-of-concept. The first mechanism simulates 'selective auditory attention' to different sound sources, by attenuating the amplitude of unattended sources. The second simulates 'enhanced sounds', by adjusting perceived brightness through filtering. The third simulates 'spatial disruptions' to perception, by dislocating sound sources from their virtual acoustic origin in 3D-space, causing them to move in oscillations around a central location. In terms of programming structure, these mechanisms are designed using scripts that are attached to the collection of assets that make up the player character, and in future developments of this type of work we foresee a more advanced, standardised interface that models the senses, emotions and state of consciousness of player avatars

Mapping media and meaning: autoethnography as an approach to designing personal heritage soundscapes

The paper presents reflections on understanding the issues of designing of locative sonic memory-scapes. As physical space and digital media become ever more intertwined, together forming and augmenting meaning and experience, we need methods to further explore possible ways in which physical places and intangible personal content can be used to develop meaningful experiences. The paper explores the use of autoethnography as a method for soundscape design in the fields of personal heritage and locative media. Specifically, we explore possible connections between digital media, space and ‘meaning making’, suggesting how autoethnographies might help discover design opportunities for merging digital media and places. These are methods that are more personally relevant than those typically associated with a more system-based design approaches that we often find are less sensitive to the way that emotion, relationships, memory and meaning come together. As a way to expand upon these relationships we also reflect on relations between personal and community-based responses

Are the robots coming? Designing with autonomy & control for musical creativity & performance

This paper1 expands upon our previous work, and starts to unpack notions of autonomy and control in musical composition and performance-based systems. The term autonomous has become synonymous with technologies such as “autonomous vehicles” and “drones”, while notions of control have mainly been raised in respect to the “control” of industrial systems and in respect to protocols. This position piece disrupts these notions and provides a platform, introducing a more radical proposition in respect to the representation of autonomy and control of features that can be used to design systems that support musical composition and performance. This paper supports a growing interest within the Design, HCI and Artificial Intelligence communities, leading us to think about Human Like Computing systems and the development of a Computational Creativity Continuum

Enabling Audiovisual User Interfaces

“Enabling Audiovisual User Interfaces” was a 2-year project, supported by a Marie Curie EU fellowship, conducted at EAVI, Goldsmiths. During the project, I investigated how human-computer interactions can be audiovisualized in order to improve user experience and usability. To address this issue, a new UI paradigm was proposed – AVUI (AudioVisual User Interface). AVUI links interaction, sound and image, building upon the concept of Graphical User Interface (GUI) by adding interconnected sound and image. The research hypothesis was: the introduction of AVUI, integrating interrelated sonic and visual feedback, reacting to user interactions, would lead to more usable, accessible, playful and engaging UIs, as compared to a traditional GUI – particularly in use cases where accessibility and/or engagement were determinant. I applied AVUIs to case studies, which was the object of user testing. After reaching conclusions from these, I proposed an AVUI framework, including software modules and a set of best practices. Dissemination activities were also implemented

NON-VERBAL COMMUNICATION WITH PHYSIOLOGICAL SENSORS. THE AESTHETIC DOMAIN OF WEARABLES AND NEURAL NETWORKS

Historically, communication implies the transfer of information between bodies, yet this phenomenon is constantly adapting to new technological and cultural standards. In a digital context, it’s commonplace to envision systems that revolve around verbal modalities. However, behavioural analysis grounded in psychology research calls attention to the emotional information disclosed by non-verbal social cues, in particular, actions that are involuntary. This notion has circulated heavily into various interdisciplinary computing research fields, from which multiple studies have arisen, correlating non-verbal activity to socio-affective inferences. These are often derived from some form of motion capture and other wearable sensors, measuring the ‘invisible’ bioelectrical changes that occur from inside the body. This thesis proposes a motivation and methodology for using physiological sensory data as an expressive resource for technology-mediated interactions. Initialised from a thorough discussion on state-of-the-art technologies and established design principles regarding this topic, then applied to a novel approach alongside a selection of practice works to compliment this. We advocate for aesthetic experience, experimenting with abstract representations. Atypically from prevailing Affective Computing systems, the intention is not to infer or classify emotion but rather to create new opportunities for rich gestural exchange, unconfined to the verbal domain. Given the preliminary proposition of non-representation, we justify a correspondence with modern Machine Learning and multimedia interaction strategies, applying an iterative, human-centred approach to improve personalisation without the compromising emotional potential of bodily gesture. Where related studies in the past have successfully provoked strong design concepts through innovative fabrications, these are typically limited to simple linear, one-to-one mappings and often neglect multi-user environments; we foresee a vast potential. In our use cases, we adopt neural network architectures to generate highly granular biofeedback from low-dimensional input data. We present the following proof-of-concepts: Breathing Correspondence, a wearable biofeedback system inspired by Somaesthetic design principles; Latent Steps, a real-time auto-encoder to represent bodily experiences from sensor data, designed for dance performance; and Anti-Social Distancing Ensemble, an installation for public space interventions, analysing physical distance to generate a collective soundscape. Key findings are extracted from the individual reports to formulate an extensive technical and theoretical framework around this topic. The projects first aim to embrace some alternative perspectives already established within Affective Computing research. From here, these concepts evolve deeper, bridging theories from contemporary creative and technical practices with the advancement of biomedical technologies.Historicamente, os processos de comunicação implicam a transferência de informação entre organismos, mas este fenómeno está constantemente a adaptar-se a novos padrões tecnológicos e culturais. Num contexto digital, é comum encontrar sistemas que giram em torno de modalidades verbais. Contudo, a análise comportamental fundamentada na investigação psicológica chama a atenção para a informação emocional revelada por sinais sociais não verbais, em particular, acções que são involuntárias. Esta noção circulou fortemente em vários campos interdisciplinares de investigação na área das ciências da computação, dos quais surgiram múltiplos estudos, correlacionando a actividade nãoverbal com inferências sócio-afectivas. Estes são frequentemente derivados de alguma forma de captura de movimento e sensores “wearable”, medindo as alterações bioeléctricas “invisíveis” que ocorrem no interior do corpo. Nesta tese, propomos uma motivação e metodologia para a utilização de dados sensoriais fisiológicos como um recurso expressivo para interacções mediadas pela tecnologia. Iniciada a partir de uma discussão aprofundada sobre tecnologias de ponta e princípios de concepção estabelecidos relativamente a este tópico, depois aplicada a uma nova abordagem, juntamente com uma selecção de trabalhos práticos, para complementar esta. Defendemos a experiência estética, experimentando com representações abstractas. Contrariamente aos sistemas de Computação Afectiva predominantes, a intenção não é inferir ou classificar a emoção, mas sim criar novas oportunidades para uma rica troca gestual, não confinada ao domínio verbal. Dada a proposta preliminar de não representação, justificamos uma correspondência com estratégias modernas de Machine Learning e interacção multimédia, aplicando uma abordagem iterativa e centrada no ser humano para melhorar a personalização sem o potencial emocional comprometedor do gesto corporal. Nos casos em que estudos anteriores demonstraram com sucesso conceitos de design fortes através de fabricações inovadoras, estes limitam-se tipicamente a simples mapeamentos lineares, um-para-um, e muitas vezes negligenciam ambientes multi-utilizadores; com este trabalho, prevemos um potencial alargado. Nos nossos casos de utilização, adoptamos arquitecturas de redes neurais para gerar biofeedback altamente granular a partir de dados de entrada de baixa dimensão. Apresentamos as seguintes provas de conceitos: Breathing Correspondence, um sistema de biofeedback wearable inspirado nos princípios de design somaestético; Latent Steps, um modelo autoencoder em tempo real para representar experiências corporais a partir de dados de sensores, concebido para desempenho de dança; e Anti-Social Distancing Ensemble, uma instalação para intervenções no espaço público, analisando a distância física para gerar uma paisagem sonora colectiva. Os principais resultados são extraídos dos relatórios individuais, para formular um quadro técnico e teórico alargado para expandir sobre este tópico. Os projectos têm como primeiro objectivo abraçar algumas perspectivas alternativas às que já estão estabelecidas no âmbito da investigação da Computação Afectiva. A partir daqui, estes conceitos evoluem mais profundamente, fazendo a ponte entre as teorias das práticas criativas e técnicas contemporâneas com o avanço das tecnologias biomédicas

VR Storytelling

The question of cinematic VR production has been on the table for several years. This is due to the peculiarity of VR language which, even if it is de ned by an image that surrounds and immerses the viewer rather than placing them, as in the classic cinematic situation, in front of a screen, relies decisively on an audiovisual basis that cannot help but refer to cinematic practices of constructing visual and auditory experience. Despite this, it would be extremely reductive to consider VR as the mere transposition of elements of cinematic language. The VR medium is endowed with its own speci city, which inevitably impacts its forms of narration. We thus need to investigate the narrative forms it uses that are probably related to cinematic language, and draw their strength from the same basis, drink from the same well, but develop according to di erent trajectories, thus displaying di erent links and a nities

Yorkshire’s folk culture in digital spaces during the COVID-19 pandemic : cultural practice, digitalisation and localisation

This thesis critically assesses the digital spaces where Yorkshire’s folk community practiced sonic and material cultures and developed identities of communal belonging during the initial phase of the COVID-19 pandemic. Using data gathered throughout the UK’s first national ‘lockdown’ in 2020, it examines how a social group previously reliant on physical space and face-to-face community began to use digital spaces in a period of rapid cultural shift. By doing so, it reflects on the process of enacting folk culture in digital spaces, and the extent to which these spaces can nurture localised and collective perceptions of being part of a balanced, tightknit and tangible folk community.Defining the term ‘folk’ as established, localised and grassroots cultural practices, with emphasis on sound (folk music and voice), visual (text and artworks) and material (craft), this thesis is positioned between the established field of folk geographies and emerging studies into digital geographies. It aims to revitalise folk geographies by modernising research into the spatial variations of localised cultures within developing digital worlds, such as social media. In order to explore these issues, the research used an experimental multimethodology including (online) ethnography, art-elicitation and interviews within social media groups with strong connections to Yorkshire folk culture. This enabled the thesis to present an empathic but critical narrative of the Yorkshire folk community’s cultural shift to digital spaces, based on researcher and participant experiences and the expressions represented through various sonic, visual and material digital artforms.The thesis makes three key arguments. Firstly, that there are spatial frictions between physical spaces and emerging digital spaces as cultural practices are digitally reproduced, and that these frictions are powerful enough to reshape spaces of folk culture. Secondly, that the COVID-19 pandemic created unique digital geographies, which encouraged experimental practices of culture and redesigned cultural groups by placing new importance on online identities. Thirdly, that tensions between different geographical scales (the local to the global) developed as Yorkshire’s folk culture was digitally reproduced, with both productive and destructive consequences for folk communities. Collectively, the frictions, experimentations and tensions related to digital spaces substantially restructured and drove the practices of small-scale cultures into alternative networks of access, control, inclusion, intimacy, heterogeneity and hybridity