Integrating art into bodily interactions : exploring digital art in HCI design to foster somaesthetic experiences

PhD ThesisMy interdisciplinary doctoral research of this thesis explored how interaction design – with a combination of digital art, body-centred practice and biophysical sensing technology – cultivates self-awareness and self-reflection to foster somaesthetic experiences in everyday walking. My research followed a Research through Design (RtD) approach to provide design artefacts as examples of research in the expanded territory of Somaesthetic Design, technology-enhanced body-centred practices and digital art applied in interaction design. Background research included a critical review of Affective Computing, the concept of somaesthetic experience, existing body-centred practices (e.g. mindfulness and deep listening), HCI designs for somaesthetic experiences, and interactive digital art applications (using biophysical data as input) to express bodily activities. In methodological terms the research could be summarized as a process of ‘making design theories’ (Redström, 2017) that draws upon a Research through Design (RtD) approach. The whole research process could be described with a ‘bucket’ model in making design theories (Redström, 2017): identified initial design space as the initial ‘bucket’; derived the first design artefact ‘Ambient Walk’ as a ‘fact’ to represent the initial design space and the cause of transitioning, re-accenting process from mindfulness to ‘adding a sixth-sense’ (i.e. to extend the initial ‘bucket’); the making of second design artefact ‘Hearing the Hidden’ as a ‘fact’ to represent the re-accented research rationale in designing for somaesthetic experience by ‘adding a sixth sense’. I followed a qualitative approach to evaluate individual user feedbacks on enhancing somaesthetic experiences, the aspects to be considered in designing for experiences, and how my design process contributed to refining design for experiences. At the end of this thesis, I discuss the findings from the two practical projects regarding the somaesthetic experiences that have been provoked during users’ engagement with ‘Ambient Walk’ and ‘Hearing the Hidden’; the inclusion of bodily interactions with surroundings in somaesthetic design; the use of ‘provotypes’ in experience-centred design practices; and the benefit of integrating digital art into technology for body-centred practices

Using Sound to Represent Uncertainty in Spatial Data

There is a limit to the amount of spatial data that can be shown visually in an effective manner, particularly when the data sets are extensive or complex. Using sound to represent some of these data (sonification) is a way of avoiding visual overload. This thesis creates a conceptual model showing how sonification can be used to represent spatial data and evaluates a number of elements within the conceptual model. These are examined in three different case studies to assess the effectiveness of the sonifications. Current methods of using sonification to represent spatial data have been restricted by the technology available and have had very limited user testing. While existing research shows that sonification can be done, it does not show whether it is an effective and useful method of representing spatial data to the end user. A number of prototypes show how spatial data can be sonified, but only a small handful of these have performed any user testing beyond the authors’ immediate colleagues (where n > 4). This thesis creates and evaluates sonification prototypes, which represent uncertainty using three different case studies of spatial data. Each case study is evaluated by a significant user group (between 45 and 71 individuals) who completed a task based evaluation with the sonification tool, as well as reporting qualitatively their views on the effectiveness and usefulness of the sonification method. For all three case studies, using sound to reinforce information shown visually results in more effective performance from the majority of the participants than traditional visual methods. Participants who were familiar with the dataset were much more effective at using the sonification than those who were not and an interactive sonification which requires significant involvement from the user was much more effective than a static sonification, which did not provide significant user engagement. Using sounds with a clear and easily understood scale (such as piano notes) was important to achieve an effective sonification. These findings are used to improve the conceptual model developed earlier in this thesis and highlight areas for future research

The digitally 'Hand Made' object

This article will outline the author’s investigations of types of computer interfaces in practical three-dimensional design practice. The paper contains a description of two main projects in glass and ceramic tableware design, using a Microscribe G2L digitising arm as an interface to record three-dimensional spatial\ud design input.\ud \ud The article will provide critical reflections on the results of the investigations and will argue that new approaches in digital design interfaces could have relevance in developing design methods which incorporate more physical ‘human’ expressions in a three-dimensional design practice. The research builds on concepts indentified in traditional craft practice as foundations for constructing new types of creative practices based on the use of digital technologies, as outlined by McCullough (1996)

Immersive analytics for oncology patient cohorts

This thesis proposes a novel interactive immersive analytics tool and methods to interrogate the cancer patient cohort in an immersive virtual environment, namely Virtual Reality to Observe Oncology data Models (VROOM). The overall objective is to develop an immersive analytics platform, which includes a data analytics pipeline from raw gene expression data to immersive visualisation on virtual and augmented reality platforms utilising a game engine. Unity3D has been used to implement the visualisation. Work in this thesis could provide oncologists and clinicians with an interactive visualisation and visual analytics platform that helps them to drive their analysis in treatment efficacy and achieve the goal of evidence-based personalised medicine. The thesis integrates the latest discovery and development in cancer patients’ prognoses, immersive technologies, machine learning, decision support system and interactive visualisation to form an immersive analytics platform of complex genomic data. For this thesis, the experimental paradigm that will be followed is in understanding transcriptomics in cancer samples. This thesis specifically investigates gene expression data to determine the biological similarity revealed by the patient's tumour samples' transcriptomic profiles revealing the active genes in different patients. In summary, the thesis contributes to i) a novel immersive analytics platform for patient cohort data interrogation in similarity space where the similarity space is based on the patient's biological and genomic similarity; ii) an effective immersive environment optimisation design based on the usability study of exocentric and egocentric visualisation, audio and sound design optimisation; iii) an integration of trusted and familiar 2D biomedical visual analytics methods into the immersive environment; iv) novel use of the game theory as the decision-making system engine to help the analytics process, and application of the optimal transport theory in missing data imputation to ensure the preservation of data distribution; and v) case studies to showcase the real-world application of the visualisation and its effectiveness

Enhancing the Quality and Motivation of Physical Exercise Using Real-Time Sonification

This research project investigated the use of real-time sonification as a way to improve the quality and motivation of biceps curl exercise among healthy young participants. A sonification system was developed featuring an elec- tromyography (EMG) sensor and Microsoft Kinect camera. During exercise, muscular and kinematic data were collected and sent to custom design sonifi- cation software developed using Max to generate real-time auditory feedback. The software provides four types of output sound in consideration of personal preference and long-term use. Three experiments were carried out. The pilot study examined the sonifi- cation system and gathered the users’ comments about their experience of each type of sound in relation to its functionality and aesthetics. A 3-session between-subjects test and an 8-session within-subjects comparative test were conducted to compared the exercise quality and motivation between two conditions: with and without the real-time sonification. Overall, several conclusions are drawn based on the experimental results: The sonification improved participants’ pace of biceps curl significantly. No significant effect was found for the effect on vertical movement range. Participants expended more effort in training with the presence of sonification. Analysis of sur- veys indicated a higher motivation and willingness when exercising with the sonification. The results reflect a wider potential for applications including general fitness, physiotherapy and elite sports training

Studies on customisation-driven digital music instruments

From John Cage’s Prepared Piano to the turntable, the history of musical instruments is scattered with examples of musicians who deeply customised their instruments to fit personal artistic objectives, objectives that differed from the ones the instruments have been designed for. In their digital counterpart however, musical instruments are often presented in the form of closed, finalised systems with apriori symbolic rules set by their designer that leave very little room for the artists to customise the technologies for their unique art practices; in these cases the only possibility to change the mode of interaction with digital instrument is to reprogram them, a possibility available to programmers but not to musicians. This thesis presents two digital music instruments designed with the explicit goal of being highly customisable by musicians and to provide different modes of interactions, whilst keeping simplicity and immediateness of use. The first one leverages real-time gesture recognition to provide continuous feedback to users as guidance in defining the behaviour of the system and the gestures it recognises. The second one is a novel tangible user interface which allows to transform everyday objects into expressive digital music instruments, and whose sound generated strongly depends by the particular nature of the physical object selected

Amplifying Actions - Towards Enactive Sound Design

Recently, artists and designers have begun to use digital technologies in order to stimulate bodily interaction, while scientists keep revealing new findings about sensorimotor contingencies, changing the way in which we understand human knowledge. However, implicit knowledge generated in artistic projects can become difficult to transfer and scientific research frequently remains isolated due to specific disciplinary languages and methodologies. By mutually enriching holistic creative approaches and highly specific scientific ways of working, this doctoral dissertation aims to set the foundation for Enactive Sound Design. It is focused on sound that engages sensorimotor experience that has been neglected within the existing design practices. The premise is that such a foundation can be best developed if grounded in transdisciplinary methods that bring together scientific and design approaches. The methodology adopted to achieve this goal is practice-based and supported by theoretical research and project analysis. Three different methodologies were formulated and evaluated during this doctoral study, based on a convergence of existing methods from design, psychology and human-computer interaction. First, a basic design approach was used to engage in a reflective creation process and to extend the existing work on interaction gestalt through hands-on activities. Second, psychophysical experiments were carried out and adapted to suit the needed shift from reception-based tests to a performance-based quantitative evaluation. Last, a set of participatory workshops were developed and conducted, within which the enactive sound exercises were iteratively tested through direct and participatory observation, questionnaires and interviews. A foundation for Enactive Sound Design developed in this dissertation includes novel methods that have been generated by extensive explorations into the fertile ground between basic design education, psychophysical experiments and participatory design. Combining creative practices with traditional task analysis further developed this basic design approach. The results were a number of abstract sonic artefacts conceptualised as the experimental apparatuses that can allow psychologists to study enactive sound experience. Furthermore, a collaboration between designers and scientists on a psychophysical study produced a new methodology for the evaluation of sensorimotor performance with tangible sound interfaces.These performance experiments have revealed that sonic feedback can support enactive learning. Finally, participatory workshops resulted in a number of novel methods focused on a holistic perspective fostered through a subjective experience of self-producing sound. They indicated the influence that such an approach may have on both artists and scientists in the future. The role of designer, as a scientific collaborator within psychological research and as a facilitator of participatory workshops, has been evaluated. Thus, this dissertation recommends a number of collaborative methods and strategies that can help designers to understand and reflectively create enactive sound objects. It is hoped that the examples of successful collaborations between designers and scientists presented in this thesis will encourage further projects and connections between different disciplines, with the final goal of creating a more engaging and a more aware sonic future.European Commission 6th Framework and European Science Foundation (COST Action