XR, music and neurodiversity: design and application of new mixed reality technologies that facilitate musical intervention for children with autism spectrum conditions

This thesis, accompanied by the practice outputs,investigates sensory integration, social interaction and creativity through a newly developed VR-musical interface designed exclusively for children with a high-functioning autism spectrum condition (ASC).The results aim to contribute to the limited expanse of literature and research surrounding Virtual Reality (VR) musical interventions and Immersive Virtual Environments (IVEs) designed to support individuals with neurodevelopmental conditions. The author has developed bespoke hardware, software and a new methodology to conduct field investigations. These outputs include a Virtual Immersive Musical Reality Intervention (ViMRI) protocol, a Supplemental Personalised, immersive Musical Experience(SPiME) programme, the Assisted Real-time Three-dimensional Immersive Musical Intervention System’ (ARTIMIS) and a bespoke (and fully configurable) ‘Creative immersive interactive Musical Software’ application (CiiMS). The outputs are each implemented within a series of institutional investigations of 18 autistic child participants. Four groups are evaluated using newly developed virtual assessment and scoring mechanisms devised exclusively from long-established rating scales. Key quantitative indicators from the datasets demonstrate consistent findings and significant improvements for individual preferences (likes), fear reduction efficacy, and social interaction. Six individual case studies present positive qualitative results demonstrating improved decision-making and sensorimotor processing. The preliminary research trials further indicate that using this virtual-reality music technology system and newly developed protocols produces notable improvements for participants with an ASC. More significantly, there is evidence that the supplemental technology facilitates a reduction in psychological anxiety and improvements in dexterity. The virtual music composition and improvisation system presented here require further extensive testing in different spheres for proof of concept

Unfold—An interactive experience on mixed reality platform to solve communication problems faced by children with ASD in the age group of 4-7 years.

Improving communication with a child with ASD has become an indispensable need in order to make the life smooth for the special child. Research has shown that communication gets driven by mood dynamics, non-verbal interactions, gestural communication and expressions. With the growth in technology, mixed reality and eye tracking are playing a major role in imparting experiences and making us learn about an environment without having to physically be in them. This research work explores how mixed reality can take a child with ASD through an experience that could help them open up and communicate well with others both with and without ASD. A series of training involved aims at an interaction and understanding gestures of others followed by initial training with child himself and animated creature through interaction. This overall experience integrates interactive play and human gestures in the way it happens in the real world. This will train and help enhance their willingness to share, interact and communicate with others besides training the motor skills. The resulting interactive environment addresses communication problem by introduction of a communicative and adaptive medium between a child with ASD and others

Attention-Based Applications in Extended Reality to Support Autistic Users: A Systematic Review

With the rising prevalence of autism diagnoses, it is essential for research to understand how to leverage technology to support the diverse nature of autistic traits. While traditional interventions focused on technology for medical cure and rehabilitation, recent research aims to understand how technology can accommodate each unique situation in an efficient and engaging way. Extended reality (XR) technology has been shown to be effective in improving attention in autistic users given that it is more engaging and motivating than other traditional mediums. Here, we conducted a systematic review of 59 research articles that explored the role of attention in XR interventions for autistic users. We systematically analyzed demographics, study design and findings, including autism screening and attention measurement methods. Furthermore, given methodological inconsistencies in the literature, we systematically synthesize methods and protocols including screening tools, physiological and behavioral cues of autism and XR tasks. While there is substantial evidence for the effectiveness of using XR in attention-based interventions for autism to support autistic traits, we have identified three principal research gaps that provide promising research directions to examine how autistic populations interact with XR. First, our findings highlight the disproportionate geographic locations of autism studies and underrepresentation of autistic adults, evidence of gender disparity, and presence of individuals diagnosed with co-occurring conditions across studies. Second, many studies used an assortment of standardized and novel tasks and self-report assessments with limited tested reliability. Lastly, the research lacks evidence of performance maintenance and transferability.Comment: [Accepted version] K. Wang, S. J. Julier and Y. Cho, "Attention-Based Applications in Extended Reality to Support Autistic Users: A Systematic Review," in IEEE Access, vol. 10, pp. 15574-15593, 2022, doi: 10.1109/ACCESS.2022.314772

SELF-IMAGE MULTIMEDIA TECHNOLOGIES FOR FEEDFORWARD OBSERVATIONAL LEARNING

This dissertation investigates the development and use of self-images in augmented reality systems for learning and learning-based activities. This work focuses on self- modeling, a particular form of learning, actively employed in various settings for therapy or teaching. In particular, this work aims to develop novel multimedia systems to support the display and rendering of augmented self-images. It aims to use interactivity (via games) as a means of obtaining imagery for use in creating augmented self-images. Two multimedia systems are developed, discussed and analyzed. The proposed systems are validated in terms of their technical innovation and their clinical efficacy in delivering behavioral interventions for young children on the autism spectrum

A Conceptual Approach in Designing Interactive Multimedia Application for Children With ASD

The purpose of this article is to create a suitable treatment program for autistic children that is based on the use of many media formats. This research utilizes a review of the relevant literature, an examination of the issue in the field, interviews, and surveys in order to obtain the data that is needed for the study. A concept to slightly practical approach has been taken to develop the app in a way that makes it possible for them to fit the strategies and lesson plans that are used throughout the treatment process for people who have autism. This has been carried out in a way that has been carried out in a way that makes it possible for them to fit these strategies and lesson plans. Children who have autism may benefit from interactive treatment programs that can be developed with the help of the Lovaas approach. These programs can make use of modern computer technology in addition to different forms of multimedia. These programs would be tailored to meet the needs of autistic children and adolescents

Tangible user interfaces and social interaction in children with autism

Tangible User Interfaces (TUIs) offer the potential for new modes of social interaction for children with Autism Spectrum Conditions (ASC). Familiar objects that are embedded with digital technology may help children with autism understand the actions of others by providing feedback that is logical and predictable. Objects that move, playback sound or create sound – thus repeating programmed effects – offer an exciting way for children to investigate objects and their effects. This thesis presents three studies of children with autism interacting with objects augmented with digital technology. Study one looked at Topobo, a construction toy augmented with kinetic memory. Children played with Topobo in groups of three of either Typically Developing (TD) or ASC children. The children were given a construction task, and were also allowed to play with the construction sets with no task. Topobo in the task condition showed an overall significant effect for more onlooker, cooperative, parallel, and less solitary behaviour. For ASC children significantly less solitary and more parallel behaviour was recorded than other play states. In study two, an Augmented Knights Castle (AKC) playset was presented to children with ASC. The task condition was extended to allow children to configure the playset with sound. A significant effect in a small sample was found for configuration of the AKC, leading to less solitary behaviour, and more cooperative behaviour. Compared to non-digital play, the AKC showed reduction of solitary behaviour because of augmentation. Qualitative analysis showed further differences in learning phase, user content, behaviour oriented to other children, and system responsiveness. Tangible musical blocks (‘d-touch’) in study three focused on the task. TD and ASC children were presented with a guided/non-guided task in pairs, to isolate effects of augmentation. Significant effects were found for an increase in cooperative symbolic play in the guided condition, and more solitary functional play was found in the unguided condition. Qualitative analysis highlighted differences in understanding blocks and block representation, exploratory and expressive play, understanding of shared space and understanding of the system. These studies suggest that the structure of the task conducted with TUIs may be an important factor for children’s use. When the task is undefined, play tends to lose structure and the benefits of TUIs decline. Tangible technology needs to be used in an appropriately structured manner with close coupling (the distance between digital housing and digital effect), and works best when objects are presented in familiar form

Interaction Design Applied to the Design of Sensory Integration Experiences

Therapies and treatments for autistic children have always been a challenging to set up due to the difficulty of diagnosing properly their difficulties and correctly addressing these. In fact, the spectrum comprehension changed over the years to a more accurate diagnosis. Thus, nowadays, treatments are more successful. Autism Spectrum Disorder (ASD) is a complex neurological disor der that information processing, social interaction, and communication. Attending to multiple cues present in the environment is demanding and might lead to a child not responding to a stimulus which, according to some researchers, might be a central aspect of ASD. Play-based therapies (gamification) have been one of the most fortunate approaches in achieving better results in developing cognitive and motor skills. Autistic children are increasingly exposed to technology in their daily lives, and digital games have been yielding more impact in this area since the last decade. In fact, studies demonstrate that autistic children engage more easily with technology than with people because of their impairments. Designing games for autistic children can be challenging, since as users, they may process information differently compared to a typically developing child. Therefore, many factors must be considered and conceptualized with their difficulties in mind (e.g., colors, sounds, mechanics, interface, levels) to keep the user engaged in the task. Since autistic children are interested in technology, video games might be a promising way of multiple cue treatment and developing their cognitive skills.Terapias e tratamentos para crianças autistas têm sido sempre um desafio devido à natureza do problema. A compreensão do espectro mudou ao longo dos anos para um diagnóstico mais preciso. Portanto, nos dias de hoje, os tratamentos são mais bem-sucedidos. O Transtorno do Espectro Autista (TEA) é uma desordem neurológica complexa que afeta o processamento de informações, interação social e comunicação. Atender a múltiplos estímulos presentes no ambiente é exigente e pode levar a uma criança não responder a um estímulo, o que, segundo alguns investigadores, pode ser um aspecto central do TEA. Terapias baseadas em jogos (gamificação) têm sido uma das abordagens mais bem-sucedidas para desenvolver habilidades cognitivas e motoras. Crianças autistas estão cada vez mais expostas à tecnologia em suas vidas diárias, e jogos digitais têm tido mais impacto nessa área desde a última década. De fato, estudos demonstram que crianças autistas se envolvem mais facilmente com a tecnologia do que com pessoas, devido às suas limitações. O design de jogos para crianças autistas possa ser desafiador, pois são utilizadores que podem processar informação de maneira diferente comparado com outra cria com um desenvolvimento normal, por isso muitos fatores devem ser considerados e conceptualizados pensando nas suas dificuldades (por exemplo, cores, sons, mecânicas, interface, níveis) para manter o usuário envolvido na tarefa. Visto que as crianças autistas têm interesse na tecnologia, os videojogos podem ser uma forma promissora de tratamento com múltiplos estímulos e desenvolvimento de suas habilidades cogniti vas