Perceptual user interface framework for immersive information retrieval environments (An experimental framework for testing and rapid iteration)

The use of perceptual inputs is an emerging area within HCI that suggests a developing Perceptual User Interface (PUI) that may prove advantageous for those involved in mobile serious games and immersive social network environments. Since there are a large variety of input devices, software platforms, possible interactions, and myriad ways to combine all of the above elements in pursuit of a PUI, we propose in this paper a basic experimental framework that will be able to standardize study of the wide range of interactive applications for testing efficacy in learning or information retrieval and also suggest improvements to emerging PUIs by enabling quick iteration. This rapid iteration will start to define a targeted range of interactions that will be intuitive and comfortable as perceptual inputs, and enhance learning and information retention in comparison to traditional GUI systems. The work focuses on the planning of the technical development of two scenarios, and the first steps in developing a framework to evaluate these and other PUIs for efficacy and pedagogy

An Evaluation of the Efficacy of a Perceptually Controlled Immersive Environment for Learning Acupuncture

This paper presents a basic but functional Perceptual User Interface (PUI) controlled immersive environment (IE) on an electronic learning platform (e-Learning) in order to deliver educational material relating to the NADA (National Acupuncture Detoxification Association) protocol for acupuncture. The purpose of this study is set out a proposed process for evaluating the learning efficacy of the PUI IE e-Learning application when compared with a typical Graphical User Interface (GUI) e-Learning IE application. Both are to be compared to a more traditional learning method. This paper evaluates user interface (UI) sentiment of the systems in advance of this proposed evaluation

16th Sound and Music Computing Conference SMC 2019 (28–31 May 2019, Malaga, Spain)

The 16th Sound and Music Computing Conference (SMC 2019) took place in Malaga, Spain, 28-31 May 2019 and it was organized by the Application of Information and Communication Technologies Research group (ATIC) of the University of Malaga (UMA). The SMC 2019 associated Summer School took place 25-28 May 2019. The First International Day of Women in Inclusive Engineering, Sound and Music Computing Research (WiSMC 2019) took place on 28 May 2019. The SMC 2019 TOPICS OF INTEREST included a wide selection of topics related to acoustics, psychoacoustics, music, technology for music, audio analysis, musicology, sonification, music games, machine learning, serious games, immersive audio, sound synthesis, etc

A Virtual Architecture Framework for Immersive Learning Environments

This thesis presents a set of experimental studies to understand the benefits of utilising architectural design to create virtual environments optimised for completing a series of cognitively demanding tasks. Each field of investigation is reviewed separately. The first field of investigation relates to spatial design and analysis from an architectural standpoint. The second is concerned with memory, spatial abilities, and embodied cognition. Two VR-based user-studies are designed to further explore the potential interactions between these fields of knowledge. An initial experiment called “Archimemory” is based on a memory palace, a historical mnemonic technique, to explore how spatial knowledge representation can enhance memory retrieval. It compares the benefits of using different architectural designs in VR to support participants’ recall accuracy of a sequence of playing cards. The main user study,called the "Immersive Virtual Architecture Studio" (IVAS), validates a new methodology to study the effect of spatial qualities on embodied cognition related tasks. A spatial analysis using the isovist technique provides an objective approach to measure spatial qualities such as openness and complexity. Participants have to perform a batch of cognitive tasks in the IVAS. Results from the spatial analysis are compared to participants subjective rating of the same spatial qualities as well as their performance.Findings suggest that a spatial performance metric can be evaluated for each room, for instance, it was the highest in the case of the more closed (fewer windows) and more complex (with columns) condition. The combination of spatial analysis and performance metrics obtained from these two novel VR applications, Archimemory and IVAS, leads this research to form a Virtual Architecture Framework. Guidelines are proposed for VR architects, UX designers and scientists to adopt this framework to support further exploration and evaluation of spatial design to enhance human cognitive abilities when experiencing immersive learning environments

"Enriching 360-degree technologies through human-computer interaction: psychometric validation of two memory tasks"

This doctoral dissertation explores the domain of neuropsychological assessment, with the objective of gaining a comprehensive understanding of an individual's cognitive functioning and detecting possible impairments. Traditional assessment tools, while possessing inherent value, frequently exhibit a deficiency in ecological validity when evaluating memory, as they predominantly concentrate on short-term, regulated tasks. To overcome this constraint, immersive technologies, specifically virtual reality and 360° videos, have surfaced as promising instruments for augmenting the ecological validity of cognitive assessments. This work examines the potential advantages of immersive technologies, particularly 360° videos, in enhancing memory evaluation. First, a comprehensive overview of contemporary virtual reality tools employed in the assessment of memory, as well as their convergence with conventional assessment measures has been provided. Then, the present study utilizes cluster and network analysis techniques to categorize 360° videos according to their content and applications, thereby offering significant insights into the potential of this nascent medium. The study introduces then a novel platform, Mindscape, that aims to address the existing technological disparity, thereby enhancing the accessibility of clinicians and researchers in developing cognitive tasks within immersive environments. The conclusion of the thesis encompasses the psychometric validation of two memory tasks, which have been specifically developed with Mindscape to assess episodic and spatial memory. The findings demonstrate disparities in cognitive performance between individuals diagnosed with Mild Cognitive Impairment and those without cognitive impairments, underscoring the interrelated nature of cognitive processes and the promising prospects of virtual reality technology in improving the authenticity of real-world experiences. Overall, this dissertation aims to respond to the demand for practical and ecologically valid neuropsychological assessments within the dynamic field of neuropsychology. It achieves this by integrating user-friendly platforms and immersive cognitive tasks into its methodology. By highlighting a shift in the field of neuropsychology towards prioritizing functional and practical assessments over theoretical frameworks, this work indicates a changing perspective within the discipline. This study highlights the potential of comprehensive and purpose-oriented assessment methods in cognitive evaluations, emphasizing the ongoing significance of research in fully comprehending the capabilities of immersive technologies

Moving sounds and sonic moves : exploring interaction quality of embodied music mediation technologies through a user-centered perspective

This research project deals with the user-experience related to embodied music mediation technologies. More specifically, adoption and policy problems surrounding new media (art) are considered, which arise from the usability issues that to date pervade new interfaces for musical expression. Since the emergence of new wireless mediators and control devices for musical expression, there is an explicit aspiration of the creative industries and various research centers to embed such technologies into different areas of the cultural industries. The number of applications and their uses have exponentially increased over the last decade. Conversely, many of the applications to date still suffer from severe usability problems, which not only hinder the adoption by the cultural sector, but also make culture participants take a rather cautious, hesitant, or even downright negative stance towards these technologies. Therefore, this thesis takes a vantage point that is in part sociological in nature, yet has a link to cultural studies as well. It combines this with a musicological frame of reference to which it introduces empirical user-oriented approaches, predominantly taken from the field of human-computer-interaction studies. This interdisciplinary strategy is adopted to cope with the complex nature of digital embodied music controlling technologies. Within the Flanders cultural (and creative) industries, opportunities of systems affiliated with embodied interaction are created and examined. This constitutes an epistemological jigsaw that looks into 1) “which stakeholders require what various levels of involvement, what interactive means and what artistic possibilities?”, 2) “the way in which artistic aspirations, cultural prerequisites and operational necessities of (prospective) users can be defined?”, 3) “how functional, artistic and aesthetic requirements can be accommodated?”, and 4) “how quality of use and quality of experience can be achieved, quantified, evaluated and, eventually, improved?”. Within this multi-facetted problem, the eventual aim is to assess the applicability of the foresaid technology, both from a theoretically and empirically sound basis, and to facilitate widening and enhancing the adoption of said technologies. Methodologically, this is achieved by 1) applied experimentation, 2) interview techniques, 3) self-reporting and survey research, 4) usability evaluation of existing devices, and 5) human-computer interaction methods applied – and attuned – to the specific case of embodied music mediation technologies. Within that scope, concepts related to usability, flow, presence, goal assessment and game enjoyment are scrutinized and applied, and both task- and experience-oriented heuristics and metrics are developed and tested. In the first part, covering three chapters, the general context of the thesis is given. In the first chapter, an introduction to the topic is offered and the current problems are enumerated. In the second chapter, a broader theoretical background is presented of the concepts that underpin the project, namely 1) the paradigm of embodiment and its connection to musicology, 2) a state of the arts concerning new interfaces for musical expression, 3) an introduction into HCI-usability and its application domain in systematic musicology, 4) an insight into user-centered digital design procedures, and 5) the challenges brought about by e-culture and digitization for the cultural-creative industries. In the third chapter, the state of the arts concerning the available methodologies related to the thesis’ endeavor is discussed, a set of literature-based design guidelines are enumerated and from this a conceptual model is deduced which is gradually presented throughout the thesis, and fully deployed in the “SoundField”-project (as described in Chapter 9). The following chapters, contained in the second part of the thesis, give a quasi-chronological overview of how methodological concepts have been applied throughout the empirical case studies, aimed specifically at the exploration of the various aspects of the complex status quaestionis. In the fourth chapter, a series of application-based tests, predominantly revolving around interface evaluation, illustrate the complex relation between gestural interfaces and meaningful musical expression, advocating a more user-centered development approach to be adopted. In the fifth chapter, a multi-purpose questionnaire dubbed “What Moves You” is discussed, which aimed at creating a survey of the (prospective) end-users of embodied music mediation technologies. Therefore, it primarily focused on cultural background, musical profile and preferences, views on embodied interaction, literacy of and attitudes towards new technology and participation in digital culture. In the sixth chapter, the ethnographical studies that accompanied the exhibition of two interactive art pieces, entitled "Heart as an Ocean" & "Lament", are discussed. In these studies, the use of interview and questionnaire methodologies together with the presentation and reception of interactive art pieces, are probed. In the seventh chapter, the development of the collaboratively controlled music-game “Sync-In-Team” is presented, in which interface evaluation, presence, game enjoyment and goal assessment are the pivotal topics. In the eighth chapter, two usability studies are considered, that were conducted on prototype systems/interfaces, namely a heuristic evaluation of the “Virtual String” and a usability metrics evaluation on the “Multi-Level Sonification Tool”. The findings of these two studies in conjunction with the exploratory studies performed in association with the interactive art pieces, finally gave rise to the “SoundField”-project, which is recounted in full throughout the ninth chapter. The integrated participatory design and evaluation method, presented in the conceptual model is fully applied over the course of the “SoundField”-project, in which technological opportunities and ecological validity and applicability are investigated through user-informed development of numerous use cases. The third and last part of the thesis renders the final conclusions of this research project. The tenth chapter sets out with an epilogue in which a brief overview is given on how the state of the arts has evolved since the end of the project (as the research ended in 2012, but the research field has obviously moved on), and attempts to consolidate the implications of the research studies with some of the realities of the Flemish cultural-creative industries. Chapter eleven continues by discussing the strengths and weaknesses of the conceptual model throughout the various stages of the project. Also, it comprises the evaluation of the hypotheses, how the assumptions that were made held up, and how the research questions eventually could be assessed. Finally, the twelfth and last chapter concludes with the most important findings of the project. Also, it discusses some of the implications on cultural production, artistic research policy and offers an outlook on future research beyond the scope of the “SoundField” project

CoMoVA - A comprehension measurement framework for visualization systems

Despite the burgeoning interest shown in visualizations by many disciplines, there yet remains the unresolved question concerning comprehension. Is the concept that is being communicated through the visual easily grasped and clearly interpreted? Visual comprehension is that characteristic of any visualization system, which deals with how efficiently and effectively users are able to grasp the underlying concepts through suitable interactions provided for exploring the visually represented information. Comprehension has been considered a very complex subject, which is intangible and subjective in nature. Assessment of comprehension can help to determine the true usefulness of visualization systems to the intended users. A principal contribution of this research is the formulation of an empirical evaluation framework for systematically assessing comprehension support provided by a visualization system to its intended users. To assess comprehension i.e. to measure this seemingly immeasurable factor of visualization systems, we propose a set of criteria based on a detailed analysis of information flow from the raw data to the cognition of information in human mind. Our comprehension criteria are adapted from the pioneering work of two eminent researchers - Donald A. Norman and Aaron Marcus, who have investigated the issues of human perception and cognition, and visual effectiveness respectively. The proposed criteria have been refined with the help of opinions from experts. To gauge and verify the efficacy of these criteria in a practical sense, they were then applied to a bioinformatics visualization study tool and an immersive art visualization environment. Given the vast variety of users and their visualization goals, it may be noted that it is difficult for one to decide on the effectiveness of different visualization tools/techniques in a context independent fashion. We therefore propose an innovative way of evaluating a visualization technique by encapsulating it in a visualization pattern where it is seen as a solution to the visualization problem in a specific context. These visualization patterns guide the tool users/evaluators to compare, understand and select appropriate visualization tools/techniques. Lastly, we propose a novel framework named as CoMoVA (Comprehension Model for Visualization Assessment) that incorporates 'context of use', visualization patterns, visual design principles and important cognitive principles into a coherent whole that can be used to effectively tell us in a more quantifiable manner the benefits of visual representations and interactions provided by a system to the intended audience. Our approach of evaluation of visualization systems is similar to other questionnaire-based approaches such as SUMI (Software Usability Measurement Inventory), where all the questions deal with the measurement of a common trait. We apply this framework to two static software visualization tools in the software visualization domain to demonstrate the practical benefits of using such a framework

Sonic Interactions in Virtual Environments

This open access book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Sonic interactions in virtual environments

This book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments