Probabilistic User Interface Design for Virtual and Augmented Reality Applications

The central hypothesis of this thesis is that probabilistic user interface design provides an effective methodology for delivering productive and enjoyable applications in virtual reality (VR) and augmented reality (AR). This investigation is timely given the recent emergence of mass-market virtual and augmented reality head-mounted displays and growing demand for tailored applications and content. The design guidance for building compelling and productive applications for these environments is, however, currently lagging the pace at which the underlying technology is maturing. This is problematic given important differences between designing conventional 2D interfaces and interactions and their embodied 3D counterparts. This dissertation investigates probabilistic user interface design as a method for solving many of the novel challenges encountered when developing applications for VR and AR. Probabilistic user interface design seeks to model the uncertain events in a system and identify, implement and validate strategies that drive improved system performance. This thesis addresses four research questions by applying a probabilistic treatment in four distinct but closely related case studies. These four case studies are selected to illustrate the flexibility and unique benefits offered by this method. Research Question 1 asks how the probabilistic qualities of an interface can be determined and how this can inform design. This question is investigated in the context of text entry in VR with a probabilistic characterisation performed on two fundamental design choices. Research Question 2 relates to the challenge of adapting AR applications to deployment contexts not knowable at design time. A study in which crowdworkers are employed to build a probabilistic understanding of the requirements for contextually adaptive AR answers this question. The text entry theme is revisited in answering Research Questions 3 which asks how high levels of input noise can be mitigated through inference. A probabilistic text entry method specifically tailored for use in AR is implemented and evaluated. Finally, Research Question 4 asks how the high dimensional design space in AR and VR applications can be efficiently explored to support ideal design choices. Interface refinement through probabilistic optimisation and crowdsourcing is shown to be highly efficient and effective for this purpose. A probabilistic treatment in the design process has many potential benefits, principle among which is increased robustness to circumstances unanticipated at design time. This thesis contributes to the toolset and guidance available to designers and supports the development of next generation user interfaces specifically tailored to virtual and augmented reality

Learning Data-Driven Models of Non-Verbal Behaviors for Building Rapport Using an Intelligent Virtual Agent

There is a growing societal need to address the increasing prevalence of behavioral health issues, such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. The statistics, worldwide and in the USA, are daunting. Excessive alcohol use is the third leading preventable cause of death in the United States (with 79,000 deaths annually), and is responsible for a wide range of health and social problems. On the positive side though, these behavioral health issues (and associated possible diseases) can often be prevented with relatively simple lifestyle changes, such as losing weight with a diet and/or physical exercise, or learning how to reduce alcohol consumption. Medicine has therefore started to move toward finding ways of preventively promoting wellness, rather than solely treating already established illness. Evidence-based patient-centered Brief Motivational Interviewing (BMI) interven- tions have been found particularly effective in helping people find intrinsic motivation to change problem behaviors after short counseling sessions, and to maintain healthy lifestyles over the long-term. Lack of locally available personnel well-trained in BMI, however, often limits access to successful interventions for people in need. To fill this accessibility gap, Computer-Based Interventions (CBIs) have started to emerge. Success of the CBIs, however, critically relies on insuring engagement and retention of CBI users so that they remain motivated to use these systems and come back to use them over the long term as necessary. Because of their text-only interfaces, current CBIs can therefore only express limited empathy and rapport, which are the most important factors of health interventions. Fortunately, in the last decade, computer science research has progressed in the design of simulated human characters with anthropomorphic communicative abilities. Virtual characters interact using humans’ innate communication modalities, such as facial expressions, body language, speech, and natural language understanding. By advancing research in Artificial Intelligence (AI), we can improve the ability of artificial agents to help us solve CBI problems. To facilitate successful communication and social interaction between artificial agents and human partners, it is essential that aspects of human social behavior, especially empathy and rapport, be considered when designing human-computer interfaces. Hence, the goal of the present dissertation is to provide a computational model of rapport to enhance an artificial agent’s social behavior, and to provide an experimental tool for the psychological theories shaping the model. Parts of this thesis were already published in [LYL+12, AYL12, AL13, ALYR13, LAYR13, YALR13, ALY14]

Studies on customisation-driven digital music instruments

From John Cage’s Prepared Piano to the turntable, the history of musical instruments is scattered with examples of musicians who deeply customised their instruments to fit personal artistic objectives, objectives that differed from the ones the instruments have been designed for. In their digital counterpart however, musical instruments are often presented in the form of closed, finalised systems with apriori symbolic rules set by their designer that leave very little room for the artists to customise the technologies for their unique art practices; in these cases the only possibility to change the mode of interaction with digital instrument is to reprogram them, a possibility available to programmers but not to musicians. This thesis presents two digital music instruments designed with the explicit goal of being highly customisable by musicians and to provide different modes of interactions, whilst keeping simplicity and immediateness of use. The first one leverages real-time gesture recognition to provide continuous feedback to users as guidance in defining the behaviour of the system and the gestures it recognises. The second one is a novel tangible user interface which allows to transform everyday objects into expressive digital music instruments, and whose sound generated strongly depends by the particular nature of the physical object selected

Advanced Occupancy Measurement Using Sensor Fusion

With roughly about half of the energy used in buildings attributed to Heating, Ventilation, and Air conditioning (HVAC) systems, there is clearly great potential for energy saving through improved building operations. Accurate knowledge of localised and real-time occupancy numbers can have compelling control applications for HVAC systems. However, existing technologies applied for building occupancy measurements are limited, such that a precise and reliable occupant count is difficult to obtain. For example, passive infrared (PIR) sensors commonly used for occupancy sensing in lighting control applications cannot differentiate between occupants grouped together, video sensing is often limited by privacy concerns, atmospheric gas sensors (such as CO2 sensors) may be affected by the presence of electromagnetic (EMI) interference, and may not show clear links between occupancy and sensor values. Past studies have indicated the need for a heterogeneous multi-sensory fusion approach for occupancy detection to address the short-comings of existing occupancy detection systems. The aim of this research is to develop an advanced instrumentation strategy to monitor occupancy levels in non-domestic buildings, whilst facilitating the lowering of energy use and also maintaining an acceptable indoor climate. Accordingly, a novel multi-sensor based approach for occupancy detection in open-plan office spaces is proposed. The approach combined information from various low-cost and non-intrusive indoor environmental sensors, with the aim to merge advantages of various sensors, whilst minimising their weaknesses. The proposed approach offered the potential for explicit information indicating occupancy levels to be captured. The proposed occupancy monitoring strategy has two main components; hardware system implementation and data processing. The hardware system implementation included a custom made sound sensor and refinement of CO2 sensors for EMI mitigation. Two test beds were designed and implemented for supporting the research studies, including proof-of-concept, and experimental studies. Data processing was carried out in several stages with the ultimate goal being to detect occupancy levels. Firstly, interested features were extracted from all sensory data collected, and then a symmetrical uncertainty analysis was applied to determine the predictive strength of individual sensor features. Thirdly, a candidate features subset was determined using a genetic based search. Finally, a back-propagation neural network model was adopted to fuse candidate multi-sensory features for estimation of occupancy levels. Several test cases were implemented to demonstrate and evaluate the effectiveness and feasibility of the proposed occupancy detection approach. Results have shown the potential of the proposed heterogeneous multi-sensor fusion based approach as an advanced strategy for the development of reliable occupancy detection systems in open-plan office buildings, which can be capable of facilitating improved control of building services. In summary, the proposed approach has the potential to: (1) Detect occupancy levels with an accuracy reaching 84.59% during occupied instances (2) capable of maintaining average occupancy detection accuracy of 61.01%, in the event of sensor failure or drop-off (such as CO2 sensors drop-off), (3) capable of utilising just sound and motion sensors for occupancy levels monitoring in a naturally ventilated space, (4) capable of facilitating potential daily energy savings reaching 53%, if implemented for occupancy-driven ventilation control

The Appreciation of Electroacoustic Music - An Empirical Study with Inexperienced Listeners

The research contained within this PhD project forms part of the Pedagogical ElectroAcoustic Resource Site project of the Music, Technology and Innovation Research Centre of De Montfort University Leicester. This thesis contributes to current research in music education and musicology related to electroacoustic music. The purpose of this research was to investigate the influence of teaching on the change in inexperienced listeners’ appreciation of electroacoustic music. A curriculum was developed to introduce electroacoustic music to 11 to 14 year old students (Key Stage 3). The curriculum was based on concepts distinguishing between electroacoustic music using (mainly) real-world sounds and generated sounds. The curriculum is presented in an online learning environment with an accompanying teacher’s handbook. The learning environment represents the prototype for the pedagogical ElectroAcoustic Resource Site offering online learning, blended learning and classroom-based learning. The website was developed following user-centred design; the curriculum was tested in a large-scale study including four Key Stage 3 classes within three schools in Leicester. In five lessons music using real-world sounds (soundscape and musique concrète) was introduced, which included the delivery of a listening training, independent research and creative tasks (composition or devising a role-play). The teaching design followed the methods of active, collaborative and self-regulated learning. Data was collected by using questionnaires, direct responses to listening experiences before and after the teaching, and summaries of the teaching written by the participants. Following a Qualitative Content Analysis, the results of the study show that the participants’ appreciation of electroacoustic music changed during the course of these lessons. Learning success could be established as well as a declining alienation towards electroacoustic music. The principal conclusion is that the appreciation of electroacoustic music can be enhanced through the acquiring of conceptual knowledge, especially through the enhancing of listening skills following the structured listening training as well as the broadening of vocabulary to describe the listening experience

Foundations and applications of human-machine-interaction

Der vorliegende Tagungsband zur 10. Berliner Werkstatt Mensch-Maschine-Systeme gibt einen Einblick in die aktuelle Forschung im Bereich der Mensch-Maschine- Interaktion. Einen besonderen Fokus stellt das Wechselspiel von Grundlagenforschung und anwendungsbezogener Forschung dar, was sich im breiten Themenspektrum widerspiegelt, welches von theoretischen und methodischen Betrachtungen bis hin zu anwendungsnahen Fragestellungen reicht. Dabei finden Inhalte aus allen Phasen des Forschungsprozesses Beachtung, sodass auch im Rahmen der 10. Berliner Werkstatt MMS wieder sowohl neue Untersuchungskonzepte als auch abschließende Befunde diskutiert werden. Zentrale Themengebiete sind u. a. Fahrer-Fahrzeug-Interaktion, Assistenzsysteme, User Experience, Usability, Ubiquitous Computing, Mixed & Virtual Reality, Robotics & Automation, Wahrnehmungsspezifika sowie Psychophysiologie und Beanspruchung in der Mensch-Maschine-Interaktion.The proceedings of the 10th Berlin Workshop Human-Machine-Systems provide an insight into the current research in the field of human-machine-interaction. The main focus lies on the interplay between basic and applied research, which is reflected in the wide range of subjects: from theoretical and methodological issues to application oriented considerations. Again all stages of the research process are represented in the contributions of the 10th Berlin Workshop HMS. This means new research concepts as well as final results are subject of this volume. Central topics include driver-vehicleinteraction, assistance systems, user experience, usability, ubiquitous computing, mixed and virtual reality, robotics & automation, perception specifics, as well as psychophysiology and workload in human-machine-interaction

Smart Technologies for Precision Assembly

This open access book constitutes the refereed post-conference proceedings of the 9th IFIP WG 5.5 International Precision Assembly Seminar, IPAS 2020, held virtually in December 2020. The 16 revised full papers and 10 revised short papers presented together with 1 keynote paper were carefully reviewed and selected from numerous submissions. The papers address topics such as assembly design and planning; assembly operations; assembly cells and systems; human centred assembly; and assistance methods in assembly