Towards Naturalistic Interfaces of Virtual Reality Systems

Interaction plays a key role in achieving realistic experience in virtual reality (VR). Its realization depends on interpreting the intents of human motions to give inputs to VR systems. Thus, understanding human motion from the computational perspective is essential to the design of naturalistic interfaces for VR. This dissertation studied three types of human motions, including locomotion (walking), head motion and hand motion in the context of VR. For locomotion, the dissertation presented a machine learning approach for developing a mechanical repositioning technique based on a 1-D treadmill for interacting with a unique new large-scale projective display, called the Wide-Field Immersive Stereoscopic Environment (WISE). The usability of the proposed approach was assessed through a novel user study that asked participants to pursue a rolling ball at variable speed in a virtual scene. In addition, the dissertation studied the role of stereopsis in avoiding virtual obstacles while walking by asking participants to step over obstacles and gaps under both stereoscopic and non-stereoscopic viewing conditions in VR experiments. In terms of head motion, the dissertation presented a head gesture interface for interaction in VR that recognizes real-time head gestures on head-mounted displays (HMDs) using Cascaded Hidden Markov Models. Two experiments were conducted to evaluate the proposed approach. The first assessed its offline classification performance while the second estimated the latency of the algorithm to recognize head gestures. The dissertation also conducted a user study that investigated the effects of visual and control latency on teleoperation of a quadcopter using head motion tracked by a head-mounted display. As part of the study, a method for objectively estimating the end-to-end latency in HMDs was presented. For hand motion, the dissertation presented an approach that recognizes dynamic hand gestures to implement a hand gesture interface for VR based on a static head gesture recognition algorithm. The proposed algorithm was evaluated offline in terms of its classification performance. A user study was conducted to compare the performance and the usability of the head gesture interface, the hand gesture interface and a conventional gamepad interface for answering Yes/No questions in VR. Overall, the dissertation has two main contributions towards the improvement of naturalism of interaction in VR systems. Firstly, the interaction techniques presented in the dissertation can be directly integrated into existing VR systems offering more choices for interaction to end users of VR technology. Secondly, the results of the user studies of the presented VR interfaces in the dissertation also serve as guidelines to VR researchers and engineers for designing future VR systems

Multimodal Content Delivery for Geo-services

This thesis describes a body of work carried out over several research projects in the area of multimodal interaction for location-based services. Research in this area has progressed from using simulated mobile environments to demonstrate the visual modality, to the ubiquitous delivery of rich media using multimodal interfaces (geo- services). To effectively deliver these services, research focused on innovative solutions to real-world problems in a number of disciplines including geo-location, mobile spatial interaction, location-based services, rich media interfaces and auditory user interfaces. My original contributions to knowledge are made in the areas of multimodal interaction underpinned by advances in geo-location technology and supported by the proliferation of mobile device technology into modern life. Accurate positioning is a known problem for location-based services, contributions in the area of mobile positioning demonstrate a hybrid positioning technology for mobile devices that uses terrestrial beacons to trilaterate position. Information overload is an active concern for location-based applications that struggle to manage large amounts of data, contributions in the area of egocentric visibility that filter data based on field-of-view demonstrate novel forms of multimodal input. One of the more pertinent characteristics of these applications is the delivery or output modality employed (auditory, visual or tactile). Further contributions in the area of multimodal content delivery are made, where multiple modalities are used to deliver information using graphical user interfaces, tactile interfaces and more notably auditory user interfaces. It is demonstrated how a combination of these interfaces can be used to synergistically deliver context sensitive rich media to users - in a responsive way - based on usage scenarios that consider the affordance of the device, the geographical position and bearing of the device and also the location of the device

Analysis of Visualisation and Interaction Tools Authors

This document provides an in-depth analysis of visualization and interaction tools employed in the context of Virtual Museum. This analysis is required to identify and design the tools and the different components that will be part of the Common Implementation Framework (CIF). The CIF will be the base of the web-based services and tools to support the development of Virtual Museums with particular attention to online Virtual Museum.The main goal is to provide to the stakeholders and developers an useful platform to support and help them in the development of their projects, despite the nature of the project itself. The design of the Common Implementation Framework (CIF) is based on an analysis of the typical workflow ofthe V-MUST partners and their perceived limitations of current technologies. This document is based also on the results of the V-MUST technical questionnaire (presented in the Deliverable 4.1). Based on these two source of information, we have selected some important tools (mainly visualization tools) and services and we elaborate some first guidelines and ideas for the design and development of the CIF, that shall provide a technological foundation for the V-MUST Platform, together with the V-MUST repository/repositories and the additional services defined in the WP4. Two state of the art reports, one about user interface design and another one about visualization technologies have been also provided in this document

Development and evaluation of mixed reality-enhanced robotic systems for intuitive tele-manipulation and telemanufacturing tasks in hazardous conditions

In recent years, with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices, there is an urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations. The value of medical telerobotic applications during the recent coronavirus pandemic has also been demonstrated and will grow in the future. This thesis investigates novel approaches to the development and evaluation of a mixed reality-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions, such as contaminated sites and undersea or extreme welding scenarios. This research aims to remove human workers from the harmful working environments by equipping complex robotic systems with human intelligence and command/control via intuitive and natural human-robot- interaction, including the implementation of MR techniques to improve the user's situational awareness, depth perception, and spatial cognition, which are fundamental to effective and efficient teleoperation. The proposed robotic mobile manipulation platform consists of a UR5 industrial manipulator, 3D-printed parallel gripper, and customized mobile base, which is envisaged to be controlled by non-skilled operators who are physically separated from the robot working space through an MR-based vision/motion mapping approach. The platform development process involved CAD/CAE/CAM and rapid prototyping techniques, such as 3D printing and laser cutting. Robot Operating System (ROS) and Unity 3D are employed in the developing process to enable the embedded system to intuitively control the robotic system and ensure the implementation of immersive and natural human-robot interactive teleoperation. This research presents an integrated motion/vision retargeting scheme based on a mixed reality subspace approach for intuitive and immersive telemanipulation. An imitation-based velocity- centric motion mapping is implemented via the MR subspace to accurately track operator hand movements for robot motion control, and enables spatial velocity-based control of the robot tool center point (TCP). The proposed system allows precise manipulation of end-effector position and orientation to readily adjust the corresponding velocity of maneuvering. A mixed reality-based multi-view merging framework for immersive and intuitive telemanipulation of a complex mobile manipulator with integrated 3D/2D vision is presented. The proposed 3D immersive telerobotic schemes provide the users with depth perception through the merging of multiple 3D/2D views of the remote environment via MR subspace. The mobile manipulator platform can be effectively controlled by non-skilled operators who are physically separated from the robot working space through a velocity-based imitative motion mapping approach. Finally, this thesis presents an integrated mixed reality and haptic feedback scheme for intuitive and immersive teleoperation of robotic welding systems. By incorporating MR technology, the user is fully immersed in a virtual operating space augmented by real-time visual feedback from the robot working space. The proposed mixed reality virtual fixture integration approach implements hybrid haptic constraints to guide the operator’s hand movements following the conical guidance to effectively align the welding torch for welding and constrain the welding operation within a collision-free area. Overall, this thesis presents a complete tele-robotic application space technology using mixed reality and immersive elements to effectively translate the operator into the robot’s space in an intuitive and natural manner. The results are thus a step forward in cost-effective and computationally effective human-robot interaction research and technologies. The system presented is readily extensible to a range of potential applications beyond the robotic tele- welding and tele-manipulation tasks used to demonstrate, optimise, and prove the concepts

Three-dimensional user interfaces for scientific visualization

The focus of this grant was to experiment with novel user interfaces for scientific visualization applications using both desktop and virtual reality (VR) systems, and thus to advance the state of the art of user interface technology for this domain. This technology has been transferred to NASA via periodic status reports and papers relating to this grant that have been published in conference proceedings. This final report summarizes the research completed over the past three years, and subsumes all prior reports

Interactive maps: What we know and what we need to know

This article provides a review of the current state of science regarding cartographic interaction a complement to the traditional focus within cartography on cartographic representation. Cartographic interaction is defined as the dialog between a human and map mediated through a computing device and is essential to the research into interactive cartography geovisualization and geovisual analytics. The review is structured around six fundamental questions facing a science of cartographic interaction: (1) what is cartographic interaction (e.g. digital versus analog interactions interaction versus interfaces stages of interaction interactive maps versus mapping systems versus map mash-ups); (2) why provide cartographic interaction (e.g. visual thinking geographic insight the stages of science the cartographic problematic); (3) when should cartographic interaction be provided (e.g. static versus interactive maps interface complexity the productivity paradox flexibility versus constraint work versus enabling interactions); (4) who should be provided with cartographic interaction (e.g. user-centered design user ability expertise and motivation adaptive cartography and geocollaboration); (5) where should cartographic interaction be provided (e.g. input capabilities bandwidth and processing power display capabilities mobile mapping and location-based services); and (6) how should cartographic interaction be provided (e.g. interaction primitives objective-based versus operator-based versus operand-based taxonomies interface styles interface design)? The article concludes with a summary of research questions facing cartographic interaction and offers an outlook for cartography as a field of study moving forward

Framework for indoor video-based augumented reality applications

Augmented Reality (AR) has been proven to be useful in many fields such as medical surgery, military training, engineering design, tourist guiding, manufacturing and maintenance. Several AR systems and tracking tools have been reviewed and examined. Taking into consideration the different shortcomings of the available AR systems, a framework for indoor video-based AR applications is proposed to integrate four main components of AR applications, which are large scale virtual environment, mobile devices, interaction methods and video-tracking, in one system. The proposed framework benefits from the rapidly evolving technology in virtual modeling by combing GIS maps and 3D virtual models of cities and building interiors in one single platform. Interaction methods for AR applications are introduced, such as the automatic 3D picking which allows for a location-based data access. In addition, a practical method is proposed for the configuration and the deployment of video tracking. This method makes use of the XML mark-up language to allow for future extensions and simplified interchangeability. An implementation of the proposed approach is developed to demonstrate the feasibility of the framework. Different case studies are carried out to validate the applicability of the system and identify its benefits and limitations

Towards an activity-oriented design method for HCI research and practice

During the last two decades, designing for usability has been the focus of attention when developing computer systems. However, the dynamic nature of human use of computer systems has meant that designing for 'usefulness' or 'fitness for purpose' is increasingly becoming the primary concern for systems developers. Central to this concern are issues underpinned by the social context in which a computer user operates.Within the field of Human-Computer Interaction (HCI), this situation led to a search for appropriate theories for conceptualising these design concerns. Whilst Activity Theory has been identified as a suitable framework for conceptualising these user perspectives, the lack of a standard methodology for applying it to HCI research and practice has meant that many systems developers have failed to benefit from the richness of this framework. The objective of this thesis was therefore to develop an Activity Theory based methodology for HCI research and practice. This thesis, contributes the `Activity-Oriented Design Methodology' (AODM) both as a practical and analytical methodology for using Activity Theory within HCI design. AODM incorporates four methodological tools namely:- The `Eight-Step-Model'- The `Activity Notation'- The technique of `Generating Research Questions'- The technique of `Mapping AODM Operational Processes'AODM tools were constructed from empirical work carried out as part of this research. Empirical analysis of work practices in two organisations was conducted for a period of two years using Activity Theory. This empirical work formed the basis for validating AODM. AODM tools support the systems design processes of gathering, analysing and communicating (through modelling) research and design insights from an Activity Theory perspective. It is argued that AODM provides a valuable practical and analytical methodology for operationalising Activity Theory within HCI so as to support early phases of systems design: namely, requirements capture and evaluation