Investigation of a holistic human-computer interaction (HCI) framework to support the design of extended reality (XR) based training simulators

In recent years, the use of Extended Reality (XR) based simulators for training has increased rapidly. In this context, there is a need to explore novel HCI-based approaches to design more effective 3D training environments. A major impediment in this research area is the lack of an HCI-based framework that is holistic and serves as a foundation to integrate the design and assessment of HCI-based attributes such as affordance, cognitive load, and user-friendliness. This research addresses this need by investigating the creation of a holistic framework along with a process for designing, building, and assessing training simulators using such a framework as a foundation. The core elements of the proposed framework include the adoption of participatory design principles, the creation of information-intensive process models of target processes (relevant to the training activities), and design attributes related to affordance and cognitive load. A new attribute related to affordance of 3D scenes is proposed (termed dynamic affordance) and its role in impacting user comprehension in data-rich 3D training environments is studied. The framework is presented for the domain of orthopedic surgery. Rigorous user-involved assessment of the framework and simulation approach has highlighted the positive impact of the HCI-based framework and attributes on the acquisition of skills and knowledge by healthcare users

Virtual Reality Based Environment for Orthopedic Surgery (Veos)

The traditional way of teaching surgery involves students observing a �live� surgery and then gradually assisting experienced surgeons. The creation of a Virtual Reality environment for orthopedic surgery (VEOS) can be beneficial in improving the quality of training while decreasing the time needed for training. Developing such virtual environments for educational and training purposes can supplement existing approaches. In this research, the design and development of a virtual reality based environment for orthopedic surgery is described. The scope of the simulation environment is restricted to an orthopedic surgery process known as Less Invasive Stabilization System (LISS) surgery. The primary knowledge source for the LISS surgical process was Miguel A. Pirela-Cruz (Head of Orthopedic Surgery and Rehabilitation, Texas Tech University Health Sciences Center (TTHSC)). The VEOS was designed and developed on a PC based platform. The developed VEOS was validated through interactions with surgical residents at TTHSC. Feedback from residents and our collaborator Miguel A. Pirela-Cruz was used to make necessary modifications to the surgical environment.Industrial Engineering & Managemen

Robot-Assisted Minimally Invasive Surgery-Surgical Robotics in the Data Age

Telesurgical robotics, as a technical solution for robot-assisted minimally invasive surgery (RAMIS), has become the first domain within medicosurgical robotics that achieved a true global clinical adoption. Its relative success (still at a low single-digit percentile total market penetration) roots in the particular human-in-the-loop control, in which the trained surgeon is always kept responsible for the clinical outcome achieved by the robot-actuated invasive tools. Nowadays, this paradigm is challenged by the need for improved surgical performance, traceability, and safety reaching beyond the human capabilities. Partially due to the technical complexity and the financial burden, the adoption of telesurgical robotics has not reached its full potential, by far. Apart from the absolutely market-dominating da Vinci surgical system, there are already 60+ emerging RAMIS robot types, out of which 15 have already achieved some form of regulatory clearance. This article aims to connect the technological advancement with the principles of commercialization, particularly looking at engineering components that are under development and have the potential to bring significant advantages to the clinical practice. Current RAMIS robots often do not exceed the functionalities deriving from their mechatronics, due to the lack of data-driven assistance and smart human–machine collaboration. Computer assistance is gradually gaining more significance within emerging RAMIS systems. Enhanced manipulation capabilities, refined sensors, advanced vision, task-level automation, smart safety features, and data integration mark together the inception of a new era in telesurgical robotics, infiltrated by machine learning (ML) and artificial intelligence (AI) solutions. Observing other domains, it is definite that a key requirement of a robust AI is the good quality data, derived from proper data acquisition and sharing to allow building solutions in real time based on ML. Emerging RAMIS technologies are reviewed both in a historical and a future perspective

The HoloLens in Medicine: A systematic Review and Taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality display, is the main player in the recent boost in medical augmented reality research. In medical settings, the HoloLens enables the physician to obtain immediate insight into patient information, directly overlaid with their view of the clinical scenario, the medical student to gain a better understanding of complex anatomies or procedures, and even the patient to execute therapeutic tasks with improved, immersive guidance. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021, were attention is shifting towards it's successor, the HoloLens 2. We identified 171 relevant publications through a systematic search of the PubMed and Scopus databases. We analyze these publications in regard to their intended use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation. We find that, although the feasibility of using the HoloLens in various medical scenarios has been shown, increased efforts in the areas of precision, reliability, usability, workflow and perception are necessary to establish AR in clinical practice.Comment: 35 pages, 11 figure

Cyber physical approach and framework for micro devices assembly

The emergence of Cyber Physical Systems (CPS) and Internet-of-Things (IoT) based principles and technologies holds the potential to facilitate global collaboration in various fields of engineering. Micro Devices Assembly (MDA) is an emerging domain involving the assembly of micron sized objects and devices. In this dissertation, the focus of the research is the design of a Cyber Physical approach for the assembly of micro devices. A collaborative framework comprising of cyber and physical components linked using the Internet has been developed to accomplish a targeted set of MDA life cycle activities which include assembly planning, path planning, Virtual Reality (VR) based assembly analysis, command generation and physical assembly. Genetic algorithm and modified insertion algorithm based methods have been proposed to support assembly planning activities. Advanced VR based environments have been designed to support assembly analysis where plans can be proposed, compared and validated. The potential of next generation Global Environment for Network Innovation (GENI) networking technologies has also been explored to support distributed collaborations involving VR-based environments. The feasibility of the cyber physical approach has been demonstrated by implementing the cyber physical components which collaborate to assemble micro designs. The case studies conducted underscore the ability of the developed Cyber Physical approach and framework to support distributed collaborative activities for MDA process contexts

Serious Games and Mixed Reality Applications for Healthcare

Virtual reality (VR) and augmented reality (AR) have long histories in the healthcare sector, offering the opportunity to develop a wide range of tools and applications aimed at improving the quality of care and efficiency of services for professionals and patients alike. The best-known examples of VR–AR applications in the healthcare domain include surgical planning and medical training by means of simulation technologies. Techniques used in surgical simulation have also been applied to cognitive and motor rehabilitation, pain management, and patient and professional education. Serious games are ones in which the main goal is not entertainment, but a crucial purpose, ranging from the acquisition of knowledge to interactive training.These games are attracting growing attention in healthcare because of their several benefits: motivation, interactivity, adaptation to user competence level, flexibility in time, repeatability, and continuous feedback. Recently, healthcare has also become one of the biggest adopters of mixed reality (MR), which merges real and virtual content to generate novel environments, where physical and digital objects not only coexist, but are also capable of interacting with each other in real time, encompassing both VR and AR applications.This Special Issue aims to gather and publish original scientific contributions exploring opportunities and addressing challenges in both the theoretical and applied aspects of VR–AR and MR applications in healthcare

Diverse approaches to learning with immersive Virtual Reality identified from a systematic review

To investigate how learning in immersive Virtual Reality was designed in contemporary educational studies, this systematic literature review identified nine design features and analysed 219 empirical studies on the designs of learning activities with immersive Virtual Reality. Overall, the technological features for physical presence were more readily implemented and investigated than pedagogical features for learning engagement. Further analysis with k-means clustering revealed five approaches with varying levels of interactivity and openness in learning tasks, from watching virtual worlds passively to responding to personalised prompts. Such differences in the design appeared to stem from different practical and educational priorities, such as accessibility, interactivity, and engagement. This review highlights the diversity in the learning task designs in immersive Virtual Reality and illustrates how researchers are navigating practical and educational concerns. We recommend future empirical studies recognise the different approaches and priorities when designing and evaluating learning with immersive Virtual Reality. We also recommend that future systematic reviews investigate immersive Virtual Reality-based learning not only by learning topics or learner demographics, but also by task designs and learning experiences

Diverse approaches to learning with immersive Virtual Reality identified from a systematic review

To investigate how learning in immersive Virtual Reality was designed in contemporary educational studies, this systematic literature review identified nine design features and analysed 219 empirical studies on the designs of learning activities with immersive Virtual Reality. Overall, the technological features for physical presence were more readily implemented and investigated than pedagogical features for learning engagement. Further analysis with k-means clustering revealed five approaches with varying levels of interactivity and openness in learning tasks, from watching virtual worlds passively to responding to personalised prompts. Such differences in the design appeared to stem from different practical and educational priorities, such as accessibility, interactivity, and engagement. This review highlights the diversity in the learning task designs in immersive Virtual Reality and illustrates how researchers are navigating practical and educational concerns. We recommend future empirical studies recognise the different approaches and priorities when designing and evaluating learning with immersive Virtual Reality. We also recommend that future systematic reviews investigate immersive Virtual Reality-based learning not only by learning topics or learner demographics, but also by task designs and learning experiences

Augmented Reality and Health Informatics: A Study based on Bibliometric and Content Analysis of Scholarly Communication and Social Media

Healthcare outcomes have been shown to improve when technology is used as part of patient care. Health Informatics (HI) is a multidisciplinary study of the design, development, adoption, and application of IT-based innovations in healthcare services delivery, management, and planning. Augmented Reality (AR) is an emerging technology that enhances the user’s perception and interaction with the real world. This study aims to illuminate the intersection of the field of AR and HI. The domains of AR and HI by themselves are areas of significant research. However, there is a scarcity of research on augmented reality as it applies to health informatics. Given both scholarly research and social media communication having contributed to the domains of AR and HI, research methodologies of bibliometric and content analysis on scholarly research and social media communication were employed to investigate the salient features and research fronts of the field. The study used Scopus data (7360 scholarly publications) to identify the bibliometric features and to perform content analysis of the identified research. The Altmetric database (an aggregator of data sources) was used to determine the social media communication for this field. The findings from this study included Publication Volumes, Top Authors, Affiliations, Subject Areas and Geographical Locations from scholarly publications as well as from a social media perspective. The highest cited 200 documents were used to determine the research fronts in scholarly publications. Content Analysis techniques were employed on the publication abstracts as a secondary technique to determine the research themes of the field. The study found the research frontiers in the scholarly communication included emerging AR technologies such as tracking and computer vision along with Surgical and Learning applications. There was a commonality between social media and scholarly communication themes from an applications perspective. In addition, social media themes included applications of AR in Healthcare Delivery, Clinical Studies and Mental Disorders. Europe as a geographic region dominates the research field with 50% of the articles and North America and Asia tie for second with 20% each. Publication volumes show a steep upward slope indicating continued research. Social Media communication is still in its infancy in terms of data extraction, however aggregators like Altmetric are helping to enhance the outcomes. The findings from the study revealed that the frontier research in AR has made an impact in the surgical and learning applications of HI and has the potential for other applications as new technologies are adopted