Trends in virtual reality technologies for the learning patient

NextMed convened the Medicine Meets Virtual Reality 22 (MMVR 22) conference in 2016. Since 1992, the conference has brought together a diverse group of researchers to share creative solutions for the evolving challenge of integrating virtual reality tools into medical education. Virtual reality (VR) and its enabling technologies utilize hardware and software to simulate environments and encounters where users can interact and learn. The MMVR 22 symposium proceedings contain projects that support a variety of learners: medical students, practitioners, soldiers, and patients. This report will contemplate the trends in virtual reality technologies for patients navigating their medical and healthcare learning. The learning patient seeks more than intervention; they seek prevention. From virtual humans and environments to motion sensors and haptic devices, patients are surrounded by increasingly rich and transformative data-driven tools. Applied data enables VR applications to simulate experience, predict health outcomes, and motivate new behavior. The MMVR 22 presents investigations into the usability of wearable devices, the efficacy of avatar inclusion, and the viability of multi-player gaming. With increasing need for individualized and scalable programming, only committed open source efforts will align instructional designers, technology integrators, trainers, and clinicians. Curriculum and InstructionCurriculum and Instructio

Augmented reality (AR) for surgical robotic and autonomous systems: State of the art, challenges, and solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future

Developing the Next Generation of Augmented Reality Games for Pediatric Healthcare: An Open-Source Collaborative Framework Based on ARCore for Implementing Teaching, Training and Monitoring Applications

[Abstract] Augmented Reality (AR) provides an alternative to the traditional forms of interaction between humans and machines, and facilitates the access to certain technologies to groups of people with special needs like children. For instance, in pediatric healthcare, it is important to help children to feel comfortable during medical procedures and tests that may be performed on them. To tackle such an issue with the help of AR-based solutions, this article presents the design, implementation and evaluation of a novel open-source collaborative framework that enables to develop teaching, training, and monitoring pediatric healthcare applications. Specifically, such a framework allows for building collaborative applications and shared experiences for AR devices, providing functionalities for connecting with other AR devices and enabling real-time visualization and simultaneous interaction with virtual objects. Since all the communications involved in AR interactions are handled by AR devices, the proposed collaborative framework is able to operate autonomously through a Local Area Network (LAN), thus requiring no cloud or external servers. In order to demonstrate the potential of the proposed framework, a practical use case application is presented. Such an application has been designed to motivate pediatric patients and to encourage them to increase their physical activity through AR games. The presented games do not require any previous configuration, as they use ARCore automatic surface detection technology. Moreover, the AR mobile gaming framework allows multiple players to engage in the same AR experience, so children can interact and collaborate among them sharing the same AR content. In addition, the proposed AR system provides a remote web application that is able to collect and to visualize data on patient use, aiming to provide healthcare professionals with qualified data about the mobility and mood of their patients through an intuitive and user-friendly web tool. Finally, to determine the performance of the proposed AR system, this article presents its evaluation in terms of latency and processing time. The results show that both times are low enough to provide a good user experience.This work has been funded by the Xunta de Galicia (by grant ED431C 2020/15, and grant ED431G 2019/01 to support the Centro de Investigación de Galicia “CITIC”), the Agencia Estatal de Investigación of Spain (by grants RED2018-102668-T and PID2019-104958RB-C42) and ERDF funds of the EU (FEDER Galicia 2014-2020 & AEI/FEDER Programs, UE)Xunta de Galicia; ED431C 2020/15Xunta de Galicia; ED431G 2019/0

VR welding kit: welding training simulation in mobile virtual reality using multiple marker tracking method

Welding simulation design using virtual reality (VR) is a challenge, as numerous developments and research in the mechanical engineering fields are involved. One of the key challenges is the improvement of realism by considering a mixed system of real and virtual equipment. A conceptual design and research management framework is currently lacking which leveraging the combination of VR and marker tracking techniques. This study seeks to examine and evaluating the use of mobile VR in welding training and how multiple markers tracking methods can be incorporated to overcome the current problems in VR for welding training simulation. In this study, the VR Welding Kit application is created by utilizing the Vuforia tracking engine to provide an alternative interaction for mobile devices. The results of the experiment revealed a benchmark comparison with Oculus Quest, the high-end VR system, to investigate the efficiency of the proposed multiple marker interaction technique. Performance for both devices was recorded. The System Usability Scales (SUS) have also been used to obtain users' acceptance rates using these devices. The Simulator Sickness Questionnaire (SSQ) was used to assess the cybersickness of participants. The performance results show that mobile VR have a moderate gap completion time in seconds if compared to Oculus Quest. The SUS scored a satisfactory result which is 73.33. Besides, SSQ surveys result shows that most of the participant felt the simulation sickness was minimal

Computer Mediated Reality Technologies: A Conceptual Framework and Survey of the State of the Art in Healthcare Intervention

The trend of an ageing and growing world population, particularly in developed countries, is expected to continue for decades to come causing an increase in demand for healthcare resources and services. Consequently, demand is growing faster than rises in funding. The UK government, in partnership with the European Commission’s Vision for 2020, propose a paradigm shift towards the delivery of more patient-centred self-care interventions, facilitated by novel ubiquitous computer mediated reality technology applications, as a key strategy to overcome the scarcity of health resources gap. If this vision is to become a reality, it is crucial that state of the art research focuses efforts on the development of applications that support the delivery of patient-centred self-care interventions

Evaluation of learning outcomes using an educational iPhone game vs. traditional game

In this paper, we present an initial study to determine the subject preferences for educational computer games for children, in which 150 education professionals participated. From the results of this first study, we have developed an iPhone game for transmitting knowledge as part of multiculturalism, solidarity and tolerance following established learning theories, several design principles, and the objectives and competences of the Spanish law for primary education. We also report on a second study to determine whether the iPhone game has better learning outcomes than a traditional game by analyzing the participation of 84 children ranging in age from 8 to 10 years old. The frequency of playing with consoles or computer games was also taken into account in this second study, and the worldwide trend of previous studies has been corroborated. For learning outcomes, the results did not show significant differences between the two groups. However, 96% of the children indicated that they would like to play with the iPhone game again, and 90% indicated that they preferred the experience with the iPhone game over the traditional one. From these results, we can conclude that the children achieved similar knowledge improvements using both the autonomous game (iPhone game) and the custom, guided game (traditional game). This could facilitate versatility in the learning process since the learning activity could be performed at any place and time without requiring supervision. Therefore, it could be a useful tool in the learning process and help teachers to fulfill students' training needs. 2013 Elsevier Ltd. All rights reserved.This work was funded by the Spanish APRENDRA project (TIN2009-14319-C02).Furió Ferri, D.; González Gancedo, S.; Juan, M.; Seguí, I.; Rando, N. (2013). Evaluation of learning outcomes using an educational iPhone game vs. traditional game. Computers and Education. 64:1-23. https://doi.org/10.1016/j.compedu.2012.12.001S1236

Augmented Reality

Augmented Reality (AR) is a natural development from virtual reality (VR), which was developed several decades earlier. AR complements VR in many ways. Due to the advantages of the user being able to see both the real and virtual objects simultaneously, AR is far more intuitive, but it's not completely detached from human factors and other restrictions. AR doesn't consume as much time and effort in the applications because it's not required to construct the entire virtual scene and the environment. In this book, several new and emerging application areas of AR are presented and divided into three sections. The first section contains applications in outdoor and mobile AR, such as construction, restoration, security and surveillance. The second section deals with AR in medical, biological, and human bodies. The third and final section contains a number of new and useful applications in daily living and learning

Augmented reality for computer assisted orthopaedic surgery

In recent years, computer-assistance and robotics have established their presence in operating theatres and found success in orthopaedic procedures. Benefits of computer assisted orthopaedic surgery (CAOS) have been thoroughly explored in research, finding improvements in clinical outcomes, through increased control and precision over surgical actions. However, human-computer interaction in CAOS remains an evolving field, through emerging display technologies including augmented reality (AR) – a fused view of the real environment with virtual, computer-generated holograms. Interactions between clinicians and patient-specific data generated during CAOS are limited to basic 2D interactions on touchscreen monitors, potentially creating clutter and cognitive challenges in surgery. Work described in this thesis sought to explore the benefits of AR in CAOS through: an integration between commercially available AR and CAOS systems, creating a novel AR-centric surgical workflow to support various tasks of computer-assisted knee arthroplasty, and three pre–clinical studies exploring the impact of the new AR workflow on both existing and newly proposed quantitative and qualitative performance metrics. Early research focused on cloning the (2D) user-interface of an existing CAOS system onto a virtual AR screen and investigating any resulting impacts on usability and performance. An infrared-based registration system is also presented, describing a protocol for calibrating commercial AR headsets with optical trackers, calculating a spatial transformation between surgical and holographic coordinate frames. The main contribution of this thesis is a novel AR workflow designed to support computer-assisted patellofemoral arthroplasty. The reported workflow provided 3D in-situ holographic guidance for CAOS tasks including patient registration, pre-operative planning, and assisted-cutting. Pre-clinical experimental validation on a commercial system (NAVIO®, Smith & Nephew) for these contributions demonstrates encouraging early-stage results showing successful deployment of AR to CAOS systems, and promising indications that AR can enhance the clinician’s interactions in the future. The thesis concludes with a summary of achievements, corresponding limitations and future research opportunities.Open Acces

Mobile depth sensing technology and algorithms with application to occupational therapy healthcare

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonThe UK government is striving to shift its current healthcare delivery model from clini-cian–oriented services, to that of patient and self–care–oriented intervention strategies. It seeks to do so through Information Communication (ICT) and Computer Mediated Re-ality Technologies (CMRT) as a key strategy to overcome the ever–increasing scarcity of healthcare resources and costs. To this end, in the UK the use of paper–based information systems have exhibited their limitations in providing apposite care. At the national level, The Royal College of Occupational Therapists (RCOT) identify home visits and modifica-tions as key levers in a multifactorial health programme to evaluate interventions for older people with a history of falling or are identified as being prone to falling. Prescribing Assistive Equipment (AE) is one such mechanism that seeks to reduce the risk of falling whilst promoting the continued independence of physical dexterity and mobility in older adults at home. In the UK, the yearly cost of falls is estimated at £2.3 billion. Further evidence places a 30% to 60% abandonment rate on prescribed AE by and large due to a ‘poor fit’ and measurement inaccuracies. To remain aligned with the national strategy, and assist in the eradication of measurement inaccuracies, this thesis employs Mobile Depth Sensing and Motion Track-ing Devices (MDSMTDs) to assist OTs in in the process of digitally measuring the extrin-sic fall–risk factors for the provision of AE. The quintessential component in this assess-ment lies in the measurement of fittings and furniture items in the home. To digitise and aid in this process, the artefact presented in this thesis employs stereo computer–vision and camera calibration algorithms to extract edges in 3D space. It modifies the Sobel–Feldman convolution filter by reducing the magnitude response and employs the camera intrinsic parameters as a mechanism to calculate the distortion matrix for interpolation between the edges and the 3D point cloud. Further Augmented Reality User Experience (AR-UX) facets are provided to digitise current state of the art clinical guidance and over-lay its instructions onto the real world (i.e., 3D space). Empirical mixed methods assessment revealed that in terms of accuracy, the arte-fact exhibited enhanced performance gains over current paper–based guidance. In terms of accuracy consistency, the artefact can rectify measurement consistency inaccuracies, but there are still a wide range of factors that can influence the integrity of the point-cloud in respect of the device’s point-of-view, holding positions and measurement speed. To this end, OTs usability, and adoption preferences materialise in favour of the artefact. In conclusion, this thesis demonstrates that MDSMTDs are a promising alterna-tive to existing paper–based measurement practices as OTs appear to prefer the digital–based system and that they can take measurements more efficiently and accurately