Natural freehand grasping of virtual objects for augmented reality

Grasping is a primary form of interaction with the surrounding world, and is an intuitive interaction technique by nature due to the highly complex structure of the human hand. Translating this versatile interaction technique to Augmented Reality (AR) can provide interaction designers with more opportunities to implement more intuitive and realistic AR applications. The work presented in this thesis uses quantifiable measures to evaluate the accuracy and usability of natural grasping of virtual objects in AR environments, and presents methods for improving this natural form of interaction. Following a review of physical grasping parameters and current methods of mediating grasping interactions in AR, a comprehensive analysis of natural freehand grasping of virtual objects in AR is presented to assess the accuracy, usability and transferability of this natural form of grasping to AR environments. The analysis is presented in four independent user studies (120 participants, 30 participants for each study and 5760 grasping tasks in total), where natural freehand grasping performance is assessed for a range of virtual object sizes, positions and types in terms of accuracy of grasping, task completion time and overall system usability. Findings from the first user study in this work highlighted two key problems for natural grasping in AR; namely inaccurate depth estimation and inaccurate size estimation of virtual objects. Following the quantification of these errors, three different methods for mitigating user errors and assisting users during natural grasping were presented and analysed; namely dual view visual feedback, drop shadows and additional visual feedback when adding user based tolerances during interaction tasks. Dual view visual feedback was found to significantly improve user depth estimation, however this method also significantly increased task completion time. Drop shadows provided an alternative, and a more usable solution, to dual view visual feedback through significantly improving depth estimation, task completion time and the overall usability of natural grasping. User based tolerances negated the fundamental problem of inaccurate size estimation of virtual objects, through enabling users to perform natural grasping without the need of being highly accurate in their grasping performance, thus providing evidence that natural grasping can be usable in task based AR environments. Finally recommendations for allowing and further improving natural grasping interaction in AR environments are provided, along with guidelines for translating this form of natural grasping to other AR environments and user interfaces

Methods and Tools for Objective Assessment of Psychomotor Skills in Laparoscopic Surgery

Training and assessment paradigms for laparoscopic surgical skills are evolving from traditional mentor–trainee tutorship towards structured, more objective and safer programs. Accreditation of surgeons requires reaching a consensus on metrics and tasks used to assess surgeons’ psychomotor skills. Ongoing development of tracking systems and software solutions has allowed for the expansion of novel training and assessment means in laparoscopy. The current challenge is to adapt and include these systems within training programs, and to exploit their possibilities for evaluation purposes. This paper describes the state of the art in research on measuring and assessing psychomotor laparoscopic skills. It gives an overview on tracking systems as well as on metrics and advanced statistical and machine learning techniques employed for evaluation purposes. The later ones have a potential to be used as an aid in deciding on the surgical competence level, which is an important aspect when accreditation of the surgeons in particular, and patient safety in general, are considered. The prospective of these methods and tools make them complementary means for surgical assessment of motor skills, especially in the early stages of training. Successful examples such as the Fundamentals of Laparoscopic Surgery should help drive a paradigm change to structured curricula based on objective parameters. These may improve the accreditation of new surgeons, as well as optimize their already overloaded training schedules

Head Mounted Display Interaction Evaluation: Manipulating Virtual Objects in Augmented Reality

Augmented Reality (AR) is getting close to real use cases,which is driving the creation of innovative applications and the unprecedented growth of Head-Mounted Display (HMD) devices in consumer availability. However, at present there is a lack of guidelines, common form factors and standard interaction paradigms between devices, which has resulted in each HMD manufacturer creating their own specifications. This paper presents the first experimental evaluation of two AR HMDs evaluating their interaction paradigms, namely we used the HoloLens v1 (metaphoric interaction) and Meta2 (isomorphic interaction). We report on precision, interactivity and usability metrics in an object manipulation task-based user study. 20 participants took part in this study and significant differences were found between interaction paradigms of the devices for move tasks, where the isomorphic mapped interaction outperformed the metaphoric mapped interaction in both time to completion and accuracy, while the contrary was found for the resize task. From an interaction perspective, the isomorphic mapped interaction (using the Meta2) was perceived as more natural and usable with a significantly higher usability score and a significantly lower task-load index. However, when task accuracy and time to completion is key mixed interaction paradigms need to be considered

All Hands on Deck: Choosing Virtual End Effector Representations to Improve Near Field Object Manipulation Interactions in Extended Reality

Extended reality, or XR , is the adopted umbrella term that is heavily gaining traction to collectively describe Virtual reality (VR), Augmented reality (AR), and Mixed reality (MR) technologies. Together, these technologies extend the reality that we experience either by creating a fully immersive experience like in VR or by blending in the virtual and real worlds like in AR and MR. The sustained success of XR in the workplace largely hinges on its ability to facilitate efficient user interactions. Similar to interacting with objects in the real world, users in XR typically interact with virtual integrants like objects, menus, windows, and information that convolve together to form the overall experience. Most of these interactions involve near-field object manipulation for which users are generally provisioned with visual representations of themselves also called self-avatars. Representations that involve only the distal entity are called end-effector representations and they shape how users perceive XR experiences. Through a series of investigations, this dissertation evaluates the effects of virtual end effector representations on near-field object retrieval interactions in XR settings. Through studies conducted in virtual, augmented, and mixed reality, implications about the virtual representation of end-effectors are discussed, and inferences are made for the future of near-field interaction in XR to draw upon from. This body of research aids technologists and designers by providing them with details that help in appropriately tailoring the right end effector representation to improve near-field interactions, thereby collectively establishing knowledge that epitomizes the future of interactions in XR

A review and consideration on the kinematics of reach-to-grasp movements in macaque monkeys

The bases for understanding the neuronal mechanisms that underlie the control of reach-to-grasp movements among nonhuman primates, particularly macaques, has been widely studied. However, only a few kinematic descriptions of their prehensile actions are available. A thorough understanding of macaques' prehensile movements is manifestly critical, in light of their role in biomedical research as valuable models for studying neuromotor disorders and brain mechanisms, as well as for developing brain-machine interfaces to facilitate arm control. This article aims to review the current state of knowledge on the kinematics of grasping movements that macaques perform in naturalistic, semi-naturalistic, and laboratory settings, to answer the following questions: Are kinematic signatures affected by the context within which the movement is performed? In what ways is kinematics of humans' and macaques' prehensile actions similar/dissimilar? Our analysis reflects the challenges involved in making comparisons across settings and species due to the heterogeneous picture in terms of the number of subjects, stimuli, conditions, and hands used. The kinematics of free-ranging macaques are characterized by distinctive features that are exhibited neither by macaques in laboratory setting nor human subjects. The temporal incidence of key kinematic landmarks diverges significantly between species, indicating disparities in the overall organization of movement. Given such complexities, we attempt a synthesis of extant body of evidence, intending to generate some significant implications for directions that future research might take, to recognize the remaining gaps and pursue the insights and resolutions to generate an interpretation of movement kinematics that accounts for all settings and subjects

A Comparison of Myoelectric Control Modes for an Assistive Robotic Virtual Platform

In this paper, we propose a daily living situation where objects in a kitchen can be grasped and stored in specific containers using a virtual robot arm operated by different myoelectric control modes. The main goal of this study is to prove the feasibility of providing virtual environments controlled through surface electromyography that can be used for the future training of people using prosthetics or with upper limb motor impairments. We propose that simple control algorithms can be a more natural and robust way to interact with prostheses and assistive robotics in general than complex multipurpose machine learning approaches. Additionally, we discuss the advantages and disadvantages of adding intelligence to the setup to automatically assist grasping activities. The results show very good performance across all participants who share similar opinions regarding the execution of each of the proposed control modes

Development and Usability Evaluation of Low-cost Virtual Reality Rehabilitation Games for Patients with Upper Limb Impairment

Stroke is one of the primary causes of long-term disability in adults in the United States which leads to mild to severe sensorimotor impairments. Long-term continuous rehabilitation therapies are needed to facilitate sensorimotor recovery and empower patients in performing daily living activities. Currently, the opportunity of receiving post stroke rehabilitation in the chronic stage (\u3e 6 months post stroke) is limited due to a lack of insurance and the high cost of therapy. Low-cost virtual rehabilitation games with motion tracking devices have tremendous potential to assist physical rehabilitation. Motion tracking devices such as Kinect (Microsoft, Redmond, CA; $100) and P5 Glove (Essential Reality, LLC, NY;$40) have become available to enable development of low-cost virtual rehabilitation games. Such low-cost games may encourage continuous, repeated, and intensive rehabilitation therapies thereby enhancing recovery post stroke. However, current virtual rehabilitation games emphasize on gross arm movements using Kinect or fine finger movements using P5 Glove, but not both at the same time. Since most daily living activities require coordination of the gross shoulder/elbow movement and fine finger movement such as reaching to grasp and transferring a jar to a shelf, effective upper limb rehabilitation must involve coordination of the arm and finger movements. In addition, many virtual rehabilitation games have been developed without user input and feedback, which may be the primary reason why virtual rehabilitation games are not prominently used at home by patients. This thesis presents the development and usability evaluation of low-cost virtual rehabilitation games. In addition to the archery and puzzle games previously developed in the laboratory, a low-cost rehabilitation kitchen game was developed to encourage patients to practice various functional tasks involving coordinated arm and finger movements that were detected by using Kinect and P5 Glove, respectively. Usability of the three games was assessed with ten chronic stroke survivors using pre-game and post-game surveys. The games met patients\u27 expectations of providing challenging movements. The House of Quality analysis revealed that technical characteristic needing the most improvement was device reliability. The future research should address device reliability by developing a better instruction manual to facilitate device set-up and use. In addition, filtering data can also improve quality of virtual arm movements in future versions of the games. In summary, this thesis presents promising evidence for low-cost rehabilitation games using commercially available motion tracking devices of Kinect and P5 Glove together with free Blender software