MetaSpace II: Object and full-body tracking for interaction and navigation in social VR

MetaSpace II (MS2) is a social Virtual Reality (VR) system where multiple users can not only see and hear but also interact with each other, grasp and manipulate objects, walk around in space, and get tactile feedback. MS2 allows walking in physical space by tracking each user's skeleton in real-time and allows users to feel by employing passive haptics i.e., when users touch or manipulate an object in the virtual world, they simultaneously also touch or manipulate a corresponding object in the physical world. To enable these elements in VR, MS2 creates a correspondence in spatial layout and object placement by building the virtual world on top of a 3D scan of the real world. Through the association between the real and virtual world, users are able to walk freely while wearing a head-mounted device, avoid obstacles like walls and furniture, and interact with people and objects. Most current virtual reality (VR) environments are designed for a single user experience where interactions with virtual objects are mediated by hand-held input devices or hand gestures. Additionally, users are only shown a representation of their hands in VR floating in front of the camera as seen from a first person perspective. We believe, representing each user as a full-body avatar that is controlled by natural movements of the person in the real world (see Figure 1d), can greatly enhance believability and a user's sense immersion in VR.Comment: 10 pages, 9 figures. Video: http://living.media.mit.edu/projects/metaspace-ii

Remote Real-Time Collaboration Platform enabled by the Capture, Digitisation and Transfer of Human-Workpiece Interactions

In this highly globalised manufacturing ecosystem, product design and verification activities, production and inspection processes, and technical support services are spread across global supply chains and customer networks. Therefore, a platform for global teams to collaborate with each other in real-time to perform complex tasks is highly desirable. This work investigates the design and development of a remote real-time collaboration platform by using human motion capture technology powered by infrared light based depth imaging sensors borrowed from the gaming industry. The unique functionality of the proposed platform is the sharing of physical contexts during a collaboration session by not only exchanging human actions but also the effects of those actions on the task environment. This enables teams to remotely work on a common task problem at the same time and also get immediate feedback from each other which is vital for collaborative design, inspection and verifications tasks in the factories of the future

Robotic Cameraman for Augmented Reality based Broadcast and Demonstration

In recent years, a number of large enterprises have gradually begun to use vari-ous Augmented Reality technologies to prominently improve the audiences’ view oftheir products. Among them, the creation of an immersive virtual interactive scenethrough the projection has received extensive attention, and this technique refers toprojection SAR, which is short for projection spatial augmented reality. However,as the existing projection-SAR systems have immobility and limited working range,they have a huge difficulty to be accepted and used in human daily life. Therefore,this thesis research has proposed a technically feasible optimization scheme so thatit can be practically applied to AR broadcasting and demonstrations. Based on three main techniques required by state-of-art projection SAR applica-tions, this thesis has created a novel mobile projection SAR cameraman for ARbroadcasting and demonstration. Firstly, by combining the CNN scene parsingmodel and multiple contour extractors, the proposed contour extraction pipelinecan always detect the optimal contour information in non-HD or blurred images.This algorithm reduces the dependency on high quality visual sensors and solves theproblems of low contour extraction accuracy in motion blurred images. Secondly, aplane-based visual mapping algorithm is introduced to solve the difficulties of visualmapping in these low-texture scenarios. Finally, a complete process of designing theprojection SAR cameraman robot is introduced. This part has solved three mainproblems in mobile projection-SAR applications: (i) a new method for marking con-tour on projection model is proposed to replace the model rendering process. Bycombining contour features and geometric features, users can identify objects oncolourless model easily. (ii) a camera initial pose estimation method is developedbased on visual tracking algorithms, which can register the start pose of robot to thewhole scene in Unity3D. (iii) a novel data transmission approach is introduced to establishes a link between external robot and the robot in Unity3D simulation work-space. This makes the robotic cameraman can simulate its trajectory in Unity3D simulation work-space and project correct virtual content. Our proposed mobile projection SAR system has made outstanding contributionsto the academic value and practicality of the existing projection SAR technique. Itfirstly solves the problem of limited working range. When the system is running ina large indoor scene, it can follow the user and project dynamic interactive virtualcontent automatically instead of increasing the number of visual sensors. Then,it creates a more immersive experience for audience since it supports the user hasmore body gestures and richer virtual-real interactive plays. Lastly, a mobile systemdoes not require up-front frameworks and cheaper and has provided the public aninnovative choice for indoor broadcasting and exhibitions

Development of Virtual Reality Games for Motor Rehabilitation

Motor rehabilitation is a long term, labor intensive and patient-specific process that requires one-on-one care from skilled clinicians and physiotherapists. Virtual rehabilitation is an alternative rehabilitation technology that can provide intensive motor training with minimal supervision from physiotherapists. However, virtual rehabilitation exercises lack of realism and less connected with Activities of Daily Livings. In this paper, we present six Virtual Reality games that we developed for 5DT data glove, 1-DOF IntelliStretch robot and Xbox Kinect to improve the accessibility of motor rehabilitation

Real-time simulation of construction workers using combined human body and hand tracking for robotic construction worker system

Construction is an inherently less safe sector than other sectors because it exposes workers to harsh and dangerous working environments. The nature of the construction industry results in a comparatively high incidence of serious injuries and death caused by falls from a height, musculoskeletal disorders and being struck by objects. This paper presents a new concept that can tackle this problem in the future. The central hypothesis of this study is that it is possible to eliminate injuries if we move the human construction worker off-site and remotely link his/her motions to a Robotic Construction Worker (RCW) on-site. As a first steppingstone towards this ultimate goal, two systems essential for the RCW were developed in this study. First, a novel system that combines 3D body and hand position tracking was developed to capture the movements of human construction worker. This combination of tracking enables the capture of changes in the orientations and articulations of the entire human body. Second, a real-time simulation system that connects a human construction worker off-site to a virtual RCW was developed to demonstrate the proposed concept in a variety of construction scenarios. The simulation results demonstrate the future viability of the RCW concept and indicate the promise of this system for eliminating the health and safety risks faced by human construction workers

The Use of lmmersive Virtual Reality in Stroke Rehabilitation by Using Microsoft Kinect™

This report is a preliminary step onto developing a new way of rehabilitation technique for stroke patient using Microsoft Kinect as the input device. Currently, many typical virtual reality based stroke rehab has been practices to treat these patient such as treadmill, pointing device, hands glove, body suit and many more whereby they have been identified as costly, bulky and also not feasible for distance patients to have their regular set of rehab exercises. With such problems in place, it is crucial to have a better solution which could help the medical industry to stay innovative, creative and effective. Moreover, this project aims to produce a low cost home-based physical VR therapy application intended for hand stroke patients. The project will involves several phases which follow the incremental model whereby firstly the construction of the basic game environment followed by the analysis and design of the system or rehab activities requirements, the implementation of the prototype which enhanced with the usability testing result, further testing and finally the increment routines to complete the milestones

Mixed reality simulators

A dissertation submitted to the Faculty of Science, University of the Witwatersrand, Johannesburg, in fulfilment of the requirements for the degree of Master of Science Johannesburg, May 2017.Virtual Reality (VR) is widely used in training simulators of dangerous or expensive vehicles such as aircraft or heavy mining machinery. The vehicles often have very complicated controls that users need to master before attempting to operate a real world version of the machine. VR allows users to safely train in a simulated environment without the risk of injury or damaging expensive equipment in the field. VR however visually cuts off the user from the real environment,whichmayobtainobstructions. Usersareunabletosafelymoveorgesturewhilewearing aVRheadset. Additionallyusersareunabletousestandardinputdevicessuchasmiceandkeyboards. Bymixinginaliveviewofthetherealworld,theusercanstillseeandinteractwiththe physical environment. The contribution of this research is presenting ways of using Mixed RealitytoenhancetheuserexperienceoftraditionalVRbasedsimulators. MixedRealityimproves on traditional VR simulators by allowing the user the safety and freedom of not being cut off from the real world, allowing interaction and the tactile feedback of interacting with complex physical controls, while still allowing simultaneous use of virtual controls and by adding a real world reference point to aid in diminishing simulator sickness caused by visual motionA dissertation submitted to the Faculty of Science, University of the Witwatersrand, Johannesburg, in fulfilment of the requirements for the degree of Master of ScienceGR201

The Smart Stage: Designing 3D interaction metaphors for immersive and ubiquitous music systems

This conceptual paper describes a work in progress in the process of design and implementation of the Smart Stage, an interactive music system prototype for collaborative musical creativity in immersive and ubiquitous environments. This system is intended to have a low entry barrier, thus more forgiving to users with lesser experience or knowledge in music, and it is designed with affordances to support intuitive progress in improvisational performance in a collaborative setting. We present a preliminary technical overview of the system and a first case study of a 3D interaction metaphor for granular synthesis, developed for this environment.Innovation Agency (Agência de Inovação, ADI, Portugal) and Quadro de Referência Estratégico Nacional (QREN, Portugal): VisualYzARt: Visual programming framework for augmented reality and ubiquitous natural user interfaces (QREN-ADI ref: 23201) and COMPETE - Programa Operacional Factores de Competitividade (POFC

PRACTICA. A Virtual Reality Platform for Specialized Training Oriented to Improve the Productivity

With the proliferation of Virtual reality headset that are emerging into a consumer-oriented market for video games, it will open new possibilities for exploiting the virtual reality (VR). Therefore, the PRACTICA project is defined as a new service aimed to offering a system for creating courses based on a VR simulator for specialized training companies that allows offering to the students an experience close to reality. The general problem of creating these virtual courses derives from the need to have programmers that can generate them. Therefore, the PRACTICA project allows the creation of courses without the need to program source code. In addition, elements of virtual interaction have been incorporated that cannot be used in a real environment due to risks for the staff, such as the introduction of fictional characters or obstacles that interact with the environment. So to do this, artificial intelligence techniques have been incorporated so these elements can interact with the user, as it may be, the movement of these fictional characters on stage with a certain behavior. This feature offers the opportunity to create situations and scenarios that are even more complex and realistic.This project aims to create a service to bring virtual reality technologies closer and artificial intelligence for non-technological companies, so that they can generate (or acquire) their own content and give it the desired shape for their purposes

Collaborative visualization and virtual reality in construction projects

In the Colombian construction industry it is recognized as a general practice that di!erent designers deliver 2D drawings to the project construction team -- Some 3D modeling applications are used but only with commercial intentions, thus wasting visualization tools that facilitate the understanding of the project, that allow the coordination of plans between di!erent specialists, and that can prevent errors with high impact on costs in the construction phase of the project -- As a continuation of the project "immersive virtual reality for construction" developed by EAFIT University, the present work intends to demonstrate how a collaborative virtual environment can be helpful in order to improve visualization of construction projects and achieve the interaction of di!erent specialties, evaluating the impact of collaborative work in the design process of the same -- The end result of this research is an application created using freely available tools and a use case scenario on how this application can be used to perform review meetings by di!erent specialist in real time -- Initial test on the system has been made with civil engineering students showing that this virtual reality tool ease the burden of performing reviews where traditionally plans and sharing the same geographical space were neede