Finger tracking and hand recognition technologies in virtual reality maritime safety training applications

The competitiveness and development of the maritime sector together with the continuous effort on increasing operations performance while reducing operations costs, drives the needs for on-board effective and qualitative training safety related issues. Virtual reality (VR) has been considered by classification societies and training organizations as a technology that can significantly improve seafarer's performance and competence with the adaptation of maritime applications developed for design simulation and gaming. This paper presents the evolution of the MarSEVR (Maritime Safety Education with VR) technology as a new concept and technology by integrating finger tracking and hand recognition technologies that increase immersiveness and user engagement within the MarISOT technology, a Green Ocean innovation composed of VR safety applications. The paper approaches this integration by addressing game design, pedagogic and cognitive neuroscience principles and challenges on the use of hand recognition and finger tracking in the MarSEVR learning episodes

Feasibility and effect of low-cost haptics on user immersion in virtual environments

Since the later 1990s research into Immersion, Presence and Interactivity in the context of digital media has been steadily evolving into an exciting area of experimentation, fuelled by advances in the visual, audio and tracking capabilities of Virtual Reality (VR) equipment, thanks to these improvements studies into the effectiveness of this equipment in producing an immersive experience are now possible. This is most commonly achieved by measuring the perceived level of Presence experienced by participants in virtual environments, with the higher the sense of Presence created, the more effective a VR system is deemed to be. However, due to the current limitations of Haptic interaction methods investigation into the role that touch plays in generating this sense of Presence is somewhat restricted. Following a structured process of design and research work, this project presents a new approach to creating Haptic Interaction by deploying a Haptic Prototyping Toolkit that enables Passive Haptic Interactions in Virtual Environments. The findings of this work provide the foundations for future research into the development of interaction methods of this type

The Perception/Action loop: A Study on the Bandwidth of Human Perception and on Natural Human Computer Interaction for Immersive Virtual Reality Applications

Virtual Reality (VR) is an innovating technology which, in the last decade, has had a widespread success, mainly thanks to the release of low cost devices, which have contributed to the diversification of its domains of application. In particular, the current work mainly focuses on the general mechanisms underling perception/action loop in VR, in order to improve the design and implementation of applications for training and simulation in immersive VR, especially in the context of Industry 4.0 and the medical field. On the one hand, we want to understand how humans gather and process all the information presented in a virtual environment, through the evaluation of the visual system bandwidth. On the other hand, since interface has to be a sort of transparent layer allowing trainees to accomplish a task without directing any cognitive effort on the interaction itself, we compare two state of the art solutions for selection and manipulation tasks, a touchful one, the HTC Vive controllers, and a touchless vision-based one, the Leap Motion. To this aim we have developed ad hoc frameworks and methodologies. The software frameworks consist in the creation of VR scenarios, where the experimenter can choose the modality of interaction and the headset to be used and set experimental parameters, guaranteeing experiments repeatability and controlled conditions. The methodology includes the evaluation of performance, user experience and preferences, considering both quantitative and qualitative metrics derived from the collection and the analysis of heterogeneous data, as physiological and inertial sensors measurements, timing and self-assessment questionnaires. In general, VR has been found to be a powerful tool able to simulate specific situations in a realistic and involving way, eliciting user\u2019s sense of presence, without causing severe cybersickness, at least when interaction is limited to the peripersonal and near-action space. Moreover, when designing a VR application, it is possible to manipulate its features in order to trigger or avoid triggering specific emotions and voluntarily create potentially stressful or relaxing situations. Considering the ability of trainees to perceive and process information presented in an immersive virtual environment, results show that, when people are given enough time to build a gist of the scene, they are able to recognize a change with 0.75 accuracy when up to 8 elements are in the scene. For interaction, instead, when selection and manipulation tasks do not require fine movements, controllers and Leap Motion ensure comparable performance; whereas, when tasks are complex, the first solution turns out to be more stable and efficient, also because visual and audio feedback, provided as a substitute of the haptic one, does not substantially contribute to improve performance in the touchless case

A Utility Framework for Selecting Immersive Interactive Capability and Technology for Virtual Laboratories

There has been an increase in the use of virtual reality (VR) technology in the education community since VR is emerging as a potent educational tool that offers students with a rich source of educational material and makes learning exciting and interactive. With a rise of popularity and market expansion in VR technology in the past few years, a variety of consumer VR electronics have boosted educators and researchers’ interest in using these devices for practicing engineering and science laboratory experiments. However, little is known about how such devices may be well-suited for active learning in a laboratory environment. This research aims to address this gap by formulating a utility framework to help educators and decision-makers efficiently select a type of VR device that matches with their design and capability requirements for their virtual laboratory blueprint. Furthermore, a framework use case is demonstrated by not only surveying five types of VR devices ranging from low-immersive to full-immersive along with their capabilities (i.e., hardware specifications, cost, and availability) but also considering the interaction techniques in each VR device based on the desired laboratory task. To validate the framework, a research study is carried out to compare these five VR devices and investigate which device can provide an overall best-fit for a 3D virtual laboratory content that we implemented based on the interaction level, usability and performance effectiveness

Measuring user experience for virtual reality

In recent years, Virtual Reality (VR) and 3D User Interfaces (3DUI) have seen a drastic increase in popularity, especially in terms of consumer-ready hardware and software. These technologies have the potential to create new experiences that combine the advantages of reality and virtuality. While the technology for input as well as output devices is market ready, only a few solutions for everyday VR - online shopping, games, or movies - exist, and empirical knowledge about performance and user preferences is lacking. All this makes the development and design of human-centered user interfaces for VR a great challenge. This thesis investigates the evaluation and design of interactive VR experiences. We introduce the Virtual Reality User Experience (VRUX) model based on VR-specific external factors and evaluation metrics such as task performance and user preference. Based on our novel UX evaluation approach, we contribute by exploring the following directions: shopping in virtual environments, as well as text entry and menu control in the context of everyday VR. Along with this, we summarize our findings by design spaces and guidelines for choosing optimal interfaces and controls in VR.In den letzten Jahren haben Virtual Reality (VR) und 3D User Interfaces (3DUI) stark an Popularität gewonnen, insbesondere bei Hard- und Software im Konsumerbereich. Diese Technologien haben das Potenzial, neue Erfahrungen zu schaffen, die die Vorteile von Realität und Virtualität kombinieren. Während die Technologie sowohl für Eingabe- als auch für Ausgabegeräte marktreif ist, existieren nur wenige Lösungen für den Alltag in VR - wie Online-Shopping, Spiele oder Filme - und es fehlt an empirischem Wissen über Leistung und Benutzerpräferenzen. Dies macht die Entwicklung und Gestaltung von benutzerzentrierten Benutzeroberflächen für VR zu einer großen Herausforderung. Diese Arbeit beschäftigt sich mit der Evaluation und Gestaltung von interaktiven VR-Erfahrungen. Es wird das Virtual Reality User Experience (VRUX)- Modell eingeführt, das auf VR-spezifischen externen Faktoren und Bewertungskennzahlen wie Leistung und Benutzerpräferenz basiert. Basierend auf unserem neuartigen UX-Evaluierungsansatz leisten wir einen Beitrag, indem wir folgende interaktive Anwendungsbereiche untersuchen: Einkaufen in virtuellen Umgebungen sowie Texteingabe und Menüsteuerung im Kontext des täglichen VR. Die Ergebnisse werden außerdem mittels Richtlinien zur Auswahl optimaler Schnittstellen in VR zusammengefasst

Animal-assisted therapy for Alzheimer patients using virtual reality

A doença de Alzheimer é a principal causa de demência. Trata-se de uma doença neurodegenerativa incurável que conduz a falhas cognitivas e problemas comportamentais. A demência constitui um enorme fardo social e económico para os doentes e para os cuidadores. A medicação ajuda a prevenir alguns sintomas da doença, mas não cura podendo revelar efeitos secundários indesejáveis. Algumas terapias alternativas têm vindo a ser exploradas, por exemplo a terapia animal (AAT). Para evitar as desvantagens da terapia animal, como a necessidade de treino, foi desenvolvida uma solução com realidade virtual. Realizamos um estudo com 12 pacientes diagnosticados com demência para perceber que tecnologias e modos de interação são mais adequados para esta população. Os resultados indicam que tecnologias com interação direta são mais adequadas (por exemplo Tablet ou AR). Foi desenvolvido um jogo de interação com animal para tablets. Realizou-se um estudo com 10 pacientes diagnosticados com demência num centro de dia. Este estudo tinha o objetivo de examinar a aplicabilidade desta ferramenta nesta população e perceber se uma sessão de terapia animal virtual poderia ter efeitos na disposição do participante, através da avaliação das suas respostas emocionais. Os participantes responderam a questionários pré- e pós-intervenção para obtenção de informação relativa ao seu estado emocional e à sua perceção da sessão de jogo. Foi utilizado um questionário destinado ao terapeuta que registou a sua opinião relativa aos efeitos no participante. Os resultados indicam melhorias na disposição dos participantes, uma boa aceitação e usabilidade desta abordagem e respostas emocionais positivas.The Alzheimer’s Disease is the main cause of dementia. It is incurable and regarded as a neurodegenerative disease that leads to cognitive and behavioral impairment. Dementia brings along a heavy burden from both a social and economic perspective. Traditional medication helps in slowing down the disease but is not able to cure it, and it may also bring undesirable side-effects. Alternative therapies are being further explored, being one of them Animal-Assisted Therapy (AAT). To avoid the disadvantages of conventional AAT, such as the need for trained animals, or hygiene issues, a Virtual Reality approach was adopted. We conducted a pilot study with 12 dementia patients to understand which technologies and interaction modalities were preferred when developing for this population. Devices that promote direct interaction were preferred (Tablet and AR) so, combining the positive aspects of traditional AAT and the promising aid of VR, an animal interaction tablet game was developed for this population. This study, conducted with 10 dementia patients in a daily-care center, aimed to examine whether a session of virtual AAT was feasible and able to produce any changes in the mood of participants, evaluating the acceptance and emotional responses as well. Participants completed pre- and post-intervention questionnaires to assess their mood and perception of the game session and a therapist was also asked to provide insight on each participant. Therapist’s reports and observations reveal positive mood changes, good acceptance by the patients and positive emotional responses

Um jogo digital em ambientes imersivos no apoio às vítimas do acidente vascular cerebral

A sociedade moderna está a testemunhar um aumento do envelhecimento médio populacional, graças à melhoria da qualidade dos serviços de saúde e de medicação. No entanto, o envelhecimento cria outros problemas como doenças físicas ou mentais com grandes taxas de incidência. O acidente vascular cerebral (AVC) é uma das doenças que afeta sobretudo a população idosa, e o processo de reabilitação é doloroso e difícil de percorrer, sendo que a forma mais eficaz de tratar o doente é na atuação rápida e eficaz da fisioterapia. O consumo de videojogos pela população sénior está a aumentar, sendo que é cada vez mais viável a introdução de novos artefactos digitais no processo de recuperação cerebral e motora pela vítima de AVC. Os programas tradicionais de recuperação para um paciente que tenha sofrido um AVC são organizados em tratamento fisioterapêutico longo e monótono, com a possibilidade de envolver tarefas domésticas desmotivadoras. No entanto existem soluções tecnológicas que monitorizam as tarefas repetitivas de movimento. O aparelho de monitorização aliado a um jogo digital tem a possibilidade de estimular o paciente nas melhorias motoras e cognitivas como uma alternativa ao tratamento fisioterapêutico tradicional. As soluções desenvolvidas até ao momento são escassas, sendo que existe uma grande margem para mudar essa realidade. O principal objetivo desta pesquisa é o de explorar caraterísticas relacionadas com o display, interface gestual, narrativa, género, estilo gráfico, dificuldade, e linguagem que um jogo digital possa ter, para complementar as sessões de fisioterapia na recuperação do AVC pela população sénior, através da criação de um protótipo experimental. Esta investigação empírica tem um carácter exploratório e tem como base a metodologia Development Research (Van den Akker, Branch, Gustafson, Nieveen, & Plomp, 1999). Os resultados indicam que o controlador de movimento – leap motion – é um dispositivo que pode ser adaptado à fisioterapia orientada ao AVC, através de movimentos específicos e contextualizados no ambiente de jogo. Adicionalmente, foi possível observar uma rejeição elevada no uso de Head Mounted Displays devido a dores oculares e perda de orientação.Modern society is witnessing a general population ageing increase in average life expectancy thanks to better health services and medication. However, ageing creates life quality problems, such has several disabilities, diseases, or mental illness with high incidence rates. Stroke patients are a main concern for such ages, and the rehabilitation process is painful and shows very small recovery improvements over time, unless treated in a fast manner. The consumption of videogames by the senior population is increasing, and it is feasible to introduce new digital artefacts for the process of recovering from brain damage and low motricity for the stroke victim. Typical rehabilitation programs for stroke patients are organized in long and monotonous physiotherapy treatment, with the possibility of involving domestic tasks, which can increase the risk of treatment withdrawal derived from low motivation. However, there are some technological solutions that can effectively help in the supervision of those repetitive tasks. A monitoring device connected to a digital game can effectively stimulate a person in cognitive and physical improvements as an alternative to traditional physiotherapy treatment. There is room for improvement in order to change the reality of stroke rehabilitation. The main objective of this research is to explore characteristics related to display, gesture interface device, narrative, genre, game art design, difficulty, and language that can be included in a digital game to complement physiotherapy sessions for stroke rehabilitation, through the creation of a functional prototype. The empirical research has an exploratory character and is based on the methodology “Development Research” (Van den Akker et al., 1999). The results indicate that the motion controller - leap motion - is a device that can be adapted to stroke-oriented physiotherapy through specific movements and contextualized in the game environment. Additionally, it was possible to observe a high rejection in the use of Head Mounted Displays due to ocular pain and orientation loss.Mestrado em Comunicação Multimédi