Using the Microsoft Kinect to assess human bimanual coordination

Optical marker-based systems are the gold-standard for capturing three-dimensional (3D) human kinematics. However, these systems have various drawbacks including time consuming marker placement, soft tissue movement artifact, and are prohibitively expensive and non-portable. The Microsoft Kinect is an inexpensive, portable, depth camera that can be used to capture 3D human movement kinematics. Numerous investigations have assessed the Kinect\u27s ability to capture postural control and gait, but to date, no study has evaluated it\u27s capabilities for measuring spatiotemporal coordination. In order to investigate human coordination and coordination stability with the Kinect, a well-studied bimanual coordination paradigm (Kelso, 1984, Kelso; Scholz, & Schöner, 1986) was adapted. ^ Nineteen participants performed ten trials of coordinated hand movements in either in-phase or anti-phase patterns of coordination to the beat of a metronome which was incrementally sped up and slowed down. Continuous relative phase (CRP) and the standard deviation of CRP were used to assess coordination and coordination stability, respectively.^ Data from the Kinect were compared to a Vicon motion capture system using a mixed-model, repeated measures analysis of variance and intraclass correlation coefficients (2,1) (ICC(2,1)).^ Kinect significantly underestimated CRP for the the anti-phase coordination pattern (p \u3c.0001) and overestimated the in-phase pattern (p\u3c.0001). However, a high ICC value (r=.097) was found between the systems. For the standard deviation of CRP, the Kinect exhibited significantly higher variability than the Vicon (p \u3c .0001) but was able to distinguish significant differences between patterns of coordination with anti-phase variability being higher than in-phase (p \u3c .0001). Additionally, the Kinect was unable to accurately capture the structure of coordination stability for the anti-phase pattern. Finally, agreement was found between systems using the ICC (r=.37).^ In conclusion, the Kinect was unable to accurately capture mean CRP. However, the high ICC between the two systems is promising and the Kinect was able to distinguish between the coordination stability of in-phase and anti-phase coordination. However, the structure of variability as movement speed increased was dissimilar to the Vicon, particularly for the anti-phase pattern. Some aspects of coordination are nicely captured by the Kinect while others are not. Detecting differences between bimanual coordination patterns and the stability of those patterns can be achieved using the Kinect. However, researchers interested in the structure of coordination stability should exercise caution since poor agreement was found between systems

3-D Hand Pose Estimation from Kinect's Point Cloud Using Appearance Matching

We present a novel appearance-based approach for pose estimation of a human hand using the point clouds provided by the low-cost Microsoft Kinect sensor. Both the free-hand case, in which the hand is isolated from the surrounding environment, and the hand-object case, in which the different types of interactions are classified, have been considered. The hand-object case is clearly the most challenging task having to deal with multiple tracks. The approach proposed here belongs to the class of partial pose estimation where the estimated pose in a frame is used for the initialization of the next one. The pose estimation is obtained by applying a modified version of the Iterative Closest Point (ICP) algorithm to synthetic models to obtain the rigid transformation that aligns each model with respect to the input data. The proposed framework uses a "pure" point cloud as provided by the Kinect sensor without any other information such as RGB values or normal vector components. For this reason, the proposed method can also be applied to data obtained from other types of depth sensor, or RGB-D camera

SCALABLE AR FOR BIM ON TELECOMMUNICATION NETWORK SITES

A growing number of research works, experiments and applications is investigating the potential at the intersection of augmented reality (AR) and the architecture, engineering, and construction (AEC) industry. Project management, project communication, collaborative design, maintenance and construction progress documentation, construction site safety, and training are some of the cases that can benefit from blending real and virtual views through mobile devices. In recent years, research also highlighted how mixed reality and building information modeling (BIM) could cooperate to provide effective communication between multiple agents and closer interaction between digital information and the building site. Nevertheless, consolidated applications in these fields are still limited, especially when compared to other areas of AR adoption. This paper presents the development of an AR-based mobile app for exploring telecommunications tower sites and interacting with a related BIM database. The project aims to provide easy-to-use tools to maintain both the physical assets and an up-to-date model. We discuss critical issues in developing a scalable and interoperable application, supporting the feasibility study of similar solutions in the AEC sector

DENTAL AUGMENTED REALITY – AUGMENTED REALITY (AR) BASED ON DEEP LEARNING FOR DENTAL EDUCATION

A project has been carried out involving dental students, deep learning, augmented reality (AR), and markerless AR to identify the challenges that dental students face and provide a new type of learning based on the observation. COVID-19 has cause major disruptions on the educational institutions in forcing the student to adapt to online learning and cancellation of overall physical assessment. This project suggests a way for dental students to proceed with their education virtually or via online

Integração de localização baseada em movimento na aplicação móvel EduPARK

More and more, mobile applications require precise localization solutions in a variety of environments. Although GPS is widely used as localization solution, it may present some accuracy problems in special conditions such as unfavorable weather or spaces with multiple obstructions such as public parks. For these scenarios, alternative solutions to GPS are of extreme relevance and are widely studied recently. This dissertation studies the case of EduPARK application, which is an augmented reality application that is implemented in the Infante D. Pedro park in Aveiro. Due to the poor accuracy of GPS in this park, the implementation of positioning and marker-less augmented reality functionalities presents difficulties. Existing relevant systems are analyzed, and an architecture based on pedestrian dead reckoning is proposed. The corresponding implementation is presented, which consists of a positioning solution using the sensors available in the smartphones, a step detection algorithm, a distance traveled estimator, an orientation estimator and a position estimator. For the validation of this solution, functionalities were implemented in the EduPARK application for testing purposes and usability tests performed. The results obtained show that the proposed solution can be an alternative to provide accurate positioning within the Infante D. Pedro park, thus enabling the implementation of functionalities of geocaching and marker-less augmented reality.Cada vez mais, as aplicações móveis requerem soluções de localização precisa nos mais variados ambientes. Apesar de o GPS ser amplamente usado como solução para localização, pode apresentar alguns problemas de precisão em condições especiais, como mau tempo, ou espaços com várias obstruções, como parques públicos. Para estes casos, soluções alternativas ao GPS são de extrema relevância e veem sendo desenvolvidas. A presente dissertação estuda o caso do projeto EduPARK, que é uma aplicação móvel de realidade aumentada para o parque Infante D. Pedro em Aveiro. Devido à fraca precisão do GPS nesse parque, a implementação de funcionalidades baseadas no posionamento e de realidade aumentada sem marcadores apresenta dificuldades. São analisados sistemas relevantes existentes e é proposta uma arquitetura baseada em localização de pedestres. Em seguida é apresentada a correspondente implementação, que consiste numa solução de posicionamento usando os sensores disponiveis nos smartphones, um algoritmo de deteção de passos, um estimador de distância percorrida, um estimador de orientação e um estimador de posicionamento. Para a validação desta solução, foram implementadas funcionalidades na aplicação EduPARK para fins de teste, e realizados testes com utilizadores e testes de usabilidade. Os resultados obtidos demostram que a solução proposta pode ser uma alternativa para a localização no interior do parque Infante D. Pedro, viabilizando desta forma a implementação de funcionalidades baseadas no posicionamento e de realidade aumenta sem marcadores.EduPARK é um projeto financiado por Fundos FEDER através do Programa Operacional Competitividade e Internacionalização - COMPETE 2020 e por Fundos Nacionais através da FCT - Fundação para a Ciência e a Tecnologia no âmbito do projeto POCI-01-0145-FEDER-016542.Mestrado em Engenharia Informátic

Exploring the use of smart glasses, gesture control, and environmental data in augmented reality games

Abstract. In the last decade, augmented reality has become a popular trend. Big corporations like Microsoft, Facebook, and Google started to invest in augmented reality because they saw the potential that it has especially with the rising of the consumer version of the head mounted displays such as Microsoft’s HoloLens and the ODG’s R7. However, there is a gap in the knowledge about the interaction with such devices since they are fairly new and an average consumer cannot yet afford them due to their relatively high prices. In this thesis, the Ghost Hunters game is described. The game is a mobile augmented reality pervasive game that uses the environment light data to charge the in-game “goggles”. The game has two different versions, a smartphone and smart glasses version. The Ghost Hunters game was implemented for exploring the use of two different types of interactions methods, buttons and natural hand gestures for both smartphones and smart glasses. In addition to that, the thesis sought to explore the use of ambient light in augmented reality games. First, the thesis defines the essential concepts related to games and augmented reality based on the literature and then describes the current state of the art of pervasive games and smart glasses. Second, both the design and implementation of the Ghost Hunters game are described in detail. Afterwards, the three rounds of field trials that were conducted to investigate the suitability of the two previously mentioned interaction methods are described and discussed. The findings suggest that smart glasses are more immersive than smartphones in context of pervasive AR games. Moreover, prior AR experience has a significant positive impact on the immersion of smart glasses users. Similarly, males were more immersed in the game than females. Hand gestures were proven to be more usable than the buttons on both devices. However, the interaction method did not affect the game engagement at all, but surprisingly it did affect the way users perceive the UI with smart glasses. Users that used the physical buttons were more likely to notice the UI elements than the users who used the hand gestures

Augmented Reality Mobile App for Children Learning on Colour

This document describes the process developing an Augmented Reality (AR) of mobile application on colour by using storybook. By using camera on the smartphone or tablet, the user or the child could view the superimposed virtual three dimensional (3D) objects and images in a fun and interactive manner using the marker-less physical colour book as the interaction tool. Learning colour since young age helps children to distinguish colours and they can choose their favourite colours. In addition, it helps them to express themselves, build the confident and more easy-going by choosing the desired colours. Otherwise, they will be tend to have difficulties in making choices in their future life. Many researchers and journalists have done their research on the topic and emphasized on the importance colour for children in the early age is beneficial for their successful in future