Multimodal sensor fusion for low-power wearable human motion tracking systems in sports applications

This paper presents a prototype human motion tracking system for wearable sports applications. It can be particularly applicable for tracking human motion during executing certain strength training exercises, such as the barbell squat, where an inappropriate technique could result in an injury. The key novelty of the proposed system is twofold. Firstly, it is an inside-out, multimodal, motion tracker that incorporates two complementary sensor modalities, i.e. a camera and an inertial motion sensor, as well as two externally-mounted points of reference. Secondly, it incorporates a novel multimodal sensor fusion algorithm which uses the complementary nature of vision and inertial sensor modalities to perform a computationally efficient 3-Dimensional (3-D) pose detection of the wearable device. The 3-D pose is determined by fusing information about the two external reference points captured by the camera together with the orientation angles captured by the inertial motion sensor. The accuracy of the prototype was experimentally validated in laboratory conditions. The main findings are as follows. The Root Mean Square Error (RMSE) in 3-D position calculation was 36.7 mm and 13.6 mm in the static and mobile cases, respectively. Whereas the static case was aimed at determining the system’s performance at all 3-D poses within the work envelope, the mobile case was used to determine the error in tracking human motion that is involved in the barbell squat, i.e. a mainly repeated vertical motion pattern

Surface Appearance Estimation from Video Sequences

The realistic virtual reproduction of real world objects using Computer Graphics techniques requires the accurate acquisition and reconstruction of both 3D geometry and surface appearance. Unfortunately, in several application contexts, such as Cultural Heritage (CH), the reflectance acquisition can be very challenging due to the type of object to acquire and the digitization conditions. Although several methods have been proposed for the acquisition of object reflectance, some intrinsic limitations still make its acquisition a complex task for CH artworks: the use of specialized instruments (dome, special setup for camera and light source, etc.); the need of highly controlled acquisition environments, such as a dark room; the difficulty to extend to objects of arbitrary shape and size; the high level of expertise required to assess the quality of the acquisition. The Ph.D. thesis proposes novel solutions for the acquisition and the estimation of the surface appearance in fixed and uncontrolled lighting conditions with several degree of approximations (from a perceived near diffuse color to a SVBRDF), taking advantage of the main features that differentiate a video sequences from an unordered photos collections: the temporal coherence; the data redundancy; the easy of the acquisition, which allows acquisition of many views of the object in a short time. Finally, Reflectance Transformation Imaging (RTI) is an example of widely used technology for the acquisition of the surface appearance in the CH field, even if limited to single view Reflectance Fields of nearly flat objects. In this context, the thesis addresses also two important issues in RTI usage: how to provide better and more flexible virtual inspection capabilities with a set of operators that improve the perception of details, features and overall shape of the artwork; how to increase the possibility to disseminate this data and to support remote visual inspection of both scholar and ordinary public

Virtual Guidance using Mixed Reality in Historical Places and Museums

Mixed Reality (MR) is one of the most disruptive technologies that shows potential in many application domains, particularly in the tourism and cultural heritage sector. MR using the latest headsets with the highest capabilities introduces a new visual platform that can change people’s visual experience. This thesis introduces a HoloLens-based mixed reality guidance system for museums and historical places. This new guidance form considers the inclusiveness of the necessary and optimised functionalities, visual and audio guiding abilities, essential roles of a guide, and the related social interactions in the real-time. A mixed reality guide, dubbed ‘MuseumEye’ was designed and developed for the Egyptian Museum in Cairo, to overcome challenges currently facing the museum, e.g. lack of guiding methods, limited information signposted on the exhibits, lack of visitor engagement resulting in less time spent in the museum compared to other museums with similar capacity and significance. These problems motivated the researcher to conduct an exploratory study to investigate the museum environment and guiding methods by interviewing 10 participants and observing 20 visitors. ‘MuseumEye’ was built based on a literature review of immersive systems in museums and the findings of an exploratory study that reveals visitor behaviours and the nature of guidance in the museum. This project increased levels of engagement and the length of time visitors spend in museums, the Egyptian Museum in Cairo in particular, using the mixed reality technology that provides visitors with additional visual, audio information and computer-generated images at various levels of details and via different media. This research introduces the guidelines of designing immersive reality guide applications using the techniques of spatial mapping, designing the multimedia and UI, and designing interactions for exploratory purposes. The main contributions of this study include various theoretical contributions: 1) creating a new form of guidance that enhances the museum experience through developing a mixed reality system; 2) a theoretical framework that assesses mixed reality guidance systems in terms of perceived usefulness, ease of use, enjoyment, interactivity, the roles of a guide and the likelihood of future use; 3) the Ambient Information Visualisation Concept for increasing visitor engagement through better presenting information and enhancing communication and interaction between visitors and exhibits; and a practical contribution in creating a mixed reality guidance system that reshapes the museum space, enhances visitors’ experience and significantly increases the length of time they spend in the museum. The evaluation comprised of quantitative surveys (171 participants and 9 experts) and qualitative observation (51 participants) using MuseumEye in their tours. The results showed positive responses for all measured aspects and compares these to similar studies. The observation results showed that visitors who use MuseumEye spent four times the duration visitors spent without guides or with human guides in front of exhibited items. The quantitative results showed significant correlations between the measured constructs (perceived usefulness, ease of use, enjoyment, multimedia and UI, interactivity) and the likelihood of future use when the roles of guide mediate the relations. Moreover, the ‘perceived guidance’ is the most influential construct on the likelihood of future use of MuseumEye. The results also revealed a high likelihood of future use, which ensures the sustainability of adopting mixed reality technology in museums. This thesis shows the potential of mixed reality guides in the museum sector that reshape the museum space and offers endless possibilities for museums and heritage sites

Virtual Guidance using Mixed Reality in Historical Places and Museums

Mixed Reality (MR) is one of the most disruptive technologies that shows potential in many application domains, particularly in the tourism and cultural heritage sector. MR using the latest headsets with the highest capabilities introduces a new visual platform that can change people’s visual experience. This thesis introduces a HoloLens-based mixed reality guidance system for museums and historical places. This new guidance form considers the inclusiveness of the necessary and optimised functionalities, visual and audio guiding abilities, essential roles of a guide, and the related social interactions in the real-time. A mixed reality guide, dubbed ‘MuseumEye’ was designed and developed for the Egyptian Museum in Cairo, to overcome challenges currently facing the museum, e.g. lack of guiding methods, limited information signposted on the exhibits, lack of visitor engagement resulting in less time spent in the museum compared to other museums with similar capacity and significance. These problems motivated the researcher to conduct an exploratory study to investigate the museum environment and guiding methods by interviewing 10 participants and observing 20 visitors. ‘MuseumEye’ was built based on a literature review of immersive systems in museums and the findings of an exploratory study that reveals visitor behaviours and the nature of guidance in the museum. This project increased levels of engagement and the length of time visitors spend in museums, the Egyptian Museum in Cairo in particular, using the mixed reality technology that provides visitors with additional visual, audio information and computer-generated images at various levels of details and via different media. This research introduces the guidelines of designing immersive reality guide applications using the techniques of spatial mapping, designing the multimedia and UI, and designing interactions for exploratory purposes. The main contributions of this study include various theoretical contributions: 1) creating a new form of guidance that enhances the museum experience through developing a mixed reality system; 2) a theoretical framework that assesses mixed reality guidance systems in terms of perceived usefulness, ease of use, enjoyment, interactivity, the roles of a guide and the likelihood of future use; 3) the Ambient Information Visualisation Concept for increasing visitor engagement through better presenting information and enhancing communication and interaction between visitors and exhibits; and a practical contribution in creating a mixed reality guidance system that reshapes the museum space, enhances visitors’ experience and significantly increases the length of time they spend in the museum. The evaluation comprised of quantitative surveys (171 participants and 9 experts) and qualitative observation (51 participants) using MuseumEye in their tours. The results showed positive responses for all measured aspects and compares these to similar studies. The observation results showed that visitors who use MuseumEye spent four times the duration visitors spent without guides or with human guides in front of exhibited items. The quantitative results showed significant correlations between the measured constructs (perceived usefulness, ease of use, enjoyment, multimedia and UI, interactivity) and the likelihood of future use when the roles of guide mediate the relations. Moreover, the ‘perceived guidance’ is the most influential construct on the likelihood of future use of MuseumEye. The results also revealed a high likelihood of future use, which ensures the sustainability of adopting mixed reality technology in museums. This thesis shows the potential of mixed reality guides in the museum sector that reshape the museum space and offers endless possibilities for museums and heritage sites

Encoded LED System for Optical Trackers

Since introducing a hybrid vision-inertial tracker using passive fiducial markers, we have encountered several applications in which the use of encoded LEDs would be preferable to paper fiducials. This includes small-scale applications requiring higher precision, and applications where the size of the target, or the range of viewing angles, or operation in the dark is important. We present a novel technique to encode LEDs without any need for synchronization between the LEDs and cameras. By using amplitude modulation codes instead of blinking binary codes, the LED is always on, and can therefore be tracked after it has been decoded at an arbitrary or even non-periodic frame rate with no missed data. 1