Cameras and Inertial/Magnetic Sensor Units Alignment Calibration

Due to the external acceleration interference/ magnetic disturbance, the inertial/magnetic measurements are usually fused with visual data for drift-free orientation estimation, which plays an important role in a wide variety of applications, ranging from virtual reality, robot, and computer vision to biomotion analysis and navigation. However, in order to perform data fusion, alignment calibration must be performed in advance to determine the difference between the sensor coordinate system and the camera coordinate system. Since orientation estimation performance of the inertial/magnetic sensor unit is immune to the selection of the inertial/magnetic sensor frame original point, we therefore ignore the translational difference by assuming the sensor and camera coordinate systems sharing the same original point and focus on the rotational alignment difference only in this paper. By exploiting the intrinsic restrictions among the coordinate transformations, the rotational alignment calibration problem is formulated by a simplified hand–eye equation AX = XB (A, X, and B are all rotation matrices). A two-step iterative algorithm is then proposed to solve such simplified handeye calibration task. Detailed laboratory validation has been performed and the good experimental results have illustrated the effectiveness of the proposed alignment calibration method

RGB-D-based Action Recognition Datasets: A Survey

Human action recognition from RGB-D (Red, Green, Blue and Depth) data has attracted increasing attention since the first work reported in 2010. Over this period, many benchmark datasets have been created to facilitate the development and evaluation of new algorithms. This raises the question of which dataset to select and how to use it in providing a fair and objective comparative evaluation against state-of-the-art methods. To address this issue, this paper provides a comprehensive review of the most commonly used action recognition related RGB-D video datasets, including 27 single-view datasets, 10 multi-view datasets, and 7 multi-person datasets. The detailed information and analysis of these datasets is a useful resource in guiding insightful selection of datasets for future research. In addition, the issues with current algorithm evaluation vis-\'{a}-vis limitations of the available datasets and evaluation protocols are also highlighted; resulting in a number of recommendations for collection of new datasets and use of evaluation protocols

SMART EQUIPMENT DESIGN CHALLENGES FOR REAL TIME FEEDBACK SUPPORT IN SPORT

Smart equipment can support feedback in motor learning process. Smart equipment with integrated sensors can be used as a standalone system or complemented with body-attached wearable sensors. Our work focuses on real-time biofeedback system design, particularly on the application of a specific sensor selection. The main goal of our research is to prepare the technical conditions to prove efficiency and benefits of the real-time biofeedback when used in selected motion-learning processes. The most used wireless technologies that are used or are expected to be used in real-time biofeedback systems are listed. The tests performed on two prototypes, smart golf club and smart ski, show an appropriate sensor selection and feasibility of implementation of the real-time biofeedback concept in golf and skiing practice. We are confident that the concept can be expanded for use in other sports and rehabilitation. It has been learned that at this time none of the existing wireless technologies can satisfy all possible demands of different real-time biofeedback applications in sport

Artificial Intelligence Of Things For Ubiquitous Sports Analytics

To enable mobile devices to perform in-the-wild sports analytics, particularly swing tracking, remains an open question. A crucial challenge is to develop robust methods that can operate across various sports (e.g., golf and tennis), different sensors (cameras and IMU), and diverse human users. Traditional approaches typically rely on vision-based or IMU-based methods to extract key points from subjects in order to estimate trajectory predictions. However, these methods struggle to generate accurate swing tracking, as vision-based techniques are susceptible to occlusion, and IMU sensors are notorious for accumulated errors. In this thesis, we propose several innovative solutions by leveraging AIoT, including the IoT with ubiquitous wearable devices such as smartphones and smart wristbands, and harnessing the power of AI such as deep neural networks, to achieve ubiquitous sports analytics. We make three main technical contributions: a tailored deep neural network design, network model automatic search, and model domain adaptation to address the problem of heterogeneity among devices, human subjects, and sports for ubiquitous sports analytics. In Chapter 2, we begin with the design of a prototype that combines IMU and depth sensor fusion, along with a tailored deep neural network, to address the occlusion problems faced by depth sensors during swings. To recover swing trajectories with fine-grained details, we propose a CNN-LSTM architecture that learns multi-modalities within depth and IMU sensor fusion. In Chapter 3, we develop a framework to reduce the overhead of model design for new devices, sports, and human users. By designing a regression-based stochastic NAS method, we improve swing-tracking algorithms through automatic model generation. We also extend our studies to include unseen human users, sensor devices, and sports. Leveraging a domain adaptation method, we propose a framework that eliminates the need for tedious training data collection and labeling for new users, devices, and sports via adversarial learning. In Chapter 4, we present a framework to alleviate the model parameter selection process in NAS, as introduced in Chapter 3. By employing zero-cost proxies, we search for the optimal swing tracking architecture without training, in a significantly larger candidate model pool. We demonstrate that the proposed method outperforms state-of-the-art approaches in swing tracking, as well as in adapting to different subjects, sports, and devices. Overall, this thesis develops a series of innovative machine learning algorithms to enable ubiquitous IoT wearable devices to perform accurate swing analytics (e.g., tracking, analysis, and assessment) in real-world conditions

A CALIBRATION PROCEDURE FOR MIMU SENSORS ALLOWING FOR THE CALCULATION OF ELBOW ANGLES

Non-optical wearable sensors such as magnetic and inertial measurement units (MIMUs) are gaining popularity in sport and clinical settings owing to their ease of application, relative affordability and potential for improved ecological validity. We propose a method for the standardised reference calibration of a simple two-sensor MIMU system for the estimation of anatomically meaningful elbow kinematics. The participant poses with the elbow at 90° flexion and neutral (0°) pronation, allowing for the relative orientation of the MIMU on the forearm to be determined with reference to the MIMU located on the arm. Comparisons were with traditional kinematic marker method results. Root mean squared errors of less than 1° in flex/ext and < 2° (pro/sup) found in simple movements. Results with simple movements provide rationale to expand research to complex movements

Enhanced 6D Measurement by Integrating an Inertial Measurement Unit (IMU) with a 6D Sensor Unit of a Laser Tracker

Six-degree-of-freedom (6D) sensors enhance the measurement capability of traditional three-degree-of-freedom (3D) laser trackers. However, the classical 6D measurement techniques still have shortcomings in actual use, such as the problem of line of sight and relatively low data acquisition rate. The proposed approach by integrating an Inertial Measurement Unit (IMU) with a 6D sensor unit of a laser tracker is effective to overcome these limitations. The error is corrected by the combination of a Kalman filter and a backward smoothing algorithm. The Kalman filter only works when the 6D sensor's data is being sent through, while the backward smoothing algorithm works during the whole process. The experiments are performed to compare the error in three positions and three rotational orientations between the proposed method and the Kalman filter and evaluate the effects of different rates and IMU frequencies on the algorithm. The simulations are also performed to estimate the maximum outage time. The results verify that the proposed method can solve the problem of line of sight and low data acquisition rate effectively.</p

Human Action Recognition with RGB-D Sensors

none3noHuman action recognition, also known as HAR, is at the foundation of many different applications related to behavioral analysis, surveillance, and safety, thus it has been a very active research area in the last years. The release of inexpensive RGB-D sensors fostered researchers working in this field because depth data simplify the processing of visual data that could be otherwise difficult using classic RGB devices. Furthermore, the availability of depth data allows to implement solutions that are unobtrusive and privacy preserving with respect to classic video-based analysis. In this scenario, the aim of this chapter is to review the most salient techniques for HAR based on depth signal processing, providing some details on a specific method based on temporal pyramid of key poses, evaluated on the well-known MSR Action3D dataset.Cippitelli, Enea; Gambi, Ennio; Spinsante, SusannaCippitelli, Enea; Gambi, Ennio; Spinsante, Susann

Human Action Recognition with RGB-D Sensors

Human action recognition, also known as HAR, is at the foundation of many different applications related to behavioral analysis, surveillance, and safety, thus it has been a very active research area in the last years. The release of inexpensive RGB-D sensors fostered researchers working in this field because depth data simplify the processing of visual data that could be otherwise difficult using classic RGB devices. Furthermore, the availability of depth data allows to implement solutions that are unobtrusive and privacy preserving with respect to classic video-based analysis. In this scenario, the aim of this chapter is to review the most salient techniques for HAR based on depth signal processing, providing some details on a specific method based on temporal pyramid of key poses, evaluated on the well-known MSR Action3D dataset

Exploring the role of wearable technology in sport kinematics and kinetics: a systematic review

The aim of this review was to understand the use of wearable technology in sport in order to enhance performance and prevent injury. Understanding sports biomechanics is important for injury prevention and performance enhancement and is traditionally assessed using optical motion capture. However, such approaches are limited by capture volume restricting assessment to a laboratory environment, a factor that can be overcome by wearable technology. A systematic search was carried out across seven databases where wearable technology was employed to assess kinetic and kinematic variables in sport. Articles were excluded if they focused on sensor design and did not measure kinetic or kinematic variables or apply the technology on targeted participants. A total of 33 articles were included for full-text analysis where participants took part in a sport and performed dynamic movements relating to performance monitored by wearable technologies. Inertial measurement units, flex sensors and magnetic field and angular rate sensors were among the devices used in over 15 sports to quantify motion. Wearable technology usage is still in an exploratory phase, but there is potential for this technology to positively influence coaching practice and athletes’ technique

Bridging the Gap between Detection and Tracking for 3D Human Motion Recovery

The aim of this thesis is to build a system able to automatically and robustly track human motion in 3–D starting from monocular input. To this end two approaches are introduced, which tackle two different types of motion: The first is useful to analyze activities for which a characteristic pose, or key-pose, can be detected, as for example in the walking case. On the other hand the second can be used for cases in which such pose is not defined but there is a clear relation between some easily measurable image quantities and the body configuration, as for example in the skating case where the trajectory followed by a subject is highly correlated to how the subject articulates. In the first proposed technique we combine detection and tracking techniques to achieve robust 3D motion recovery of people seen from arbitrary viewpoints by a single and potentially moving camera. We rely on detecting key postures, which can be done reliably, using a motion model to infer 3D poses between consecutive detections, and finally refining them over the whole sequence using a generative model. We demonstrate our approach in the cases of golf motions filmed using a static camera and walking motions acquired using a potentially moving one. We will show that this approach, although monocular, is both metrically accurate because it integrates information over many frames and robust because it can recover from a few misdetections. The second approach is based on the fact that the articulated body models used to represent human motion typically have many degrees of freedom, usually expressed as joint angles that are highly correlated. The true range of motion can therefore be represented by latent variables that span a low-dimensional space. This has often been used to make motion tracking easier. However, learning the latent space in a problem independent way makes it non trivial to initialize the tracking process by picking appropriate initial values for the latent variables, and thus for the pose. In this thesis, it will be shown that by directly using observable quantities as latent variables, this issue can be eliminated