Toward human-centered automated driving: a novel spatial-temporal vision transformer-enabled head tracker

Accurate dynamic driver head pose tracking is of great importance for driver–automotive collaboration, intelligent copilot, head-up display (HUD), and other human-centered automated driving applications. To further advance this technology, this article proposes a low-cost and markerless headtracking system using a deep learning-based dynamic head pose estimation model. The proposed system requires only a red, green, blue (RGB) camera without other hardware or markers. To enhance the accuracy of the driver’s head pose estimation, a spatiotemporal vision transformer (ST-ViT) model, which takes an image pair as the input instead of a single frame, is proposed. Compared to a standard transformer, the ST-ViT contains a spatial–convolutional vision transformer and a temporal transformer, which can improve the model performance. To handle the error fluctuation of the head pose estimation model, this article proposes an adaptive Kalman filter (AKF). By analyzing the error distribution of the estimation model and the user experience of the head tracker, the proposed AKF includes an adaptive observation noise coefficient; this can adaptively moderate the smoothness of the curve. Comprehensive experiments show that the proposed system is feasible and effective, and it achieves a state-of-the-art performance.Agency for Science, Technology and Research (A*STAR)Nanyang Technological UniversityThis work was supported in part by in part by the A*STAR National Robotics Program under grant W1925d0046, the Start-Up Grant, Nanyang Assistant Professorship under grant M4082268.050, Nanyang Technological University, Singapore, and the State Key Laboratory of Automotive Safety and Energy under project KF2021

Application and Evaluation of Lighthouse Technology for Precision Motion Capture

This thesis presents the development towards a system that can capture and quantify motion for applications in biomechanical and medical fields demanding precision motion tracking using the lighthouse technology. Commercially known as SteamVR tracking, the lighthouse technology is a motion tracking system developed for virtual reality applications that makes use of patterned infrared light sources to highlight trackers (objects embedded with photodiodes) to obtain their pose or spatial position and orientation. Current motion capture systems such as the camera-based motion capture are expensive and not readily available outside of research labs. This thesis provides a case for low-cost motion capture systems. The technology is applied to quantify motion to draw inferences about biomechanics capture and analysis, quantification of gait, and prosthetic alignment. Possible shortcomings for data acquisition using this system for the stated applications have been addressed. The repeatability of the system has been established by determining the standard deviation error for multiple trials based on a motion trajectory using a seven degree-of-freedom robot arm. The accuracy testing for the system is based on cross-validation between the lighthouse technology data and transformations derived using joint angles by developing a forward kinematics model for the robot’s end-effector pose. The underlying principle for motion capture using this system is that multiple trackers placed on limb segments allow to record the position and orientation of the segments in relation to a set global frame. Joint angles between the segments can then be calculated from the recorded positions and orientations of each tracker using inverse kinematics. In this work, inverse kinematics for rigid bodies was based on calculating homogeneous transforms to the individual trackers in the model’s reference frame to find the respective Euler angles as well as using the analytical approach to solve for joint variables in terms of known geometric parameters. This work was carried out on a phantom prosthetic limb. A custom application-specific motion tracker was also developed using a hardware development kit which would be further optimized for subsequent studies involving biomechanics motion capture

Développement d'un système en réalité augmentée d'assistance aux opérateurs de machinerie lourde dans une cour à bois

De nos jours, la Réalité augmentée (RA) prend une place de plus en plus prépondérante dans l’industrie en fournissant de l’assistance contextuelle à un niveau jamais atteint précédemment. En particulier, dans le domaine de la foresterie, les cours à bois sont des zones de stockage de paquets de bois, où évoluent des engins de machinerie lourde réalisant la manutention. Dans notre étude, on s’attache à développer une application en RA capable d’assister les opérateurs des machines évoluant dans la cour. On utilise pour cela le visiocasque HoloLens de Microsoft. Ce visiocasque possède un dispositif interne pour mesurer sa position basé sur une centrale inertielle et des caméras analysant l’environnement. Mais nos tests ont montré que ce dispositif était incapable de fournir une position fiable à l’intérieur d’un véhicule en mouvement. En conséquence nous avons conçu un système capable de mesurer la position et l’orientation d’un visiocasque à l’intérieur d’un véhicule évoluant en extérieur avec la précision requise pour obtenir une qualité d’expérience satisfaisante en RA. Nous n’avons pas pu mettre en pratique ce système car nous n’avions pas accès au matériel haute précision nécessaire. Nous avons cependant développé un prototype applicatif fonctionnant en intérieur en utilisant les informations du système de positionnement embarqué de l’HoloLens afin de mettre en pratique et valider les solutions de design d’interface que nous avons conçues. En parallèle, nous avons aussi mené une réflexion pour déterminer comment instrumenter les chargeurs à fourche servant à la manutention dans la cour à bois afin qu’ils puissent s’interfacer avec le reste du système d’assistance aux opérateurs. La manutention dans une cour à bois étant rarement la tâche d’un seul chargeur, nous avons ainsi conçu et mis en pratique une architecture réseau permettant d’interconnecter tous les chargeurs instrumentés de la cour avec un serveur d’inventaire qui était développé en parallèle avec notre étude. De plus, nous nous sommes assurés de la robustesse du système face à des connexions instables telles que rencontrées en utilisant des liens sans-fil à bord de véhicules en mouvement. Nous avons donc conçu un système complet d’assistance en RA aux opérateurs de machinerie lourde dans une cour à bois, et nous en avons mis en pratique l’interface ainsi que l’ensemble de l’architecture réseautique.----------ABSTRACT: Nowadays, augmented reality (AR) has taken a central role in industrial settings by providing contextual help at a level never reached before. For example, in the domain of forestry, timber yards are storage areas for wood stacks. In these yards, one can find heavy equipment tasked with handling material. In our study, we aim to develop an AR application that can assist the operators working in the timber yard. For this purpose, we elected to use the HoloLens head-mounted display (HMD) sold by Microsoft. The HMD has an embedded positioning device that uses an inertial measurement unit (IMU) and environment sensing cameras. However, our results showed this device is not able to provide a reliable position when used inside a moving vehicle. Thus we designed an attitude measurement system able to locate and give the orientation of an HMD sitting inside a vehicle driving outdoors, with enough precision to provide a satisfying experience in AR. However, we were not able to fully implement this system due to lacking access the high performance equipment required. Nonetheless, we developed an application prototype designed to work indoors using the embedded HoloLens positioning system. We did so in order to implement and validate the user interface we designed. We also lead a reflection in parallel, to determine a way to instrument the fork-loaders used for handling in the timber yard. That way, they would be able to interface with the rest of the operator assistance system. Handling in a timber yard is rarely – if ever – a one-loader task. Thus we designed and implemented a network architecture allowing for connections between the instrumented loaders and an inventory server that was developed in parallel of our project. Moreover, we made sure the system was robust against unstable connections, as those can be common when using a wireless link in a moving vehicle. In summary, we designed an entire system to assist heavy equipment operators in a timber yard, and we implemented the interface and user experience, as well as the whole network architecture