Learning from the Dark Side: a parallel time series modelling framework for forecasting and fault detection on intelligent vehicles

Time series vehicle state modelling is crucial in various real-world applications, such as fault detection, fault tolerance, optimization, and cyber security for intelligent vehicles (IVs). In this study, we propose a novel parallel time series modeling framework (PTSM) to forecast and detect vehicle braking cylinder pressure states, thereby enhancing the safety of the braking system. Specifically, the PTSM consists of two branches: LightNet and DarkNet. The LightNet learns time-series (TS) representations of real-world signals to forecasts and identifies vehicle states. On the other hand, the DarkNet employs a novel multi-task learning and dual Relativistic Generative Adversarial Network (dual-RaGAN) framework to reconstructs healthy sequential states, detects faults, and forecasts future vehicle states using synthesized faulty sequences. To develop the PTSM framework, we introduce a novel data processing and random fault synthesizing method. We evaluate the performance of the dual-RaGAN model using real-world data and compare it with non-adversarial approaches, demonstrating the efficiency of the multi-task generative sequential representation. Extensive experimental results show that by integrating knowledge from the dark side, real-world time-series modelling (TSM) for forecasting and fault detection can be significantly improved, with a 34.7% enhancement in forecasting and an 11% improvement in fault recognition. The results indicate that signal reconstruction leads to more accurate sequence forecasting and fault recognition in both the dark and light sides. This proposed study not only introduces a novel time-series modelling framework but also establishes a new approach for vehicle testing, fault detection, and cyber security research for intelligent vehicles. Data and Codes are available at: https://github.com/YXING-CC/Dark-Light

Driver anomaly quantification for intelligent vehicles: a contrastive learning approach with representation clustering

Driver anomaly quantification is a fundamental capability to support human-centric driving systems of intelligent vehicles. Existing studies usually treat it as a classification task and obtain discrete levels for abnormalities. Meanwhile, the existing data-driven approaches depend on the quality of dataset and provide limited recognition capability for unknown activities. To overcome these challenges, this paper proposes a contrastive learning approach with the aim of building a model that can quantify driver anomalies with a continuous variable. In addition, a novel clustering supervised contrastive loss is proposed to optimize the distribution of the extracted representation vectors to improve the model performance. Compared with the typical contrastive loss, the proposed loss can better cluster normal representations while separating abnormal ones. The abnormality of driver activity can be quantified by calculating the distance to a set of representations of normal activities rather than being produced as the direct output of the model. The experiment results with datasets under different modes demonstrate that the proposed approach is more accurate and robust than existing ones in terms of recognition and quantification of unknown abnormal activities

An integrated framework for multi-state driver monitoring using heterogeneous loss and attention-based feature decoupling

Multi-state driver monitoring is a key technique in building human-centric intelligent driving systems. This paper presents an integrated visual-based multi-state driver monitoring framework that incorporates head rotation, gaze, blinking, and yawning. To solve the challenge of head pose and gaze estimation, this paper proposes a unified network architecture that tackles these estimations as soft classification tasks. A feature decoupling module was developed to decouple the extracted features from different axis domains. Furthermore, a cascade cross-entropy was designed to restrict large deviations during the training phase, which was combined with the other features to form a heterogeneous loss function. In addition, gaze consistency was used to optimize its estimation, which also informed the model architecture design of the gaze estimation task. Finally, the proposed method was verified on several widely used benchmark datasets. Comprehensive experiments were conducted to evaluate the proposed method and the experimental results showed that the proposed method could achieve a state-of-the-art performance compared to other methods

Aemulus ν\nu: Precise Predictions for Matter and Biased Tracer Power Spectra in the Presence of Neutrinos

We present the Aemulus $\nu$ simulations: a suite of 150 $(1.05 h^{-1}\rm Gpc)^3$ $N$-body simulations with a mass resolution of $3.51\times 10^{10} \frac{\Omega_{cb}}{0.3} ~ h^{-1} M_{\odot}$ in a $w\nu$CDM cosmological parameter space. The simulations have been explicitly designed to span a broad range in $\sigma_8$ to facilitate investigations of tension between large scale structure and cosmic microwave background cosmological probes. Neutrinos are treated as a second particle species to ensure accuracy to $0.5\, \rm eV$, the maximum neutrino mass that we have simulated. By employing Zel'dovich control variates, we increase the effective volume of our simulations by factors of $10-10^5$ depending on the statistic in question. As a first application of these simulations, we build new hybrid effective field theory and matter power spectrum surrogate models, demonstrating that they achieve $\le 1\%$ accuracy for $k\le 1\, h\,\rm Mpc^{-1}$ and $0\le z \le 3$, and $\le 2\%$ accuracy for $k\le 4\, h\,\rm Mpc^{-1}$ for the matter power spectrum. We publicly release the trained surrogate models, and estimates of the surrogate model errors in the hope that they will be broadly applicable to a range of cosmological analyses for many years to come.Comment: 37 pages, 15 figures, matching version accepted by JCA

Toward human-centered automated driving: a novel spatial-temporal vision transformer-enabled head tracker

Accurate dynamic driver head pose tracking is of great importance for driver–automotive collaboration, intelligent copilot, head-up display (HUD), and other human-centered automated driving applications. To further advance this technology, this article proposes a low-cost and markerless headtracking system using a deep learning-based dynamic head pose estimation model. The proposed system requires only a red, green, blue (RGB) camera without other hardware or markers. To enhance the accuracy of the driver’s head pose estimation, a spatiotemporal vision transformer (ST-ViT) model, which takes an image pair as the input instead of a single frame, is proposed. Compared to a standard transformer, the ST-ViT contains a spatial–convolutional vision transformer and a temporal transformer, which can improve the model performance. To handle the error fluctuation of the head pose estimation model, this article proposes an adaptive Kalman filter (AKF). By analyzing the error distribution of the estimation model and the user experience of the head tracker, the proposed AKF includes an adaptive observation noise coefficient; this can adaptively moderate the smoothness of the curve. Comprehensive experiments show that the proposed system is feasible and effective, and it achieves a state-of-the-art performance.Agency for Science, Technology and Research (A*STAR)Nanyang Technological UniversityThis work was supported in part by in part by the A*STAR National Robotics Program under grant W1925d0046, the Start-Up Grant, Nanyang Assistant Professorship under grant M4082268.050, Nanyang Technological University, Singapore, and the State Key Laboratory of Automotive Safety and Energy under project KF2021

Review and perspectives on driver digital twin and its enabling technologies for intelligent vehicles

Digital Twin (DT) is an emerging technology and has been introduced into intelligent driving and transportation systems to digitize and synergize connected automated vehicles. However, existing studies focus on the design of the automated vehicle, whereas the digitization of the human driver, who plays an important role in driving, is largely ignored. Furthermore, previous driver-related tasks are limited to specific scenarios and have limited applicability. Thus, a novel concept of a driver digital twin (DDT) is proposed in this study to bridge the gap between existing automated driving systems and fully digitized ones and aid in the development of a complete driving human cyber-physical system (H-CPS). This concept is essential for constructing a harmonious human-centric intelligent driving system that considers the proactivity and sensitivity of the human driver. The primary characteristics of the DDT include multimodal state fusion, personalized modeling, and time variance. Compared with the original DT, the proposed DDT emphasizes on internal personality and capability with respect to the external physiological-level state. This study systematically illustrates the DDT and outlines its key enabling aspects. The related technologies are comprehensively reviewed and discussed with a view to improving them by leveraging the DDT. In addition, the potential applications and unsettled challenges are considered. This study aims to provide fundamental theoretical support to researchers in determining the future scope of the DDT system