A lightweight temporal attention-based convolution neural network for driver's activity recognition in edge

Low inference latency and accurate response to environment changes play a crucial role in the automated driving system, especially in the current Level 3 automated driving. Achieving the rapid and reliable recognition of driver's non-driving related activities (NDRAs) is important for designing an intelligent takeover strategy that ensures a safe and quick control transition. This paper proposes a novel lightweight temporal attention-based convolutional neural network (LTA-CNN) module dedicated to edge computing platforms, specifically for NDRAs recognition. This module effectively learns spatial and temporal representations at a relatively low computational cost. Its superiority has been demonstrated in an NDRA recognition dataset, achieving 81.01% classification accuracy and an 8.37% increase compared to the best result of the efficient network (MobileNet V3) found in the literature. The inference latency has been evaluated to demonstrate its effectiveness in real applications. The latest NVIDIA Jetson AGX Orin could complete one inference in only 63 ms

Learning spatio-temporal representations with a dual-stream 3-D residual network for nondriving activity recognition

Accurate recognition of non-driving activity (NDA) is important for the design of intelligent Human Machine Interface to achieve a smooth and safe control transition in the conditionally automated driving vehicle. However, some characteristics of such activities like limited-extent movement and similar background pose a challenge to the existing 3D convolutional neural network (CNN) based action recognition methods. In this paper, we propose a dual-stream 3D residual network, named D3D ResNet, to enhance the learning of spatio-temporal representation and improve the activity recognition performance. Specifically, a parallel 2-stream structure is introduced to focus on the learning of short-time spatial representation and small-region temporal representation. A 2-feed driver behaviour monitoring framework is further build to classify 4 types of NDAs and 2 types of driving behaviour based on the drivers head and hand movement. A novel NDA dataset has been constructed for the evaluation, where the proposed D3D ResNet achieves 83.35% average accuracy, at least 5% above three selected state-of-the-art methods. Furthermore, this study investigates the spatio-temporal features learned in the hidden layer through the saliency map, which explains the superiority of the proposed model on the selected NDAs

A refined non-driving activity classification using a two-stream convolutional neural network

It is of great importance to monitor the driver’s status to achieve an intelligent and safe take-over transition in the level 3 automated driving vehicle. We present a camera-based system to recognise the non-driving activities (NDAs) which may lead to different cognitive capabilities for take-over based on a fusion of spatial and temporal information. The region of interest (ROI) is automatically selected based on the extracted masks of the driver and the object/device interacting with. Then, the RGB image of the ROI (the spatial stream) and its associated current and historical optical flow frames (the temporal stream) are fed into a two-stream convolutional neural network (CNN) for the classification of NDAs. Such an approach is able to identify not only the object/device but also the interaction mode between the object and the driver, which enables a refined NDA classification. In this paper, we evaluated the performance of classifying 10 NDAs with two types of devices (tablet and phone) and 5 types of tasks (emailing, reading, watching videos, web-browsing and gaming) for 10 participants. Results show that the proposed system improves the averaged classification accuracy from 61.0% when using a single spatial stream to 90.5

Driver behaviour characterization using artificial intelligence techniques in level 3 automated vehicle.

Brighton, James L. - Associate SupervisorAutonomous vehicles free drivers from driving and allow them to engage in some non-driving related activities. However, the engagement in such activities could reduce their awareness of the driving environment, which could bring a potential risk for the takeover process in the current automation level of the intelligent vehicle. Therefore, it is of great importance to monitor the driver's behaviour when the vehicle is in automated driving mode. This research aims to develop a computer vision-based driver monitoring system for autonomous vehicles, which characterises driver behaviour inside the vehicle cabin by their visual attention and hand movement and proves the feasibility of using such features to identify the driver's non-driving related activities. This research further proposes a system, which employs both information to identify driving related activities and non-driving related activities. A novel deep learning- based model has been developed for the classification of such activities. A lightweight model has also been developed for the edge computing device, which compromises the recognition accuracy but is more suitable for further in-vehicle applications. The developed models outperform the state-of-the-art methods in terms of classification accuracy. This research also investigates the impact of the engagement in non-driving related activities on the takeover process and proposes a category method to group the activities to improve the extendibility of the driving monitoring system for unevaluated activities. The finding of this research is important for the design of the takeover strategy to improve driving safety during the control transition in Level 3 automated vehicles.PhD in Manufacturin

Body Pose and Context Information for Driver Secondary Task Detection

Distraction of the driver by secondary tasks is already dangerous while driving manually but especially in handover situations in an automated mode this can lead to critical situations. Currently, these tasks are not taken into account in most modern cars. We present a system that detects typical distracting secondary tasks in an efficient modular way. We first determine the body pose of the driver and afterwards use recurrent neuronal networks to estimate actions based on sequences of the captured body poses. Our system uses knowledge about the surroundings of the driver that is unique to the car environment. Our evaluation shows that this approach achieves better results than other state of the art systems for action recognition on our dataset