Understanding operator engagement in safety-critical Chinese motorway traffic control rooms [Redacted]

The influence of motorway control rooms on public safety is far-reaching. Finding ways to improve the safety of such control systems continues to be an important concern in human factor research. Human factor is focused on humans’ interaction with technology, and this includes their experience at present. However, current research mainly focuses on facilitating efficiency and effectiveness in interaction with technology, but there is a lack of research from the perspective of considering enhancing operator engagement while using technologies. If an operator uses a system that has state-of-the-art technology and high usability with a negative working attitude, the intended effectiveness of the system will be reduced. Furthermore, this lack also leads to an incomplete research perspective, missing the opportunity for enhancing system safety from the perspective of experience. Because engagement is beneficial for improving operator performance and being willing to go the extra mile at work (e.g., doing additional work), which is very crucial to respond to unexpected risks in safety-critical control rooms. Moreover, existing research often emphasizes that the solution lies in excluding the defects of a system, such as reducing the error rate of the system and the fatigue caused by operator’s workload. Instead of merely reducing the occurrence of negative aspects, such as failure and inefficiency, close attention should also be given to resilience and positive psychology encouraging the occurrence of more positive outcomes. This research proposes that work performance can also be enhanced by considering the operators’ well-being and their experience of engagement. A key concept that is used in this research is full operator engagement, which can be defined as a positive, fulfilling, confident, concentrated and even enjoyable experience in the safety-critical control room. This work experience is beneficial to both the utilitarian aspects (e.g., work performance) and the well-being aspects (e.g., positive working mentality and personal development). In order to enhance full operator engagement in a detailed and targeted manner, this PhD project emphasizes the importance of interpretation and measurement of the dynamic nature of engagement. The research has been conducted over five years, across several motorway control rooms in southwest China and involved other institutions related to motorway control rooms, such as the company that designed the monitoring software and the department that managers motorway control rooms. The participants consisted of motorway operators, supervisors, monitoring software engineers and designers. The research approach combines qualitative and quantitative methods, and considering the role of Artificial Intelligence (AI) in the investigation and improvement of operator engagement. Chapters 2 (Literature review) and 3 (Methodology) help to identify the ontology and epistemology of full operator engagement, and the corresponding approaches and methodologies used in this research. Building on the comprehensive literature review and a qualitative study (Chapter 4), two groups of important factors that can affect the level of operator engagement are identified, namely organizational factors (i.e., psychological safety, psychological meaningfulness and psychological availability) and individual factors (e.g., personal characteristics and traits). Following the task analysis in Chapter 5 and the optimization of some research methods in Chapter 4, Chapter 6 identifies five incremental levels of engagement, and Chapter 7 develops the measurement tool in a unique qualitative approach, named Video-based Measurement for Operator Engagement (VMOE), for locating examples of observed behaviour in the appropriate engagement level. VMOE enables a manual evaluation of operator engagement by assessing video tapes of operators at work. In Chapter 8, according to the ratings of operator engagement made by volunteers using VMOE, an AI model based on supervised learning algorithms was developed. This model can automatically measure operator engagement in real time by using an operator’s biometric data recorded on video. The qualitative investigations showed that among all Chinese motorway control rooms surveyed in this doctoral project, the operators rarely experienced full operator engagement. To address this problem, the research has combined theory and empirical evidence to indicate a range of design opportunities. An operator's working performance and working attitudes are two factors that are closely related to system safety. Full operator engagement has a significant impact on these two factors. Some achievements of this study suggest a number of important steps for optimizing full operator engagement including: •Defining the concept of full operator engagement, in order to set an important target for this thesis and related research.  •Clarifying the factors that influence full operator engagement, thereby indicating the way to improve the degree of engagement from a theoretical perspective.  •The identification of five typical operator engagement levels that assist understanding of the actual status of operator engagement, and this identification indicates potential opportunities for improving engagement.  •An application for AI that can efficiently and effectively inform operator engagement design.  These research results broaden the vision for optimizing system safety in the domain of safety-critical control rooms, that is, enhancing system safety by emphasizing and advocating full operator engagement that is beneficial to both operators’ well-being and control room work. Moreover, this research indicates design opportunities by considering the dynamic nature of operator engagement. Dividing engagement into typical levels may offer problem solving opportunities that are not provided if engagement is considered to be a single concept (i.e., a holistic approach). This study also expands AI application in the area of safety-critical domain, such as helping to optimize the positive experience of operators in safety-critical control rooms, suggesting a direction for future research.</p

A video processing and machine learning based method for evaluating safety-critical operator engagement in a motorway control room

In safety-critical automatic systems, safety can be compromised if operators lack engagement. Effective detection of undesirable engagement states can inform the design of interventions for enhancing engagement. However, the existing engagement measurement methods suffer from several limitations which damage their effectiveness in the work environment. A novel engagement evaluation methodology, which adopts Artificial Intelligence (AI) technologies, has been proposed. It was developed using motorway control room operators as subjects. Openpose and Open Source Computer Vision Library (OpenCV) were used to estimate the body postures of operators, then a Support Vector Machine (SVM) was utilised to build the engagement evaluation model based on discrete states of operator engagement. The average accuracy of the evaluation results reached 0.89 and the weighted average precision, recall, and F1-score were all above 0.84. This study emphasises the importance of specific data labelling when measuring typical engagement states, forming the basis for potential control room improvements.  Practitioner summary: This study demonstrates an automatic, real-time, objective, and relatively unobtrusive method for measuring dynamic operator engagement states. Computer vision technologies were used to estimate body posture, then machine learning (ML) was utilised to build the engagement evaluation model. The overall evaluation shows the effectiveness of this framework.  Abbreviations: AI: Artificial Intelligence; OpenCV: Open Source Computer Vision Library; SVM: Support Vector Machine; UWES: Utrecht Work Engagement Scale; ISA Engagement Scale: Intellectual, Social, Affective Engagement Scale; DSSQ: Dundee Stress State Questionnaire; SSSQ: Short Stress State Questionnaire; EEG: electroencephalography; ECG: Electrocardiography; VMOE: Video-based Measurement for Operator Engagement; CMU: Carnegie Mellon University; CNN: Convolutional Neural Network; 2D: two dimensional; ML: Machine learning.</p