An evaluation of the low-event task Subjective Situation Awareness (LETSSA) Technique

Abstract

Introduction: Accurate situation awareness is an important part of driving a train and it is important to have effective measurement techniques to assess drivers’ levels of situation awareness. The most widely used subjective measure of situation awareness is the Situation Awareness Rating Technique (SART) (Taylor, 1990). Although SART was developed using information from military flight crews, Endsley, Selcon, Hardiman and Croft (1998) state that it is suitable for any domain without need for customisation, although there appears to be little or no research confirming this. Based on comparisons between military aviation operations and long-haul train driving and on the researchers’ experience of using SART in long-haul train driving, it is argued that a new measure needs to be developed to assess situation awareness for low-event tasks. Based on a task analysis of the train driving task produced by Rose and Bearman (2012) and on Endsley’s three stages of situation awareness, a new measure of low-event task subjective situation awareness was developed (LETSSA). This paper describes an initial study that seeks to provide a basic evaluation of the new LETSSA technique. Method: To test the new measure, simulator experiments were conducted using participants with no train driving experience. Twenty-three volunteers (20 males, 3 females, aged 22-70y), attended two sessions in a full-cab, high-fidelity train simulator. In the first session, information provided to assist situation awareness was low and in the second session, information provided was high. Measures included a summary measure of train driving violations, subjective performance ratings, NASA Task Load Index (NASA-TLX) workload measure, Situation Awareness Global Rating Technique (SAGAT) and LETSSA. Results: Wilcoxon rank-sum tests revealed significant differences in scores of subjective and actual performance, LETSSA, SAGAT, and workload between the first and second sessions (p<.01). Investigating session-to-session change for each individual participant revealed higher consistency between LETSSA (22 indicated improvement) and actual performance (23 improved) than between SAGAT (only 19 indicated improvement) and actual performance. Correlational analysis of the differences between sessions (high awareness score minus low awareness score) for LETSSA and objective performance, and SAGAT and objective performance found low correlations that were not significant. Discussion: The study was designed to produce an increase in situation awareness in the second, compared to the first experimental session. Overall, this was clearly reflected by all measures, including SAGAT and LETSSA. Inspection of change in individual participants revealed that the direction of change was more consistent in LETSSA than in SAGAT (22 versus 19 participants), suggesting that LETSSA may be a better measure in this instance. The lack of correlations with performance in terms of the change between sessions suggests that the magnitude of the change was not well-reflected by either measure. These findings represent the first step in a program of studies that will investigate use of LETSSA in more detail, and compare these results in novices with those in experienced train drivers