Operational testing of a figure of merit for overall task performance

Abstract

An overall indicator or figure of merit (FOM), for the quality of pilot performance is needed to define optimal workload levels, predict system failure, measure the impact of new automation in the cockpit, and define the relative contributions of subtasks to overall task performance. A normative FOM was developed based on the calculation of a standard score for each component of a complex task. It reflected some effects, detailed in an earlier study, of the introduction of new data link technology into the cockpit. Since the technique showed promise, further testing was done. A new set of data was obtained using the recently developed Multi-Attribute Task Battery. This is a complex battery consisting of four tasks which can be varied in task demand, and on which performance measures can be obtained. This battery was presented to 12 subjects in a 20 minute trial at each of three levels of workload or task demand, and performance measures collected on all four tasks. The NASA-TLX workload rating scale was presented at minutes 6, 12, and 18, of each trial. A figure of merit was then obtained for each run of the battery by calculating a mean, SD, and standard score for each task. Each task contributed its own proportion to the overall FOM, and relative contributions changed with increasing workload. Thus, the FOM shows the effect of task changes, not only on the individual task that is changed, but also on the performance of other tasks and of the whole task. The cost to other tasks of maintaining constant performance on an individual task can be quantified