A large-scale experimental study was conducted involving the training and testing of 123 human subjects on the Criterion Task Set (Version 1.0). Testing was performed under baseline and stressor conditions. The performance data and Subjective Workload Assessment Technique ratings for the first baseline trial (Trial 6) were analyzed using the SAS VARCLUS procedure to evaluate the structure of the CTS. Seven clusters of response time variables were identified for the nine tasks. In general, the Memory Search, Linguistic Processing and Mathematical Processing tasks were grouped in one cluster with each of the other clusters representing a single task. Five clusters were identified for the SWAT ratings with clusters differentiated along the dimensions of task difficulty and processing stage.Yeshttps://us.sagepub.com/en-us/nam/manuscript-submission-guideline

Betina Schlegel

Kirby Gilliland

Robert E. Schlegel

 A large-scale experimental study was conducted involving the training and testing of 123 human subjects on the Criterion Task Set (Version 1.0). Testing was performed under baseline and stressor conditions. The performance data and Subjective Workload Assessment Technique ratings for the first baseline trial (Trial 6) were analyzed using the SAS VARCLUS procedure to evaluate the structure of the CTS. Seven clusters of response time variables were identified for the nine tasks. In general, the Memory Search, Linguistic Processing and Mathematical Processing tasks were grouped in one cluster with each of the other clusters representing a single task. Five clusters were identified for the SWAT ratings with clusters differentiated along the dimensions of task difficulty and processing stage. </jats:p

Crossref

Proceedings of the Human Factors Society Annual Meeting

Factor Structure of the Criterion Task Set

SHAREOK repository

PROCEEDINGS OF THE HUMAN FACTORS SOCIETY-3lst ANNUAL MEETING-1987 FACTOR STRUCTURE OF THE CRITERION TASK SET Robert E. Schlegel Kirby Gilliland* Betina Schlegel School of Industrial Engineering and *Department of Psychology The University of Oklahoma Norman, OK 73019 ABSTRACT A large-scale experimental study was conducted involving the training and testing of 123 human subjects on the Criterion Task Set (Version 1.0). Testing was performed under baseline and stressor conditions. The perfor- mance data and Subjective Workload Assessment Technique ratings for the first baseline trial (Trial 6) were analyzed using the SAS VARCLUS pro- cedure to evaluate the structure of the CTS. Seven clusters of response time variables were identified for the nine tasks. In general, the Memory Search, Linguistic Processing and Mathematical Processing tasks were grouped in one cluster with each of the other clusters representing a sin- gle task. Five clusters were identified for the SWAT ratings with clusters differentiated along the dimensions of task difficulty and processing stage. INTRODUCTION From its initial development in 1983 and subsequent release in 1984, the Criterion Task Set (CTS) has been widely disseminated as a research tool for evaluating workload assessment tech- niques and as an instrument for human performance assessment. Literature exists concerning the content of the task battery and the development of the individual tasks (Shingledecker, 1984), training characteristics of the battery (Schlegel and Shingledecker, 1985), and preliminary results from the development of a large-scale data base of CTS per- formance data, Subjective Workload Assessment Technique (SWAT) ratings and i nd ividual difference variables (Schlegel et al., 1986; Gilliland et al., 1986). A major selling point of the CTS is its selection of tasks based on process- ing stage and multiple resource theories . Table 1 summarizes the characteristics of the nine tasks ( 2 5  individual task levels) presented in alphabetic order. Data collection has recently been completed for the large-scale standardi- zation study of the CTS. Performance and Subjective Workload Assessment Tech- nique (SWAT) data were collected for approximately 123 subjects (96 males, 27 females) performing all nine tasks of the CTS at all levels for nine days. Five days were allocated for training, two days for baseline testing and two days for testing under various stres- sors. Details of the methodology were 389 reported in Schlegel, Gilliland and Schlegel (1986). In addition to providing standardi- zation data for the CTS Version 1.0 tasks, the study provided an opportunity to examine the relationships among the dependent measures for the nine tasks and the SWAT ratings for the tasks. Specifically, data from the first base- line trial following training (Trial 6) was analyzed using the Statistical Analysis System VARCLUS procedure to cluster the nine tasks and twenty-five individual task levels. The results of these analyses wi.11 help determine if all levels of a particular CTS task are drawing from the same resource pool and how much overlap in resource demands exists between tasks. THE SAS VARCLUS PROCEDURE Several techniques may be employed to determine the underlying structure of a set of measures. These include factor analysis, principle components and mul- tidimensional scaling techniques. Another approach is clustering analysis. The SAS VARCLUS procedure divides a set of numeric variables into either dis- joint or hierarchical clusters in such a way that each cluster can be interpreted as essentially unidimensional (SAS, 1985). The clusters are chosen to max- imize the variation accounted for by either the first principal component or the centroid component of each cluster. Specifically, the procedure attempts to maximize the sum across clusters of the variance of the original variables that is explained by the cluster components.  at UNIV OF OKLAHOMA LIBRARIES on January 20, 2016pro.sagepub.comDownloaded from PROCEEDINGS OF THE HUMAN FACTORS SOCIETY-3lst ANNUAL M E E T I N G 1 9 8 7  Table 1. Summary of CTS Task Characteristics. Task code Stage Level Description Continuous Recall CR Central continuous update memory 1 digit - next item - 1 2 2 digits - 2 items back 3 4 digits - 3 items back G r amma t j. ca 1 Re as o n i ng GR Central logic; reasoning 1 single sentence 2 two sentences - active/ positive 3 two sentences - active/ negative or passive/negative Interval. Production IP output Linguistic Processing LP Central Mathemat.ica1 Processing MP Central Memory Search MS Cen tra 1 Probability Monitoring PM Input Spatial Processing SP Cen tra 1 1 tapping at 2 taps/sec. symbol manipulation 1 physical identity match 2 vowel/consonant match 3 antonym match math operations (+ or - )  1 1 operation 2 2 operations 3 3 operations 1 positive set size 1 2 positive set size 4 3 positive set size 6 1 1 meter, 95% bias 2 3 meters, 85% bias 3 4 meters, 7 5 %  bias Sternberg memory test scanning and detection histogram shapes 1 2 bars, 0 rotation 2 4 bars, 90 or 270  rotation 3 6 bars, 1 8 0  rotation Unstable Tracking UT Input/Output manual response 1 lambda=l 2 lambda= 3 3 lambda= 5 VARCLUS is a type of oblique com- ponent analysis related to multiple group factor analysis. It can be used as a variable reduction method to replace a large set of variables by the Set of cluster components. A given number of cluster components does not typically explain as much variance as the same number of principal components, but the cluster components are usually easier to interpret than the principal componen.ts even after rotation. RESULTS Four separate analyses were per- formed, three involving the performance data and one with the SWAT data. CTS Performance Data The first clustering analysis included the response time measures for the discrete stimulus tasks and proba- bility monitoring plus the mean tapping rate (IPMN) and variability score (IPVS) for interval production and the absolute rror and edge violations for unstable 398  at UNIV OF OKLAHOMA LIBRARIES on January 20, 2016pro.sagepub.comDownloaded from PROCEEDINGS OF THE HUMAN FACTORS SOCIETY-3lst ANNUAL MEETING-1987 tracking. This provided nine variables for each of the three levels of eight of the tasks plus two variables for Inter- val Production for a total of 2.9 perfor- mance measures. Each of the task diffi- culty levels was included separately in order to determine whether the different levels tap the same resource as indi- cated by the clustering. A summary of the clusters generated from this analysis is given in Table 2. Table 2. Cluster Analysis for Response Time Measures. The second clustering analysis examined the accuracy measures of pro- portion correct in place of the response time measures for the discrete stimulus tasks. Performance measures for the non-central processing tasks were not included in the analysis. The cluster- ing exhibited more task overlap (Table 3) than in the analysis of response times. This was probably due to the relatively high level of accuracy in several tasks. Table 3 .  Cluster Analysis for Proportion Correct Measures. Cluster Cluster 1 2 3 4 5 6 7 1 2 3 4 5 CR1 CR2 GR1 IPMN PM1 SP1 All CR3 GR2 IPVS PM2 SP2 UT GR3 PM3 Var LP1 LP2 LP3 MP1 M P2 M P3 MS 1 MS 2 MS 3 1 - low level 2 - medium level 3 - high level Several significant findings are evident from the table. With one major exception, measures from different lev- els of the same task were placed in the same cluster indicating that the various workload levels or difficulty manipula- tions of any given task draw from the same resource pool. The exception is the Continuous Recall task at the low level which was placed in the largest cluster with linguistic processing, mathematical processing and memory search. This indicates the much lower difficulty of CR at the low level and its closer association with LP and MP as a symbol manipulation task. Each task occupied a separate clus- ter with the exception of LP, MP and MS. This indicates the minimal resource overlap for all tasks except these three. The overlap of these three is probably due to the relative ease of the tasks and the similarities of simple symbol manipulation whether linguistic, mathematical 01: simple memory update. 391 CR 1 CR 2 GR 1 MS 1 SP1 CR3 GR2 MS 2 SP2 MP1 GR3 MS 3 s P3 MP2 LP2 LP 1 MP3 LP3 1 - low level 2 - medium level 3 - hign level A final clustering analysis with the performance data combined the response time and accuracy measures for a total of 47 variables. Ten clusters were defined as shown in Table 4 .  It is obvious that the Probability Monitoring measures do not form a separate cluster nor do they clearly associate with other clusters. With minor exceptions, the other measures form logical clusters along the lines of the first two ana- lyses. SWAT Ratings The SWAT ratings reflect the per- ceived difficulty of the task rather than the actual performance. Cluster analysis of the SWAT ratings for the 25 task levels produced the results given in Table 5. In terms of subjective ratings, the tasks and task levels were clustered along the dimensions of task difficulty and processing stage. It is evident from the table that cluster 1 contains discrete stimulus central processing tasks of low and moderate difficulty while cluster 2 contains the more diffi- cult central processing tasks. Cluster 3 contains the easy levels of the motor output tasks, a spatial task and a math  at UNIV OF OKLAHOMA LIBRARIES on January 20, 2016pro.sagepub.comDownloaded from PROCEEDINGS OF THE HUMAN FACTORS SOClETY-3lst ANNUAL MEETlNG-1987 Table 4.  Cluster Analysis for Performance Measures. C1 us ter 1 2 3 4 5 6 7 8 9 10 CRMNl L PMN 1 LPMN2 LPMN3 M PMN 1 MPMN2 MPMN3 MSMNl MSMN2 MSMN3 G RPC 1 G RPC 2 MPPC 1 MPPC 2 MPPC 3 CRMN2 CRPCl I PMN SPMNl SPPCl All CRMN3 IPVS SPMN2 SPPC2 UT CRPC 2 LPPC 1 SPMN3 SPPC3 Vars CRPC 3 LPPC2 LPPC 3 GRMNl GRMN2 GRPC3 GRMN3 PMRT3 PMRT2 PMRTl MS PC 1 MS PC 2 MS PC 3 1 - low level 2 - medium level 3 - high level processing task. Cluster 4 contains the difficult levels of the visual/spatial tasks and cluster 5 contains the diffi- cult levels of the motor output tasks. Table 5 .  Cluster Analysis of SWAT Ratings. Cluster 1 2 3 4 5 .  CR 1 CR2 UT 1 PM1 UT2 CR3 IP1 PM2 UT3 GR1 GR2 PM3 GR3 SP1 s P2 MP 2 MP1 SP3 M P3 LP1 LP2 LP3 MS 1 MS 2 MS 3 1 - low level 2 - medium level 3 - high level DISCUSSION The results of the cluster analysis help to validate the design goals of the Criterion Task Set as a battery of tasks that tap separate information processing resources and stages. The primary area of concern is the overlap among the Linguistic Processing, Mathematical Pro- cessing and Memory Search tasks, all of which involve somewhat simple symbol manipulation. Some differences existed between the cluster structure for the performance data and that for the SWAT ratings indicating that subjects perform differently than their estimate of task difficulty as might be expected. Addi- tional analyses using different tech- niques are being performed to substan- tiate these results. ACKNOWLEDGEMENT This research was sponsored by the Workload and Ergonomics Branch of the Armstrong Aerospace Medical Research Laboratory, united States Air Force, under Contract F33615-85-D-0514 through the Southeastern Center for Electrical Engineering Education (SCEEE-HER/86-9). The United Sta.tes Government is author- ized to reproduce and distribute reprints for governmental purposes not- withstanding any copyright notation hereon. The authors wish to thank Gary B. Reid and Dr. F. Thomas Eggemeier for their interest and support. 392  at UNIV OF OKLAHOMA LIBRARIES on January 20, 2016pro. agepub.comD w loaded fr m PROCEEDINGS OF THE HUMAN FACTORS S O C l E T Y 3 l s t  ANNUAL MEETING-1987 REFERENCES Gilliland, K., Schlegel, R.E. and Dan- nels, S.A. (1986). Individual Differ- ences in Criterion Task Set Perfor- mance. In Proceedings of the Human Factors Society 30th Annual Meeting. (pp. 64-68). Dayton, Ohio: Human Fac- tors Society. SAS User's Guide: Statistics, Version 5 Cary, North Carolina: SAS Institute Inc. Edit ion. (1985). (pp. 801-816). Schlegel, R.E. and Shingledecker, C.A. (1985). Training Characteristics of the Criterion Task Set Workload Assessment Battery. In Proceedings of the Human Factors Society 29th Annual Meeting (pp. 770-773). Baltimore, Maryland: Human Factors Society. Schlegel, R.E., Gilliland, K. and Schlegel, B. (1986). Development of the Criterion Task Set Performance Data Base. In Proceedings of the Human Factors Society 30th Annual Meeting. (pp. 58-62). Dayton, Ohio: Human Factors Society. Shingledecker. C.A. (1984). A Task Bat- tery for Applied Human Performance Assessment Research (Tech. Report AFAMRL -T R-8 4- 0 7 1 ) . Wr i gh t-Pa t ter son AFB, Ohio: Medical Research Labora- tory. 393  at UNIV OF OKLAHOMA LIBRARIES on January 20, 2016pro.sagepub.comDownloaded from 

https://shareok.org/bitstream/11244/25467/1/10.1177.154193128703100401.pdf

Factor Structure of the Criterion Task Set

Abstract

Similar works

Full text

Available Versions

Crossref

SHAREOK repository