4. The Validity of ComputerBased Test Interpretations of the MMPI

With advances in computer technology, computer-based test interpretations (CBTI), first developed in the early 1960s (Fowler, 1985), have proliferated (Eyde & Kowal, 1987). CBTIs have been developed and marketed for a variety of tests used in clinical, counseling, educational, and employment settings. The largest number of commercial CBTI systems are available for the Minnesota Multiphasic Personality Inventory (MMPI; Krug, 1987), the most widely used inventory of its kind in the world, which has a continuously growing literature of more than 8,000 books and articles (Holden, 1986; Lanyon, 1984). According to Harris: CBTI refers to the automation of a set of pre-specified rules for use in analyzing, interpreting and assigning certain qualities to a response or response pattern (e .g., test score, profile pattern). The discrete rules are used to form an algorithm that guides the activity of the computer to interpret specific input data. (1987, p. 239) Consumers of CBTIs have very little information available on the development of the algorithm or the validity of the CBTI systems. Companies selling CBTIs often do not provide a user\u27s guide. The algorithms used in generating the computer interpretations are not available to CBTI users nor are they provided for scholarly review purposes. Notable exceptions to these business practices include Lachar\u27s (1974) presentation of all the rules and interpretive statements for the WPS Test Report, the MMPI CBTI sold by Western Psychological Services. National Computer Systems provided the algorithms for the Minnesota Report:Personnel Selection System, for scholarly review purposes, and gave an independent evaluation of the extent to which the interpretive statements were based on the MMPI\u27s research literature or on the clinical judgment of the CBTI\u27s author (Eyde, 1985). Numerous critics have pointed out serious problems arising from the growth of CBTIs. Mitchell (1984) observed that the advent of CBTIs presents the field of psychology with its most serious and consequential challenge of the next decade. Lanyon (1984) called attention to the exponential growth of available CBTI systems, noting that Meehl\u27s cookbook approach to MMPI interpretation (however carefully designed) has been used to justify and market many inadequate systems. Eyde and Kowal (1987) commented that the scientific basis for the C.B.T.I., namely the decision rules which codify the rationale and the evidence used to produce the computer interpretations, may wind up locked in a black box, inaccessible to test users (p. 402). Also, Matarazzo (1986) decried the lack of validity evidence for CBTIs. The problems associated with CBTIs have to do not only with the lack of validity data, but also with the problem of how to establish the validity of a computer interpretive report (Mitchell, 1984; Moreland, 1985, 1987; O\u27Dell, 1972). Mitchell (1984) notes that purists who want to do the job properly, are faced with the task of a conducting a statement-by-statement validation involving statements generated by decision rules and decision trees of almost incomprehensible complexity. Critics of prevailing practices in developing, marketing, and validating computerized applications of knowledge-based systems, may choose, as Eyde and Kowal (1985) have, to do some of the developmental work that should have been done before a computerized test product is sold. The intent of this chapter is to describe a methodology for studying the validity of the output of CBTI systems . The research focuses on a variety of CBTI systems developed as tools for interpreting the MMPI. The MMPI is the most widely used psychodiagnostic instrument with active-duty military populations (Parkison & Fishburne, 1984). Our methodology is designed so that it may be adapted to CBTIs for other tests or self-report inventories. The study involves a comparative analysis of the accuracy, relevancy, and usefulness of the output of seven CBTI systems for patients in a military hospital which draws its patients from a wide geographical area. The research design allows us to make some inferences about the relative accuracy of CBTI systems for different profile types. A secondary objective of the research was to identify racial differences, if any, in the accuracy of the CBTIs. This chapter will describe the study, provide basic data, and describe the results. Other chapters will cover (a) the Black/white differences in the accuracy of the CBTIs, which are minimal (Eyde, Kowal, & Fishburne, 1987); and (b) neuropsychological cases vs. non neuropsychological cases (Fishburne, Eyde, & Kowal, 1988)