Preface

Preface This work evolved from the 1993 Buros-Nebraska Symposium on Testing and Measurement: Multicultural Assessment, held by the Buros Institute of Mental Measurements, Department of Educational Psychology, at the University of Nebraska-Lincoln. Among the symposium presenters, Stanley Sue, Juris Draguns, Guiseppe Costantino and Thomas Malgady, Janet Helms, Robert Carter, Sandra Choney and John Behrens, Joseph Ponterotto, Gargi Roysircar Sodowsky, and Donald Pope-Davis accepted the Buros Institute\u27s invitation to contribute chapters to an envisioned multicultural assessment volume. The authorssome with the assistance of new collaborators-painstakingly revised their papers. They added measurement and statistical analyses that they have not previously published, advanced the theory-building of their respective topics, and wrote integrative literature reviews, thus providing substantive chapters for this book. With reference to the few multicultural and cross-cultural assessment books, book chapters, and journal reviews that are currently available in professional psychology, this book is an essential complement to them. Its uniqueness is that it responds to the paucity of measurement research in multicultural counseling. This collection might be characterized as recording multicultural assessment\u27s empirical beginnings, a fairly unchartered territory. Select multicultural instruments that are paper-and-pencil attitude and projective tests, developed recently and cited in refereed journals, are presented. New data are treated to multivariate statistics, exploratory and confirmatory factor analyses, structural equation modeling, cluster analyses, reliability estimation, and norm transformations. Tests are shown to be construct-related to multicultural theories and/ or criterion- related to sociocultural variables. That is, instrument validity is supported through the theoretical groundedness of obtained results, through theory-building based on data interpretation, or through criterion predictions. Test bias of mainstream instruments are discussed statistically as well as conceptually. Differences in test scores are given an interpretation different from the conclusion that deviant scores indicate deficits. Clinical judgement is shown to be subject to individual bias and to clinicians\u27 immersion in their racial and cultural contexts and their inability to see their imposed bias. Decision trees, guidelines, and assessment reports are provided to illustrate qualitative methods for contextual diagnosis and integrative clinical judgement. Methods to identify social desirability in multicultural self-reports are discussed. Classical measurement theory is argued to overlook the multiplicity of person-environment reactivity that merits investigation in a multicultural society and in a majority-minority sociopolitical system

Epilogue

The book has shown the use of a combination of approaches to understand the nature of a problem: traditional diagnosis and standardized assessment, cultural and racial explanations as alternative hypotheses, clinical judgement based on a decision-tree involving cross-cultural and indigenous frameworks, quantitative-qualitative methods of data analyses, and the use of multicultural paper-and pencil and projective tests. The attitudes and cognitive-affective tests presented or referenced in the book, in addition to being formally administered, could be used as springboards for collaborative discussions with clients and psychology trainees in order to gain a better understanding of their values and assumptions and, by inference, their modes of problem-solving in a multicultural society. We look forward to these new instruments\u27 future refinements, psychometric enhancements, and diverse sampling of subjects. The measurement of acculturation attitudes is important in counseling and clinical psychology. Its importance to applications has been affirmed by the 1994 Diagnostic and Statistical Manual of Mental Disorders (DSM-IV), and the 1993 APA Guidelines for Service Providers to Ethnic, Linguistic, and Culturally Diverse Populations, the latter stating that psychologists must document culturally relevant factors in client records, including number of generations in the country, number of years in the country, fluency in English, community resources, level of education, and level of stress related to acculturation. Because a multicultural book is incomplete without addressing issues of acculturation, Appendices A and B provide measurement and research information on acculturation scales. Appendix A summarizes select psychometric properties of and predictions for frequently referenced acculturation scales developed for Hispanic/Latino and Asian groups in the U.S. Appendix B summarizes select counseling psychology studies showing the effects of acculturation on client reactions. At the end of each Appendix is a reference list of the authors of the instruments and related research studies. We hope this work, Multicultural Measurement in Counseling and Clinical Psychology, will add to the long and colorful history of psychological assessment

Preface- Licensure Testing: Purposes, Procedures, And Practices

This book represents a unique effort for the Buros Institute of Mental Measurements and the Buros-Nebraska Series on Measurement and Testing. All of the previous books in this series have been associated with a symposium sponsored by the Buros Institute of Mental Measurements and the University of Nebraska. This book is free standing in that it is an independent effort intended to fulfill a perceived need for a book, but without preceding the book with a symposium. There are few books devoted solely to the topic of licensure testing, but each state and the federal government is involved in this form of testing. Licensure testing is far too important to be undertaken casually, because it is big business and it has implications and consequences for thousands of people annually. Every author in this book is involved in some way with licensure testing. Some authors work for professional organizations that are responsible for development of licensure tests. Other authors work for companies that contract with licensure boards to develop licensure tests. The rest serve as consultants to one or more licensure boards to assist in the development and maintenance of licensure testing programs. The intent of this book is to provide licensure board members with practical information that will help them to understand and carry out their measurement related responsibilities. Many licensure boards employ consultants to assist in developing or selecting measures to use in making the licensure decision. They employ such consultants for a number of reasons (e.g., because they do not have large numbers of employees who are trained in test development, because the occupations they are charged with regulating may have only small numbers of applicants, or because they have little funding and must issue licenses based on results of tests developed by national professional organizations). The reason for employing outside consultants is not really material, the board is the legally responsible agent and the board must make decisions about the test. This book is intended to help board members make such decisions. This book is also intended to aid measurement consultants by providing them with specific information written in the context of licensure testing. This book provides some technical guidance useful in the development of licensure tests. Most chapters contain some technical content (especially the chapters in Part Two); however, there was an attempt to make most of the content readable by board members while providing measurement consultants with guidelines and references that will assist them in their consulting role. All the authors in this book tried to walk a fine line between writing for board members and for measurement experts alike. I hope we have achieved that end. The book is divided into three parts: Part One addresses the purposes for licensure and it includes discussion of legal and policy issues in licensure. The contents of the three chapters in Part One represent essential knowledge for licensure board members. These three chapters constitute the basis for licensure testing and are entirely nontechnical. Part Two provides the details of setting up and operating a licensure testing program. This part represents the bulk of the book and it consists of the rationale for the various steps involved in developing a licensure test and those activities necessary for operating an ongoing licensure testing program. It provides the licensure board member with an understanding of why certain activities must be undertaken (such as a job analysis and conducting an analysis of differential item functioning for different candidate populations) and it contains information helpful in making decisions about such issues as which testing strategies and which methods of setting the cut score are most appropriate under different conditions. The basis for doing many of the technical tasks is also explained in ways a board member can understand. The measurement consultant will also appreciate this part of the book. It provides the details and rationale for undertaking many of the technical steps used in developing and maintaining a licensure testing program. It does not always go into the level of detail necessary for the measurement expert, but when that level of detail is missing the reference list should provide citations to aid the expert. In Part Three some futuristic looks at the practice of licensure testing m·e taken

Section Two: Overview of the Procedures for Developing a Licensure Examination

There are a variety of strategies that may be employed in the development of a licensure examination. The following list of activities illustrates typical procedures. Depending on the needs and conditions of the particular occupation, certain variations in specific activities may take place or changes in the sequence may be appropriate. In addition to the procedures listed, many decisions will be made that may add activities. For example, the decision to use a computerized item bank or to enter into a computerized adaptive testing format will require procedures in addition to those described briefly below. 1. Conducting a job (or practice) analysis. Often practitioners in the occupation are surveyed to assess the nature of the job; the essential knowledge, skills, and abilities (KSAs) associated with the job; the extent that the KSAs are critical to performance in the profession for the purpose of protecting the public; and the extent that these critical KSAs are at the entry level of practice. An initial list of KSAs is often developed by a committee (perhaps supplemented by shadowing some practitioners and seeking additional insights from interviews with others) and a survey questionnaire is drafted and piloted. The pilot testing leads to expansion and development of the questionnaire that is then sent to a sample of practitioners. Responses may be analyzed by staff or a consultant. 2. Developing test specifications based on the job analysis. The licensure board, or a test committee, determines the specific content dimensions and nature of the test by examining the job analysis (often assisted by quantitative analyses of the survey done by staff or a consultant). It may be possible to obtain a copy of the test specifications (or even the job analysis) of the tests currently being used in the same or related fields in other states or nationally and compare them with each other to assess the essential differences among the occupations of interest. 3. Making a decision about test development. Strategies at this stage include: (a) develop an original test from scratch, (b) use a test already developed by another state or a national organization, or (c) attempt to collaborate with other states to develop a new test. Test development is an arduous and long-term project that has far-reaching implications and costs. If the occupation to be licensed is unique this may be the only available option. If original test development is the decision, the development process could be expected to take up to 2 years and it should be done with the advice of a test specialist/consultant. Ideally, test development includes such activities as drafting original test items, reviewing the items by a testing committee, revising the items, pilot testing the items, assessing the psychometric properties of the items (DIF analysis) and the total scores (reliability and validity studies), refining the items based on the pilot test results, assembling the items into a final form (and having one or more alternate forms is desirable). 4. Arranging for test administration. The examination must be administered and scored. Various decisions need to be made (e.g., distributing the tests, employing test proctors, insuring test security, compliance with the ADA legislation) before the testing program can become operational. 5. Arranging for test scoring and data analysis. Reports need to be developed for the Board as well as for candidates who were tested. Reports to the Board often include various statistics about the test (e.g., item analyses, DIF analyses, reliability estimates, overall and subgroup score distributions). Examinees will need to know their test results, licensed or not, and the Board may decide to provide some diagnostic feedback to candidates who failed the test

Dedication- Licensure Testing: Purposes, Procedures, And Practices

The impetus for this book and for my involvement in it are due principally to the efforts of three people: Jimmie Fortune, Howard Stoker, and Barbara Plake. Jimmie edited (and contributed to) an earlier book devoted to licensure testing (Fortune, J. C., & Associates, 1985, Understanding Testing in Occupational Licensing, Jossey Bass: San Francisco.). Shortly after that book was published he told me that he felt some important topics had not been included and that another book was needed. He encouraged me on several occasions to edit a new book that picked up where his left off and expand it to include some additional topics. I discussed Jimmie\u27s suggestions with my friend and mentor, Howard Stoker and he, too, encouraged me to put the book-editing process in motion. He offered to be a contributor even though he was in the process of retiring (for the third time!). Finally, I moved to the University of Nebraska-Lincoln and to the Buros Institute of Mental Measurements where I began to work with Barbara Plake and the wonderful people here. When I mentioned the possibility of editing a book on licensure testing, she encouraged me and was instrumental in finally pushing me over the edge. Jimmie, Howard, and Barbara all made excellent recommendations of possible chapter authors and each agreed to be a contributor. Without their support and encouragement this book would not have happened. If you find this book to be helpful, they deserve much of the credit. If this book is not what you thought it should be, then the blame is mine because I made the final decisions about content and structure

MULTICULTURAL ASSESSMENT IN COUNSELING AND CLINICAL PSYCHOLOGY (complete work)

Select multicultural instruments that are paper-and-pencil attitude and projective tests, developed recently and cited in refereed journals, are presented. New data are treated to multivariate statistics, exploratory and confirmatory factor analyses, structural equation modeling, cluster analyses, reliability estimation, and norm transformations. Tests are shown to be construct-related to multicultural theories and/ or criterion- related to sociocultural variables. That is, instrument validity is supported through the theoretical groundedness of obtained results, through theory-building based on data interpretation, or through criterion predictions. Test bias of mainstream instruments are discussed statistically as well as conceptually. Differences in test scores are given an interpretation different from the conclusion that deviant scores indicate deficits. Clinical judgement is shown to be subject to individual bias and to clinicians\u27 immersion in their racial and cultural contexts and their inability to see their imposed bias. Decision trees, guidelines, and assessment reports are provided to illustrate qualitative methods for contextual diagnosis and integrative clinical judgement. Methods to identify social desirability in multicultural self-reports are discussed. Classical measurement theory is argued to overlook the multiplicity of person-environment reactivity that merits investigation in a multicultural society and in a majority-minority sociopolitical system. Thus, this work represents empiricism, ideology, and applications, the scientist-practitioner hallmark of professional psychology. True to their professional training in applications, the authors\u27 discourse, tables, and figures are reader- and consumer-friendly. The book has been designed to give counseling and clinical psychology students, researchers, and practitioners information that aims to make them multiculturally effective in their respective roles. Once again the Buros Institute tradition of being at the cutting edge of measurement concerns is continued by this book

Epilogue

The book has shown the use of a combination of approaches to understand the nature of a problem: traditional diagnosis and standardized assessment, cultural and racial explanations as alternative hypotheses, clinical judgement based on a decision-tree involving cross-cultural and indigenous frameworks, quantitative-qualitative methods of data analyses, and the use of multicultural paper-and pencil and projective tests. The attitudes and cognitive-affective tests presented or referenced in the book, in addition to being formally administered, could be used as springboards for collaborative discussions with clients and psychology trainees in order to gain a better understanding of their values and assumptions and, by inference, their modes of problem-solving in a multicultural society. We look forward to these new instruments\u27 future refinements, psychometric enhancements, and diverse sampling of subjects. The measurement of acculturation attitudes is important in counseling and clinical psychology. Its importance to applications has been affirmed by the 1994 Diagnostic and Statistical Manual of Mental Disorders (DSM-IV), and the 1993 APA Guidelines for Service Providers to Ethnic, Linguistic, and Culturally Diverse Populations, the latter stating that psychologists must document culturally relevant factors in client records, including number of generations in the country, number of years in the country, fluency in English, community resources, level of education, and level of stress related to acculturation. Because a multicultural book is incomplete without addressing issues of acculturation, Appendices A and B provide measurement and research information on acculturation scales. Appendix A summarizes select psychometric properties of and predictions for frequently referenced acculturation scales developed for Hispanic/Latino and Asian groups in the U.S. Appendix B summarizes select counseling psychology studies showing the effects of acculturation on client reactions. At the end of each Appendix is a reference list of the authors of the instruments and related research studies. We hope this work, Multicultural Measurement in Counseling and Clinical Psychology, will add to the long and colorful history of psychological assessment

Preface

Preface This work evolved from the 1993 Buros-Nebraska Symposium on Testing and Measurement: Multicultural Assessment, held by the Buros Institute of Mental Measurements, Department of Educational Psychology, at the University of Nebraska-Lincoln. Among the symposium presenters, Stanley Sue, Juris Draguns, Guiseppe Costantino and Thomas Malgady, Janet Helms, Robert Carter, Sandra Choney and John Behrens, Joseph Ponterotto, Gargi Roysircar Sodowsky, and Donald Pope-Davis accepted the Buros Institute\u27s invitation to contribute chapters to an envisioned multicultural assessment volume. The authorssome with the assistance of new collaborators-painstakingly revised their papers. They added measurement and statistical analyses that they have not previously published, advanced the theory-building of their respective topics, and wrote integrative literature reviews, thus providing substantive chapters for this book. With reference to the few multicultural and cross-cultural assessment books, book chapters, and journal reviews that are currently available in professional psychology, this book is an essential complement to them. Its uniqueness is that it responds to the paucity of measurement research in multicultural counseling. This collection might be characterized as recording multicultural assessment\u27s empirical beginnings, a fairly unchartered territory. Select multicultural instruments that are paper-and-pencil attitude and projective tests, developed recently and cited in refereed journals, are presented. New data are treated to multivariate statistics, exploratory and confirmatory factor analyses, structural equation modeling, cluster analyses, reliability estimation, and norm transformations. Tests are shown to be construct-related to multicultural theories and/ or criterion- related to sociocultural variables. That is, instrument validity is supported through the theoretical groundedness of obtained results, through theory-building based on data interpretation, or through criterion predictions. Test bias of mainstream instruments are discussed statistically as well as conceptually. Differences in test scores are given an interpretation different from the conclusion that deviant scores indicate deficits. Clinical judgement is shown to be subject to individual bias and to clinicians\u27 immersion in their racial and cultural contexts and their inability to see their imposed bias. Decision trees, guidelines, and assessment reports are provided to illustrate qualitative methods for contextual diagnosis and integrative clinical judgement. Methods to identify social desirability in multicultural self-reports are discussed. Classical measurement theory is argued to overlook the multiplicity of person-environment reactivity that merits investigation in a multicultural society and in a majority-minority sociopolitical system

7. Basic Psychometric Issues In Licensure Testing

INTRODUCTION The number of people in the United States who carry some responsibility for the writing of examination questions and the construction of tests is unknown. In the Preface to The Construction and Use of Achievement Examinations, published by the American Council on Education in 1936, the authors indicated that the number probably exceeded a million. That number has certainly grown in the past 60 years. Questions are posed to students by teachers at all levels of education; the Armed Forces have people whose job it is to construct tests which are used in the promotion of personnel; over 1,000 occupations are regulated by the states and many, ranging from the professions to the trades, require licensure or certification (Brinegar, 1990). Many licensure and certification decisions are based on test performance. Throughout the years, the types of test questions being used have changed, emphasis has changed from performance testing to multiple-choice testing and back to performance assessment. Apprenticeship programs in the trades- a kind of continuous assessment of performance-have been supplemented, or even replaced, by written examinations, or by a combination of written and performance tests. More recently, the use of technology in testing has begun to come into the picture. For example, computer administration of questions, interactive video, and CD-ROM are beginning to be used. Regardless of the type of test, whether it was written 50 years ago or last week, there are some important concerns. Fundamental among these concerns are the reliability and validity of the measures. The purpose of this chapter is to focus on the psychometric issues of reliability and validity of measures as they pertain to licensure examinations. In addition, the chapter focuses on the relationship of the measures to various guidelines- those of the Equal Employment Opportunity Commission (EEOC, 1975) and The Standards for Educational and Psychological Testing, produced by a joint committee of the American Educational Research Association (AERA), American Psychological Association (APA), and the National Council on Measurement in Education (NCME) and published by the AP A (1 985). (We will refer to the EEOC document as the EEOC Guidelines and the AERA, APA, and NCME document as the Standards.) Frequent references are made to the reliability and validity of examinations when, in reality, it is the scores and the decisions made on the basis of the scores that are, or are not, reliable and valid . In the context of licensure, scores are used to make decisions. Statistical analysis may show that the scores possess properties indicative of reliability . Studies may be conducted to show that the measures have some type of validity. However, reliable and valid scores may be used inconsistently or incorrectly, and when this happens, the decisions made on the basis of the scores may not be reliable or valid decisions. The discussion of reliability and validity in this chapter focuses on the traditional concepts of reliability and validity rather than on a more contemporary approach broadly called generalizability theory. Our reasons for the focus on the more traditional concepts are simply that most licensure and certification programs with which we are familiar have not yet made the transition to generalizability theory as their basic approach to reporting the psychometric characteristics of their tests

Issues in the Measurement of Metacognition- Contents

Contents 1. A Process-Oriented Model of Metacognition: Links Between Motivation and Executive Functioning John G. Borkowski, Lorna K. S. Chan, and Nithi Muthukrishna...........................1 2. Assessing Metacognition and Self-Regulated LearningPaul R. Pintrich, Christopher A. Wolters, and Gail P. Baxter.........................43 3. Assessing Metacognition in Children and AdultsLinda Baker and Lorraine C. Cerro............................................................99 4. Assessing Metacognitive Knowledge Monitoring Sigmund Tobias and Howard Everson.......................................................147 5. Metacognition and Computer-Based Testing Gregory Schraw, Steven L. Wise, and Linda L. Roos................................223 6. Development of Grounded Theories of Complex Cognitive Processing: Exhaustive Within- and Between Study Analyses of Think-Aloud Data Michael Pressley....................................................................................262 7. Assessing Metacognition: Implications of the Buros Symposium Gregory Schraw .....................................................................................297 Author Index........................................................................................323 Subject Index.........................................................................................33