Is it harder to know or to reason? Analyzing two-tier science assessment items using the Rasch measurement model

Two-tier multiple-choice (TTMC) items are used to assess students’ knowledge of a scientific concept for tier 1 and their reasoning about this concept for tier 2. But are the knowledge and reasoning involved in these tiers really distinguishable? Are the tiers equally challenging for students? The answers to these questions influence how we use and interpret TTMC instruments. We apply the Rasch measurement model on TTMC items to see if the items are distinguishable according to different traits (represented by the tier), or according to different content sub-topics within the instrument, or to both content and tier. Two TTMC data sets are analyzed: data from Singapore and Korea on the Light Propagation Diagnostic Instrument (LPDI), data from the United States on the Classroom Test of Scientific Reasoning (CTSR). Findings for LPDI show that tier-2 reasoning items are more difficult than tier-1 knowledge items, across content sub-topics. Findings for CTSR do not show a consistent pattern by tier or by content sub-topic. We conclude that TTMC items cannot be assumed to have a consistent pattern of difficulty by tier—and that assessment developers and users need to consider how the tiers operate when administering TTMC items and interpreting results. Researchers must check the tiers’ difficulties empirically during validation and use. Though findings from data in Asian contexts were more consistent, further study is needed to rule out differences between the LPDI and CTSR instruments

Death and Science: The Existential Underpinnings of Belief in Intelligent Design and Discomfort with Evolution

The present research examined the psychological motives underlying widespread support for intelligent design theory (IDT), a purportedly scientific theory that lacks any scientific evidence; and antagonism toward evolutionary theory (ET), a theory supported by a large body of scientific evidence. We tested whether these attitudes are influenced by IDT's provision of an explanation of life's origins that better addresses existential concerns than ET. In four studies, existential threat (induced via reminders of participants' own mortality) increased acceptance of IDT and/or rejection of ET, regardless of participants' religion, religiosity, educational background, or preexisting attitude toward evolution. Effects were reversed by teaching participants that naturalism can be a source of existential meaning (Study 4), and among natural-science students for whom ET may already provide existential meaning (Study 5). These reversals suggest that the effect of heightened mortality awareness on attitudes toward ET and IDT is due to a desire to find greater meaning and purpose in science when existential threats are activated

Applying Computerized-Scoring Models of Written Biological Explanations across Courses and Colleges: Prospects and Limitations

Our study explored the prospects and limitations of using machine-learning software to score introductory biology students’ written explanations of evolutionary change. We investigated three research questions: 1) Do scoring models built using student responses at one university function effectively at another university? 2) How many human-scored student responses are needed to build scoring models suitable for cross-institutional application? 3) What factors limit computer-scoring efficacy, and how can these factors be mitigated? To answer these questions, two biology experts scored a corpus of 2556 short-answer explanations (from biology majors and nonmajors) at two universities for the presence or absence of five key concepts of evolution. Human- and computer-generated scores were compared using kappa agreement statistics. We found that machine-learning software was capable in most cases of accurately evaluating the degree of scientific sophistication in undergraduate majors’ and nonmajors’ written explanations of evolutionary change. In cases in which the software did not perform at the benchmark of “near-perfect” agreement (kappa > 0.80), we located the causes of poor performance and identified a series of strategies for their mitigation. Machine-learning software holds promise as an assessment tool for use in undergraduate biology education, but like most assessment tools, it is also characterized by limitations