GenBio-MAPS as a Case Study to Understand and Address the Effects of Test-Taking Motivation in Low-Stakes Program Assessments

The General Biology–Measuring Achievement and Progression in Science (GenBio-MAPS) assessment measures student understanding of the Vision and Change core concepts at the beginning, middle, and end of undergraduate biology degree programs. Assessment coordinators typically administer this instrument as a low-stakes assignment for which students receive participation credit. While these conditions can elicit high participation rates, it remains unclear how to best measure and account for potential variation in the amount of effort students give to the assessment. To better understand student test-taking motivation, we analyzed GenBio-MAPS data from more than 8000 students at 20 institutions. While the majority of students give acceptable effort, some students exhibited behaviors associated with low motivation, such as low self-reported effort, short test completion time, and high levels of rapid-selection behavior on test questions. Standard least-squares regression models revealed that students’ self-reported effort predicts their observable time-based behaviors and that these motivation indices predict students’ Gen- Bio-MAPS scores. Furthermore, we observed that test-taking behaviors and performance change as students progress through the assessment. We provide recommendations for identifying and filtering out data from students with low test-taking motivation so that the filtered data set better represents student understanding

How Administration Stakes and Settings Affect Student Behavior and Performance on a Biology Concept Assessment

Biology instructors use concept assessments in their courses to gauge student understanding of important disciplinary ideas. Instructors can choose to administer concept assessments based on participation (i.e., lower stakes) or the correctness of responses (i.e., higher stakes), and students can complete the assessment in an in-class or out-of-class setting. Different administration conditions may affect how students engage with and perform on concept assessments, thus influencing how instructors should interpret the resulting scores. Building on a validity framework, we collected data from 1578 undergraduate students over 5 years under five different administration conditions. We did not find significant differences in scores between lower-stakes in-class, higher-stakes in-class, and lower-stakes out-of-class conditions, indicating a degree of equivalence among these three options. We found that students were likely to spend more time and have higher scores in the higher-stakes out-of-class condition. However, we suggest that instructors cautiously interpret scores from this condition, as it may be associated with an increased use of external resources. Taken together, we highlight the lower-stakes out-of-class condition as a widely applicable option that produces outcomes similar to in-class conditions, while respecting the common desire to preserve classroom instructional time

Evaluating Assessment Score Validity and Characterizing Undergraduate Biology Exam Content

The landscape of undergraduate biology education has been shaped by decades of reform efforts calling for instruction to integrate core concepts and scientific skills as a means of helping students become proficient in the discipline. Assessments can be used to make inferences about how these reform efforts have translated into changes in department curriculum and course practices. Such changes can be measured using student scores on researcher-developed programmatic and concept assessments. Scores on these assessments are often assumed to be accurate representations of student biology content knowledge, but my work indicates that the validity of these interpretations may be threatened when students complete the assessments in low-stakes contexts that are more likely to elicit low test-taking effort. Score validity is also threatened in high-stakes out-of-class contexts in which students may be incentivized to leverage external resources to increase their score. My findings suggest that departments and instructors using programmatic and concept assessments to evaluate the progress of their curriculum and courses in meeting the goals of reform effort should carefully interpret scores in light of the conditions in which students completed the assessment. The impacts of reform efforts may also be detected in the types of skills and content that are assessed on course exams. I studied the skills and content of lower-division undergraduate biology exams in the context of a three-dimensional framework consisting of scientific practices, interdisciplinary crosscutting concepts, and disciplinary core ideas. I found that very few exam items were three-dimensional, primarily due to the low number of items assessing scientific practices. Although there were few three-dimensional items, those items were more likely to use a constructed-response format and assess higher-order cognitive skills compared to items not aligned with all three dimensions. To achieve the goals of reform efforts in undergraduate biology education, my research indicates instructors may need time, resources, and training for writing and grading three-dimensional assessments. Altogether, this dissertation sheds critical insight into the process and content of evaluating student learning, thereby refining our understanding of the impact of education reforms. Advisor: Brian A. Couc

Evaluating Assessment Score Validity and Characterizing Undergraduate Biology Exam Content

The landscape of undergraduate biology education has been shaped by decades of reform efforts calling for instruction to integrate core concepts and scientific skills as a means of helping students become proficient in the discipline. Assessments can be used to make inferences about how these reform efforts have translated into changes in department curriculum and course practices. Such changes can be measured using student scores on researcher-developed programmatic and concept assessments. Scores on these assessments are often assumed to be accurate representations of student biology content knowledge, but my work indicates that the validity of these interpretations may be threatened when students complete the assessments in low-stakes contexts that are more likely to elicit low test-taking effort. Score validity is also threatened in high-stakes out-of-class contexts in which students may be incentivized to leverage external resources to increase their score. My findings suggest that departments and instructors using programmatic and concept assessments to evaluate the progress of their curriculum and courses in meeting the goals of reform effort should carefully interpret scores in light of the conditions in which students completed the assessment. The impacts of reform efforts may also be detected in the types of skills and content that are assessed on course exams. I studied the skills and content of lower-division undergraduate biology exams in the context of a three-dimensional framework consisting of scientific practices, interdisciplinary crosscutting concepts, and disciplinary core ideas. I found that very few exam items were three-dimensional, primarily due to the low number of items assessing scientific practices. Although there were few three-dimensional items, those items were more likely to use a constructed-response format and assess higher-order cognitive skills compared to items not aligned with all three dimensions. To achieve the goals of reform efforts in undergraduate biology education, my research indicates instructors may need time, resources, and training for writing and grading three-dimensional assessments. Altogether, this dissertation sheds critical insight into the process and content of evaluating student learning, thereby refining our understanding of the impact of education reforms