Knowing what students know: How question formats differ in their abilities to reveal student thinking

Instructors use assessment ubiquitously throughout undergraduate STEM courses to gauge student understanding of important concepts. The utility of an assessment can be partly judged by the degree to which it can detect the presence of correct and incorrect understandings. We will discuss a series of within-subjects controlled experiments to determine how student responses differ for questions posed in multiple-choice, multiple-true-false, or free-response formats as well as implications of this research for instructional practice

Multiple-true-false questions reveal more thoroughly the complexity of student thinking than multiple-choice questions: a Bayesian item response model comparison

Background: Within undergraduate science courses, instructors often assess student thinking using closed-ended question formats, such as multiple-choice (MC) and multiple-true-false (MTF), where students provide answers with respect to predetermined response options. While MC and MTF questions both consist of a question stem followed by a series of options, MC questions require students to select just one answer, whereas MTF questions enable students to evaluate each option as either true or false. We employed an experimental design in which identical questions were posed to students in either format and used Bayesian item response modeling to understand how responses in each format compared to inferred student thinking regarding the different options. Results: Our data support a quantitative model in which students approach each question with varying degrees of comprehension, which we label as mastery, partial mastery, and informed reasoning, rather than uniform random guessing. MTF responses more closely estimate the proportion of students inferred to have complete mastery of all the answer options as well as more accurately identify students holding misconceptions. The depth of instructional information elicited by MTF questions is demonstrated by the ability of MTF results to predict the MC results, but not vice-versa. We further discuss how MTF responses can be processed and interpreted by instructors. Conclusions: This research supports the hypothesis that students approach MC and MTF questions with varying levels of understanding and demonstrates that the MTF format has a greater capacity to characterize student thinking regarding the various response options

GenBio-MAPS as a Case Study to Understand and Address the Effects of Test-Taking Motivation in Low-Stakes Program Assessments

The General Biology–Measuring Achievement and Progression in Science (GenBio-MAPS) assessment measures student understanding of the Vision and Change core concepts at the beginning, middle, and end of undergraduate biology degree programs. Assessment coordinators typically administer this instrument as a low-stakes assignment for which students receive participation credit. While these conditions can elicit high participation rates, it remains unclear how to best measure and account for potential variation in the amount of effort students give to the assessment. To better understand student test-taking motivation, we analyzed GenBio-MAPS data from more than 8000 students at 20 institutions. While the majority of students give acceptable effort, some students exhibited behaviors associated with low motivation, such as low self-reported effort, short test completion time, and high levels of rapid-selection behavior on test questions. Standard least-squares regression models revealed that students’ self-reported effort predicts their observable time-based behaviors and that these motivation indices predict students’ Gen- Bio-MAPS scores. Furthermore, we observed that test-taking behaviors and performance change as students progress through the assessment. We provide recommendations for identifying and filtering out data from students with low test-taking motivation so that the filtered data set better represents student understanding

Weed Suppressive Groundcovers

Plants found to be weed suppressive through Cornell research can be used to naturally suppress weeds.   These attractive plants represent a new market potential for NY greenhouse producers.  We have published a color brochure and website to describe these plants and promote their use to suppress weeds.  We also coordinated the establishment of demonstration gardens at several locations across the state.  The recently published brochures have been well received.   The increase in sales of these plants will be tracked prior to and during the two year period of this project.  An analysis of the results will be conducted.  The results will be shared state wide

Nursery IPM Practices - Survey

A survey of New York Nursery growers was conducted in 2007 to assess current cultural and pest management practices.  Nearly 150 growers responded to a series of questions regarding production systems, pest problems, management approaches, information and training sources and educational needs.  The results indicate an opportunity and interest on the growers’ behalf to provide IPM information particularly around pest identification, and biological control.   There also appear to be opportunities to increase knowledge of the timing of controls and access to good decision making resources

Student Buy-In Toward Formative Assessments: The Influence of Student Factors and Importance for Course Success

Formative assessment (FA) techniques, such as pre-class assignments, in-class activities, and post-class homework, have been shown to improve student learning. While many students find these techniques beneficial, some students may not understand how they support learning or may resist their implementation. Improving our understanding of FA buy-in has important implications, since buy-in can potentially affect whether students fully engage with and learn from FAs. We investigated FAs in 12 undergraduate biology courses to understand which student characteristics influenced buy-in toward FAs and whether FA buy-in predicted course success. We administered a mid-semester survey that probed student perceptions toward several different FA types, including activities occurring before, during, and after class. The survey included closed-ended questions aligned with a theoretical framework outlining key FA objectives. We used factor analysis to calculate an overall buy-in score for each student and general linear models to determine whether certain characteristics were associated with buy-in and whether buy-in predicted exam scores and course grades. We found that unfixed student qualities, such as perceptions, behaviors, and beliefs, consistently predicted FA buy-in, while fixed characteristics, including demographics, previous experiences, and incoming performance metrics, had more limited effects. Importantly, we found that higher buy-in toward most FA types predicted higher exam scores and course grades, even when controlling for demographic characteristics and previous academic performance. We further discuss steps that instructors can take to maximize student buy-in toward FAs

Survey of IPM Practice Adoption in New York State Greenhouses

A survey of New York greenhouses was run to evaluate the level of adoption of IPM practices and to compare the results with a baseline survey completed in 2000.  Questions covered important crops and pests, cultural practices related to IPM, scouting, disease/weed/insect management, and where respondents receive their IPM information.  Three hundred and ninety four surveys were completed; a 44% response rate.  Most respondents use at least some preseason sanitation methods.  Analysis of pH and nutrient levels in water, growing media, or foliage is not a common practice.  Also, improvement could be made in calibration of sprayers. Scouting is widely accepted as an IPM practice and most growers use some additional elements of IPM in their pest management.  Cornell Cooperative Extension is the most common source of information cited.  Overall response rates were similar to those in 2000, although the adoption of scouting and use of scouting information has increased by 30 percentage points

How Administration Stakes and Settings Affect Student Behavior and Performance on a Biology Concept Assessment

Biology instructors use concept assessments in their courses to gauge student understanding of important disciplinary ideas. Instructors can choose to administer concept assessments based on participation (i.e., lower stakes) or the correctness of responses (i.e., higher stakes), and students can complete the assessment in an in-class or out-of-class setting. Different administration conditions may affect how students engage with and perform on concept assessments, thus influencing how instructors should interpret the resulting scores. Building on a validity framework, we collected data from 1578 undergraduate students over 5 years under five different administration conditions. We did not find significant differences in scores between lower-stakes in-class, higher-stakes in-class, and lower-stakes out-of-class conditions, indicating a degree of equivalence among these three options. We found that students were likely to spend more time and have higher scores in the higher-stakes out-of-class condition. However, we suggest that instructors cautiously interpret scores from this condition, as it may be associated with an increased use of external resources. Taken together, we highlight the lower-stakes out-of-class condition as a widely applicable option that produces outcomes similar to in-class conditions, while respecting the common desire to preserve classroom instructional time

Multiple–True–False Questions Reveal the Limits of the Multiple–Choice Format for Detecting Students with Incomplete Understandings

By having students select one answer among several plausible options, multiple–choice (MC) questions capture a student’s preferred answer but provide little information regarding a student’s thinking on the remaining options. We conducted a crossover design experiment in which similar groups of introductory biology students were assigned verbatim questions in the MC format or multiple–true–false (MTF) format, which requires students to separately evaluate each option as either true or false. Our data reveal that nearly half of the students who select the correct MC answer likely hold incorrect understandings of the other options and that the selection rates for individual MC options provide inaccurate estimations of how many students separately endorse each option. These results suggest that MC questions systematically overestimate question mastery and underestimate the prevalence of mixed and partial conceptions, whereas MTF questions enable students and instructors to gain a more nuanced portrait of student thinking with little additional effort

Knowing your own: A classroom case study using the scientific method to investigate how birds learn to recognize their offspring

Understanding the scientific method provides students with a necessary foundation for careers in science-related fields. Moreover, students can apply scientific inquiry skills in many aspects of their daily lives and decision making. Thus, the ability to apply the scientific method represents an essential skill that students should learn during undergraduate science education. We designed an interrupted case study in which students learn about and apply the scientific method to investigate and recapitulate the findings of a published research article. This research article addresses the question of how parents recognize their own young in a system where birds of the same species lay eggs in each other’s nests. The researchers approach the question through three experiments in which the bird’s own offspring and unrelated offspring hatch in different orders. This experiment specifically tests for the effect of hatching order on the bird’s ability to recognize its own offspring. In the case study, students form hypotheses based on behavioral observations made while watching a video clip, together with background information provided by the instructor. With additional information about the experimental design, students make graphical predictions for the three related experiments, compare their predictions to the results, and draw conclusions based on evidence. This lesson is designed for introductory undergraduate students, and we provide suggestions on how to adjust the lesson for more advanced students. This case study helps students differentiate between hypotheses and predictions, introduces them to constructing and interpreting graphs, and provides a clear example of the scientific method in action