A novel examination of none-of-the-above as it influences examinee item responses

It is imperative to collect validity evidence prior to interpreting and using test scores. During the process of collecting validity evidence, test developers should consider whether test scores are contaminated by sources of extraneous information. This is referred to as construct irrelevant variance, or the “degree to which test scores are affected by processes that are extraneous to the test’s intended purpose” (AERA et al., 2014, p. 12). One possible source of construct irrelevant variance is violating item-writing guidelines, such as to “avoid the use of none-of-the-above” in multiple-choice items (Rodriguez, 2016, p. 268). Numerous studies have been conducted with regards to how none-of-the-above (NOTA) impacts item statistics, such as item difficulty, item discrimination, and test score reliability. The impacts of NOTA on item statistics are mixed and often depend on whether NOTA is the correct or incorrect option. In the case of NOTA as the incorrect option, NOTA tends to be more frequently selected by examinees (Garcia-Perez, 1993; Frary, 1991). This increased selection is hypothesized to be due to a potential selection tendency that examinees possess toward NOTA (Butler, 2018). While this tendency toward selecting NOTA is hypothesized in the literature, there has not yet been a study which tests this hypothesis. In the current study, I extended previous NOTA literature to explore whether item difficulty varies across groups of examinees who receive a test with NOTA and a test without NOTA, after controlling for examinee ability. I also tested whether there is a hypothesized selection tendency toward NOTA. Overall, as described in previous research, NOTA resulted in mixed results. I discuss these results, as well as future areas of NOTA research

The effects of undesirable distractors on estimates of ability

Distractors, or the incorrect options, are an important part of the multiple-choice item. Previous literature has supported the inclusion of distractors when estimating abilities. While the effects of well-functioning distractors on estimates of ability have been examined, research has neglected to examine the effects of undesirable distractors on estimates of ability. Undesirable distractors are defined as distractors that are opposite of what test-developers expect or want distractors to behave. For instance, an upper lure distractor is one that high ability examinees select rather than selecting the correct answer. A simulation study was employed to determine these effects by varying undesirable distractor type, percentage of items containing undesirable distractors, and test length. Item responses were generated using the Thissen-Steinberg multiple-choice model for simulating undesirable distractors and the three-parameter logistic model for simulating normal items. Following data generation, item responses were analyzed using the three-parameter logistic model in SAS. An analysis of covariance (ANCOVA) was used to examine the effects of undesirable distractors on estimates of ability for bias and standard error. Multiple significant interactions were identified for bias and standard error. One type of undesirable distractor that was especially problematic was the lower lure distractor, where high ability examinees have a slightly lower, but still high, probability of being selected in comparison to the correct answer. Additionally, a longer test resulted in the least amount of bias and standard error. Overall, test-developers should pay attention to the functioning of distractors, as there are effects of these undesirable distractors on estimates of ability

Seeking Compassion

The creative non-fiction essay “Seeking Compassion” is submitted. The essay centers on the meaning of ethics, through the topics of compassion and charity. The essay follows the author’s experiences and failings in ethics and charity. Braided throughout are quotes on the topic from the works of C.S. Lewis