Constructed-response (CR) questions are a mainstay of introductory physics
textbooks and exams. However, because of time, cost, and scoring reliability
constraints associated with this format, CR questions are being increasingly
replaced by multiple-choice (MC) questions in formal exams. The integrated
testlet (IT) is a recently-developed question structure designed to provide a
proxy of the pedagogical advantages of CR questions while procedurally
functioning as set of MC questions. ITs utilize an answer-until-correct
response format that provides immediate confirmatory or corrective feedback,
and they thus allow not only for the granting of partial credit in cases of
initially incorrect reasoning, but furthermore the ability to build cumulative
question structures. Here, we report on a study that directly compares the
functionality of ITs and CR questions in introductory physics exams. To do
this, CR questions were converted to concept-equivalent ITs, and both sets of
questions were deployed in midterm and final exams. We find that both question
types provide adequate discrimination between stronger and weaker students,
with CR questions discriminating slightly better than the ITs. Meanwhile, an
analysis of inter-rater scoring of the CR questions raises serious concerns
about the reliability of the granting of partial credit when this traditional
assessment technique is used in a realistic (but non optimized) setting.
Furthermore, we show evidence that partial credit is granted in a valid manner
in the ITs. Thus, together with consideration of the vastly reduced costs of
administering IT-based examinations compared to CR-based examinations, our
findings indicate that ITs are viable replacements for CR questions in formal
examinations where it is desirable to both assess concept integration and to
reward partial knowledge, while efficiently scoring examinations.Comment: 14 pages, 3 figures, with appendix. Accepted for publication in
PRST-PER (August 2014