Improving Survey Methods with Cognitive Interviews in Small- and Medium-scale Evaluations

Findings derived from self-reported, structured survey questionnaires are commonly used in evaluation and applied research to inform policy-making and program decisions. Although there are a variety of issues related to the quality of survey evidence (e.g., sampling precision), the validity of response processes–how respondents process thoughts and perceptions when answering questionnaires–is crucial. We assess the extent to which cognitive interviews, used to test survey questions as part of the process of questionnaire design and refinement, can strengthen the quality of survey evidence for small- and medium-scale evaluations. To illustrate how cognitive interviews can reveal respondents' response processes and improved questionnaire interpretations, we present excerpts from two types of evaluations (e.g., needs assessment) in two domains (e.g., education). The paper concludes with a brief summary about how the use of cognitive interviews can be augmented in survey development, refinement, and adaptation to improve survey questionnaire interpretations in evaluations with limited resources.Ope

Reframing teachers' conceptions of assessment and data use

Teachers’ use of assessment data for instructional decision making is one of several essential components of data-driven decision making (DDDM) as a reform strategy to improve teaching and learning. This thesis explores three different issues related to the use of assessment data for instructional decision making. The first paper examines the way instructional decision making is currently conceived in the DDDM discourse. The second paper uses qualitative data to highlight the broad spectrum of assessment strategies a group of high school science teachers employ to collect data for instruction, and it illustrates the complexity of using that data for instructional decision making. The third paper is a study of an instrument that measures preservice teachers’ conceptions of assessment. Qualitative pre-tests of the instrument and quantitative data analyses post-administration suggest that pre-service teachers’ beliefs about assessment are conditional on the tensions present in the cultural and political assessment climate. Collectively, these papers support reframing conceptions of assessment and data in DDDM research to provider clearer explanations of what “use” of data for instructional decision making actually means