Using Cognitive Interviewing to Test Youth Survey and Interview Items in Evaluation: A Case Example

Background: Cognitive interviewing is a pretesting tool used by evaluators to increase item and response option validity. Cognitive interviewing techniques are used to assess the cognitive processes utilized by participants to respond to items. This approach is particularly appropriate for testing items with children and adolescents who have more limited cognitive capacities than adults, vary in their cognitive development, and have a unique perspective on their life experiences and context. Purpose: This paper presents a case example of cognitive interviewing with youth as part of a national program evaluation, and aims to expand the use of cognitive interviewing as a pretesting tool for both quantitative and qualitative items in evaluation studies involving youth. Setting: Youth participants were located in four regions of the United States: Northeast, Central, Southern, and Western. Interviewers were located at Montclair State University. Intervention: Not applicable. Research design: A cognitive interview measure was designed to include a subset of survey items, interview questions, and verbal probes, to evaluate if these items and questions would be understood as intended by both younger and older youth participants. An iterative design was used with cognitive interviewing testing rounds, analysis, and revisions. Data Collection and Analysis: The cognitive interview was administered by phone to 10 male youth, five from the 10-13-year-old age range and five from the 15-17-year-old age range. Interviews were audio-recorded, transcribed, reviewed, and coded. Survey items and interview questions were revised based on feedback from the participants and consensus agreement among the evaluation team. Item revisions were included in further testing rounds with new participants. Findings: As a result of using cognitive interviewing to pretest survey and interview items with youth, response errors were identified. Participants did not understand some of the items and response options as intended, indicating problems with validity. These findings support the use of cognitive interviewing for testing and modifying survey items adapted for use with youth, as well as qualitative interview items. Additionally, the perspective of the youth participants was valuable for informing decisions to modify items and helping the evaluators learn the participants’ program culture and experiences. Based on the findings and limitations of the study, we give practice recommendations for future studies using cognitive interviewing with a youth sample. Keywords: cognitive interviewing; item validity; response error; verbal probes; pre-testing surveys; qualitative evaluation; interviewing children and adolescents; survey developmen

The impact of motivational interviewing on initial treatment attendance for participants with dual disorders

Poor treatment attendance and high drop out rates are pervasive problems among clients with dual disorders. As a result, these clients are commonly described as having low motivation for treatment. Motivation is a dynamic state, or a state of readiness for change, that has been conceptualized as stages within the stages of change model (Prochaska & DiClemente, 1986). Clients with dual disorders are typically in the early stages of change for attending outpatient treatment, or have low motivation to attend, due to ambivalence regarding the benefits of treatment. Motivational interviewing is a client-centered, directive method for increasing motivation to change by exploring and resolving ambivalence (Miller & Rollnick, 2002). Initial findings from studies of motivational interviewing with participants with dual disorders have included increased rates of outpatient treatment attendance. Additional research is needed, however, as the majority of these studies have been limited by small to moderate sample sizes and some have lacked comparison groups. The present study aimed to use a motivational interviewing intervention to increase attendance at the first outpatient therapy appointment for participants with dual disorders enrolling in treatment. It was hypothesized that participants who received motivational interviewing would attend the first appointment at higher rates than individuals who were in a video control condition or a retrospective waiting list control condition. A total of 24 participants were assigned to the intervention conditions, and 35 individuals were selected for the retrospective control group. The small obtained sample limited the study to exploratory analyses, and the findings indicated no significant differences in attendance among the groups. The MI group did have a significantly greater increase in motivation after the intervention than the video group. Additionally, an order effect was found as the interviewer\u27s empathy scores improved over time in the motivational interviewing intervention. Possible explanations for the change in motivation but the lack of a related attendance finding are discussed, including: the waiting list for treatment; the conceptualization and operationalization of motivation; intervention implementation; the small sample size and recruitment problems. Implications for research and practice are discussed, as well as limitations and directions for future research

Using Cognitive Interviewing to Test Youth Survey and Interview Items in Evaluation: A Case Example

Background: Cognitive interviewing is a pretesting tool used by evaluators to increase item and response option validity. Cognitive interviewing techniques are used to assess the cognitive processes utilized by participants to respond to items. This approach is particularly appropriate for testing items with children and adolescents who have more limited cognitive capacities than adults, vary in their cognitive development, and have a unique perspective on their life experiences and context. Purpose: This paper presents a case example of cognitive interviewing with youth as part of a national program evaluation, and aims to expand the use of cognitive interviewing as a pretesting tool for both quantitative and qualitative items in evaluation studies involving youth. Setting: Youth participants were located in four regions of the United States: Northeast, Central, Southern, and Western. Interviewers were located at Montclair State University. Intervention: Not applicable. Research design: A cognitive interview measure was designed to include a subset of survey items, interview questions, and verbal probes, to evaluate if these items and questions would be understood as intended by both younger and older youth participants. An iterative design was used with cognitive interviewing testing rounds, analysis, and revisions. Data Collection and Analysis: The cognitive interview was administered by phone to 10 male youth, five from the 10-13-year-old age range and five from the 15-17-year-old age range. Interviews were audio-recorded, transcribed, reviewed, and coded. Survey items and interview questions were revised based on feedback from the participants and consensus agreement among the evaluation team. Item revisions were included in further testing rounds with new participants. Findings: As a result of using cognitive interviewing to pretest survey and interview items with youth, response errors were identified. Participants did not understand some of the items and response options as intended, indicating problems with validity. These findings support the use of cognitive interviewing for testing and modifying survey items adapted for use with youth, as well as qualitative interview items. Additionally, the perspective of the youth participants was valuable for informing decisions to modify items and helping the evaluators learn the participants’ program culture and experiences. Based on the findings and limitations of the study, we give practice recommendations for future studies using cognitive interviewing with a youth sample. Keywords: cognitive interviewing; item validity; response error; verbal probes; pre-testing surveys; qualitative evaluation; interviewing children and adolescents; survey developmen