Appendix 6A Medical Expenditure Panel Survey key statistics for prescribed medicines for those 65 and older in the U.S.
Appendix 6B CAPI screenshots
Appendix 6C Westat’s CARI code screenshots
Appendix 6D Coding scheme
Appendix 6E Flowchart of CARI rapid feedback process
Appendix 6F Multilevel multinomial logistic regression models
Appendix 6G Multilevel multinomial logistic regression models with interviewer experience
Appendix 6H Alert frequenc

Edwards, Brad

Hubbard, Ryan

Sun, Hanyu

English

UNL | Libraries

University of Nebraska - Lincoln 
DigitalCommons@University of Nebraska - Lincoln 
2019 Workshop: Interviewers and Their Effects 
from a Total Survey Error Perspective Sociology, Department of 
2020 
Chapter 6: Behavior Change Techniques for Reducing Interviewer 
Contributions to Total Survey Error Appendix 6 
Brad Edwards 
Hanyu Sun 
Ryan Hubbard 
Follow this and additional works at: https://digitalcommons.unl.edu/sociw 
 Part of the Quantitative, Qualitative, Comparative, and Historical Methodologies Commons 
This Article is brought to you for free and open access by the Sociology, Department of at 
DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in 2019 Workshop: 
Interviewers and Their Effects from a Total Survey Error Perspective by an authorized administrator of 
DigitalCommons@University of Nebraska - Lincoln. 
1 
 
Chapter 6: Behavior Change Techniques for Reducing Interviewer Contributions  
to Total Survey Error 
Appendix 6 
 
 
 
Brad Edwards, Hanyu Sun, and Ryan Hubbard 
 
 
 
 
 
 
 
 
Full book reference: 
Olson, K., J. D. Smyth, J. Dykema, A. L. Holbrook, F. Kreuter, B. T. West. 2020. Interviewer 
Effects from a Total Survey Error Perspective. Boca Raton: CRC Press. 
2 
 
Appendix 6A Medical Expenditure Panel Survey key statistics for prescribed medicines for 
those 65 and older in the U.S. 
 
Some key MEPS statistics for prescribed medicine utilization and expenditures are illustrated in 
the table below. The vast majority (just over 90%) of people aged 65 and above in the U.S. who 
were not institutionalized obtained prescribed medicines in 2016. Medicines were obtained about 
25 times per person on average, and the average total expenditure (including public and private 
insurance and out-of-pocket payments) for each medicine purchase was $117. The median 
annual expenditures per person who had at least one prescribed medicine purchase was $1,100, 
but the mean was much higher, $3,289. Two factors explain this large difference: some people 
have many prescribed medicine purchases, and some prescribed medicines are extremely 
expensive – the tails of both distributions are very long. This pattern is typical of most types of 
medical utilization and expenditures. The key statistics are quite sensitive to data from outliers. 
This puts a premium on high quality interviewing.      
 
Table A6A.1 Key MEPS statistics, prescribed medicines for the non-institutionalized 65+ 
population, 2016 
Description                                                   
Statistic 
Population with expense            46,409,000 
Proportion of total population with expense                    90.2% 
Number of prescription events 1,304,000 
Mean events per person          25.4 
Mean expenditures per event         $117 
Mean expenditures per person with at least one event $3,289 
Median expenditures per person with at least one 
event 
   $1,100 
Total expenditures $152,602,000,000 
Source: Medical Expenditure Panel Survey, Agency for Healthcare Research and Quality. 
https://meps.ahrq.gov/mepsweb/. 
  
3 
 
Appendix 6B CAPI screenshots 
 
 
 
4 
 
 
 
 
5 
 
Appendix 6C Westat’s CARI code screenshots 
 
6 
 
 
 
7 
 
8 
 
 
9 
 
 
 
10 
 
 
 
11 
 
Appendix 6D Coding scheme  
Calendar Series:  
1. How well can you hear the interviewer on this recording?  
1=Very Clear  
2=Somewhat clear  
3=Not very clear   
4=Cannot hear the interviewer  
 
2. How well can you hear the respondent on this recording?  
1=Very Clear  
2=Somewhat clear  
3=Not very clear  
4=Cannot hear the interviewer 
 
3. What can you hear on this recording?  
1=Question and Answer   
2=Question only  
3=Answer only  
4=Neither the question nor answer 
4. Did the interviewer read the question as it is worded  
1=Yes (End of coding) 
2=No  
3=Unclear 
 
5. (If Q4 =No or Q4=Unclear ) Did the interviewer maintain the question meaning?  
1=Yes 
2=No 
3=Unclear 
 
 
  
12 
 
Provider Probes 
1. How well can you hear the interviewer on this recording?  
1=Very clear  
2=Somewhat clear  
3=Not very clear   
4=Cannot hear the interviewer  
 
2. How well can you hear the respondent on this recording?  
1=Very clear  
2=Somewhat clear  
3=Not very clear  
4=Cannot hear the interviewer 
 
3. What can you hear on this recording? 
1=Question and answer   
2=Question only  
3=Answer only  
4=Neither the question nor answer  
 
4. Did the interviewer read the question as it is worded? 
1=Yes (Skip to Q7) 
2=No  
3=Unclear 
 
5. (If Q4 =No or Q4 =Unclear) Did the interviewer read the examples provided for the type of 
doctors or health care professionals?  
1=Yes  
2=No  
3=Unclear 
 
6. (If Q4 =No or Q4=Unclear) Did the interviewer maintain the question meaning?  
1=Yes 
2=No 
3=Unclear 
 
7. Did the interviewer back up to or wait for a more appropriate probe to enter a respondent’s 
answer about health care events?  
1=Yes 
2=No 
3=Unclear 
4=NA 
 
13 
 
Appendix 6E  Flowchart of CARI rapid feedback process 
 
  
14 
 
Appendix 6F Multilevel multinomial logistic regression models 
Table A6F.1 Parameter estimates in the multilevel multinomial logistic regression models, 
predicting interviewer behavior (verbatim vs. did maintain the meaning, maintained the meaning 
vs. did not maintain the) 
 Model 1 (Figure 6.4) Model 2 (Figure 6.5) 
Parameter Estimate SE Estimate SE 
Intercept (1) 4.3*** 0.5 3.9*** 0.6 
Intercept (2) 4.5*** 0.4 4.4*** 0.5 
Before Feedback (vs. After Feedback) 
(1) 
-0.8* 0.4 -0.4 0.5 
Before Feedback (vs. After Feedback) 
(2) 
-0.2 0.4 0.2 0.5 
Calendar Series (vs. Provider Probes) (1) -1.9*** 0.2 -0.6 0.5 
Calendar Series (vs. Provider Probes) (2) -0.3 0.2 -0.1 0.5 
Asked for Clarification (vs. Did Not Ask 
for Clarification) (1) 
-0.9 0.4 0.7 0.6 
Asked for Clarification (vs. Did Not Ask 
for Clarification (2) 
-0.2 0.4 1.0 0.6 
Two-way Interaction     
Before Feedback * Calendar Series (1)    -0.8 0.6 
Before Feedback * Calendar Series (2)    -0.5 0.6 
Before Feedback * Asked for 
Clarification (1) 
  -2.1* 0.8 
Before Feedback * Asked for 
Clarification (2) 
  -3.5*** 0.8 
Calendar Series * Asked for Clarification 
(1) 
-0.6 0.7 -4.1*** 0.7 
Calendar Series * Asked for Clarification 
(2) 
-1.6* 0.7 -2.3*** 0.7 
Three-way Interaction     
Before Feedback * Calendar Series * 
Asked for Clarification (1)  
  -14.0 375.6 
Before Feedback * Calendar Series * 
Asked for Clarification (2)  
  3.4*** 0.8 
Covariance Parameter       
𝜎𝑖𝑛𝑡:𝑞𝑢𝑒𝑠𝑡𝑖𝑜𝑛
2  (1) 14.6 3.2 15.9 3.5 
𝜎𝑖𝑛𝑡:𝑞𝑢𝑒𝑠𝑡𝑖𝑜𝑛
2  (2) 7.2 1.6 7.7 1.7 
Note: * p<0.05, ** p<0.01, *** p<0.001. There were 112 interviewers. The number of questions each 
interviewer had ranges from 15 to 121 with the mean of 50. The number of observations used is 4626 
questions of calendar series and provider probes. The dependent variable is whether the interviewer 
followed the interviewing protocol including three categories: verbatim, maintained the meaning, and did 
not maintain the meaning (reference group). The comparison between verbatim vs. did not maintain the 
meaning is denoted as (1), and the comparison between maintained the meaning vs. did not maintain the 
15 
 
meaning is denoted as (2). The unconditional model with random effects only show that interviewers 
accounted for 80% of the variance for verbatim vs. did not maintain the meaning and 63% of the variance 
for maintained the meaning vs. did not maintain the meaning.  
  
16 
 
Appendix 6G Multilevel multinomial logistic regression models with interviewer experience 
Table A6G.1 Parameter estimates in the multilevel multinomial logistic regression model with the 
interviewer’s experience, predicting interviewer behavior (verbatim vs. did maintain the meaning, 
maintained the meaning vs. did not maintain the meaning) 
 Model 1 (Experience in 
Years) 
Model 2 (Experienced Yes 
vs. No) 
Parameter Estimate Standard 
Error 
Estimate Standard 
Error 
Intercept (1) 4.0*** 0.8 4.1*** 0.6 
Intercept (2) 3.9*** 0.6 4.5*** 0.5 
Before Feedback (vs. After Feedback) (1) -0.4 0.5 -0.4 0.5 
Before Feedback (vs. After Feedback) (2) 0.2 0.5 0.2 0.5 
Calendar Series (vs. Provider Probes) (1) -0.6 0.5 -0.6 0.5 
Calendar Series (vs. Provider Probes) (2) -0.1 0.5 -0.1 0.5 
Asked for Clarification (vs. Did Not Ask 
for Clarification) (1) 
0.7 0.6 0.7 0.6 
Asked for Clarification (vs. Did Not Ask 
for Clarification (2) 
1.1 0.6 1.0 0.6 
Two-way Interaction     
Before Feedback (1) * Calendar Series (1)  -0.8 0.6 -0.8 0.6 
Before Feedback (2) * Calendar Series (2)  -0.5 0.6 -0.5 0.6 
Before Feedback (1) * Asked for 
Clarification (1) 
-2.1* 0.8 -2.0* 0.8 
Before Feedback (2) * Asked for 
Clarification (2) 
-3.5*** 0.8 -3.4*** 0.8 
Calendar Series (1) * Asked for 
Clarification (1) 
-4.2*** 0.7 -4.1*** 0.7 
Calendar Series (2) * Asked for 
Clarification (2) 
-2.3*** 0.7 -2.3*** 0.7 
Three-way Interaction     
Before Feedback (1) * Calendar Series (1) 
* Asked for Clarification (1)  
-5.2 4.9 -12.7 194.5 
Before Feedback (2) * Calendar Series (2) 
* Asked for Clarification (2)  
3.5*** 0.8 3.4*** 0.8 
Experience in Years (1) 0.0 0.1   
Experience in Years (2) 0.1 0.1   
Experienced: Yes (vs. No) (1)   -1.1 1.4 
Experienced: Yes (vs. No) (2)   -0.8 1.0 
Covariance Parameter       
𝜎𝑖𝑛𝑡:𝑞𝑢𝑒𝑠𝑡𝑖𝑜𝑛
2  (1) 16.1 3.6 16.0 3.5 
𝜎𝑖𝑛𝑡:𝑞𝑢𝑒𝑠𝑡𝑖𝑜𝑛
2  (2) 7.6 1.7 7.8 1.7 
Note: * p<0.05, ** p<0.01, *** p<0.001. There were 112 interviewers. The number of questions each 
interviewer had ranges from 15 to 121 with the mean of 50. The number of observations used is 4626 
17 
 
questions of calendar series and provider probes. The dependent variable is whether the interviewer 
followed the interviewing protocol including three categories: verbatim, maintained the meaning, and did 
not maintain the meaning (reference group). The comparison between verbatim vs. did not maintain the 
meaning is denoted as (1), and the comparison between maintained the meaning vs. did not maintain the 
meaning is denoted as (2). The interviewer’s experience was used as a continuous predictor in Model 1 
and a dichotomous predictor in Model 2.  
 
  
18 
 
Appendix 6H Alert frequency 
Table A6H.1 Data quality alerts, frequency by type  
Alert Type Count Percent 
Record usage (general), all medical 
events 
1,968 84.4% 
Record usage prescribed medicines (65+) 243 10.4% 
Zero night hospital stays                     
(admission and discharge on same day) 
117 5.0% 
Respondent under 18 4 0.2% 
Total 2,332 100.0% 
 
 
 


Chapter 6: Behavior Change Techniques for Reducing Interviewer Contributions
to Total Survey Error
Appendix 6

https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1041&amp;context=sociw

Chapter 6: Behavior Change Techniques for Reducing Interviewer Contributions to Total Survey Error Appendix 6

Abstract

Similar works

Full text

Available Versions

UNL | Libraries