Krosnick, Jon A.

National Centre for Research Methods: NCRM EPrints Repository

1Effects of Survey Data Collection Mode on Response Quality: Implications for Mixing Modes in Cross-National StudiesJon A. KrosnickStanford UniversitySeptember 2005Copyright KrosnickAmerican National Election Study Face-to-face national surveys since 1948. Sharply rising costs. Consumed an increasingly large portion of the NSF Political Science. Recently declining response rates. Pressure from the community to switch to telephone to save money.2American National Election Study Shrinking grants from the National Science Foundation. Coped by reducing sample size. Community dissatisfaction with shrinking sample size. Conduct experiments with telephone.American National Election Study NES commissioned an outside mode review committee. Conclusion: Changing Mode Ends the Time Series NSF commissioned an advisory committee. Conclusion: NES is an under-funded gold standard of quality – protect and nurture it.3American National Election Study Net Result: Nearly tripling of the budget for the next 4 years. Maintain Super-Quality Face-to-Face Fieldwork: RTI, International. Minimum guaranteed response rate: 70%General Social Survey Similar national face-to-face study for decades. Similar experiences with declining response rates. Similarly shrinking grants. Coped by selling questionnaire modules. Coped by double sampling in 2004. Coped by mixing modes: 16% telephone in 2004.4Reasonable Solutions for Increasing Costs Switch modes between waves Shift from face-to-face to telephone Shift from telephone or mail to Internet Mix modes Begin with face-to-face, end with telephone Begin with telephone, end with face-to-face Begin with mail, end with telephoneReasonable Solutions for Decreasing Response RatesEven if cost is not an issue, exhausting case recruitment with one mode and then adding a new mode increases the response rate.So if high response rate is a goal, mixing modes is desirable.5Costs of Switching Modes Between or Within Studies? Is the response process different? Does accuracy change? Does the nature of response bias change? Do you get what you pay for? If reporting accuracy declines, is it offset by an improvement in sample representativeness ?Realistically ….. Increasing a response rate from 60% to 100% has the potential to change representativeness substantially and is guaranteed to eliminate unit non-response error. BUT … Increasing a response rate from 60% to 68% might even *decrease* sample representativeness. That’s the most we can hope for with mode mixing.6So what does mode switching do to response quality?A Starting Point:Some Stereotypes of Modes Face-to-face Increasingly super-expensive Very long field periods Powerful social desirability pressures Telephone Declining response rates Increasing barriers to contact (call blocking) Collapsing sampling frame (cell phones) Mailed Self-Administered Low response rates Slow turnaround Internet  Non-probability samples7Data Collection ModeFace to faceTelephonePaper and pencilComputer/Internet Rapport and trust Confidentiality Modeling of commitment Accountability Pace Discomfort with silence One screen at a time Time of day Literacy Working memory burdenTwo kinds of studies Comparisons of modes confounded with sampling methods: E.g., telephone = RDD Decide which mode to use first Comparisons of modes unconfounded with sampling methods: E.g., random assignment experiments8Comparing Face-to-Face vs. Telephone(Mode confounded with sampling method)(Control for Sample Composition Differences in Analysis)9Comparing Face-to-Face vs. Telephone 1976 SRC Study (RR=72% vs. 62%) 1982 National Election Study Methods Comparison Project (RR=74 vs. 65%) 2000 National Election Study Experiment (RR=64% vs. 57%)General Public Samples – 1 hour questionnaireSatisficing100.340.350.360.370.380.390.40.410.420.43Full Sample Low Education High EducationFace-to-faceTelephoneNon-Differentiation0.10.150.20.250.30.350.40.450.5Full Sample Low Education High EducationFace-to-faceTelephoneNo-Opinion Responses110.250.270.290.310.330.350.370.390.410.43Full Sample Low Education High EducationFace-to-faceTelephoneAcquiescence00.020.040.060.080.10.12Face-to-Face TelephoneAsked How Much Longer the Interview Will Be120.490.50.510.520.530.540.550.560.570.58Face-to-Face TelephoneExpressed Dissatisfaction with Interview Length00.010.020.030.040.050.060.070.080.09Face-to-Face TelephoneSaid Interview Was Too Long1362636465666768697071Face-to-Face TelephoneInterview Length (minutes)0.670.680.690.70.710.720.73Face-to-Face TelephoneInterviewer Rating: Respondent Interest140.860.870.880.890.90.91Face-to-Face TelephoneInterviewer Rating: Respondent CooperationSocial Desirability Response Bias150.450.460.470.480.490.50.510.520.530.54Face-to-Face TelephoneSocially Desirable Responding00.020.040.060.080.10.120.14Face-to-Face TelephoneUnease Discussing Sensitive Topics1600.020.040.060.080.10.12Face-to-Face TelephoneInterviewer Rating: Respondent SuspicionTelephone vs. Paper and Pencil(Mode Unconfounded with Sampling)17National Aviation Operations Monitoring System Field Experiment Licensed Pilots Random assignment Telephone (RR=73%) Self-administered mailed questionnaires (RR=70%) Questionnaire Measure number of safety-related events witnessed during a specified recall period. FACTUAL QUESTIONSValidity Measurement Random assignment to recall period 1 week 2 weeks 4 weeks 8 weeks 16 weeks 24 weeks As recall period increases, so should number of events reported.180204060801001201401601801 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Weeks in Recall Period0.30.350.40.450.50.550.6Telephone PaperValidity1905101520253035Telephone PaperTotal Number of Events Recalled00.10.20.30.40.50.60.70.8Telephone PaperPerceived Accuracy20051015202530Telephone PaperCompletion TimeTelephone vs. the Internet(Both: Mode Confounded AND Unconfounded with Sampling Method)21XReduced memory burdenXFlexibility in completion timeXSelf-pacingXHonest respondingXNo interviewer effectsXPractice effectsXNo long-term panel conditioningXComputer proficiency not neededXLiteracy not requiredXAccountabilityXModeling of task involvementXPositive reinforcementInternetTelephoneCriterionWhich has the advantage?First Study Knowledge Networks (Internet) Ohio State University Center for Survey Research (RDD Telephone)General Public Samples – 30 minute questionnaire2270%43%RR3CompletionRate5849Days in the FieldKnowledge NetworksSRBIPredictor’s      Latent   ConstructReport ofPredictor1Report ofVote Choice2True Vote Choiceε1 ε2r12r12~ λ1 λ223Logistic Regressions Predicting Vote Choice Using Clinton Approval RatingsCSRCSRCSRKNKNKN0246Pollution Crime Race RelationsDifferences Between KN and CSRModel 2(Controlling for Demographics and Knowledge)Model 1 (No control variables)Single Predictors3%6%19%71%0%6%23%71%0%3%16%77%KN < CSR(sig)KN < CSR(non-sig)KN > CSR(non-sig)KN > CSR(sig)24Lab ExperimentRandom assignment to computer vs. intercom mode Interviewer training & supervision332 respondentsConcurrent Validity<>0%0%0%50%71%67%BushGoreClintonSignificant Mode DifferencesTotal number of predictors = 3925Moderators Past experience Respondents with no past experience completing surveys may benefit more from visual presentation and self-pacing Cognitive skills Respondents with low cognitive skills may benefit more from visual presentation and self-pacingDV = Clinton ThermometerPast Experience with Surveys.07-.06ModeMilitary Spending x ModeMilitary Spending-.19.35*.17*-.08YES(n=246)NO (n=83)Significant moderator effect present on 17% of predictors26DV = Bush ThermometerCognitive Skills.02.02ModeExpectations x ModeExpectations for Foreign Relations.06.23*.36**.22**HIGH(n=107)LOW(n=98)Significant moderator effect present on 25% of predictorsResponse Order Effects27Response Order Effects54%51%48%63%40%50%60%70%Computer IntercomPercent Selecting "Crime & Violence" as the Most Important Problem Facing the CountryPresented first Presented lastResponse Order Studies Using Categorical ItemsPrimacy Recency Non-SignificantCampbell & Mohr (1950) 1Becker (1954) 1Kalton et al. (1978) 1Schuman & Presser (1981) 2 5 8McClendon (1986) 4 3Bishop (1987) 3 3Krosnick & Alwin (1987) 1Bishop et al. (1988) 1 1Israel & Taylor (1990) 2 8Ayidiya & McClendon (1990) 2 1McClendon (1991) 7 9Krosnick (1992) 1 3Schwarz et al. (1992) 1 4Krosnick et al. (1996) 1 328Primacy Recency Non-SignificantVisual                                                          Campbell & Mohr (1950) 1Becker (1954) 1Krosnick & Alwin (1987) 1Bishop et al. (1988) 1 1Israel & Taylor (1990) 2 8Ayidiya & McClendon (1990) 2 1Schwarz et al. (1992) 1OralKalton et al. (1978) 1Schuman & Presser (1981) 2 5 8McClendon (1986) 4 3Bishop (1987) 3 3McClendon (1991) 7 9Krosnick (1992) 1 3Krosnick et al. (1996) 1 3Schwarz et al. (1992) 4Voting in Elections 3% average primacy effect in the voting booth. 3% average recency effect in pre-election telephone surveys. Last 4 presidential election exit polls over-estimated Democratic share of the vote. All exit poll questionnaires listed the Democratic candidate first.29Social DesirabilityHelp for Black Americans38%16%26%35% 36%49%0%20%40%60%LESS SAME MORE(White respondents only)Computer Intercom30New Study Knowledge Networks (Internet) SRBI (RDD Telephone)FACTUAL QUESTIONSGeneral Public Samples – 25 minute questionnaire73%36%RR3CompletionRate1584Days in the FieldKnowledge NetworksSRBI31-6-4-20246810Married Workedfor PayOwn Home 3Bedrooms2 VehiclesKnowledgeNetworks -Av = 2.1SRBI -Telephone -Av = 4.5Demographics-4-2024681012Last Trip -AmericanAirlinesSmokeCigarettesPassport Driver'sLicenseMagazineSubscriptionsKnowledgeNetworks -Av = 2.9SRBI -Telephone -Av = 4.7Life-Style32012345Knowledge NetworksSRBIAverage Absolute Error2.24.3Conclusions Face-to-face beats telephone Internet beats telephone  Telephone beats paper-and-pencil A guess: F-to-F > Internet > Telephone > Paper-and-pencilYou (sort of) get what you pay for.33So … Mixing modes can compromise response quality: Satisficing Social Desirability Response Bias Inaccuracy Mixing modes can reverse response order effects: Visual: primacy Oral: recency Mixing modes can compromise correlational analyses: Respondents self-select into second mode Type of person is confounded with measurement artifacts Correlation between two person attributes could be due to mode differences instead Control for mode? What if mode effects are interactive (different in different subpopulations)?34Cross-National ComparisonsIf mode is confounded with country, we cannot simply control for mode during the analysis.Test impact on sample representativeness of mixing modes intended to increase response rates.Even if sample representativeness is improved, is response quality/comparability compromised?

Effects of Survey Data Collection Mode on Response Quality: Implications for Mixing Modes in Cross-National Studies

https://eprints.ncrm.ac.uk/81/1/Krosnick.pdf

Effects of Survey Data Collection Mode on Response Quality: Implications for Mixing Modes in Cross-National Studies

Abstract

Similar works

Full text

Available Versions

National Centre for Research Methods: NCRM EPrints Repository