Jann, Ben

Müller, Barbara

Sacchi, Stefan

von Rotz, Christina

Bern Open Repository and Information System (BORIS)

Prepaid incentives and survey quality in youth surveysExperimental evidence from the TREE panelStefan Sacchi, Christina von Rotz, Barbara Müller, Ben JannUniversity of Bern, ben.jann@soz.unibe.chIII. Midterm Conference of the ESA RN 21Cracow, October 3–6, 2018Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 1Outline1 Introduction2 Literature and hypotheses3 Study design4 ResultsAverage incentive effectsIncentive effects by cooperation probability5 ConclusionsSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 2IntroductionHigh response rates and low panel attrition are key for studies suchas TREE (Transitions from Education to Employment).A subpopulation of special interest in TREE are adolescents fromdisadvantaged backgrounds, as these adolescents have elevated risksof remaining without upper secondary education.The very same group, however, is difficult to study because of highnonresponse and panel attrition.In the pretest sample of the second TREE cohort we thereforeimplemented a survey experiment to evaluate the effect of (small)monetary incentives.We expected that such incentives may work particularly well for thecritical subpopulation of disadvantaged adolescents.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 3Outline1 Introduction2 Literature and hypotheses3 Study design4 ResultsAverage incentive effectsIncentive effects by cooperation probability5 ConclusionsSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 4HypothesesVarious studies show that (unconditional) monetary incentives dohave an effect on response rates. Less clear, however, are the exactmechanisms behind these effects and the conditions under whichincentive work well or not.One theoretical angle to look at survey participation is toconceptualize participation as a rational decision. In this case,respondents balance costs of participation (time and effort) withbenefits that may have individual or social and intrinsic or extrinsiccomponents.intrinsic extrinsicindividual enjoyment, curiosity, interest incontent of surveysymbolic appreciation,material/monetary gratificationsocial contribution to public good,increase scientific knowledge,etc.benefits for in-groupMonetary incentives work by increasing the individual-extrinsic utilitycomponent.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 5HypothesesAn alternative approach sees survey participation as a specific formof social exchange and highlights the importance of reciprocity.According to this view also symbolic gifts without a high materialvalue can be effective. Furthermore, monetary incentives can becounterproductive because they change the nature of the exchangeand crowd out intrinsic motivation.In any case, it is reasonable to assume that the motivation toparticipate in a survey may differ between subpopulations and thatmonetary incentives will have differential effects.For example, we expect intrinsic motivation to be low amongadolescents from disadvantaged backgrounds. At the same time, dueto their low economic resources, monetary incentives should workbetter in this subpopulation.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 6HypothesesH1: Incentive effects will be generally low in our study because weuse a sample of respondents with relatively high intrinsic motivation.H2: A gift certificate will have a stronger effect in our study thanplain money because it has more of a symbolic character and is lesslikely to crowd out intrinsic motivation.H3: Monetary incentives will be effective primarily for adolescentsfrom disadvantaged backgrounds. This is relevant because responserates are particularly low for these adolescents.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 7Outline1 Introduction2 Literature and hypotheses3 Study design4 ResultsAverage incentive effectsIncentive effects by cooperation probability5 ConclusionsSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 8Sample and experimental groupsOur study focuses on a pretest sample of 541 adolescents whoparticipated in 2015 in a (class-room) pretest for a Swiss large scaleassessment study and who provided valid contact details so thatthey can be contacted one year later.Providing these contact details was voluntary, so our sample is apositive selection of adolescents with high intrinsic motivation.The respondents were randomly divided into three (evenly sized)experimental groups:I No incentive (control group)I Unconditional incentive: rail check (10 CHF)I Unconditional incentive: cash (10 CHF)Respondents were then contacted for a first follow-up interview in2016. They were interviewed by CATI or, as a secondary mode, bypaper-and-pencil questionnaire.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 9OutcomesWe look at effects on various outcomes.I response rateF proportion of respondents who participated in the interviews (CATI ormail questionnaire)I CATI proportionF proportion of telephone interviews among all interviewsI panel drop-out rateF proportion of respondents who actively refused participation in futurewaves or did not agree to data linkage across wavesI etc.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 10Types of resultsAs indicated, our sample is selective. We therefore report raw resultsas well as results for which the sample composition has beenadjusted using reweighting.Raw (unweighted) pretest sample resultsI Descriptive incentive effects in the pretest sample at hand.Population reweighted resultsI Estimates of the incentive effects for the relevant population ofschool leavers. The weights compensate selectivity due to sampledesign (overrepresentation of small cantons) as well as differentialcooperation (providing contact details).Main sample reweighted resultsI Estimates of the incentive effects to be expected in the main sampleof the TREE study. The composition of the pretest sample and themain sample is quite different because the cooperation rate(respondents providing contact details) could be substantiallyincreased in the main study. Weights are computed using entropybalancing (Hainmueller 2012).Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 11Outline1 Introduction2 Literature and hypotheses3 Study design4 ResultsAverage incentive effectsIncentive effects by cooperation probability5 ConclusionsSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 12Incentive effects on response rateNoneRailcheckCashRailcheckor cashCash vs.Railcheck.7 .75 .8 .85 .9 .95 1 -.15 -.1 -.05 0 .05 .1 .15 .2 .25Pretest samplePopulation estimateMain study sampleLevels DifferencesSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 13Incentive effects on CATI proportionNoneRailcheckCashRailcheckor cashCash vs.Railcheck.65 .7 .75 .8 .85 .9 .95 1 -.25 -.2 -.15 -.1 -.05 0 .05 .1 .15 .2 .25Pretest samplePopulation estimateMain study sampleLevels DifferencesSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 14Incentive effects on panel drop-outNoneRailcheckCashRailcheckor cashCash vs.Railcheck-.05 0 .05 .1 .15 .2 -.2 -.15 -.1 -.05 0 .05 .1 .15 .2Pretest samplePopulation estimateMain study sampleLevels DifferencesSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 15Outline1 Introduction2 Literature and hypotheses3 Study design4 ResultsAverage incentive effectsIncentive effects by cooperation probability5 ConclusionsSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 16Incentive effects by cooperation probabilityWith respect to H3 we are interested in how incentive effects dependon characteristics of respondents.Adolescents from disadvantaged backgrounds are underrepresentedin our sample, but for many research questions they are very relevantand it would be good to have good coverage of this group.In general, there are characteristics that are related to low surveyparticipation and we are interested in whether incentives can be usedto counterbalance such selectivity.We use heterogeneous treatment effect analysis as suggested by Xieet al. (2012) to evaluate whether incentive effects are related to thepropensity score of cooperation.The propensity score of cooperation is the estimated probability ofproviding contact details (see above). Important predictors are, forexample, gender, migration background, parental education, andsatisfaction with school.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 17Incentive effects on response rate-.3-.2-.10.1.2.3.4.5.6.7.8.1 .15 .2 .25 .3 .35 .4 .45 .5 .55 .1 .15 .2 .25 .3 .35 .4 .45 .5 .55 .1 .15 .2 .25 .3 .35 .4 .45 .5 .55Cash or railcheck Cash RailcheckTreatment effectPropensity score of cooperation (providing contact details)Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 18Incentive effects on CATI proportion-.6-.5-.4-.3-.2-.10.1.2.3.4.5.1 .15 .2 .25 .3 .35 .4 .45 .5 .55 .1 .15 .2 .25 .3 .35 .4 .45 .5 .55 .1 .15 .2 .25 .3 .35 .4 .45 .5 .55Cash or railcheck Cash RailcheckTreatment effectPropensity score of cooperation (providing contact details)Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 19Incentive effects on panel drop-out-.6-.5-.4-.3-.2-.10.1.2.3.4.5.1 .15 .2 .25 .3 .35 .4 .45 .5 .55 .1 .15 .2 .25 .3 .35 .4 .45 .5 .55 .1 .15 .2 .25 .3 .35 .4 .45 .5 .55Cash or railcheck Cash RailcheckTreatment effectPropensity score of cooperation (providing contact details)Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 20Outline1 Introduction2 Literature and hypotheses3 Study design4 ResultsAverage incentive effectsIncentive effects by cooperation probability5 ConclusionsSacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 21ConclusionsCompared to the literature, incentive effects indeed seem to berelatively low in our sample (H1; effect of about 5 percentage pointson the participation rate).We could not confirm our expectation, that gift certificates have astronger effect than plain cash (H2).With respect to H3, however, we find clear evidence forheterogeneous effects. Incentive effects are stronger for adolescentswith low cooperation probabilities; for adolescents with highcooperation the effects vanish.I From a practical perspective this raises the question whether targetedincentives should be used. Such targeted incentives seem to beeffective and cost efficient, but there might also be good argumentsagainst their usage.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 22ReferencesHainmueller, Jens (2012). Entropy Balancing for Causal Effects: A MultivariateReweighting Method to Produce Balanced Samples in Observational Studies.Political Analysis 20(1):25–46.Xie, Yu, Jennie E. Brand, Ben Jann (2012). Estimating HeterogeneousTreatment Effects with Observational Data. Sociological Methodology42(1):314–347.Sacchi et al. (University of Bern) Incentives Cracow, 4.10.2018 23

Prepaid incentives and survey quality in youth surveys. Experimental evidence from the TREE panel.

https://boris.unibe.ch/125509/1/Sacchi-et-al-2018-cracow.pdf

Prepaid incentives and survey quality in youth surveys. Experimental evidence from the TREE panel.

Abstract

Similar works

Full text

Available Versions

Bern Open Repository and Information System (BORIS)