Diekmann, Andreas

Höglinger, Marc

Jann, Ben

BORIS Portal  Bern Open Repository and Information System

Andreas Diekmann (ETH Zurich) Marc Höglinger (ETH Zurich) Ben Jann (University of Bern) Asking Sensitive Questions: Testing A New Alternative to the Randomized Response Technique The 2011 ASA Spring Methodology Conference at Tilburg University, Mai 18 to 20 source: https://doi.org/10.7892/boris.9020 | downloaded: 8.5.2016Outline §  Sensitive Questions and Truthful Answers in Surveys  §  Indirect Approaches to Elicit Truthful Answers §  a. The Randomized Response Technique (RRT) §  b. Crosswise: A New Alternative to RRT  §  Experimental Comparison of the Different Approaches §  Study A: P&P Survey: Crosswise vs. Direct Questioning §  Study B: Online- Survey: Crosswise vs. Direct Questioning vs. RRT §  Conclusions 12/03/15 2 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Sensitive Questions and Truthful Answers §  Challenge in survey research on sensitive topics: Getting a truthful answer to questions that have a clear socially desirable answer (e.g. ‘No’) §  For example, whether respondents… §  …have evaded tax payments. §  …have committed an act of violence. §  …have cheated in exams. §  …have plagiarized a research paper. §  Some respondents give no truthful (‘Yes’-)answer because they fear consequences, deem it a too intrusive question or just feel uncomfortable answering truthfully (Tourangeau & Yan 2007).  §  This leads to underestimation of the surveyed behavior (social desirability bias).  12/03/15 3 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Eliciting Truthful Answers to Sensitive Questions – No Easy Task with Direct Questioning (DQ)  Share of respondents with false negative response (“liars”) from validation studies: §  Penal conviction: 42.5% (F2F, Wolter 2010) §  Welfare and unemployment benefit fraud: 75% (F2F, van der Heijden et al. 2000) §  Driving under influence: 54% (P&P, Locander et al. 1976) §  Bankruptcy: 32% (Ibid.) 12/03/15 4 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique An Indirect Approach: The Randomized Response Technique (RRT, forced response variant) §  Main principle: 100% privacy protection through randomization (‘random noise in the answering process’) §  A randomizing device, the outcome of which is only known to the respondent, decides whether… 1.  the sensitive question has to be answered truthfully 2.  a forced ‘yes’ has to be given 3.  a forced ‘no’ has to be given.   §  An example:  12/03/15 5 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique §  Respondents give the true answer only with probability Psens. quest.<1 (and a forced yes/no with Pforced yes/no). §  Therefore, a ‘yes’ could be the true answer to the sensitive Question (Pyes|sens. quest.) but it could also just be a forced ‘yes’ (Pforced yes). 12/03/15 6 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Critical Issues with RRT (forced response) §  Complicated procedure, low respondents’ understanding of RRT’s principle ‘protection through randomization’. §  Reluctance of respondents to give a forced-yes (or a ‘false’ answer in general) (Edgell et al. 1982, Lensvelt-Mulders & Boeije 2007) §  Self-protective ‘no’-bias (Jann, Jerke, Krumpal fc.) §  Seemingly bad performance in self-administered online-mode: §  Lower prevalence estimates than DQ, even negative estimates (Böcherer  et al. 2005, Holbrook & Krosnick 2010, Coutts & Jann 2011) §  Not superior to DQ (but at least not worse…) (Coutts & Jann 2011, Peeters 2006, Snijders & Weesie 2008)     12/03/15 7 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique A New Alternative: The Crosswise Model (Yu et al. 2008) §  Simple idea: Ask a sensitive question and a non-sensitive question and let the respondent indicate whether… §  A: the answer is ‘yes’ to both questions or ‘no’ to both questions §  B: the answer is ‘yes’ to one question and ‘no’ to the other question §  In either case (A,B) the researcher does not know whether a particular respondent’s answer to the sens. question is ‘yes’ or ‘no’. §  The prevalence of the non-sens. question must be unequal to 0.5 and known (furthermore, it must be independent of the sensitive question) §  Examples:             12/03/15 8 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique non-sens. question no yes sens. question no A B yes B A §  Let… §  X be the sensitive question with  §  Y be the non-sensitive question with §  X and Y are independent: Cov(X,Y) = 0 §  Prevalence for observed answer option A (‚same answers‘): §  Prevalence estimate for yes to sensitive question: §  Variance for prevalence estimate:  è Note that, formally, the crosswise model is identical to Warner‘s RRT. Prevalence Estimation for the Crosswise Model 12/03/15 9 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Study A: Experimental Comparison of Crosswise vs. Direct Questioning (Jann, Jerke & Krumpal fc.)  §  Paper & pencil classroom survey on plagiarism at different Universities:   §  ETH Zurich, University of Leipzig, LMU Munich, Spring/Summer 2009 §  Total sample size approx. 500 students §  Experimental conditions: §  ¼ direct questioning §  ¾ crosswise model §  Thanks	  to	  Norman	  Braun	  and	  Jochen	  Gross	  for	  suppor6ng	  the	  data	  collec6on	  at	  LMU	  Munich   12/03/15 10 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Crosswise Item Study A 12/03/15 11 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Results Study A: Prevalence Estimates by Experimental Condition (in %) 12/03/15 12 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Exp.	  Condi6on	   Par6al	  plagiarism	  	   Severe	  plagiarism	  	  1	  Direct	  ques6oning	   7.3	   1.0	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	   se	   2.7	   1.0	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	   N	   96	   96	  2	  Crosswise	   22.3	   1.6	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	   se	   5.5	   5.0	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	   N	   310	   310	  Difference	   15.0	   .6	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	   se	   6.1	   5.1	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	   N	   406	   406	  Study B: Experimental Comparison of Crosswise vs. DQ vs. RRT 12/03/15 13 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique §  Online survey on student cheating and plagiarism:   §  University of Bern, Switzerland, March/April 2011 §  Mailing to all 8’610 students §  2379	  completed questionnaires è response rate of 27.6%  §  Random, balanced assignment of respondents to different experimental conditions: 1.  Direct questioning 2.  RRT forced resp. “Number picking” 3.  RRT forced resp. “Virtual random wheel” 4.  RRT forced resp. innoc. question “Benford” 5.  Crosswise “Number picking” 6.  Crosswise “Innocuous question”  Crosswise Item Study B Donnerstag 12 März 15 14 Asking sensitive questions in online surveys Results Study B: Prevalence Estimates by Experimental Condition (in %) Exp.	  condi6on	   Copying	  in	  exam	  Cheat	  sheet	  in	  exam	  Perf.	  enhancing	  drugs	  for	  exam	  Par6al	  plagiarism	  Severe	  plagiarism	  Direct	  Quest.	   0.20	   0.08	   0.05	   0.02	   0.01	  SE	  	   0.02	   0.01	   0.01	   0.01	   0	  N	  	   393	   393	   392	   288	   289	  RRT	  forced	  resp.	   	  	   0.21	   0.10	   0.00	   0.03	   -­‐0.01	  SE	  	   0.02	   0.02	   0.02	   0.02	   0.02	  N	  	   1139	   1140	   1137	   836	   835	  Crosswise	   0.29	   0.14	   0.12	   0.07	   0.02	  SE	  	   0.03	   0.03	   0.03	   0.03	   0.03	  N	  	   765	   767	   760	   564	   562	  Difference	  CW	  -­‐	  DQ	   0.10	   0.06	   0.08	   0.04	   0.01	  SE	  	   0.04	   0.03	   0.03	   0.03	   0.03	  N	  	   1158	   1160	   1152	   852	   851	  Difference	  CW	  -­‐	  RRT	   0.08	   0.04	   0.12	   0.04	   0.03	  SE	  	   0.04	   0.03	   0.03	   0.04	   0.04	  N	  	   1904	   1907	   1897	   1400	   1397	  CW	  inn.	  quest.	   prev.	  	   0.32	   0.19	   0.15	   0.06	   0.06	  SE	  	   0.05	   0.04	   0.04	   0.05	   0.05	  N	  	   384	   384	   378	   281	   280	  CW	  nbr.	  pick	   prev.	  	   0.26	   0.09	   0.10	   0.07	   -­‐0.02	  SE	  	   0.04	   0.04	   0.04	   0.05	   0.04	  N	  	   381	   383	   382	   283	   282	  12/03/15 15 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Results Study B: Prevalence Estimates for 2 Different Crosswise-Versions – NSQ & RD (in %) Exp.	  condi6on	   Copying	  in	  exam	  Cheat	  sheet	  in	  exam	  Perf.	  enhancing	  drugs	  for	  exam	  Par6al	  plagiarism	  Severe	  plagiarism	  Direct	  Quest.	   prev.	  	   0.20	   0.08	   0.05	   0.02	   0.01	  SE	  	   0.02	   0.01	   0.01	   0.01	   0	  N	  	   393	   393	   392	   288	   289	  Crosswise_NSQ	  	   prev.	  	   0.32	   0.19	   0.15	   0.06	   0.06	  (non-­‐sensi6ve	  ques6on	  version	  of	  Crosswise)	  SE	  	   0.05	   0.04	   0.04	   0.05	   0.05	  N	  	   384	   384	   378	   281	   280	  Crosswise_RD	   prev.	  	   0.26	   0.09	   0.10	   0.07	   -­‐0.02	  (random-­‐device	  version	  of	  Crosswise)	  	  SE	  	   0.04	   0.04	   0.04	   0.05	   0.04	  N	  	   381	   383	   382	   283	   282	  Difference	  CW_NSQ	  -­‐	  DQ	   0.13	   0.10	   0.10	   0.04	   0.05	  SE	  	   0.05	   0.05	   0.05	   0.05	   0.05	  N	  	   777	   777	   770	   569	   569	  Difference	  CW_RD	  -­‐	  DQ	   0.07	   0.01	   0.05	   0.05	   -­‐0.03	  SE	  	   0.05	   0.04	   0.04	   0.05	   0.04	  N	  	   774	   776	   774	   571	   571	  12/03/15 16 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Conclusions §  Crosswise Model clearly outperforms DQ in both studies: Respondents report significantly higher prevalence for sensitive behaviors with the Crosswise Model. §  An exception are items with very low prevalence (plagiarism), where SE are just too high to estimate differences precisely. §  Also the Randomized Response Technique (RRT) in the forced response variant is sign. outperformed by the Crosswise Model in 2 of 5 cheating behaviors and always shows higher prevalence estimates (Study B). §  Contrary to RRT, Crosswise does not seem to suffer from the self-protective no-bias which leads to lower prevalence estimates in some cheating behaviors for the RRT compared to DQ.   12/03/15 17 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Conclusions §  The Crosswise Model, therefore, is a promising and probably superior alternative to RRT in self-administered modes such as P&P or Online. §  We also tested the Crosswise Model in a version with an (explicit) randomizing device instead of the non-sensitive question. However, the result for this variant of the Crosswise Model are somewhat less promising than for the non-sensitive question version of the Crosswise Model. §  Another survey is currently under way and will bring about more evidence to clarify whether this finding is robust.   12/03/15  18 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique Thank you! marc.hoeglinger@soz.gess.ethz.ch jann@soz.unibe.ch    12/03/15 19 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique References Böcherer, G., S. Senn, P. Stadelmann und D. Stutzer. 2005. Wissenschaftliches Arbeiten an der ETH Zürich und die Randomized Response Technik (RRT) bei Online-Umfragen, MTU-Arbeit an der ETH Zürich, Coutts, E. und B. Jann. 2011. Sensitive Questions in Online Surveys: Experimental Results for the Randomized Response Technique (RRT) and the Unmatched Count Technique (UCT), Sociological Methods & Research, Vol. 40, 196-193. Edgell, S. E., S. Himmelfarb und K. L. Duchan. 1982. Validity of forced responses in a randomized-response model, Sociological Methods & Research, Vol. 11, 89-100. Fox, J. A. und P. E. Tracy. 1986. Randomized response: A method for sensitive surveys, Newbury Park, CA: Sage. Holbrook, A. L. und J. A. Krosnick. 2010. Measuring Voter Turnout By Using The Randomized Response Technique: Evidence Calling Into Question The Method's Validity, Public Opin Q, Vol. 74, 328-343. Jann, B., J. Jerke und I. Krumpal. forthcoming. Asking Sensitive Questions Using the Crosswise Model: Some Experimental Results, Public Opinion Quarterly. Lensvelt-Mulders, G. J. L. M. und H. R. Boeije. 2007. Evaluating compliance with a computer assisted randomized response technique: a qualitative study into the origins of lying and cheating, Computers in Human Behavior, Vol. 23, 591-608. Lensvelt-Mulders, G. J. L. M., J. J. Hox, P. G. M. van der Heijden und C. J. M. Maas. 2005. Meta-Analysis of Randomized Response Research: Thirty-Five Years of Validation, Sociological Methods & Research, Vol. 33, 319-348. Locander, W., S. Sudman und N. Bradburn. 1976. An Investigation of Interview Method, Threat and Response Distortion, Journal of the American Statistical Association, Vol. 71, 269-275. Tourangeau, R. und T. Yan. 2007. Sensitive Questions in Surveys, Psychological Bulletin, Vol. 133, 859-883. van der Heijden, P. G. M., G. van Gils, J. Bouts und J. J. Hox. 2000. A Comparison of Randomized Response, Computer-Assisted Self-Interview, and Face-to-Face Direct Questioning. Eliciting Sensitive Information in the Context of Welfare and Unemployment Benefit, Sociological Methods and Research, Vol. 28, 505–537. Warner, S. L. 1965. Randomized-response: A survey technique for eliminating evasive answer bias, Journal of the American Statistical Association, Vol. 60, 63-69. Yu, Jun-Wu, Guo-Liang Tian, Man-Lai Tang (2008). Two new models for survey sampling with sensitive characteristic: design and analysis. Metrika 67(3): 251-263.  §     12/03/15 20 Diekmann, Höglinger, Jann: A New Alternative to the Randomized Response Technique 

Asking sensitive questions: Testing a new alternative to the Randomized Response Technique

http://boris.unibe.ch/9020/1/2011_ASA_Tilburg_V2.pptx.pdf

Asking sensitive questions: Testing a new alternative to the Randomized Response Technique

Abstract

Similar works

Full text

Available Versions

BORIS Portal Bern Open Repository and Information System