Questionnaire development, evaluation, and pretesting research is critical for ensuring that survey
questions, materials, and data collection procedures produce the highest quality data possible.
Interviewer-administered cognitive interviews is a common pretesting method used to collect rich,
qualitative data. As technology has advanced, researchers can conduct similar research online in
self-administered modes (Behr 2016), allowing for pretesting with larger samples. Each approach
has strengths and limitations that researchers can leverage to address their pretesting goals. This
research presents a multi-study, iterative project using traditional and online pretesting to evaluate
new confidentiality language. Study 1 used traditional cognitive interviews to collect information on
respondents’ qualitative reactions to, and comprehension of, the new language, but was limited by
a small sample of prior survey respondents. Study 2 used online testing to help verify the previous
findings with a larger sample, but was limited to hypothetical respondent behaviour. Study 3 used
online testing over two waves of data collection to evaluate actual behaviour over time and
expanded on the previous two studies by using an experimental design. We discuss the utility of
using multiple pretesting methods to complement each other, providing research findings that
would not be possible when using one alone
