Presenting Survey Items One at a Time Compared to All at Once Decreases Missing Data without Sacrificing Validity in Research with Internet Volunteers

In two large web-based studies, across five distinct criteria, presenting survey items one-at-a-time was psychometrically either the same or better than presenting survey items all-at-once on a single web page to volunteer participants. In the one-at-a-time format, participants were no more likely to drop-out of the study (Criterion 1), and were much more likely to provide answers for the survey items (Criterion 2). Rehabilitating participants who otherwise would not have provided survey responses with the one-at-a-time format did not damage internal consistency of the measures (Criterion 3) nor did it negatively affect criterion validity (Criterion 4). Finally, the one-at-a-time format was more efficient with participants completing it more quickly than the all-at-once format (Criterion 5). In short, the one-at-a-time format results in less missing data with a shorter presentation time, and ultimately more power to detect relations among variables

Visual representation of the <i>one-at-a-time</i> format for a single item from the Social Dominance Orientation survey.

<p>Visual representation of the <i>one-at-a-time</i> format for a single item from the Social Dominance Orientation survey.</p

Time to complete study materials (in seconds) for each survey format separately for sessions that included the MFQ or did not.

<p>Notes: Total time to complete the study included instructions and the IAT. <i>SD</i>s in parentheses. Sample includes only participants that finished the study and completed all items in all surveys.</p

Zero-order correlations with criterion validity variables for the five subscales of the Moral Foundations Questionnaire by survey format condition.

<p>Notes: Gender (Male = 1, Female = 2), Social politics (1 = strongly liberal, 7 = strongly conservative). Samples include only participants that completed all MFQ items. All correlations >.05 were significant <i>p</i><.0001. Testing significant differences in correlations between conditions showed only 3 of 10 tests being significantly different: politics effect for purity subscale being stronger for <i>all-at-once</i> versus <i>one-at-a-time</i> (<i>p</i> = .006), and gender effect for authority (<i>p</i> = .025) and purity (<i>p</i><.001) subscales being stronger for <i>one-at-a-time</i> than <i>all-at-once</i>.</p

Alpha internal consistency coefficients for the five subscales of the Moral Foundations Questionnaire by survey format condition.

<p>Note: Samples include only participants that completed all MFQ items. A bootstrap comparison of alphas with 5000 runs each demonstrated that all five subscales had higher reliability in the one-at-a-time compared to all-at-once conditions (<i>p</i>’s <.001).</p