We present an increasingly stringent set of replications of Ghitza & Gelman
(2013), a multilevel regression and poststratification analysis of polls from
the 2008 U.S. presidential election campaign, focusing on a set of plots
showing the estimated Republican vote share for whites and for all voters, as a
function of income level in each of the states.
We start with a nearly-exact duplication that uses the posted code and
changes only the model-fitting algorithm; we then replicate using
already-analyzed data from 2004; and finally we set up preregistered
replications using two surveys from 2008 that we had not previously looked at.
We have already learned from our preliminary, non-preregistered replication,
which has revealed a potential problem with the published analysis of Ghitza &
Gelman (2013); it appears that our model may not sufficiently account for
nonsampling error, and that some of the patterns presented in that earlier
paper may simply reflect noise.
In addition to the substantive interest in validating earlier findings about
demographics, geography, and voting, the present project serves as a
demonstration of preregistration in a setting where the subject matter is
historical (and thus the replication data exist before the preregistration plan
is written) and where the analysis is exploratory (and thus a replication
cannot be simply deemed successful or unsuccessful based on the statistical
significance of some particular comparison).Comment: This article is a review and preregistration plan. It will be
published, along with a new Section 5 describing the results of the
preregistered analysis, in Statistics and Public Polic
