Predicting the replicability of social science lab experiments

We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman rho of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to rho = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative

Predicting the replicability of social science lab experiments

We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative

Evaluating replicability of laboratory experiments in economics

The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone

Demand effects of consumers' stated and revealed preferences

Knowledge of how consumers react to different quality signals is fundamental for understanding how markets work. We study the online marketplace for Android apps where we compare the causal effects on demand from two quality related signals; other consumers' stated and revealed preferences toward an app. Our main result is that consumers are much more responsive to other consumers' revealed preferences, compared to others' stated preferences. A 10 percentile increase in displayed average rating only increases downloads by about 3 percent, while a 10 percentile increase in displayed number of downloads increases downloads by about 20 percent

Demand effects of consumers' stated and revealed preferences

Knowledge of how consumers react to different quality signals is fundamental for understanding how markets work. We study the online marketplace for Android apps where we compare the causal effects on demand from two quality related signals; other consumers' stated and revealed preferences toward an app. Our main result is that consumers are much more responsive to other consumers' revealed preferences, compared to others' stated preferences. A 10 percentile increase in displayed average rating only increases downloads by about 3 percent, while a 10 percentile increase in displayed number of downloads increases downloads by about 20 percent

Bostadstillägg för pensionärer : ett randomiserat informationsexperiment

Många äldre med låga inkomster ansöker inte om bostadstillägg för pensionärer trots att de kan ha rätt till det. En viktig fråga är därför hur man kan få fler berättigade att ansöka. Vi har tillsammans med Pensionsmyndigheten genomfört ett randomiserat informationsexperiment riktat till populationen av potentiellt berättigade pensionärer. Ungefär var tionde pensionär som fick ett brev (behandlingsgruppen) ansökte om bostadstillägg inom fyra månader jämfört med drygt en av hundra som inte fick ett brev (kontrollgruppen). Andelen avslag i behandlingsgruppen var dock något högre

Increasing the Take-Up of the Housing Allowance Among Swedish Pensioners : A Field Experiment

Using a randomized field experiment in the Swedish pension system, we investigate whether receiving an information letter affects the take-up rate of the housing allowance for pensioners. We also investigate whether the framing of the information letter affects take-up. The results show that simple information letters had a dramatic effect on the application rate and subsequent take-up rate: the baseline application rate in the targeted control population was only 1.4 percent while the corresponding rates in the different treatment groups were between 9.9 and 12.1 percent. The letter that addressed common misconceptions about the benefit caused significantly higher submission and acceptance rates. The letters had a substantial economic effect on the applicants. We estimate that the applicants, induced by the treatment, increased their monthly income by around 10 percent