The Crowdsourced Replication Initiative: Investigating Immigration and Social Policy Preferences. Executive Report.

In an era of mass migration, social scientists, populist parties and social movements raise concerns over the future of immigration-destination societies. What impacts does this have on policy and social solidarity? Comparative cross-national research, relying mostly on secondary data, has findings in different directions. There is a threat of selective model reporting and lack of replicability. The heterogeneity of countries obscures attempts to clearly define data-generating models. P-hacking and HARKing lurk among standard research practices in this area.This project employs crowdsourcing to address these issues. It draws on replication, deliberation, meta-analysis and harnessing the power of many minds at once. The Crowdsourced Replication Initiative carries two main goals, (a) to better investigate the linkage between immigration and social policy preferences across countries, and (b) to develop crowdsourcing as a social science method. The Executive Report provides short reviews of the area of social policy preferences and immigration, and the methods and impetus behind crowdsourcing plus a description of the entire project. Three main areas of findings will appear in three papers, that are registered as PAPs or in process

Demonstration-based training (DBT) in the design of a video tutorial for software training

This study investigates the design and effectiveness of a video tutorial for software training. In accordance with demonstration-based training, the tutorial consisted of a series of task demonstrations, with instructional features added to enhance learning. An experiment is reported in which a demonstration-only control condition was compared with a demonstration-with-review condition. The review videos provided a recap to support retention of the procedure. Strict viewing conditions were imposed during training. Both learning and motivation were assessed. Participants were 55 students from primary and secondary schools in Germany. The outcomes showed that both tutorials significantly contributed to development of procedural knowledge and to motivation, and that the review tutorial surpassed the control. The discussion addresses the role of user control. In addition, with the effectiveness of reviews apparently becoming better established, the need arises for measures that can reveal the effects of reviews on observational learning processes

The Crowdsourced Replication Initiative: Investigating Immigration and Social Policy Preferences. Executive Report

Breznau N, Rinke EM, Wuttke A, et al. The Crowdsourced Replication Initiative: Investigating Immigration and Social Policy Preferences. Executive Report. 2019

How Many Replicators Does It Take to Achieve Reliability? Investigating Researcher Variability in a Crowdsourced Replication

The paper reports findings from a crowdsourced replication. Eighty-four replicator teams attempted to verify results reported in an original study by running the same models with the same data. The replication involved an experimental condition. A “transparent” group received the original study and code, and an “opaque” group received the same underlying study but with only a methods section and description of the regression coefficients without size or significance, and no code. The transparent group mostly verified the original study (95.5%), while the opaque group had less success (89.4%). Qualitative investigation of the replicators’ workflows reveals many causes of non-verification. Two categories of these causes are hypothesized, routine and non-routine. After correcting non-routine errors in the research process to ensure that the results reflect a level of quality that should be present in ‘real-world’ research, the rate of verification was 96.1 in the transparent group and 92.4 in the opaque group. Two conclusions follow: (1) Although high, the verification rate suggests that it would take a minimum of three replicators per study to achieve replication reliability of at least 95 confidence assuming ecological validity in this controlled setting, and (2) like any type of scientific research, replication is prone to errors that derive from routine and undeliberate actions in the research process. The latter suggests that idiosyncratic researcher variability might provide a key to understanding part of the “reliability crisis” in social and behavioral science and is a reminder of the importance of transparent and well documented workflows

NSF 19-501 AccelNet Proposal: Community of Open Scholarship Grassroots Networks (COSGN)

The Community of Open Scholarship Grassroots Networks (COSGN), includes 120 grassroots networks, representing virtually every region of the world and every research discipline. These networks communicate and coordinate on topics of common interest. We propose, using an NSF 19-501 Full-Scale implementation grant, to formalize governance and coordination of the networks to maximize impact and establish standard practices for sustainability. In the project period, we will increase the capacity of COSGN to advance the research and community goals of the participating networks individually and collectively, and establish governance, succession planning, shared resources, andcommunication pathways to ensure an active, community-sustained network of networks. By the end of the project period, we will have established a self-sustaining network of networks that leverages disciplinary and regional diversity, actively collaborates across networks for grassroots organizing, and shares resources for maximum impact on culture change for open scholarship