Realistic precision and accuracy of online experiment platforms, web browsers, and devices

Funder: University of CambridgeAbstract: Due to increasing ease of use and ability to quickly collect large samples, online behavioural research is currently booming. With this popularity, it is important that researchers are aware of who online participants are, and what devices and software they use to access experiments. While it is somewhat obvious that these factors can impact data quality, the magnitude of the problem remains unclear. To understand how these characteristics impact experiment presentation and data quality, we performed a battery of automated tests on a number of realistic set-ups. We investigated how different web-building platforms (Gorilla v.20190828, jsPsych v6.0.5, Lab.js v19.1.0, and psychoJS/PsychoPy3 v3.1.5), browsers (Chrome, Edge, Firefox, and Safari), and operating systems (macOS and Windows 10) impact display time across 30 different frame durations for each software combination. We then employed a robot actuator in realistic set-ups to measure response recording across the aforementioned platforms, and between different keyboard types (desktop and integrated laptop). Finally, we analysed data from over 200,000 participants on their demographics, technology, and software to provide context to our findings. We found that modern web platforms provide reasonable accuracy and precision for display duration and manual response time, and that no single platform stands out as the best in all features and conditions. In addition, our online participant analysis shows what equipment they are likely to use

Gorilla in our midst: An online behavioral experiment builder

Abstract: Behavioral researchers are increasingly conducting their studies online, to gain access to large and diverse samples that would be difficult to get in a laboratory environment. However, there are technical access barriers to building experiments online, and web browsers can present problems for consistent timing—an important issue with reaction-time-sensitive measures. For example, to ensure accuracy and test–retest reliability in presentation and response recording, experimenters need a working knowledge of programming languages such as JavaScript. We review some of the previous and current tools for online behavioral research, as well as how well they address the issues of usability and timing. We then present the Gorilla Experiment Builder (gorilla.sc), a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues and make reliable online experimentation open and accessible to a wider range of technical abilities. To demonstrate the platform’s aptitude for accessible, reliable, and scalable research, we administered a task with a range of participant groups (primary school children and adults), settings (without supervision, at home, and under supervision, in both schools and public engagement events), equipment (participant’s own computer, computer supplied by the researcher), and connection types (personal internet connection, mobile phone 3G/4G). We used a simplified flanker task taken from the attentional network task (Rueda, Posner, & Rothbart, 2004). We replicated the “conflict network” effect in all these populations, demonstrating the platform’s capability to run reaction-time-sensitive experiments. Unresolved limitations of running experiments online are then discussed, along with potential solutions and some future features of the platform

What over 1,000,000 participants tell us about online research protocols

With the ever-increasing adoption of tools for online research, for the first time we have visibility on macro-level trends in research that were previously unattainable. However, until now this data has been siloed within company databases and unavailable to researchers. Between them, the online study creation and hosting tool Gorilla Experiment Builder and the recruitment platform Prolific hold metadata gleaned from millions of participants and over half a million studies. We analyzed a subset of this data (over 1 million participants and half a million studies) to reveal critical information about the current state of the online research landscape that researchers can use to inform their own study planning and execution. We analyzed this data to discover basic benchmarking statistics about online research that all researchers conducting their work online may be interested to know. In doing so, we identified insights related to: the typical study length, average completion rates within studies, the most frequent sample sizes, the most popular participant filters, and gross participant activity levels. We present this data in the hope that it can be used to inform research choices going forward and provide a snapshot of the current state of online research

Tracking seasonal changes in North Sea zooplankton trophic dynamics using stable isotopes

Trophodynamics of meso-zooplankton in the North Sea (NS) were assessed at a site in the southern NS, and at a shallow and a deep site in the central NS. Offshore and neritic species from different ecological niches, including Calanus spp., Temora spp. and Sagitta spp., were collected during seven cruises over 14 months from 2007 to 2008. Bulk stable isotope (SI) analysis, phospholipid-derived fatty acid (PLFA) compositions, and δ 13CPLFA data of meso-zooplankton and particulate organic matter (POM) were used to describe changes in zooplankton relative trophic positions (RTPs) and trophodynamics. The aim of the study was to test the hypothesis that the RTPs of zooplankton in the North Sea vary spatially and seasonally, in response to hydrographic variability, with the microbial food web playing an important role at times. Zooplankton RTPs tended to be higher during winter and lower during the phytoplankton bloom in spring. RTPs were highest for predators such as Sagitta sp. and Calanus helgolandicus and lowest for small copepods such as Pseudocalanus elongatus and zoea larvae (Brachyura). δ 15NPOM-based RTPs were only moderate surrogates for animals’ ecological niches, because of the plasticity in source materials from the herbivorous and the microbial loop food web. Common (16:0) and essential (eicosapentaenoic acid, EPA and docosahexaenoic acid, DHA) structural lipids showed relatively constant abundances. This could be explained by incorporation of PLFAs with δ 13C signatures which followed seasonal changes in bulk δ 13CPOM and PLFA δ 13CPOM signatures. This study highlighted the complementarity of three biogeochemical approaches for trophodynamic studies and substantiated conceptual views of size-based food web analysis, in which small individuals of large species may be functionally equivalent to large individuals of small species. Seasonal and spatial variability was also important in altering the relative importance of the herbivorous and microbial food webs

What over 1,000,000 participants tells us about online research protocols

Dataset shared from Gorilla Experiment Builder and Prolific's data warehouses which provides a snapshot of online research methods

GC-MS method for determining faecal sterols as biomarkers of human and pastoral animal presence in freshwater sediments

In order to determine sterols and stanols in freshwater sediments to reconstruct the past presence of humans and pastoral animals, we developed an analytical method based on pressurised liquid extraction (PLE), clean-up performed using solid phase extraction (SPE) and sterol determination using gas chromatography-mass spectrometry (GCMS) analysis. PLE extraction conditions were optimised using dichloromethane (DCM) and DCM/methanol mixtures. Clean-up was performed with 2 g silica SPE cartridges, and the concentrated extracts were eluted with 70 mL DCM. Extraction yield was evaluated using an in-house reference material spiked with C-13-labelled cholesterol and aged for 10 days. In comparison with pre-extraction, where the sediment is extracted and then spiked with a known analyte concentration, this approach preserves the original composition of the sediment. DCM and DCM/methanol mixtures resulted in high extraction yields ranging from 86 to 92 % with good reproducibility (relative standard deviation (RSD) 5-8 %). PLE extraction yields obtained with DCM as the extracting solvent were about 1.5 times higher than extractions using an ultrasonic bath. The solvent extraction mixture and matrix composition strongly affected the solvent extraction composition where higher overall recoveries (70-80 %) for each compound were obtained with DCM. The extraction mixture and matrix composition also affected the analyte concentrations, resulting in a method precision ranging from 1 to 18 %. Diatomaceous earth spiked with 10 to 100 ng of sterols, and environmental samples fortified with suitable amounts of sterols provided apparent recovery values ranging from 90 to 110 %. We applied the method to environmental samples both close to and upstream from sewage discharge zones, resulting in substantially higher faecal sterol (FeSt) concentrations near the sewage. In addition, we also applied the method to a 37cm freshwater sediment core in order to evaluate its applicability for obtaining vertical sterol profiles

New Insights into the Evolution of the Human Diet from Faecal Biomarker Analysis in Wild Chimpanzee and Gorilla Faeces.

Our understanding of early human diets is based on reconstructed biomechanics of hominin jaws, bone and teeth isotopic data, tooth wear patterns, lithic, taphonomic and zooarchaeological data, which do not provide information about the relative amounts of different types of foods that contributed most to early human diets. Faecal biomarkers are proving to be a valuable tool in identifying relative proportions of plant and animal tissues in Palaeolithic diets. A limiting factor in the application of the faecal biomarker approach is the striking absence of data related to the occurrence of faecal biomarkers in non-human primate faeces. In this study we explored the nature and proportions of sterols and stanols excreted by our closest living relatives. This investigation reports the first faecal biomarker data for wild chimpanzee (Pan troglodytes) and mountain gorilla (Gorilla beringei). Our results suggest that the chemometric analysis of faecal biomarkers is a useful tool for distinguishing between NHP and human faecal matter, and hence, it could provide information for palaeodietary research and early human diets