Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p <.05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δr =.002 or.014, depending on analytic approach). The median effect size for the revised protocols (r =.05) was similar to that of the RP:P protocols (r =.04) and the original RP:P replications (r =.11), and smaller than that of the original studies (r =.37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r =.07, range =.00–.15) were 78% smaller, on average, than the original effect sizes (median r =.37, range =.19–.50)

Many Labs 5:Testing pre-data collection peer review as an intervention to increase replicability

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3?9; median total sample = 1,279.5, range = 276?3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (?r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00?.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19?.50)

Magnetic resonance spectroscopy detects differential lipid composition in mammary glands on low fat, high animal fat versus high fructose diets.

The effects of consumption of different diets on the fatty acid composition in the mammary glands of SV40 T-antigen (Tag) transgenic mice, a well-established model of human triple-negative breast cancer, were investigated with magnetic resonance spectroscopy and spectroscopic imaging. Female C3(1) SV40 Tag transgenic mice (n = 12) were divided into three groups at 4 weeks of age: low fat diet (LFD), high animal fat diet (HAFD), and high fructose diet (HFruD). MRI scans of mammary glands were acquired with a 9.4 T scanner after 8 weeks on the diet. 1H spectra were acquired using point resolved spectroscopy (PRESS) from two 1 mm3 boxes on each side of inguinal mammary gland with no cancers, lymph nodes, or lymph ducts. High spectral and spatial resolution (HiSS) images were also acquired from nine 1-mm slices. A combination of Gaussian and Lorentzian functions was used to fit the spectra. The percentages of poly-unsaturated fatty acids (PUFA), mono-unsaturated fatty acids (MUFA), and saturated fatty acids (SFA) were calculated from each fitted spectrum. Water and fat peak height images (maps) were generated from HiSS data. The results showed that HAFD mice had significantly lower PUFA than both LFD (p < 0.001) and HFruD (p < 0.01) mice. The mammary lipid quantity calculated from 1H spectra was much larger in HAFD mice than in LFD (p = 0.03) but similar to HFruD mice (p = 0.10). The average fat signal intensity over the mammary glands calculated from HiSS fat maps was ~60% higher in HAFD mice than in LFD (p = 0.04) mice. The mean or median of calculated parameters for the HFruD mice were between those for LFD and HAFD mice. Therefore, PRESS spectroscopy and HiSS MRI demonstrated water and fat composition changes in mammary glands due to a Western diet, which was low in potassium, high in sodium, animal fat, and simple carbohydrates. Measurements of PUFA with MRI could be used to evaluate cancer risk, improve cancer detection and diagnosis, and guide preventative therapy

Magnetic resonance spectroscopy detects differential lipid composition in mammary glands on low fat, high animal fat versus high fructose diets

The effects of consumption of different diets on the fatty acid composition in the mammary glands of SV40 T-antigen (Tag) transgenic mice, a well-established model of human triple-negative breast cancer, were investigated with magnetic resonance spectroscopy and spectroscopic imaging. Female C3(1) SV40 Tag transgenic mice (n = 12) were divided into three groups at 4 weeks of age: low fat diet (LFD), high animal fat diet (HAFD), and high fructose diet (HFruD). MRI scans of mammary glands were acquired with a 9.4 T scanner after 8 weeks on the diet. 1H spectra were acquired using point resolved spectroscopy (PRESS) from two 1 mm3 boxes on each side of inguinal mammary gland with no cancers, lymph nodes, or lymph ducts. High spectral and spatial resolution (HiSS) images were also acquired from nine 1-mm slices. A combination of Gaussian and Lorentzian functions was used to fit the spectra. The percentages of poly-unsaturated fatty acids (PUFA), mono-unsaturated fatty acids (MUFA), and saturated fatty acids (SFA) were calculated from each fitted spectrum. Water and fat peak height images (maps) were generated from HiSS data. The results showed that HAFD mice had significantly lower PUFA than both LFD (p 1H spectra was much larger in HAFD mice than in LFD (p = 0.03) but similar to HFruD mice (p = 0.10). The average fat signal intensity over the mammary glands calculated from HiSS fat maps was ~60% higher in HAFD mice than in LFD (p = 0.04) mice. The mean or median of calculated parameters for the HFruD mice were between those for LFD and HAFD mice. Therefore, PRESS spectroscopy and HiSS MRI demonstrated water and fat composition changes in mammary glands due to a Western diet, which was low in potassium, high in sodium, animal fat, and simple carbohydrates. Measurements of PUFA with MRI could be used to evaluate cancer risk, improve cancer detection and diagnosis, and guide preventative therapy

Example of T2W images (top row), water peak height (PH) images (maps) (middle row), and fat peak height (PH) images (maps) (bottom row) for LFD (left panel), HAFD (middle panel), and HFruD (right panel) fed SV40 mice.

<p>The color bar indicates the normalized signal intensity values, which were obtained by dividing the average of water peak height intensity over the muscle ROI.</p