Response to Comment on “Estimating the reproducibility of psychological science”

Gilbert et al. conclude that evidence from the Open Science Collaboration's Reproducibility Project: Psychology indicates high reproducibility, given the study methodology. Their very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data. Using the Reproducibility Project: Psychology data, both optimistic and pessimistic conclusions about reproducibility are possible, and neither are yet warranted.status: publishe

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely highpowered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.UCR::Vicerrectoría de Investigación::Unidades de Investigación::Ciencias Sociales::Instituto de Investigaciones Psicológicas (IIP

The International Index of Erectile Function (IIEF)-A Systematic Review of Measurement Properties

Introduction: The International Index of Erectile Function (IIEF) is a patient-reported outcome measure to evaluate erectile dysfunction and other sexual problems in men. Aim: To perform a systematic review of the measurement properties of the 15-item patient-reported outcome measure (IIEF-15) and the shortened 5-item version (IIEF-5). Methods: A systematic search of scientific literature up to April 2018 was performed. Data were extracted and analyzed according to COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) guidelines for structural validity, internal consistency, reliability, measurement error, hypothesis testing for construct validity, and responsiveness. Evidence of measurement properties was categorized into sufficient, insufficient, inconsistent, or indeterminate, and quality of evidence as very high, high, moderate, or low. Results: 40 studies were included. The evidence for criterion validity (of the Erectile Function subscale), and responsiveness of the IIEF-15 was sufficient (high quality), but inconsistent (moderate quality) for structural validity, internal consistency, construct validity, and test-retest reliability. Evidence for structural validity, test-retest reliability, construct validity, and criterion validity of the IIEF-5 was sufficient (moderate quality) but indeterminate for internal consistency, measurement error, and responsiveness. Clinical Implications: Lack of evidence for and evidence not supporting some of the measurement properties of the IIEF-15 and IIEF-5 shows the importance of further research on the validity of these questionnaires in clinical research and clinical practice. Strengths & Limitations: A strength of the current review is the use of predefined guidelines (COSMIN). A limitation of this review is the use of a precise rather than a sensitive search filter regarding measurement properties to identify studies to be included. Conclusion: The IIEF requires more research on structural validity (IIEF-15), internal consistency (IIEF-15 and IIEF-5), construct validity (IIEF-15), measurement error (IIEF-15 and IIEF-5), and responsiveness (IIEF-5). The most pressing matter for future research is determining the unidimensionality of the IIEF-5 and the exact factor structure of the IIEF-15. Neijenhuijs KI, Holtmaat K, Aaronson NK, et al. The International Index of Erectile Function (IIEF)—A Systematic Review of Measurement Properties. J Sex Med 2019;16:1078–1091

The Female Sexual Function Index (FSFI)—A Systematic Review of Measurement Properties

Introduction: The Female Sexual Function Index (FSFI) is a patient-reported outcome measure measuring female sexual dysfunction. The FSFI-19 was developed with 6 theoretical subscales in 2000. In 2010, a shortened version became available (FSFI-6). Aim: To investigate the measurement properties of the FSFI-19 and FSFI-6. Methods: A systematic search was performed of Embase, Medline, and Web of Science for studies that investigated measurement properties of the FSFI-19 or FSFI-6 up to April 2018. Data were extracted and analyzed according to COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) guidelines. Evidence was categorized into sufficient, insufficient, inconsistent, or indeterminate, and quality of evidence as very high, high, moderate, or low. Main Outcome Measures: The Main Outcome Measure is the evidence of a measurement property, and the quality of evidence based on the COSMIN guidelines. Results: 83 studies were included. Concerning the FSFI-19, the evidence for internal consistency was sufficient and of moderate quality. The evidence for reliability was sufficient but of low quality. The evidence for criterion validity was sufficient and of high quality. The evidence for structural validity was inconsistent of low quality. The evidence for construct validity was inconsistent of moderate quality. Concerning the FSFI-6, the evidence for criterion validity was sufficient of moderate quality. The evidence for internal consistency was rated as indeterminate. The evidence for reliability was inconsistent of low quality. The evidence for construct validity was inconsistent of very low quality. No information was available on structural validity of the FSFI-6, and measurement error, responsiveness, and cross-cultural validity of both FSFI-6 and FSFI-19. Clinical Implications: Conflicting and lack of evidence for some of the measurement properties of the FSFI-19 and FSFI-6 indicates the importance of further research on the validity of these patient-reported outcome measures. We advise researchers who use the FSFI-19 to perform confirmatory factor analyses and report the factor structure found in their sample. Regardless of these concerns, the FSFI-19 and FSFI-6 have strong criterion validity. Pragmatically, they are good screening tools for the current definition of female sexual dysfunction. Strength & Limitation: A strong point of the review is the use of predefined guidelines. A limitation is the use of a precise rather than a sensitive search filter. Conclusions: The FSFI requires more research on structural validity (FSFI-19 and FSFI-6), reliability (FSFI-6), construct validity (FSFI-19), measurement error (FSFI-19 and FSFI-6), and responsiveness (FSFI-19 and FSFI-6). Further corroboration of measurement invariance (both across cultures and across subpopulations) in the factor structure of the FSFI-19 is necessary, as well as tests for the unidimensionality of the FSFI-6. Neijenhuijs KI, Hooghiemstra N, Holtmaat K, et al. The Female Sexual Function Index (FSFI)—A Systematic Review of Measurement Properties. J Sex Med 2019;16:640–660

Challenges in Recruiting University Students for Web-Based Indicated Prevention of Depression and Anxiety: Results From a Randomized Controlled Trial (ICare Prevent)

©Felix Bolinski, Annet Kleiboer, Koen Neijenhuijs, Eirini Karyotaki, Reinout Wiers, Lisa de Koning, Corinna Jacobi, Anna-Carlotta Zarski, Kiona K Weisel, Pim Cuijpers, Heleen Riper.Background: Depression and anxiety are common mental health conditions in college and university student populations. Offering transdiagnostic, web-based prevention programs such as ICare Prevent to those with subclinical complaints has the potential to reduce some barriers to receiving help (eg, availability of services, privacy considerations, and students’ desire for autonomy). However, uptake of these interventions is often low, and accounts of recruitment challenges are needed to complement available effectiveness research in student populations. Objective: The aims of this study were to describe recruitment challenges together with effective recruitment strategies for ICare Prevent and provide basic information on the intervention’s effectiveness. Methods: A 3-arm randomized controlled trial was conducted in a student sample with subclinical symptoms of depression and anxiety on the effectiveness of an individually guided (human support and feedback on exercises provided after each session, tailored to each participant) and automatically guided (computer-generated messages provided after each session, geared toward motivation) version of ICare Prevent, a web-based intervention with transdiagnostic components for the indicated prevention of depression and anxiety. The intervention was compared with care as usual. Descriptive statistics were used to outline recruitment challenges and effective web-based and offline strategies as well as students’ use of the intervention. A basic analysis of intervention effects was conducted using a Bayesian linear mixed model, with Bayes factors reported as the effect size. Results: Direct recruitment through students’ email addresses via the central student administration was the most effective strategy. Data from 35 participants were analyzed (individually guided: n=14, 40%; automatically guided: n=8, 23%; care as usual: n=13, 37%). Use of the intervention was low, with an average of 3 out of 7 sessions (SD 2.9) completed. The analyses did not suggest any intervention effects other than anecdotal evidence (all Bayes factors10≤2.7). Conclusions: This report adds to the existing literature on recruitment challenges specific to the student population. Testing the feasibility of recruitment measures and the greater involvement of the target population in their design, as well as shifting from direct to indirect prevention, can potentially help future studies in the field. In addition, this report demonstrates an alternative basic analytical strategy for underpowered randomized controlled trials. Trial Registration: International Clinical Trials Registry Platform NTR6562; https://tinyurl.com/4rbexzrk International Registered Report Identifier (IRRID): RR2-10.1186/s13063-018-2477-

A systematic review of the measurement properties of the European Organisation for Research and Treatment of Cancer In-patient Satisfaction with Care Questionnaire, the EORTC IN-PATSAT32

PURPOSE: The EORTC IN-PATSAT32 is a patient-reported outcome measure (PROM) to assess cancer patients' satisfaction with in-patient health care. The aim of this study was to investigate whether the initial good measurement properties of the IN-PATSAT32 are confirmed in new studies. METHODS: Within the scope of a larger systematic review study (Prospero ID 42017057237), a systematic search was performed of Embase, Medline, PsycINFO, and Web of Science for studies that investigated measurement properties of the IN-PATSAT32 up to July 2017. Study quality was assessed, data were extracted, and synthesized according to the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) methodology. RESULTS: Nine studies were included in this review. The evidence on reliability and construct validity were rated as sufficient and of the quality of the evidence as moderate. The evidence on structural validity was rated as insufficient and of low quality. The evidence on internal consistency was indeterminate. Measurement error, responsiveness, criterion validity, and cross-cultural validity were not reported in the included studies. Measurement error could be calculated for two studies and was judged indeterminate. CONCLUSION: In summary, the IN-PATSAT32 performs as expected with respect to reliability and construct validity. No firm conclusions can be made yet whether the IN-PATSAT32 also performs as well with respect to structural validity and internal consistency. Further research on these measurement properties of the PROM is therefore needed as well as on measurement error, responsiveness, criterion validity, and cross-cultural validity. For future studies, it is recommended to take the COSMIN methodology into account

Challenges in Recruiting University Students for Web-Based Indicated Prevention of Depression and Anxiety: Results From a Randomized Controlled Trial (ICare Prevent)

Background: Depression and anxiety are common mental health conditions in college and university student populations. Offering transdiagnostic, web-based prevention programs such as ICare Prevent to those with subclinical complaints has the potential to reduce some barriers to receiving help (eg, availability of services, privacy considerations, and students’ desire for autonomy). However, uptake of these interventions is often low, and accounts of recruitment challenges are needed to complement available effectiveness research in student populations.  Objective: The aims of this study were to describe recruitment challenges together with effective recruitment strategies for ICare Prevent and provide basic information on the intervention’s effectiveness.  Methods: A 3-arm randomized controlled trial was conducted in a student sample with subclinical symptoms of depression and anxiety on the effectiveness of an individually guided (human support and feedback on exercises provided after each session, tailored to each participant) and automatically guided (computer-generated messages provided after each session, geared toward motivation) version of ICare Prevent, a web-based intervention with transdiagnostic components for the indicated prevention of depression and anxiety. The intervention was compared with care as usual. Descriptive statistics were used to outline recruitment challenges and effective web-based and offline strategies as well as students’ use of the intervention. A basic analysis of intervention effects was conducted using a Bayesian linear mixed model, with Bayes factors reported as the effect size. , Results : Direct recruitment through students’ email addresses via the central student administration was the most effective strategy. Data from 35 participants were analyzed (individually guided: n=14, 40%; automatically guided: n=8, 23%; care as usual: n=13, 37%). Use of the intervention was low, with an average of 3 out of 7 sessions (SD 2.9) completed. The analyses did not suggest any intervention effects other than anecdotal evidence (all Bayes factors10≤2.7).   Conclusions : This report adds to the existing literature on recruitment challenges specific to the student population. Testing the feasibility of recruitment measures and the greater involvement of the target population in their design, as well as shifting from direct to indirect prevention, can potentially help future studies in the field. In addition, this report demonstrates an alternative basic analytical strategy for underpowered randomized controlled trial