Exploring new avenues for the meta-analysis method in personality and social psychology research

This dissertation addresses theoretical validity and bias in meta-analytic research in personality and social psychology research. The conceptual starting point of the dissertation is research on ego depletion (Baumeister et al., 1998). In this line of research, hundreds of studies documented an experimental effect that probably does not exist, as was later revealed by extensive replication work (Hagger et al., 2010, 2016). This debacle has presumably been caused by dysfunctional structures and procedures in psychological science, such as widespread publication bias (Carter & McCullough, 2014). Unfortunately, these dysfunctionalities were (and in some cases still are) also prevalent in other areas of psychological research beside ego depletion (Ferguson & Brannick, 2012; Open Science Collaboration, 2015). Because extensive replication research is too costly to be repeated for all past work, it has been a contentious question what to do with research data that has been generated during an era of questionable research practices: should this research be abandoned or can some of it be salvaged? In four research papers, this dissertation project attempts to address these questions. In part I of the dissertation project, two papers highlight and analyze challenges when summarizing past research in social psychology and personality research. Paper 1 (Friese et al., 2017) attempted to find summary evidence for the effectiveness of self-control training, a research field related to ego depletion, but came to a sobering conclusion: The summary effect was small, likely inflated by publication bias, and could not be attributed beyond doubt to a theoretical mechanism. Paper 2 (Friese & Frankenbach, 2020) reported on a simulation study that showed how multiple sources of bias (publication bias, p-hacking) can interact with contextual factors and each other to create significant meta-analytic evidence from very small or even zero true effects. Part II of the dissertation project is an attempt to advance social-psychological and personality theory with meta-scientific work despite an unknowable risk of bias in the literature. In part II, two papers (Frankenbach et al., 2020, 2022) make use of one key idea: Re-using existing raw research data to test novel theoretical ideas in secondary (meta-)analyses. Results revealed that this idea helps towards both goals of the dissertation project, that is, advancing theory while reducing risk-of-bias. The general discussion analyses promises and limitations of such secondary data analyses in more detail and attempts to situate the idea more broadly in the psychological research toolkit by contrasting integrative versus innovative research. Further discussion covers how conceptual and technological innovations may facilitate more secondary data analyses in the future, and how such advances may pave the way for a slower, more incremental, but truly valid and cumulative psychological science.Die vorliegende Dissertation behandelt theoretischen Validität und Verzerrung (Bias) von meta-analytischer Forschung in der Persönlichkeits- und Sozialpsychologie. Der konzeptuelle Ausgangspunkt der Dissertation ist die Forschung zu „Ego Depletion“ (Baumeister et al., 1998). In dieser Forschungslinie haben hunderte von Studien einen Effekt belegt, der, wie sich später durch umfangreiche Replikationsarbeiten (Hagger et al., 2010, 2016) herausstellte, vermutlich nicht existiert. Dieses Debakel wurde mutmaßlich mitverursacht durch dysfunktionale Strukturen und Prozesse in der psychologischen Forschung, insbesondere Publikationsbias („publication bias“). Unglücklicherweise lagen (und liegen) diese Dysfunktionalitäten neben Ego Depletion auch in anderen psychologischen Forschungsbereichen vor (Ferguson & Brannick, 2012; Open Science Collaboration, 2015). Da aus Kostengründen nicht alle Forschungsarbeiten der Vergangenheit repliziert werden können, ergibt sich eine kritische Frage: Wie soll mit psychologischer Forschung umgegangen werden, die unter mutmaßlich verzerrenden Bedingungen generiert wurde? Sollte diese Forschung ad acta gelegt werden oder können Teile davon weiterverwendet werden? Das vorliegende Dissertationsprojekt versucht im Rahmen von vier Forschungsbeiträgen sich diesen Fragen anzunähern. Im ersten Teil der Dissertation beleuchten und analysieren zwei Forschungsbeiträge Probleme und Herausforderungen, die sich bei der Zusammenfassung von bestehender Forschung der Sozial- und Persönlichkeitspsychologie ergeben. Der erste Beitrag (Friese et al., 2017) versucht in einer Meta-Analyse Evidenz für die Wirksamkeit von Selbstkontrolltrainings zu finden, aber kommt zu einem ernüchternden Ergebnis: Die Gesamteffekte sind klein, mutmaßlich durch Publikationsbias fälschlich überhöht und können überdies nicht zweifelsfrei einem theoretischen Kausalmechanismus zugeordnet werden. Der zweite Beitrag (Friese & Frankenbach, 2020) umfasst eine Simulationsstudie, die aufzeigt, wie verschiedene Formen von Bias (Publikationsbias und sog. „p-hacking“) miteinander und mit Kontextfaktoren interagieren können, wodurch signifikante, meta-analytische Effekte aus sehr kleinen wahren Effekten oder sogar Nulleffekten entstehen können. Der zweite Teil der Dissertation versucht, trotz eines unbestimmbaren Bias-Risikos, Fortschritte in der sozial- und persönlichkeitspsychologischen Theorie zu erzielen. Zu diesem Zweck wird in zwei Forschungsbeiträgen (Frankenbach et al., 2020, 2022) auf eine Schlüssel-Idee zurückgegriffen: Die Testung von neuen theoretischen Hypothesen unter Wiederverwendung von existierenden Forschungsdaten in Sekundärdatenanalysen. Die Ergebnisse zeigen, dass dieser Ansatz tatsächlich dazu beitragen kann, theoretische Fortschritte mit vermindertem Verzerrungsrisiko zu machen. Die anschließende, übergreifende Diskussion behandelt Möglichkeiten und Limitationen solcher Sekundärdatenanalysen und versucht, den Ansatz in einer Gegenüberstellung von integrativer und innovativer Forschung übergreifender in die psychologische Forschungsmethodik einzuordnen. Im Weiteren wird diskutiert, wie konzeptuelle und technologische Entwicklungen in der Zukunft Sekundärdatenanalysen erleichtern könnten und wie solche Fortschritte den Weg ebnen könnten für eine langsamere, inkrementelle, aber wahrhaft valide und kumulative psychologische Wissenschaft.German Research Foundation (DFG): "Die Rolle mentaler Anstrengung bei Ego Depletion

Nostalgia and Neuroticism Meta-Analysis

Nostalgia, a sentimental longing or wistful affection for the past, conveys self-oriented, existential, and social benefits. We examined whether nostalgic engagement is less beneficial for individuals who are high in neuroticism (i.e., those who are emotionally unstable and prone to negative affect). Specifically, we tested whether the benefits of experimentally induced nostalgia are moderated by trait-level neuroticism. To address this, we conducted a high-powered meta-analysis (N = 3556, k = 19), and found that the benefits of nostalgia were not significantly moderated by neuroticism, as they emerged for both high- and low-neurotics. This conclusion was supported when the self-oriented, existential, and social benefits of nostalgia were analyzed jointly and when they were analyzed separately. Taken together, individuals high and low in neuroticism are equally likely to benefit psychologically from engagement in nostalgic reverie

p-Hacking and Publication Bias Interact to Distort Meta-Analytic Effect Size Estimates

Science depends on trustworthy evidence. Thus, a biased scientific record is of questionable value because it impedes scientific progress, and the public receives advice on the basis of unreliable evidence that has the potential to have far-reaching detrimental consequences. Meta-analysis is a valid and reliable technique that can be used to summarize research evidence. However, meta-analytic effect size estimates may themselves be biased, threatening the validity and usefulness of meta-analyses to promote scientific progress. Here, we offer a large-scale simulation study to elucidate how p-hacking and publication bias distort meta-analytic effect size estimates under a broad array of circumstances that reflect the reality that exists across a variety of research areas. The results revealed that, first, very high levels of publication bias can severely distort the cumulative evidence. Second, p-hacking and publication bias interact: At relatively high and low levels of publication bias, p-hacking does comparatively little harm, but at medium levels of publication bias, p-hacking can considerably contribute to bias, especially when the true effects are very small or are approaching zero. Third, p-hacking can severely increase the rate of false positives. A key implication is that, in addition to preventing p-hacking, policies in research institutions, funding agencies, and scientific journals need to make the prevention of publication bias a top priority to ensure a trustworthy base of evidence

Sex drive: Theoretical conceptualization and meta-analytic review of gender differences

Few spheres in life are as universally relevant for (almost) all individuals past puberty as sexuality. One important aspect of sexuality concerns individuals’ sex drive—their dispositional sexual motivation. A vigorous scientific (and popular) debate revolves around the question of whether or not there is a gender difference in sex drive. Several theories predict a higher sex drive in men compared to women, with some theories attributing this difference to biased responding rather than true differences. Currently, there is little consensus on how to conceptualize sex drive, nor does a quantitative summary of the literature exist. In this paper, we present a theory-driven conceptualization of sex drive as the density distribution of state sex drive, where state sex drive is defined as momentary sexual motivation that manifests in sexual cognition, affect, and behavior. We conduct a comprehensive meta-analysis of gender differences in sex drive based on 211 studies, 856 effect sizes, and 621,463 persons. The meta-analysis revealed a stronger sex drive in men compared to women, with a medium-to-large effect size (g = 0.69, CI95 [0.58, 0.81]). Men more often think and fantasize about sex, more often experience sexual affect like desire, and more often engage in masturbation than women. Adjustment for biased responding reduced the gender difference (g = 0.54). Moderation analyses suggest that the effect is robust and largely invariant to contextual factors. There was no evidence of publication bias. The discussion focuses on validity considerations, limitations, and implications for psychological theory and people’s everyday lives