The Underappreciated Effects of Unreliability on Multiple Regression and Mediation

There is an increasing trend for researchers in the social sciences to draw causal conclusions from correlational data. Even researchers who use relatively causally neutral language in describing their findings, imply causation by including diagrams with arrows. Moreover, they typically make recommendations for intervention or other applications in their discussion sections, that would make no sense without an implicit assumption that the findings really do indicate causal pathways. The present manuscript commences with the generous assumption that regression-based procedures extract causation out of correlational data, with an exploration of the surprising effects of unreliability on causal conclusions. After discussing the pros and cons of correcting for unreliability, the generous assumption is questioned too. The conclusion is that researchers should be more cautious in interpreting findings based on correlational research paradigms

A defense against the alleged unreliability of difference scores

Based on a classical true score theory (classical test theory, CTT) equation, indicating that as the observed correlation between two tests increases, the reliability of the difference scores decreases, researchers have concluded that difference scores are unreliable. But CTT shows that the reliabilities of the two tests and the true correlation between them influence the observed correlation and previous analyses have not taken the true correlation sufficiently into account. In turn, the reliability of difference scores depends on the interaction of the reliabilities of the individual tests and their true correlation when the variances of the tests are equal, and on a more complicated interaction between them and the deviation ratio when the variances of the tests are not equal. The upshot is that difference scores likely are more reliable, on more occasions, than researchers have realized. I show how researchers can predict what the reliability of the difference scores is likely to be, to aid in deciding whether to carry through one’s planned use of difference scores.https://deepblue.lib.umich.edu/bitstream/2027.42/136924/1/Trafimow(2015)ADefenseAgainst.pdfDescription of Trafimow(2015)ADefenseAgainst.pdf : Main Articl

Situation-specific expectancies in person memory

Thesis (B.S.) in Liberal Arts and Sciences--University of Illinois at Urbana-Champaign, 1984.Bibliography: leaves 38-40.Microfiche of typescript. [Urbana, Ill.] : Photographic Services, University of Illinois, U of I Library, [1987]. 2 microfiches (78 frames) negative ; 11 x 15 cm

Inferential statistics as descriptive statistics: there is no replication crisis if we don't expect replication

Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a "replication crisis" may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as providing generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced

Why we habitually engage in null-hypothesis significance testing:A qualitative study

BACKGROUND: Null Hypothesis Significance Testing (NHST) is the most familiar statistical procedure for making inferences about population effects. Important problems associated with this method have been addressed and various alternatives that overcome these problems have been developed. Despite its many well-documented drawbacks, NHST remains the prevailing method for drawing conclusions from data. Reasons for this have been insufficiently investigated. Therefore, the aim of our study was to explore the perceived barriers and facilitators related to the use of NHST and alternative statistical procedures among relevant stakeholders in the scientific system. METHODS: Individual semi-structured interviews and focus groups were conducted with junior and senior researchers, lecturers in statistics, editors of scientific journals and program leaders of funding agencies. During the focus groups, important themes that emerged from the interviews were discussed. Data analysis was performed using the constant comparison method, allowing emerging (sub)themes to be fully explored. A theory substantiating the prevailing use of NHST was developed based on the main themes and subthemes we identified. RESULTS: Twenty-nine interviews and six focus groups were conducted. Several interrelated facilitators and barriers associated with the use of NHST and alternative statistical procedures were identified. These factors were subsumed under three main themes: the scientific climate, scientific duty, and reactivity. As a result of the factors, most participants feel dependent in their actions upon others, have become reactive, and await action and initiatives from others. This may explain why NHST is still the standard and ubiquitously used by almost everyone involved. CONCLUSION: Our findings demonstrate how perceived barriers to shift away from NHST set a high threshold for actual behavioral change and create a circle of interdependency between stakeholders. By taking small steps it should be possible to decrease the scientific community’s strong dependence on NHST and p-values

Predicting Faculty Intentions to Assign Writing in Their Classes

Teachers who offer undergraduate courses agree widely on the importance of writing assignments to further undergraduate education. And yet, there is a great deal of variance among teachers in their writing assignments; some teachers assign no writing whatsoever. To determine the variables that influence the decisions of teachers about whether to assign writing, we predicted their intentions to assign writing from attitudes, subjective norms, perceived control, and perceived difficulty pertaining to assigning writing. Zero-order correlations and hierarchical regression analyses implicate attitude and perceived difficulty as the most important predictors of teacher’s intentions to assign writing in two studies. We also obtained open-ended belief statements in Study 1 and used them to obtain quantitative belief data in Study 2 to find and validate the importance of the impact of particular specific beliefs on intentions to assign writing

The theory of planned behaviour predicts self-reports of walking, but does not predict step count

Objectives This paper compares multiple measures of walking in two studies, and the second study compares how well Theory of Planned Behaviour (TPB) constructs perform in predicting these different measures. Methods In Study 1, 41 participants wore a New Lifestyles NL-2000 pedometer for 1 week. Subsequently, participants completed a questionnaire containing measures of the TPB constructs and two self-report measures of walking, followed by two interview measures of walking. For Study 2, 200 RAF trainee aircraftsmen wore pedometers for 2 weeks. At the end of each week, participants completed the questionnaire and interview measures of walking. Results Both studies found no significant association between questionnaire measures of walking and pedometer measures. In Study 1, the interview measures produced significant, large correlations with the pedometer measure, but these relationships were markedly weaker in the second study. TPB variables were found to explain 22% of variance in intention to walk in Study 1 and 45% of the variance in Study 2. In Study 2, prediction of subsequent measures of behaviour was found to be weak, except when using a single-item measure of walking. Conclusions Recall of walking is poor, and accurate measurement by self-report is problematic. Although the TPB predicts intentions to walk well, it does not predict actual amount of walking, as assessed by pedometer. Possible reasons for these findings include the unique nature of walking as an activity primarily used to facilitate higher order goals. The use of single-item measures may exaggerate the effectiveness of the TPB model for walking, and possibly other forms of physical activity.</p