A defense against the alleged unreliability of difference scores

Based on a classical true score theory (classical test theory, CTT) equation, indicating that as the observed correlation between two tests increases, the reliability of the difference scores decreases, researchers have concluded that difference scores are unreliable. But CTT shows that the reliabilities of the two tests and the true correlation between them influence the observed correlation and previous analyses have not taken the true correlation sufficiently into account. In turn, the reliability of difference scores depends on the interaction of the reliabilities of the individual tests and their true correlation when the variances of the tests are equal, and on a more complicated interaction between them and the deviation ratio when the variances of the tests are not equal. The upshot is that difference scores likely are more reliable, on more occasions, than researchers have realized. I show how researchers can predict what the reliability of the difference scores is likely to be, to aid in deciding whether to carry through one’s planned use of difference scores.https://deepblue.lib.umich.edu/bitstream/2027.42/136924/1/Trafimow(2015)ADefenseAgainst.pdfDescription of Trafimow(2015)ADefenseAgainst.pdf : Main Articl

Situation-specific expectancies in person memory

Thesis (B.S.) in Liberal Arts and Sciences--University of Illinois at Urbana-Champaign, 1984.Bibliography: leaves 38-40.Microfiche of typescript. [Urbana, Ill.] : Photographic Services, University of Illinois, U of I Library, [1987]. 2 microfiches (78 frames) negative ; 11 x 15 cm

Inferential statistics as descriptive statistics: there is no replication crisis if we don't expect replication

Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a "replication crisis" may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as providing generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced

Predicting Faculty Intentions to Assign Writing in Their Classes

Teachers who offer undergraduate courses agree widely on the importance of writing assignments to further undergraduate education. And yet, there is a great deal of variance among teachers in their writing assignments; some teachers assign no writing whatsoever. To determine the variables that influence the decisions of teachers about whether to assign writing, we predicted their intentions to assign writing from attitudes, subjective norms, perceived control, and perceived difficulty pertaining to assigning writing. Zero-order correlations and hierarchical regression analyses implicate attitude and perceived difficulty as the most important predictors of teacher’s intentions to assign writing in two studies. We also obtained open-ended belief statements in Study 1 and used them to obtain quantitative belief data in Study 2 to find and validate the importance of the impact of particular specific beliefs on intentions to assign writing

The theory of planned behaviour predicts self-reports of walking, but does not predict step count

Objectives This paper compares multiple measures of walking in two studies, and the second study compares how well Theory of Planned Behaviour (TPB) constructs perform in predicting these different measures. Methods In Study 1, 41 participants wore a New Lifestyles NL-2000 pedometer for 1 week. Subsequently, participants completed a questionnaire containing measures of the TPB constructs and two self-report measures of walking, followed by two interview measures of walking. For Study 2, 200 RAF trainee aircraftsmen wore pedometers for 2 weeks. At the end of each week, participants completed the questionnaire and interview measures of walking. Results Both studies found no significant association between questionnaire measures of walking and pedometer measures. In Study 1, the interview measures produced significant, large correlations with the pedometer measure, but these relationships were markedly weaker in the second study. TPB variables were found to explain 22% of variance in intention to walk in Study 1 and 45% of the variance in Study 2. In Study 2, prediction of subsequent measures of behaviour was found to be weak, except when using a single-item measure of walking. Conclusions Recall of walking is poor, and accurate measurement by self-report is problematic. Although the TPB predicts intentions to walk well, it does not predict actual amount of walking, as assessed by pedometer. Possible reasons for these findings include the unique nature of walking as an activity primarily used to facilitate higher order goals. The use of single-item measures may exaggerate the effectiveness of the TPB model for walking, and possibly other forms of physical activity.</p

The influence of journal submission guidelines on authors' reporting of statistics and use of open research practices.

From January 2014, Psychological Science introduced new submission guidelines that encouraged the use of effect sizes, estimation, and meta-analysis (the "new statistics"), required extra detail of methods, and offered badges for use of open science practices. We investigated the use of these practices in empirical articles published by Psychological Science and, for comparison, by the Journal of Experimental Psychology: General, during the period of January 2013 to December 2015. The use of null hypothesis significance testing (NHST) was extremely high at all times and in both journals. In Psychological Science, the use of confidence intervals increased markedly overall, from 28% of articles in 2013 to 70% in 2015, as did the availability of open data (3 to 39%) and open materials (7 to 31%). The other journal showed smaller or much smaller changes. Our findings suggest that journal-specific submission guidelines may encourage desirable changes in authors' practices

The shocking implications of Bayes’ theorem for diagnosing herniated nucleus pulposus based on MRI scans

We obtain the input data for Bayes Theorem, and use the theorem to determine the probability of a patient having a lumbar HNP, given only a positive MRI. We also enumerate the potential consequences that the clinician must keep in mind when making the diagnosis of lumbar HNP. We used the theorem by Bayes, in conjunction with well-established results in the orthopedic literature, to calculate the probability of lumbar HNP given only a positive MRI finding. The necessary information provided by the orthopedic literature includes the prevalence of lumbar HNP, the probability of a positive MRI finding given that there is no HNP, and the probability of a positive MRI finding given that there is HNP. We found that the probability of lumbar HNP given only a positive MRI finding was 8%. The probability that there is no lumbar HNP, even when there is a positive MRI finding, is 92%. Clearly, MRI scans cannot be trusted as the sole source of diagnostic information