Useful scientific theories are useful: A reply to Rouder, Pratte, and Morey (2010)

In a recognition memory experiment, Mickes, Wixted, and Wais (2007) asked a simple question: Would the same result-namely, a higher mean and variance of the memory strengths for the targets as compared with the lures-be evident if one used a 20-point confidence scale and then simply computed the relevant distributional statistics from the ratings themselves instead of estimating them by fitting a Gaussian model to ROC data? And if an unequal-variance model were suggested by the ratings data, would the magnitude of the estimated ratio of the standard deviations based on the ratings (s Lure / s Target ) be similar to the magnitude of the estimated ratio obtained by fitting a Gaussian model to ROC data ( Lure / Target )? A priori, agreement between the two ratio estimates seems unlikely, because there are many reasons why they might disagree. For example, if the Gaussian assumption is not valid, then disagreement between the two estimates seems more likely than agreement. In addition, if the rating scale does not approximate an interval scale, or if it covers only a limited range of the memory strength dimension, then, again, disagreement seems more likely than agreement. Somewhat surprisingly, From these results, 1. The two experiments reported here support a conclusion that is commonly drawn from ROC analysis-namely, that the memory strengths of the targets are more variable than the memory strengths of the lures. (p. 864) 2. The close agreement between the model-based ROC analysis and the model-free ratings method supports not only an unequal-variance model, but also the idea that the memory strengths are distributed in such a way that fitting a specifically Gaussian model to the data yields accurate conclusions (even if the true underlying distributions are not strictly Gaussian). (p. 864

Receiver operating characteristic analysis of eyewitness memory: comparing the diagnostic accuracy of simultaneous versus sequential lineups.

A police lineup presents a real-world signal-detection problem because there are two possible states of the world (the suspect is either innocent or guilty), some degree of information about the true state of the world is available (the eyewitness has some degree of memory for the perpetrator), and a decision is made (identifying the suspect or not). A similar state of affairs applies to diagnostic tests in medicine because, in a patient, the disease is either present or absent, a diagnostic test yields some degree of information about the true state of affairs, and a decision is made about the presence or absence of the disease. In medicine, Receiver Operating Characteristic (ROC) analysis is the standard method for assessing diagnostic accuracy. By contrast, in the eyewitness memory literature, this powerful technique has never been used. Instead, researchers have attempted to assess the diagnostic performance of different lineup procedures using methods that cannot identify the better procedure (e.g., by computing a diagnosticity ratio). Here, we describe the basics of ROC analysis, explaining why it is needed and showing how to use it to measure the performance of different lineup procedures. To illustrate the unique advantages of this technique, we also report three ROC experiments that were designed to investigate the diagnostic accuracy of simultaneous vs. sequential lineups. According to our findings, the sequential procedure appears to be inferior to the simultaneous procedure in discriminating between the presence vs. absence of a guilty suspect in a lineup

Three Tests and Three Corrections: Comment on Koen and Yonelinas (2010)

The slope of the z-transformed receiver-operating characteristic (zROC) in recognition memory experiments is usually less than 1, which has long been interpreted to mean that the variance of the target distribution is greater than the variance of the lure distribution. The greater variance of the target distribution could arise because the different items on a list receive different increments in memory strength during study (the "encoding variability" hypothesis). In a test of that interpretation