A signal detection theory analysis of racial and ethnic disproportionality in the referral and substantiation process of the U.S. child welfare services system

Signal detection theory (SDT) was developed to analyze the behavior of a single judge but also can be used to analyze decisions made by organizations or other social systems. SDT quantifies the ability to distinguish between signal and noise by separating accuracy of the detection system from response bias—the propensity to over-warn (too many false positives) or under-warn (too many misses). We apply SDT techniques to national and state-level data sets to analyze the ability of the child welfare services systems to detect instances of child maltreatment. Blacks have higher rates of referral and the system is less accurate for them than for Whites or Hispanics. The incidence of false positives—referrals leading to unsubstantiated findings—is higher for Blacks than for other groups, as is the incidence of false negatives—children for whom no referral was made but who are in fact neglected or abused. The rate of true positives–children for whom a referral was made and for whom the allegation was substantiated–is higher for Blacks. Values of d′ (signal strength) are roughly the same for Whites, Blacks, and Hispanics but there are pronounced group differences in C (a measure of the location of the decision threshold). Analyses show that the child welfare services system treats Blacks differently from Hispanics and Whites in ways that cannot be justified readily in terms of objective measures of group differences. This study illustrates the potential for JDM techniques such as SDT to contribute to understanding of system-level decision making processes

A signal detection theory analysis of racial and ethnic disproportionality in the referral and substantiation processes of the U.S. child welfare services system

Signal detection theory (SDT) was developed to analyze the behavior of a single judge but also can be used to analyze decisions made by organizations or other social systems. SDT quantifies the ability to distinguish between signal and noise by separating accuracy of the detection system from response bias—the propensity to over-warn (too many false positives) or under-warn (too many misses). We apply SDT techniques to national and state-level data sets to analyze the ability of the child welfare services systems to detect instances of child maltreatment. Blacks have higher rates of referral and the system is less accurate for them than for Whites or Hispanics. The incidence of false positives—referrals leading to unsubstantiated findings—is higher for Blacks than for other groups, as is the incidence of false negatives—children for whom no referral was made but who are in fact neglected or abused. The rate of true positives–children for whom a referral was made and for whom the allegation was substantiated–is higher for Blacks. Values of d′ (signal strength) are roughly the same for Whites, Blacks, and Hispanics but there are pronounced group differences in C (a measure of the location of the decision threshold). Analyses show that the child welfare services system treats Blacks differently from Hispanics and Whites in ways that cannot be justified readily in terms of objective measures of group differences. This study illustrates the potential for JDM techniques such as SDT to contribute to understanding of system-level decision making processes

Predictors of the Perceived Risk of Climate Change and Preferred Resource Levels

The Version of Record of this manuscript has been published and is available in Journal of Risk Research, 20 May 2015, http://www.tandfonline.com/doi/abs/10.1080/13669877.2015.1043567#.VZ2Y7WND2Ao.In a 2013 U.S. national public opinion survey, data were collected from 1,321 adult respondents for five psychometric variables—Dread, Scientists’ Level of Understanding, Public’s Level of Understanding, Number Affected, and Likelihood—for six threats (sea level rise, increased flooding, and four others) associated with climate change. Respondents also rated Perceived Risk and indicated the Resource Level that they believed should be invested in management programs for each threat. Responses did not vary significantly across the six threats, so they were combined. The survey collected standard demographic information, as well as measuring Climate Change Knowledge (CCK) and environmental values (New Ecological Paradigm, NEP). Psychometric variables predicted Perceived Risk extremely well (R = .890, p < .001); all five psychometric variables were significant predictors. The results were generally consistent with previous research except that Scientists’ Level of Understanding was a positive, rather than negative, predictor of Perceived Risk. Jointly the demographic, knowledge and environmental values variables significantly predicted Perceived Risk (R = .504, p < .001). Consistent with previous research, significant positive predictors were Age, Democratic Party identification, and NEP score; significant negative predictors were Male gender and White ethnicity. When demographic, knowledge, and environmental values variables were added to psychometric ones, only the psychometric variables were statistically significant predictors. Perceived Risk strongly predicted Resource Level (r = .772, p < .001). Adding demographic, knowledge and environmental value variables to Perceived Risk as predictors of Resource Level did not appreciably increase overall predictive ability (r = .790, p < .001), although White ethnicity emerged as a significant negative predictor and Religiosity, Democratic Party ID, Liberal Political Ideology, and NEP score were significant positive predictors. The results demonstrate that risk perceptions of climate change and policy preferences among climate change management options are highly predictable as a function of demographic, knowledge, environmental values, and psychometric variables. Among these, psychometric variables were found to be the strongest predictors.This material is based upon research conducted by the Institute for Science, Technology and Public Policy (ISTPP) in The Bush School of Government and Public Service, Texas A&M University. The statements, findings, conclusions, and recommendations are solely those of the authors

Kenneth R. Hammond’s contributions to the study of judgment and decision making

Kenneth R. Hammond (1917--2015) made several major contributions to the science of human judgment and decision making. As a student of Egon Brunswik, he kept Brunswik’s legacy alive – advancing his theory of probabilistic functionalism and championing his method of representative design. Hammond pioneered the use of Brunswik’s lens model as a framework for studying how individuals use information from the task environment to make clinical judgments, which was the precursor to much `policy capturing’ and `judgment analysis’ research. Hammond introduced the lens model equation to the study of judgment processes, and used this to measure the utility of different forms of feedback in multiple-cue probability learning. He extended the scope of analysis to contexts in which individuals interact with one another – introducing the interpersonal learning and interpersonal conflict paradigms. Hammond developed social judgment theory which provided a comprehensive quantitative approach for describing and improving judgment processes. He proposed cognitive continuum theory which states that quasi-rationality is an important middle-ground between intuition and analysis and that cognitive performance is dictated by the match between task properties and mode of cognition. Throughout his career, Hammond moved easily from basic laboratory work to applied settings, where he resolved policy disputes, and in doing so, he pointed to the dichotomy between theories of correspondence and coherence. In this paper, we present Hammond’s legacy to a new generation of judgment and decision making scholars

A signal detection theory analysis of racial and ethnic disproportionality in the referral and substantiation processes of the U.S. child welfare services system

Signal detection theory (SDT) was developed to analyze the behavior of a single judge but also can be used to analyze decisions made by organizations or other social systems. SDT quantifies the ability to distinguish between signal and noise by separating accuracy of the detection system from response bias - the propensity to over-warn (too many false positives) or under-warn (too many misses). We apply SDT techniques to national and state-level data sets to analyze the ability of the child welfare services systems to detect instances of child maltreatment. Blacks have higher rates of referral and the system is less accurate for them than for Whites or Hispanics. The incidence of false positives - referrals leading to unsubstantiated findings - is higher for Blacks than for other groups, as is the incidence of false negatives - children for whom no referral was made but who are in fact neglected or abused. The rate of true positives - children for whom a referral was made and for whom the allegation was substantiated - is higher for Blacks. Values of d' (signal strength) are roughly the same for Whites, Blacks, and Hispanics but there are pronounced group differences in C (a measure of the location of the decision threshold). Analyses show that the child welfare services system treats Blacks differently from Hispanics and Whites in ways that cannot be justified readily in terms of objective measures of group differences. This study illustrates the potential for JDM techniques such as SDT to contribute to understanding of system-level decision making processes

Affirmative action, duality of error, and the consequences of mispredicting the academic performance of african american college applicants

The implications of different potential affirmative action policies depend on three factors: selection rate from the applicant pool, base rate of qualified applicants, and accuracy of performance predictions. A series of analyses was conducted under various assumptions concerning affirmative action plans, causes of racial differences in average college admissions test scores, and racial differences in accuracy of performance predictions. Evidence suggesting a lower level of predictive accuracy for African Americans implies that, under a program of affirmative action, both proportionately more false positives (matriculated students who do not succeed) and proportionately more false negatives (rejected applicants who could have succeeded) will be found among African American applicants. Unless equivalent levels of predictive accuracy are achieved for both groups, no admission policy can be fair simultaneously to majority group applicants and African American applicants. © 2002 by the Association for Public Policy Analysis and Management.