Measuring Risk Literacy: The Berlin Numeracy Test

We introduce the Berlin Numeracy Test, a new psychometrically sound instrument that quickly assesses statistical numeracy and risk literacy. We present 21 studies (n=5336) showing robust psychometric discriminability across 15 countries (e.g., Germany, Pakistan, Japan, USA) and diverse samples (e.g., medical professionals, general populations, Mechanical Turk web panels). Analyses demonstrate desirable patterns of convergent validity (e.g., numeracy, general cognitive abilities), discriminant validity (e.g., personality, motivation), and criterion validity (e.g., numerical and nonnumerical questions about risk). The Berlin Numeracy Test was found to be the strongest predictor of comprehension of everyday risks (e.g., evaluating claims about products and treatments; interpreting forecasts), doubling the predictive power of other numeracy instruments and accounting for unique variance beyond other cognitive tests (e.g., cognitive reflection, working memory, intelligence). The Berlin Numeracy Test typically takes about three minutes to complete and is available in multiple languages and formats, including a computer adaptive test that automatically scores and reports data to researchers (www.riskliteracy.org). The online forum also provides interactive content for public outreach and education, and offers a recommendation system for test format selection. Discussion centers on construct validity of numeracy for risk literacy, underlying cognitive mechanisms, and applications in adaptive decision support

Development and initial validation of a computer-administered health literacy assessment in Spanish and English: FLIGHT/VIDAS.

Current measures of health literacy have been criticized on a number of grounds, including use of a limited range of content, development on small and atypical patient groups, and poor psychometric characteristics. In this paper, we report the development and preliminary validation of a new computer-administered and -scored health literacy measure addressing these limitations. Items in the measure reflect a wide range of content related to health promotion and maintenance as well as care for diseases. The development process has focused on creating a measure that will be useful in both Spanish and English, while not requiring substantial time for clinician training and individual administration and scoring. The items incorporate several formats, including questions based on brief videos, which allow for the assessment of listening comprehension and the skills related to obtaining information on the Internet. In this paper, we report the interim analyses detailing the initial development and pilot testing of the items (phase 1 of the project) in groups of Spanish and English speakers. We then describe phase 2, which included a second round of testing of the items, in new groups of Spanish and English speakers, and evaluation of the new measure\u27s reliability and validity in relation to other measures. Data are presented that show that four scales (general health literacy, numeracy, conceptual knowledge, and listening comprehension), developed through a process of item and factor analyses, have significant relations to existing measures of health literacy

Predicting biases in very highly educated samples: Numeracy and metacognition

We investigated the relations between numeracy and superior judgment and decision making in two large community outreach studies in Holland (n=5408). In these very highly educated samples (e.g., 30–50% held graduate degrees), the Berlin Numeracy Test was a robust predictor of financial, medical, and metacognitive task performance (i.e., lotteries, intertemporal choice, denominator neglect, and confidence judgments), independent of education, gender, age, and another numeracy assessment. Metacognitive processes partially mediated the link between numeracy and superior performance. More numerate participants performed better because they deliberated more during decision making and more accurately evaluated their judgments (e.g., less overconfidence). Results suggest that well-designed numeracy tests tend to be robust predictors of superior judgment and decision making because they simultaneously assess (1) mathematical competency and (2) metacognitive and self-regulated learning skills

DEVELOPMENT OF THE INTELLIGENT GRAPHS FOR EVERYDAY RISKY DECISIONS TUTOR

Simple graphical visual aids have now been shown to be among the most effective means of quickly improving people’s ability to evaluate and understand risks (i.e., risk literacy), particularly for diverse and vulnerable groups (e.g., older adults, less educated, less numerate, minority and immigrant samples). Although well-developed theory and standards for user-friendly graph design exist, guidelines are often violated by designers faced with constraints like conflicts of interest (e.g., persuasion and marketing vs. informed decision making). Even when information is presented in well-designed graphs, many people struggle with appropriate data interpretation. Can basic computerized graph literacy training improve essential graph and risk evaluation skills? To begin to answer this question, I conducted three studies that developed and validated psychometric tests of three component graph literacy skills, namely (1) graph type knowledge, (2) selecting appropriate graphs, and (3) knowledge of graph distortions. I then developed a computerized graph literacy training platform and conducted a mixed-factorial experiment investigating a wide-range of training effects. Results indicate that even in a sample of tech savvy college students one hour of basic computerized training can dramatically improve graph literacy (Cohen’s d = 1.10). Results also provide some of the first evidence that graph literacy training can improve general decision making skills that involve spatial or visualization-relevant processing, such as resistance to sunk costs, framing effects, and class-inclusion illusions. Discussion focuses on practical and theoretical implications, including usability modeling that should inform continuing development of the RiskLiteracy.org Decision Making Skills Training Program

Visualization Literacy and Decision-making in Healthcare

The ability of workers in the healthcare industry to analyze, interpret and communicate with health data is critical to decision-making and impacts both health and business outcomes. Optimal decision-making requires having real-time access to information that provides useful insights and that lends itself to collaborative decision-making. Data visualizations have the potential to facilitate decision-making in healthcare when presented as a dashboard. However, dashboards have shown varying results in both effectiveness and adoption. Data or graphical literacy challenges experienced by health team members could complicate strategic decision-making through an inability to correctly interpret or summarize the information presented in a dashboard. One assumption is that visualization literacy and its impact on how people process health data visualizations play a part in the effective interpretation of information to support decision-making. To determine the impact of visualization literacy on the process of decision-making in a healthcare setting, we first developed and deployed a dashboard designed to provide important information for decision-makers on a clinical trial management team. We engaged Project Managers and Medical Managers in the project as key decision-makers on the team. The dashboard was integrated into the normal workflow of a clinical trial management team and designated as the tool used in the workflow to report on the trial status within the organization. Next, we administered a series of assessments to the key decision-makers. The assessments were designed to evaluate numeracy, visualization literacy, and the impact of both on the decision-making ability of participants. Decision-making was assessed using a common workflow scenario supported by visualizations from the deployed dashboard. Additionally, we were interested in exploring indicators related to job satisfaction that was collected during the project period through a formal engagement survey. We performed a general linear model to assess the relationship between the assessments and decision-making. Results of our project show a significant and clear relationship between visualization literacy and decision-making ability and an insignificant relationship between numeracy and decision-making ability. Job satisfaction scores for the participant group obtained through the engagement survey suggest favorable results. However, areas of opportunity for improvement illuminated through the survey included better tools and additional resources to support the execution of tasks, a better workload balance, and improvements in collaboration across departments and functions. The results of this project contribute to the informatics discipline by demonstrating that information obtained from data visualizations produced through the aggregation of multiple sources of data can be effective decision-support tools if they are designed with user skills and abilities in mind. The results of the project suggest an opportunity to develop more useful and usable tools to improve job satisfaction as well as organizational business objectives related to workforce staffing, job competencies, and learning and development initiatives

Impact of numeracy on parental self-efficacy and treatment outcome of children on complex diets.

Health numeracy, a counterpart to health literacy, can be a mediator of health disparities. This study analyzed the impact of both cognitive and affective numeracy on the pathway linking health behavior to health outcomes, and the role of self-efficacy in this relationship, based on the Health Belief Model. The context was parental management of children\u27s complex diets that require numerical calculations. Parents of children ages 12 months to 12 years with type 1 diabetes (T1D) or phenylketonuria (PKU) were recruited at clinics or community events in east-central states. Ninety-eight participants completed a standardized test of math skills, an instrument to assess attitudes and emotions towards mathematics in daily life, and a questionnaire on parental self-efficacy of caring for a child with T1D or PKU. Health outcome was evaluated via hemoglobin A1c or blood levels of phenylalanine. Engagement was measured by number of blood levels taken during glucose or phenylalanine monitoring, compared to clinic recommendations. Factor analysis indicated affective numeracy was a significant component of the overall variable numeracy. Structural equation modeling did not support a relationship between any variable and health outcome, although bivariate analysis suggested significant relationships between poor math skills, low self-efficacy, less engagement, low income, less education, or more years on the diet, and poor metabolic control. In pathway analysis, cognitive numeracy had a strong positive relationship with engagement, while affective numeracy had an equal but negative predictive effect. Adjustments to the model identified education as the ultimate driver of the relationship. Parental self-efficacy was not a mediator between numeracy and health outcomes or engagement. The relationship between self-efficacy and engagement was strongly influenced by other pathway variables, and parental self-efficacy was significantly lower when the child had been on the diet for a longer time. This study asserts the importance of affective component of numeracy along with cognitive skills, and offers a validated instrument for assessment. Treatment programs for PKU and T1D should recognize that parents with lower numeracy skills and discomfort with math are at risk for less engagement. Further research is needed to clarify the path by which numeracy impacts health outcomes

COMPONENT NUMERACY SKILLS AND DECISION MAKING

Numeracy—i.e., one’s practical understanding of mathematics in context—is one of the strongest predictors of people’s general decision making skill, independent of other cognitive abilities (e.g., intelligence, working memory, attentional control). Despite notable scientific progress on the nature of numeracy and decision making, the cognitive and decision sciences have yet to investigate individual differences in numeracy components (e.g., algebra versus probability). In this dissertation, I report on my efforts to develop new measurement technology and quantitative models of cognitive and decision skills. Analyses include the first known investigations of the relations between the major adult component numeracy skills and general decision making skills. Specifically, Study 1 built on an established theoretical framework from adult education and used a two-parameter logistic IRT model to create the Berlin Numeracy Components Test (BNT-C) for college educated samples. Behavioral and analytic results indicated that the test efficiently measured full-scale adult numeracy and component numeracy skills (i.e. operations, probability, geometry, and algebra), with superior psychometric performance (e.g., difficulty, discriminability, and sensitivity). In Study 2, I investigated the links between the BNT-C, other numeracy tests, general cognitive abilities (e.g., intelligence, cognitive impulsivity), and general decision making skills (e.g., Adult Decision Making Competencies, Risk Literacy). Predictive modeling of behavioral data revealed that with few exceptions the BNT-C explained all types of decision skill better than any other individual ability assessment (e.g., intelligence v. impulsivity v. other numeracy tests). The BNT-C additionally outperformed the optimal linear combination of all combined ability tests when predicting overall general decision making skill, a finding that is consistent with a causal cognitive account of the relations between numeracy and general decision making skill. In accord with leading theory, component analyses indicated that operations and probability skills were robustly and uniquely tied to risk literacy and to nearly all general decision making sub-skill competencies independent of the influence of other cognitive abilities (e.g., intelligence). Analyses also provided the first evidence that algebra and geometry sub-skills are uniquely linked to some essential general decision skills in educated adults (e.g., confidence calibration, sunk costs, ratio bias). Discussion focuses on theoretical implications and factor analytic modeling of the relations between numeracy, its component skills, and superior decision making. Adaptive test construction and potential applications in training and personalized decision support are also briefly discussed