Measuring quality of general reasoning

Machine learning models that automatically assess reasoning quality are trained on human-annotated written products. These “gold-standard” corpora are typically created by prompting annotators to choose, using a forced choice design, which of two products presented side by side is the most convincing, contains the strongest evidence or would be adopted by more people. Despite the increase in popularity of using a forced choice design for assessing quality of reasoning (QoR), no study to date has established the validity and reliability of such a method. In two studies, we simultaneously presented two products of reasoning to participants and asked them to identify which product was ‘better justified’ through a forced choice design. We investigated the criterion validity and inter-rater reliability of the forced choice protocol by assessing the relationship between QoR, measured using the forced choice protocol, and accuracy in objectively answerable problems using naive raters sampled from MTurk (Study 1) and experts (Study 2), respectively. In both studies products that were closer to the correct answer and products generated by larger teams were consistently preferred. Experts were substantially better at picking the reasoning products that corresponded to accurate answers. Perhaps the most surprising finding was just how rapidly raters made judgements regarding reasoning: On average, both novices and experts made reliable decisions in under 15 seconds. We conclude that forced choice is a valid and reliable method of assessing QoR

Validating a forced‑choice method for eliciting quality‑of‑reasoning judgments

In this paper we investigate the criterion validity of forced-choice comparisons of the quality of written arguments with normative solutions. Across two studies, novices and experts assessing quality of reasoning through a forced-choice design were both able to choose arguments supporting more accurate solutions—62.2% (SE = 1%) of the time for novices and 74.4% (SE = 1%) for experts—and arguments produced by larger teams—up to 82% of the time for novices and 85% for experts—with high inter-rater reliability, namely 70.58% (95% CI = 1.18) agreement for novices and 80.98% (95% CI = 2.26) for experts. We also explored two methods for increasing efficiency. We found that the number of comparative judgments needed could be substantially reduced with little accuracy loss by leveraging transitivity and producing quality-of-reasoning assessments using an AVL tree method. Moreover, a regression model trained to predict scores based on automatically derived linguistic features of participants’ judgments achieved a high correlation with the objective accuracy scores of the arguments in our dataset. Despite the inherent subjectivity involved in evaluating differing quality of reasoning, the forced-choice paradigm allows even novice raters to perform beyond chance and can provide a valid, reliable, and efficient method for producing quality-of-reasoning assessments at scale

Better together: reliable application of the post-9/11 and post-Iraq US intelligence tradecraft standards requires collective analysis

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq's Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the US Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Method: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusions: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards

Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products.Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments.Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability.Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards

Robust estimation of bacterial cell count from optical density

Optical density (OD) is widely used to estimate the density of cells in liquid culture, but cannot be compared between instruments without a standardized calibration protocol and is challenging to relate to actual cell count. We address this with an interlaboratory study comparing three simple, low-cost, and highly accessible OD calibration protocols across 244 laboratories, applied to eight strains of constitutive GFP-expressing E. coli. Based on our results, we recommend calibrating OD to estimated cell count using serial dilution of silica microspheres, which produces highly precise calibration (95.5% of residuals <1.2-fold), is easily assessed for quality control, also assesses instrument effective linear range, and can be combined with fluorescence calibration to obtain units of Molecules of Equivalent Fluorescein (MEFL) per cell, allowing direct comparison and data fusion with flow cytometry measurements: in our study, fluorescence per cell measurements showed only a 1.07-fold mean difference between plate reader and flow cytometry data

Worldwide trends in hypertension prevalence and progress in treatment and control from 1990 to 2019: a pooled analysis of 1201 population-representative studies with 104 million participants.

BACKGROUND: Hypertension can be detected at the primary health-care level and low-cost treatments can effectively control hypertension. We aimed to measure the prevalence of hypertension and progress in its detection, treatment, and control from 1990 to 2019 for 200 countries and territories. METHODS: We used data from 1990 to 2019 on people aged 30-79 years from population-representative studies with measurement of blood pressure and data on blood pressure treatment. We defined hypertension as having systolic blood pressure 140 mm Hg or greater, diastolic blood pressure 90 mm Hg or greater, or taking medication for hypertension. We applied a Bayesian hierarchical model to estimate the prevalence of hypertension and the proportion of people with hypertension who had a previous diagnosis (detection), who were taking medication for hypertension (treatment), and whose hypertension was controlled to below 140/90 mm Hg (control). The model allowed for trends over time to be non-linear and to vary by age. FINDINGS: The number of people aged 30-79 years with hypertension doubled from 1990 to 2019, from 331 (95% credible interval 306-359) million women and 317 (292-344) million men in 1990 to 626 (584-668) million women and 652 (604-698) million men in 2019, despite stable global age-standardised prevalence. In 2019, age-standardised hypertension prevalence was lowest in Canada and Peru for both men and women; in Taiwan, South Korea, Japan, and some countries in western Europe including Switzerland, Spain, and the UK for women; and in several low-income and middle-income countries such as Eritrea, Bangladesh, Ethiopia, and Solomon Islands for men. Hypertension prevalence surpassed 50% for women in two countries and men in nine countries, in central and eastern Europe, central Asia, Oceania, and Latin America. Globally, 59% (55-62) of women and 49% (46-52) of men with hypertension reported a previous diagnosis of hypertension in 2019, and 47% (43-51) of women and 38% (35-41) of men were treated. Control rates among people with hypertension in 2019 were 23% (20-27) for women and 18% (16-21) for men. In 2019, treatment and control rates were highest in South Korea, Canada, and Iceland (treatment >70%; control >50%), followed by the USA, Costa Rica, Germany, Portugal, and Taiwan. Treatment rates were less than 25% for women and less than 20% for men in Nepal, Indonesia, and some countries in sub-Saharan Africa and Oceania. Control rates were below 10% for women and men in these countries and for men in some countries in north Africa, central and south Asia, and eastern Europe. Treatment and control rates have improved in most countries since 1990, but we found little change in most countries in sub-Saharan Africa and Oceania. Improvements were largest in high-income countries, central Europe, and some upper-middle-income and recently high-income countries including Costa Rica, Taiwan, Kazakhstan, South Africa, Brazil, Chile, Turkey, and Iran. INTERPRETATION: Improvements in the detection, treatment, and control of hypertension have varied substantially across countries, with some middle-income countries now outperforming most high-income nations. The dual approach of reducing hypertension prevalence through primary prevention and enhancing its treatment and control is achievable not only in high-income countries but also in low-income and middle-income settings. FUNDING: WHO

Worldwide trends in hypertension prevalence and progress in treatment and control from 1990 to 2019: a pooled analysis of 1201 population-representative studies with 104 million participants

Background Hypertension can be detected at the primary health-care level and low-cost treatments can effectively control hypertension. We aimed to measure the prevalence of hypertension and progress in its detection, treatment, and control from 1990 to 2019 for 200 countries and territories. Methods We used data from 1990 to 2019 on people aged 30-79 years from population-representative studies with measurement of blood pressure and data on blood pressure treatment. We defined hypertension as having systolic blood pressure 140 mm Hg or greater, diastolic blood pressure 90 mm Hg or greater, or taking medication for hypertension. We applied a Bayesian hierarchical model to estimate the prevalence of hypertension and the proportion of people with hypertension who had a previous diagnosis (detection), who were taking medication for hypertension (treatment), and whose hypertension was controlled to below 140/90 mm Hg (control). The model allowed for trends over time to be non-linear and to vary by age. Findings The number of people aged 30-79 years with hypertension doubled from 1990 to 2019, from 331 (95% credible interval 306-359) million women and 317 (292-344) million men in 1990 to 626 (584-668) million women and 652 (604-698) million men in 2019, despite stable global age-standardised prevalence. In 2019, age-standardised hypertension prevalence was lowest in Canada and Peru for both men and women; in Taiwan, South Korea, Japan, and some countries in western Europe including Switzerland, Spain, and the UK for women; and in several low-income and middle-income countries such as Eritrea, Bangladesh, Ethiopia, and Solomon Islands for men. Hypertension prevalence surpassed 50% for women in two countries and men in nine countries, in central and eastern Europe, central Asia, Oceania, and Latin America. Globally, 59% (55-62) of women and 49% (46-52) of men with hypertension reported a previous diagnosis of hypertension in 2019, and 47% (43-51) of women and 38% (35-41) of men were treated. Control rates among people with hypertension in 2019 were 23% (20-27) for women and 18% (16-21) for men. In 2019, treatment and control rates were highest in South Korea, Canada, and Iceland (treatment >70%; control >50%), followed by the USA, Costa Rica, Germany, Portugal, and Taiwan. Treatment rates were less than 25% for women and less than 20% for men in Nepal, Indonesia, and some countries in sub-Saharan Africa and Oceania. Control rates were below 10% for women and men in these countries and for men in some countries in north Africa, central and south Asia, and eastern Europe. Treatment and control rates have improved in most countries since 1990, but we found little change in most countries in sub-Saharan Africa and Oceania. Improvements were largest in high-income countries, central Europe, and some upper-middle-income and recently high-income countries including Costa Rica, Taiwan, Kazakhstan, South Africa, Brazil, Chile, Turkey, and Iran. Interpretation Improvements in the detection, treatment, and control of hypertension have varied substantially across countries, with some middle-income countries now outperforming most high-income nations. The dual approach of reducing hypertension prevalence through primary prevention and enhancing its treatment and control is achievable not only in high-income countries but also in low-income and middle-income settings. Copyright (C) 2021 World Health Organization; licensee Elsevier

Worldwide trends in hypertension prevalence and progress in treatment and control from 1990 to 2019: a pooled analysis of 1201 population-representative studies with 104 million participants

Background Hypertension can be detected at the primary health-care level and low-cost treatments can effectively control hypertension. We aimed to measure the prevalence of hypertension and progress in its detection, treatment, and control from 1990 to 2019 for 200 countries and territories. Methods We used data from 1990 to 2019 on people aged 30–79 years from population-representative studies with measurement of blood pressure and data on blood pressure treatment. We defined hypertension as having systolic blood pressure 140 mm Hg or greater, diastolic blood pressure 90 mm Hg or greater, or taking medication for hypertension. We applied a Bayesian hierarchical model to estimate the prevalence of hypertension and the proportion of people with hypertension who had a previous diagnosis (detection), who were taking medication for hypertension (treatment), and whose hypertension was controlled to below 140/90 mm Hg (control). The model allowed for trends over time to be non-linear and to vary by age. Findings The number of people aged 30–79 years with hypertension doubled from 1990 to 2019, from 331 (95% credible interval 306–359) million women and 317 (292–344) million men in 1990 to 626 (584–668) million women and 652 (604–698) million men in 2019, despite stable global age-standardised prevalence. In 2019, age-standardised hypertension prevalence was lowest in Canada and Peru for both men and women; in Taiwan, South Korea, Japan, and some countries in western Europe including Switzerland, Spain, and the UK for women; and in several low-income and middle-income countries such as Eritrea, Bangladesh, Ethiopia, and Solomon Islands for men. Hypertension prevalence surpassed 50% for women in two countries and men in nine countries, in central and eastern Europe, central Asia, Oceania, and Latin America. Globally, 59% (55–62) of women and 49% (46–52) of men with hypertension reported a previous diagnosis of hypertension in 2019, and 47% (43–51) of women and 38% (35–41) of men were treated. Control rates among people with hypertension in 2019 were 23% (20–27) for women and 18% (16–21) for men. In 2019, treatment and control rates were highest in South Korea, Canada, and Iceland (treatment >70%; control >50%), followed by the USA, Costa Rica, Germany, Portugal, and Taiwan. Treatment rates were less than 25% for women and less than 20% for men in Nepal, Indonesia, and some countries in sub-Saharan Africa and Oceania. Control rates were below 10% for women and men in these countries and for men in some countries in north Africa, central and south Asia, and eastern Europe. Treatment and control rates have improved in most countries since 1990, but we found little change in most countries in sub-Saharan Africa and Oceania. Improvements were largest in high-income countries, central Europe, and some upper-middle-income and recently high-income countries including Costa Rica, Taiwan, Kazakhstan, South Africa, Brazil, Chile, Turkey, and Iran. Interpretation Improvements in the detection, treatment, and control of hypertension have varied substantially across countries, with some middle-income countries now outperforming most high-income nations. The dual approach of reducing hypertension prevalence through primary prevention and enhancing its treatment and control is achievable not only in high-income countries but also in low-income and middle-income settings

Worldwide trends in hypertension prevalence and progress in treatment and control from 1990 to 2019: a pooled analysis of 1201 population-representative studies with 104 million participants

Background Hypertension can be detected at the primary health-care level and low-cost treatments can effectively control hypertension. We aimed to measure the prevalence of hypertension and progress in its detection, treatment, and control from 1990 to 2019 for 200 countries and territories. Methods We used data from 1990 to 2019 on people aged 30–79 years from population-representative studies with measurement of blood pressure and data on blood pressure treatment. We defined hypertension as having systolic blood pressure 140 mm Hg or greater, diastolic blood pressure 90 mm Hg or greater, or taking medication for hypertension. We applied a Bayesian hierarchical model to estimate the prevalence of hypertension and the proportion of people with hypertension who had a previous diagnosis (detection), who were taking medication for hypertension (treatment), and whose hypertension was controlled to below 140/90 mm Hg (control). The model allowed for trends over time to be non-linear and to vary by age. Findings The number of people aged 30–79 years with hypertension doubled from 1990 to 2019, from 331 (95% credible interval 306–359) million women and 317 (292–344) million men in 1990 to 626 (584–668) million women and 652 (604–698) million men in 2019, despite stable global age-standardised prevalence. In 2019, age-standardised hypertension prevalence was lowest in Canada and Peru for both men and women; in Taiwan, South Korea, Japan, and some countries in western Europe including Switzerland, Spain, and the UK for women; and in several low-income and middle-income countries such as Eritrea, Bangladesh, Ethiopia, and Solomon Islands for men. Hypertension prevalence surpassed 50% for women in two countries and men in nine countries, in central and eastern Europe, central Asia, Oceania, and Latin America. Globally, 59% (55–62) of women and 49% (46–52) of men with hypertension reported a previous diagnosis of hypertension in 2019, and 47% (43–51) of women and 38% (35–41) of men were treated. Control rates among people with hypertension in 2019 were 23% (20–27) for women and 18% (16–21) for men. In 2019, treatment and control rates were highest in South Korea, Canada, and Iceland (treatment >70%; control >50%), followed by the USA, Costa Rica, Germany, Portugal, and Taiwan. Treatment rates were less than 25% for women and less than 20% for men in Nepal, Indonesia, and some countries in sub-Saharan Africa and Oceania. Control rates were below 10% for women and men in these countries and for men in some countries in north Africa, central and south Asia, and eastern Europe. Treatment and control rates have improved in most countries since 1990, but we found little change in most countries in sub-Saharan Africa and Oceania. Improvements were largest in high-income countries, central Europe, and some upper-middle-income and recently high-income countries including Costa Rica, Taiwan, Kazakhstan, South Africa, Brazil, Chile, Turkey, and Iran. Interpretation Improvements in the detection, treatment, and control of hypertension have varied substantially across countries, with some middle-income countries now outperforming most high-income nations. The dual approach of reducing hypertension prevalence through primary prevention and enhancing its treatment and control is achievable not only in high-income countries but also in low-income and middle-income settings