Setting health research priorities using the CHNRI method: VI. Quantitative properties of human collective opinion.

INTRODUCTION: Crowdsourcing has become an increasingly important tool to address many problems - from government elections in democracies, stock market prices, to modern online tools such as TripAdvisor or Internet Movie Database (IMDB). The CHNRI method (the acronym for the Child Health and Nutrition Research Initiative) for setting health research priorities has crowdsourcing as the major component, which it uses to generate, assess and prioritize between many competing health research ideas. METHODS: We conducted a series of analyses using data from a group of 91 scorers to explore the quantitative properties of their collective opinion. We were interested in the stability of their collective opinion as the sample size increases from 15 to 90. From a pool of 91 scorers who took part in a previous CHNRI exercise, we used sampling with replacement to generate multiple random samples of different size. First, for each sample generated, we identified the top 20 ranked research ideas, among 205 that were proposed and scored, and calculated the concordance with the ranking generated by the 91 original scorers. Second, we used rank correlation coefficients to compare the ranks assigned to all 205 proposed research ideas when samples of different size are used. We also analysed the original pool of 91 scorers to to look for evidence of scoring variations based on scorers' characteristics. RESULTS: The sample sizes investigated ranged from 15 to 90. The concordance for the top 20 scored research ideas increased with sample sizes up to about 55 experts. At this point, the median level of concordance stabilized at 15/20 top ranked questions (75%), with the interquartile range also generally stable (14-16). There was little further increase in overlap when the sample size increased from 55 to 90. When analysing the ranking of all 205 ideas, the rank correlation coefficient increased as the sample size increased, with a median correlation of 0.95 reached at the sample size of 45 experts (median of the rank correlation coefficient = 0.95; IQR 0.94-0.96). CONCLUSIONS: Our analyses suggest that the collective opinion of an expert group on a large number of research ideas, expressed through categorical variables (Yes/No/Not Sure/Don't know), stabilises relatively quickly in terms of identifying the ideas that have most support. In the exercise we found a high degree of reproducibility of the identified research priorities was achieved with as few as 45-55 experts

Setting health research priorities using the CHNRI method:III. Involving stakeholders

Setting health research priorities is a complex and value–driven process. The introduction of the Child Health and Nutrition Research Initiative (CHNRI) method has made the process of setting research priorities more transparent and inclusive, but much of the process remains in the hands of funders and researchers, as described in the previous two papers in this series. However, the value systems of numerous other important stakeholders, particularly those on the receiving end of health research products, are very rarely addressed in any process of priority setting. Inclusion of a larger and more diverse group of stakeholders in the process would result in a better reflection of the system of values of the broader community, resulting in recommendations that are more legitimate and acceptable

Setting health research priorities using the CHNRI method: V. Quantitative properties of human collective knowledge.

INTRODUCTION: The CHNRI method for setting health research priorities has crowdsourcing as the major component. It uses the collective opinion of a group of experts to generate, assess and prioritize between many competing health research ideas. It is difficult to compare the accuracy of human individual and collective opinions in predicting uncertain future outcomes before the outcomes are known. However, this limitation does not apply to existing knowledge, which is an important component underlying opinion. In this paper, we report several experiments to explore the quantitative properties of human collective knowledge and discuss their relevance to the CHNRI method. METHODS: We conducted a series of experiments in groups of about 160 (range: 122-175) undergraduate Year 2 medical students to compare their collective knowledge to their individual knowledge. We asked them to answer 10 questions on each of the following: (i) an area in which they have a degree of expertise (undergraduate Year 1 medical curriculum); (ii) an area in which they likely have some knowledge (general knowledge); and (iii) an area in which they are not expected to have any knowledge (astronomy). We also presented them with 20 pairs of well-known celebrities and asked them to identify the older person of the pair. In all these experiments our goal was to examine how the collective answer compares to the distribution of students' individual answers. RESULTS: When answering the questions in their own area of expertise, the collective answer (the median) was in the top 20.83% of the most accurate individual responses; in general knowledge, it was in the top 11.93%; and in an area with no expertise, the group answer was in the top 7.02%. However, the collective answer based on mean values fared much worse, ranging from top 75.60% to top 95.91%. Also, when confronted with guessing the older of the two celebrities, the collective response was correct in 18/20 cases (90%), while the 8 most successful individuals among the students had 19/20 correct answers (95%). However, when the system in which the students who were not sure of the correct answer were allowed to either choose an award of half of the point in all such instances, or withdraw from responding, in order to improve the score of the collective, the collective was correct in 19/20 cases (95%), while the 3 most successful individuals were correct in 17/20 cases (85%). CONCLUSIONS: Our experiments showed that the collective knowledge of a group with expertise in the subject should always be very close to the true value. In most cases and under most assumption, the collective knowledge will be more accurate than the knowledge of an "average" individual, but there always seems to be a small group of individuals who manage to out-perform the collective. The accuracy of collective prediction may be enhanced by allowing the individuals with low confidence in their answer to withdraw from answering

Setting research priorities to improve global newborn health and prevent stillbirths by 2025

Background: In 2013, an estimated 2.8 million newborns died and 2.7 million were stillborn. A much greater number suffer from long term impairment associated with preterm birth, intrauterine growth restriction, congenital anomalies, and perinatal or infectious causes. With the approaching deadline for the achievement of the Millennium Development Goals (MDGs) in 2015, there was a need to set the new research priorities on newborns and stillbirth with a focus not only on survival but also on health, growth and development. We therefore carried out a systematic exercise to set newborn health research priorities for 2013–2025. Methods: We used adapted Child Health and Nutrition Research Initiative (CHNRI) methods for this prioritization exercise. We identified and approached the 200 most productive researchers and 400 program experts, and 132 of them submitted research questions online. These were collated into a set of 205 research questions, sent for scoring to the 600 identified experts, and were assessed and scored by 91 experts. Results: Nine out of top ten identified priorities were in the domain of research on improving delivery of known interventions, with simplified neonatal resuscitation program and clinical algorithms and improved skills of community health workers leading the list. The top 10 priorities in the domain of development were led by ideas on improved Kangaroo Mother Care at community level, how to improve the accuracy of diagnosis by community health workers, and perinatal audits. The 10 leading priorities for discovery research focused on stable surfactant with novel modes of administration for preterm babies, ability to diagnose fetal distress and novel tocolytic agents to delay or stop preterm labour. Conclusion: These findings will assist both donors and researchers in supporting and conducting research to close the knowledge gaps for reducing neonatal mortality, morbidity and long term impairment. WHO, SNL and other partners will work to generate interest among key national stakeholders, governments, NGOs, and research institutes in these priorities, while encouraging research funders to support them. We will track research funding, relevant requests for proposals and trial registers to monitor if the priorities identified by this exercise are being addressed

Research priorities for adolescent health in low- and middle-income countries: A mixed-methods synthesis of two separate exercises.

BACKGROUND: In order to clarify priorities and stimulate research in adolescent health in low- and middle-income countries (LMICs), the World Health Organization (WHO) conducted two priority-setting exercises based on the Child Health and Nutrition Research Initiative (CHNRI) methodology related to 1) adolescent sexual and reproductive health and 2) eight areas of adolescent health including communicable diseases prevention and management, injuries and violence, mental health, non-communicable diseases management, nutrition, physical activity, substance use, and health policy. Although the CHNRI methodology has been utilized in over 50 separate research priority setting exercises, none have qualitatively synthesized the ultimate findings across studies. The purpose of this study was to conduct a mixed-method synthesis of two research priority-setting exercises for adolescent health in LMICs based on the CHNRI methodology and to situate the priority questions within the current global health agenda. METHODS: All of the 116 top-ranked questions presented in each exercise were analyzed by two independent reviewers. Word clouds were generated based on keywords from the top-ranked questions. Questions were coded and content analysis was conducted based on type of delivery platform, vulnerable populations, and the Survive, Thrive, and Transform framework from the United Nations Global Strategy for Women's, Children's, and Adolescents' Health, 2016-2030. FINDINGS: Within the 53 top-ranked intervention-related questions that specified a delivery platform, the platforms specified were schools (n = 17), primary care (n = 12), community (n = 11), parenting (n = 6), virtual media (n = 5), and peers (n = 2). Twenty questions specifically focused on vulnerable adolescents, including those living with HIV, tuberculosis, mental illness, or neurodevelopmental disorders; victims of gender-based violence; refugees; young persons who inject drugs; sex workers; slum dwellers; out-of-school youth; and youth in armed conflict. A majority of the top-ranked questions (108/116) aligned with one or a combination of the Survive (n = 39), Thrive (n = 67), and Transform (n = 28) agendas. CONCLUSIONS: This study advances the CHNRI methodology by conducting the first mixed-methods synthesis of multiple research priority-setting exercises by analyzing keywords (using word clouds) and themes (using content analysis)

Setting health research priorities using the CHNRI method: VII. a review of the first 50 applications of the CHNRI method

Background: Several recent reviews of the methods used to set research priorities have identified the CHNRI method (acronym derived from the Child Health and Nutrition Research Initiative ) as an approach that clearly became popular and widely used over the past decade. In this paper we review the first 50 examples of application of the CHNRI method, published between 2007 and 2016, and summarize the most important messages that emerged from those experiences.Methods: We conducted a literature review to identify the first 50 examples of application of the CHNRI method in chronological order. We searched Google Scholar, PubMed and so-called grey literature.Results: Initially, between 2007 and 2011, the CHNRI method was mainly used for setting research priorities to address global child health issues, although the first cases of application outside this field (eg, mental health, disabilities and zoonoses) were also recorded. Since 2012 the CHNRI method was used more widely, expanding into the topics such as adolescent health, dementia, national health policy and education. The majority of the exercises were focused on issues that were only relevant to low- and middle-income countries, and national-level applications are on the rise. The first CHNRI-based articles adhered to the five recommended priority-setting criteria, but by 2016 more than two-thirds of all conducted exercises departed from recommendations, modifying the CHNRI method to suit each particular exercise. This was done not only by changing the number of criteria used, but also by introducing some entirely new criteria (eg, low cost , sustainability , acceptability , feasibility , relevance and others).Conclusions: The popularity of the CHNRI method in setting health research priorities can be attributed to several key conceptual advances that have addressed common concerns. The method is systematic in nature, offering an acceptable framework for handling many research questions. It is also transparent and replicable, because it clearly defines the context and priority-setting criteria. It is democratic, as it relies on crowd-sourcing . It is inclusive, fostering ownership of the results by ensuring that various groups invest in the process. It is very flexible and adjustable to many different contexts and needs. Finally, it is simple and relatively inexpensive to conduct, which we believe is one of the main reasons for its uptake by many groups globally, particularly those in low- and middle-income countries

Comparison of short term surgical outcomes of male and female gastrointestinal surgeons in Japan: retrospective cohort study

男女の消化器外科医による手術成績は同等 --女性消化器外科医のさらなる活躍に向けて--. 京都大学プレスリリース. 2022-09-29.Study finds no differences in performance between male and female surgeons. 京都大学プレスリリース. 2022-10-04.[Objective] To compare short term surgical outcomes between male and female gastrointestinal surgeons in Japan. [Design] Retrospective cohort study. [Setting] Data from the Japanese National Clinical Database (includes data on >95% of surgeries performed in Japan) (2013-17) and the Japanese Society of Gastroenterological Surgery. [Participants] Male and female surgeons who performed distal gastrectomy, total gastrectomy, and low anterior resection. [Main outcome measures] Surgical mortality, surgical mortality combined with postoperative complications, pancreatic fistula (distal gastrectomy/total gastrectomy only), and anastomotic leakage (low anterior resection only). The association of surgeons’ gender with surgery related mortality and surgical complications was examined using multivariable logistic regression models adjusted for patient, surgeon, and hospital characteristics. [Results] A total of 149 193 distal gastrectomy surgeries (male surgeons: 140 971 (94.5%); female surgeons: 8222 (5.5%)); 63 417 gastrectomy surgeries (male surgeons: 59 915 (94.5%); female surgeons: 3502 (5.5%)); and 81 593 low anterior resection procedures (male surgeons: 77 864 (95.4%);female surgeons: 3729 (4.6%)) were done. On average, female surgeons had fewer post-registration years, operated on patients at higher risk, and did fewer laparoscopic surgeries than male surgeons. No significant difference was found between male and female surgeons in the adjusted risk for surgical mortality (adjusted odds ratio 0.98 (95% confidence interval 0.74 to 1.29) for distal gastrectomy; 0.83 (0.57 to 1.19) for total gastrectomy; 0.56 (0.30 to 1.05) for low anterior resection), surgical mortality combined with Clavien-Dindo grade ≥3 complications (adjusted odds ratio 1.03 (0.93 to 1.14) for distal gastrectomy; 0.92 (0.81 to 1.05) for total gastrectomy; 1.02 (0.91 to 1.15) for low anterior resection), pancreatic fistula (adjusted odds ratio 1.16 (0.97 to 1.38) for distal gastrectomy; 1.02 (0.84 to 1.23) for total gastrectomy), and anastomotic leakage (adjusted odds ratio 1.04 (0.92 to 1.18) for low anterior resection). [Conclusion] This study found no significant adjusted risk difference in the outcomes of surgeries performed by male versus female gastrointestinal surgeons. Despite disadvantages, female surgeons take on patients at high risk. Greater access to surgical training for female physicians is warranted in Japan

Neonatal mortality within 24 hours of birth in six low- and lower-middle-income countries.

Objective: To estimate neonatal mortality, particularly within 24 hours of birth, in six low- and lower-middle-income countries. Methods: We analysed epidemiological data on a total of 149 570 live births collected between 2007 and 2013 in six prospective randomized trials and a cohort study from predominantly rural areas of Bangladesh, Ghana, India, Pakistan, the United Republic of Tanzania and Zambia. The neonatal mortality rate and mortality within 24 hours of birth were estimated for all countries and mortality within 6 hours was estimated for four countries with available data. The findings were compared with published model-based estimates of neonatal mortality. Findings: Overall, the neonatal mortality rate observed at study sites in the six countries was 30.5 per 1000 live births (range: 13.6 in Zambia to 47.4 in Pakistan). Mortality within 24 hours was 14.1 per 1000 live births overall (range: 5.1 in Zambia to 20.1 in India) and 46.3% of all neonatal deaths occurred within 24 hours (range: 36.2% in Pakistan to 65.5% in the United Republic of Tanzania). Mortality in the first 6 hours was 8.3 per 1000 live births, i.e. 31.9% of neonatal mortality. Conclusion: Neonatal mortality within 24 hours of birth in predominantly rural areas of six low- and lower-middle-income countries was higher than model-based estimates for these countries. A little under half of all neonatal deaths occurred within 24 hours of birth and around one third occurred within 6 hours. Implementation of high-quality, effective obstetric and early newborn care should be a priority in these settings

Setting health research priorities using the CHNRI method:I. Involving funders

In 2007 and 2008, the World Health Organization's Department for Child and Adolescent Health and Development commissioned five large research priority setting exercises using the CHNRI (Child Health and Nutrition Research Initiative) method. The aim was to define research priorities related to the five major causes of child deaths for the period up to the year 2015. The selected causes were childhood pneumonia, diarrhoea, birth asphyxia, neonatal infections and preterm birth/low birth weight. The criteria used for prioritization in all five exercises were the “standard” CHNRI criteria: answerability, effectiveness, deliverability, potential for mortality burden reduction and the effect on equity. Having completed the exercises, the WHO officers were left with another question: how “fundable” were the identified priorities, i.e. how attractive were they to research funders