Supervised machine learning algorithms can classify open-text feedback of doctor performance with human-level accuracy

Background: Machine learning techniques may be an effective and efficient way to classify open-text reports on doctor’s activity for the purposes of quality assurance, safety, and continuing professional development. Objective: The objective of the study was to evaluate the accuracy of machine learning algorithms trained to classify open-text reports of doctor performance and to assess the potential for classifications to identify significant differences in doctors’ professional performance in the United Kingdom. Methods: We used 1636 open-text comments (34,283 words) relating to the performance of 548 doctors collected from a survey of clinicians’ colleagues using the General Medical Council Colleague Questionnaire (GMC-CQ). We coded 77.75% (1272/1636) of the comments into 5 global themes (innovation, interpersonal skills, popularity, professionalism, and respect) using a qualitative framework. We trained 8 machine learning algorithms to classify comments and assessed their performance using several training samples. We evaluated doctor performance using the GMC-CQ and compared scores between doctors with different classifications using t tests. Results: Individual algorithm performance was high (range F score=.68 to .83). Interrater agreement between the algorithms and the human coder was highest for codes relating to “popular” (recall=.97), “innovator” (recall=.98), and “respected” (recall=.87) codes and was lower for the “interpersonal” (recall=.80) and “professional” (recall=.82) codes. A 10-fold cross-validation demonstrated similar performance in each analysis. When combined together into an ensemble of multiple algorithms, mean human-computer interrater agreement was .88. Comments that were classified as “respected,” “professional,” and “interpersonal” related to higher doctor scores on the GMC-CQ compared with comments that were not classified (P.05). Conclusions: Machine learning algorithms can classify open-text feedback of doctor performance into multiple themes derived by human raters with high performance. Colleague open-text comments that signal respect, professionalism, and being interpersonal may be key indicators of doctor’s performance

‘Why haven’t I got one of those?’ A consideration regarding the need to protect non-participant children in early years research

It is widely documented that young children participating in research should be protected from harm and that ethical considerations should be applied throughout a research project. What this paper strives to assert, however, is that protecting these participants is insufficient. A research project into children’s speech and language development, using audio–-visual methods, highlighted that children who are non-participants, those on the periphery of research, can also be affected by the research process. It is acknowledged throughout this paper that although ethical procedures were adhered to whilst undertaking a specific research project, this was insufficient. It is therefore argued that all children within a research environment, whether participatory or not, should be given equal consideration with regards to ethical protection when undertaking research. It is asserted that ‘“why haven’t I got one of those’”, or the equivalent, is a phrase to be avoided at all costs when undertaking research with children

Noninvasive measures of brain edema predict outcome in pediatric cerebral malaria.

BackgroundIncreased brain volume (BV) and subsequent herniation are strongly associated with death in pediatric cerebral malaria (PCM), a leading killer of children in developing countries. Accurate noninvasive measures of BV are needed for optimal clinical trial design. Our objectives were to examine the performance of six different magnetic resonance imaging (MRI) BV quantification measures for predicting mortality in PCM and to review the advantages and disadvantages of each method.MethodsReceiver operator characteristics were generated from BV measures of MRIs of children admitted to an ongoing research project with PCM between 2009 and 2014. Fatal cases were matched to the next available survivor. A total of 78 MRIs of children aged 5 months to 13 years (mean 4.0 years), of which 45% were males, were included.ResultsAreas under the curve (AUC) with 95% confidence interval on measures from the initial MRIs were: Radiologist-derived score = 0.69 (0.58-0.79; P = 0.0037); prepontine cistern anteroposterior (AP) dimension = 0.70 (0.56-0.78; P = 0.0133); SamKam ratio [Rt. parietal lobe height/(prepontine AP dimension + fourth ventricle AP dimension)] = 0.74 (0.63-0.83; P = 0.0002); and global cerebrospinal fluid (CSF) space ascertained by ClearCanvas = 0.67 (0.55-0.77; P = 0.0137). For patients with serial MRIs (n = 37), the day 2 global CSF space AUC was 0.87 (0.71-0.96; P P ConclusionAll noninvasive measures of BV performed well in predicting death and providing a proxy measure for brain volume. Initial MRI assessment may inform future clinical trials for subject selection, risk adjustment, or stratification. Measures of temporal change may be used to stage PCM

Diagnosis, prevalence estimation and burden measurement in population surveys of headache: presenting the HARDSHIP questionnaire

The global burden of headache is very large, but knowledge of it is far from complete and needs still to be gathered. Published population-based studies have used variable methodology, which has influenced findings and made comparisons difficult. The Global Campaign against Headache is undertaking initiatives to improve and standardize methods in use for cross-sectional studies. One requirement is for a survey instrument with proven cross-cultural validity. This report describes the development of such an instrument. Two of the authors developed the initial version, which was used with adaptations in population-based studies in China, Ethiopia, India, Nepal, Pakistan, Russia, Saudi Arabia, Zambia and 10 countries in the European Union. The resultant evolution of this instrument was reviewed by an expert consensus group drawn from all world regions. The final output was the Headache-Attributed Restriction, Disability, Social Handicap and Impaired Participation (HARDSHIP) questionnaire, designed for application by trained lay interviewers. HARDSHIP is a modular instrument incorporating demographic enquiry, diagnostic questions based on ICHD-3 beta criteria, and enquiries into each of the following as components of headache-attributed burden: symptom burden; health-care utilization; disability and productive time losses; impact on education, career and earnings; perception of control; interictal burden; overall individual burden; effects on relationships and family dynamics; effects on others, including household partner and children; quality of life; wellbeing; obesity as a comorbidity. HARDSHIP already has demonstrated validity and acceptability in multiple languages and cultures. Modules may be included or not, and others (eg, on additional comorbidities) added, according to the purpose of the study and resources (especially time) available

Drone-based Water Sampling and Characterization of Three Freshwater Harmful Algal Blooms in the United States

Freshwater harmful algal blooms (HABs), caused mostly by toxic cyanobacteria, produce a range of cyanotoxins that threaten the health of humans and domestic animals. Climate conditions and anthropogenic influences such as agricultural run-off can alter the onset and intensity of HABs. Little is known about the distribution and spread of freshwater HABs. Current sampling protocols in some lakes involve teams of researchers that collect samples by hand from a boat and/or from the shoreline. Water samples can be collected from the surface, from discrete-depth collections, and/or from depth-integrated intervals. These collections are often restricted to certain months of the year, and generally are only performed at a limited number of collection sites. In lakes with active HABs, surface samples are generally sufficient for HAB water quality assessments. We used a unique DrOne Water Sampling SystEm (DOWSE) to collect water samples from the surface of three different HABs in Ohio (Grand Lake St Marys, GLSM and Lake Erie) and Virginia (Lake Anna), United States in 2019. The DOWSE consisted of a 3D-printed sampling device tethered to a drone (uncrewed aerial system, or UAS), and was used to collect surface water samples at different distances (10–100 m) from the shore or from an anchored boat. One hundred and eighty water samples (40 at GLSM, 20 at Lake Erie, and 120 at Lake Anna) were collected and analyzed from 18 drone flights. Our methods included testing for cyanotoxins, phycocyanin, and nutrients from surface water samples. Mean concentrations of microcystins (MCs) in drone water samples were 15.00, 1.92, and 0.02 ppb for GLSM, Lake Erie, and Lake Anna, respectively. Lake Anna had low levels of anatoxin in nearly all (111/120) of the drone water samples. Mean concentrations of phycocyanin in drone water samples were 687, 38, and 62 ppb for GLSM, Lake Erie, and Lake Anna, respectively. High levels of total phosphorus were observed in the drone water samples from GLSM (mean of 0.34 mg/L) and Lake Erie (mean of 0.12 mg/L). Lake Anna had the highest variability of total phosphorus with concentrations that ranged from 0.01 mg/L to 0.21 mg/L, with a mean of 0.06 mg/L. Nitrate levels varied greatly across sites, inverse with bloom biomass, ranging from below detection to 3.64 mg/L, with highest mean values in Lake Erie followed by GLSM and Lake Anna, respectively. Drones offer a rapid, targeted collection of water samples from virtually anywhere on a lake with an active HAB without the need for a boat which can disturb the surrounding water. Drones are, however, limited in their ability to operate during inclement weather such as rain and heavy winds. Collectively, our results highlight numerous opportunities for drone-based water sampling technologies to track, predict, and respond to HABs in the future

The impact of the Calman–Hine report on the processes and outcomes of care for Yorkshire's colorectal cancer patients

The 1995 Calman–Hine plan outlined radical reform of the UK's cancer services with the aim of improving outcomes and reducing inequalities in NHS cancer care. Its main recommendation was to concentrate care into the hands of site-specialist, multi-disciplinary teams. This study aimed to determine if the implementation of Calman–Hine cancer teams was associated with improved processes and outcomes of care for colorectal cancer patients. The design included longitudinal survey of 13 colorectal cancer teams in Yorkshire and retrospective study of population-based data collected by the Northern and Yorkshire Cancer Registry and Information Service. The population was all colorectal cancer patients diagnosed and treated in Yorkshire between 1995 and 2000. The main outcome measures were: variations in the use of anterior resection and preoperative radiotherapy in rectal cancer, chemotherapy in Dukes stage C and D patients, and five-year survival. Using multilevel models, these outcomes were assessed in relation to measures of the extent of Calman–Hine implementation throughout the study period, namely: (i) each team's degree of adherence to the Manual of Cancer Service Standards (which outlines the specification of the ‘ideal’ colorectal cancer team) and (ii) the extent of site specialisation of each team's surgeons. Variation was observed in the extent to which the colorectal cancer teams in Yorkshire had conformed to the Calman–Hine recommendations. An increase in surgical site specialisation was associated with increased use of preoperative radiotherapy (OR=1.43, 95% CI=1.04–1.98, P<0.04) and anterior resection (OR=1.43, 95% CI=1.16–1.76, P<0.01) in rectal cancer patients. Increases in adherence to the Manual of Cancer Service Standards was associated with improved five-year survival after adjustment for the casemix factors of age, stage of disease, socioeconomic status and year of diagnosis, especially for colon cancer (HR=0.97, 95% CI=0.94–0.99 P<0.01). There was a similar trend of improved survival in relation to increased surgical site specialisation for rectal cancer, although the effect was not statistically significant (HR=0.93, 95% CI=0.84–1.03, P=0.15). In conclusion, the extent of implementation of the Calman–Hine report has been variable and its recommendations are associated with improvements in processes and outcomes of care for colorectal cancer patients

Automated Detection of Malarial Retinopathy in Digital Fundus Images for Improved Diagnosis in Malawian Children with Clinically Defined Cerebral Malaria

Cerebral malaria (CM), a complication of malaria infection, is the cause of the majority of malaria-associated deaths in African children. The standard clinical case definition for CM misclassifies ~25% of patients, but when malarial retinopathy (MR) is added to the clinical case definition, the specificity improves from 61% to 95%. Ocular fundoscopy requires expensive equipment and technical expertise not often available in malaria endemic settings, so we developed an automated software system to analyze retinal color images for MR lesions: retinal whitening, vessel discoloration, and white-centered hemorrhages. The individual lesion detection algorithms were combined using a partial least square classifier to determine the presence or absence of MR. We used a retrospective retinal image dataset of 86 pediatric patients with clinically defined CM (70 with MR and 16 without) to evaluate the algorithm performance. Our goal was to reduce the false positive rate of CM diagnosis, and so the algorithms were tuned at high specificity. This yielded sensitivity/specificity of 95%/100% for the detection of MR overall, and 65%/94% for retinal whitening, 62%/100% for vessel discoloration, and 73%/96% for hemorrhages. This automated system for detecting MR using retinal color images has the potential to improve the accuracy of CM diagnosis

Lessons Learned Developing a Diagnostic Tool for HIV-Associated Dementia Feasible to Implement in Resource-Limited Settings: Pilot Testing in Kenya

Objective: To conduct a preliminary evaluation of the utility and reliability of a diagnostic tool for HIV-associated dementia (HAD) for use by primary health care workers (HCW) which would be feasible to implement in resource-limited settings. Background: In resource-limited settings, HAD is an indication for anti-retroviral therapy regardless of CD4 T-cell count. Anti-retroviral therapy, the treatment for HAD, is now increasingly available in resource-limited settings. Nonetheless, HAD remains under-diagnosed likely because of limited clinical expertise and availability of diagnostic tests. Thus, a simple diagnostic tool which is practical to implement in resource-limited settings is an urgent need. Methods: A convenience sample of 30 HIV-infected outpatients was enrolled in Western Kenya. We assessed the sensitivity and specificity of a diagnostic tool for HAD as administered by a primary HCW. This was compared to an expert clinical assessment which included examination by a physician, neuropsychological testing, and in selected cases, brain imaging. Agreement between HCW and an expert examiner on certain tool components was measured using Kappa statistic. Results: The sample was 57 % male, mean age was 38.6 years, mean CD4 T-cell count was 323 cells/mL, and 54 % had less than a secondary school education. Six (20%) of the subjects were diagnosed with HAD by expert clinical assessment. The diagnostic tool was 63 % sensitive and 67 % specific for HAD. Agreement between HCW and expert examiners was poor for many individual items of the diagnostic tool (K =.03–.65). This diagnostic tool had moderate sensitivity and specificity fo