Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies

<p>Abstract</p> <p>Background</p> <p>Verbal autopsy (VA) is an important method for obtaining cause of death information in settings without vital registration and medical certification of causes of death. An array of methods, including physician review and computer-automated methods, have been proposed and used. Choosing the best method for VA requires the appropriate metrics for assessing performance. Currently used metrics such as sensitivity, specificity, and cause-specific mortality fraction (CSMF) errors do not provide a robust basis for comparison.</p> <p>Methods</p> <p>We use simple simulations of populations with three causes of death to demonstrate that most metrics used in VA validation studies are extremely sensitive to the CSMF composition of the test dataset. Simulations also demonstrate that an inferior method can appear to have better performance than an alternative due strictly to the CSMF composition of the test set.</p> <p>Results</p> <p>VA methods need to be evaluated across a set of test datasets with widely varying CSMF compositions. We propose two metrics for assessing the performance of a proposed VA method. For assessing how well a method does at individual cause of death assignment, we recommend the average chance-corrected concordance across causes. This metric is insensitive to the CSMF composition of the test sets and corrects for the degree to which a method will get the cause correct due strictly to chance. For the evaluation of CSMF estimation, we propose CSMF accuracy. CSMF accuracy is defined as one minus the sum of all absolute CSMF errors across causes divided by the maximum total error. It is scaled from zero to one and can generalize a method's CSMF estimation capability regardless of the number of causes. Performance of a VA method for CSMF estimation by cause can be assessed by examining the relationship across test datasets between the estimated CSMF and the true CSMF.</p> <p>Conclusions</p> <p>With an increasing range of VA methods available, it will be critical to objectively assess their performance in assigning cause of death. Chance-corrected concordance and CSMF accuracy assessed across a large number of test datasets with widely varying CSMF composition provide a robust strategy for this assessment.</p

Performance of InterVA for assigning causes of death to verbal autopsies: multisite validation study using clinical diagnostic gold standards

Background: InterVA is a widely disseminated tool for cause of death attribution using information from verbal autopsies. Several studies have attempted to validate the concordance and accuracy of the tool, but the main limitation of these studies is that they compare cause of death as ascertained through hospital record review or hospital discharge diagnosis with the results of InterVA. This study provides a unique opportunity to assess the performance of InterVA compared to physician-certified verbal autopsies (PCVA) and alternative automated methods for analysis.Methods: Using clinical diagnostic gold standards to select 12,542 verbal autopsy cases, we assessed the performance of InterVA on both an individual and population level and compared the results to PCVA, conducting analyses separately for adults, children, and neonates. Following the recommendation of Murray et al., we randomly varied the cause composition over 500 test datasets to understand the performance of the tool in different settings. We also contrasted InterVA with an alternative Bayesian method, Simplified Symptom Pattern (SSP), to understand the strengths and weaknesses of the tool.Results: Across all age groups, InterVA performs worse than PCVA, both on an individual and population level. On an individual level, InterVA achieved a chance-corrected concordance of 24.2% for adults, 24.9% for children, and 6.3% for neonates (excluding free text, considering one cause selection). On a population level, InterVA achieved a cause-specific mortality fraction accuracy of 0.546 for adults, 0.504 for children, and 0.404 for neonates. The comparison to SSP revealed four specific characteristics that lead to superior performance of SSP. Increases in chance-corrected concordance are attained by developing cause-by-cause models (2%), using all items as opposed to only the ones that mapped to InterVA items (7%), assigning probabilities to clusters of symptoms (6%), and using empirical as opposed to expert probabilities (up to 8%).Conclusions: Given the widespread use of verbal autopsy for understanding the burden of disease and for setting health intervention priorities in areas that lack reliable vital registrations systems, accurate analysis of verbal autopsies is essential. While InterVA is an affordable and available mechanism for assigning causes of death using verbal autopsies, users should be aware of its suboptimal performance relative to other methods

Genome-Wide Association Studies in an Isolated Founder Population from the Pacific Island of Kosrae

It has been argued that the limited genetic diversity and reduced allelic heterogeneity observed in isolated founder populations facilitates discovery of loci contributing to both Mendelian and complex disease. A strong founder effect, severe isolation, and substantial inbreeding have dramatically reduced genetic diversity in natives from the island of Kosrae, Federated States of Micronesia, who exhibit a high prevalence of obesity and other metabolic disorders. We hypothesized that genetic drift and possibly natural selection on Kosrae might have increased the frequency of previously rare genetic variants with relatively large effects, making these alleles readily detectable in genome-wide association analysis. However, mapping in large, inbred cohorts introduces analytic challenges, as extensive relatedness between subjects violates the assumptions of independence upon which traditional association test statistics are based. We performed genome-wide association analysis for 15 quantitative traits in 2,906 members of the Kosrae population, using novel approaches to manage the extreme relatedness in the sample. As positive controls, we observe association to known loci for plasma cholesterol, triglycerides, and C-reactive protein and to a compelling candidate loci for thyroid stimulating hormone and fasting plasma glucose. We show that our study is well powered to detect common alleles explaining ≥5% phenotypic variance. However, no such large effects were observed with genome-wide significance, arguing that even in such a severely inbred population, common alleles typically have modest effects. Finally, we show that a majority of common variants discovered in Caucasians have indistinguishable effect sizes on Kosrae, despite the major differences in population genetics and environment

Current and prospective pharmacological targets in relation to antimigraine action

Migraine is a recurrent incapacitating neurovascular disorder characterized by unilateral and throbbing headaches associated with photophobia, phonophobia, nausea, and vomiting. Current specific drugs used in the acute treatment of migraine interact with vascular receptors, a fact that has raised concerns about their cardiovascular safety. In the past, α-adrenoceptor agonists (ergotamine, dihydroergotamine, isometheptene) were used. The last two decades have witnessed the advent of 5-HT1B/1D receptor agonists (sumatriptan and second-generation triptans), which have a well-established efficacy in the acute treatment of migraine. Moreover, current prophylactic treatments of migraine include 5-HT2 receptor antagonists, Ca2+ channel blockers, and β-adrenoceptor antagonists. Despite the progress in migraine research and in view of its complex etiology, this disease still remains underdiagnosed, and available therapies are underused. In this review, we have discussed pharmacological targets in migraine, with special emphasis on compounds acting on 5-HT (5-HT1-7), adrenergic (α1, α2, and β), calcitonin gene-related peptide (CGRP 1 and CGRP2), adenosine (A1, A2, and A3), glutamate (NMDA, AMPA, kainate, and metabotropic), dopamine, endothelin, and female hormone (estrogen and progesterone) receptors. In addition, we have considered some other targets, including gamma-aminobutyric acid, angiotensin, bradykinin, histamine, and ionotropic receptors, in relation to antimigraine therapy. Finally, the cardiovascular safety of current and prospective antimigraine therapies is touched upon

The primary headaches: genetics, epigenetics and a behavioural genetic model

The primary headaches, migraine with (MA) and without aura (MO) and cluster headache, all carry a substantial genetic liability. Familial hemiplegic migraine (FHM), an autosomal dominant mendelian disorder classified as a subtype of MA, is due to mutations in genes encoding neural channel subunits. MA/MO are considered multifactorial genetic disorders, and FHM has been proposed as a model for migraine aetiology. However, a review of the genetic studies suggests that the FHM genes are not involved in the typical migraines and that FHM should be considered as a syndromic migraine rather than a subtype of MA. Adopting the concept of syndromic migraine could be useful in understanding migraine pathogenesis. We hypothesise that epigenetic mechanisms play an important role in headache pathogenesis. A behavioural model is proposed, whereby the primary headaches are construed as behaviours, not symptoms, evolutionarily conserved for their adaptive value and engendered out of a genetic repertoire by a network of pattern generators present in the brain and signalling homeostatic imbalance. This behavioural model could be incorporated into migraine genetic research

Evaluation of appendicitis risk prediction models in adults with suspected appendicitis

Background Appendicitis is the most common general surgical emergency worldwide, but its diagnosis remains challenging. The aim of this study was to determine whether existing risk prediction models can reliably identify patients presenting to hospital in the UK with acute right iliac fossa (RIF) pain who are at low risk of appendicitis. Methods A systematic search was completed to identify all existing appendicitis risk prediction models. Models were validated using UK data from an international prospective cohort study that captured consecutive patients aged 16–45 years presenting to hospital with acute RIF in March to June 2017. The main outcome was best achievable model specificity (proportion of patients who did not have appendicitis correctly classified as low risk) whilst maintaining a failure rate below 5 per cent (proportion of patients identified as low risk who actually had appendicitis). Results Some 5345 patients across 154 UK hospitals were identified, of which two‐thirds (3613 of 5345, 67·6 per cent) were women. Women were more than twice as likely to undergo surgery with removal of a histologically normal appendix (272 of 964, 28·2 per cent) than men (120 of 993, 12·1 per cent) (relative risk 2·33, 95 per cent c.i. 1·92 to 2·84; P < 0·001). Of 15 validated risk prediction models, the Adult Appendicitis Score performed best (cut‐off score 8 or less, specificity 63·1 per cent, failure rate 3·7 per cent). The Appendicitis Inflammatory Response Score performed best for men (cut‐off score 2 or less, specificity 24·7 per cent, failure rate 2·4 per cent). Conclusion Women in the UK had a disproportionate risk of admission without surgical intervention and had high rates of normal appendicectomy. Risk prediction models to support shared decision‐making by identifying adults in the UK at low risk of appendicitis were identified