A Syndrome Definition Validation Approach for Zika Virus

ObjectiveTo develop and validate a Zika virus disease syndrome definitionwithin the GUARDIAN (Geographic Utilization of ArtificialIntelligence in Real-Time for Disease Identification and AlertNotification) surveillance system.IntroductionIn 2016, the World Health Organization declared Zika virus aglobal public health emergency. Zika infection during pregnancycan cause microcephaly and other fetal brain defects. To facilitateclinicians’ ability to detect Zika, various syndrome definitions havebeen developed.MethodsTo create and validate a detailed syndrome definition for Zika,we utilized the literature based methodology developed anddocumented by GUARDIAN researchers.1,2The syndrome definitionutilized clinical signs and symptoms that were documented inhistorical Zika cases.A testing sample of 1000 randomly selected emergency departmentcases (i.e., true negative cases) and 200 synthetically generated cases(i.e., true positive cases) was created. These 1,200 sample cases wereevaluated by the GUARDIAN surveillance system to determine theprobability of matching the Zika syndrome definition. A probabilityof≥90% was utilized to designate positive Zika cases.We identified the main signs and symptoms contributing to theidentification of Zika cases and conducted statistical performancemetrics. Clinical review of the false positive and false negative casesalong with a sample of true positive and true negative cases wasconducted by a board certified emergency physician.ResultsThe Zika syndrome definition was developed with eleven articles(six used for developing the syndrome definition, and five used fortesting the definition). The sample size for these articles was between1 and 72 positive Zika cases, with a total of 139 cases across the11 articles. The article with the most number of Zika cases wasbased on pregnant women with rash. The publication timeframefor the articles was from 1962 to 2016. Some of the main signsand symptoms from the historical cases that contribute to the Zikasyndrome definition are presented in Table 1. The initial results forthe sample testing data showed accuracy, sensitivity, and specificitywere 94.7%, 93%, and 95% respectively. There were a total of14 false negative and 50 false positive cases.ConclusionsThe initial Zika syndrome definition utilized by the GUARDIANsurveillance system contains similar signs and symptoms to thecurrent CDC case definition, but also includes additional signs andsymptoms such as pruritus/itching, malaise/fatigue/generalizedweakness, headache, retro-orbital pain, myalgia/muscle pain, andlymphadenopathy In addition, the GUARDIAN system provides therelative importance of identified signs and symptoms and allows forproactive surveillance of emergency department patients in real-time.Though we did not include epidemiologic risk factors, such as travel toan infected region or contact with an infected person in the syndromedefinition, GUARDIAN has above 90% sensitivity and specificity.Thus, inclusion of epidemiologic risk factors would further enhancethe early detection of Zika, when used with the appropriate high riskpopulation.Table 1. Main signs and symptoms of Zika syndrome definition*Signs and symptoms included in the Centers for Disease Control andPrevention (CDC)’s Zika clinical case definitio

A Syndrome Definition Validation Approach for Zika Virus

ObjectiveTo develop and validate a Zika virus disease syndrome definitionwithin the GUARDIAN (Geographic Utilization of ArtificialIntelligence in Real-Time for Disease Identification and AlertNotification) surveillance system.IntroductionIn 2016, the World Health Organization declared Zika virus aglobal public health emergency. Zika infection during pregnancycan cause microcephaly and other fetal brain defects. To facilitateclinicians’ ability to detect Zika, various syndrome definitions havebeen developed.MethodsTo create and validate a detailed syndrome definition for Zika,we utilized the literature based methodology developed anddocumented by GUARDIAN researchers.1,2The syndrome definitionutilized clinical signs and symptoms that were documented inhistorical Zika cases.A testing sample of 1000 randomly selected emergency departmentcases (i.e., true negative cases) and 200 synthetically generated cases(i.e., true positive cases) was created. These 1,200 sample cases wereevaluated by the GUARDIAN surveillance system to determine theprobability of matching the Zika syndrome definition. A probabilityof≥90% was utilized to designate positive Zika cases.We identified the main signs and symptoms contributing to theidentification of Zika cases and conducted statistical performancemetrics. Clinical review of the false positive and false negative casesalong with a sample of true positive and true negative cases wasconducted by a board certified emergency physician.ResultsThe Zika syndrome definition was developed with eleven articles(six used for developing the syndrome definition, and five used fortesting the definition). The sample size for these articles was between1 and 72 positive Zika cases, with a total of 139 cases across the11 articles. The article with the most number of Zika cases wasbased on pregnant women with rash. The publication timeframefor the articles was from 1962 to 2016. Some of the main signsand symptoms from the historical cases that contribute to the Zikasyndrome definition are presented in Table 1. The initial results forthe sample testing data showed accuracy, sensitivity, and specificitywere 94.7%, 93%, and 95% respectively. There were a total of14 false negative and 50 false positive cases.ConclusionsThe initial Zika syndrome definition utilized by the GUARDIANsurveillance system contains similar signs and symptoms to thecurrent CDC case definition, but also includes additional signs andsymptoms such as pruritus/itching, malaise/fatigue/generalizedweakness, headache, retro-orbital pain, myalgia/muscle pain, andlymphadenopathy In addition, the GUARDIAN system provides therelative importance of identified signs and symptoms and allows forproactive surveillance of emergency department patients in real-time.Though we did not include epidemiologic risk factors, such as travel toan infected region or contact with an infected person in the syndromedefinition, GUARDIAN has above 90% sensitivity and specificity.Thus, inclusion of epidemiologic risk factors would further enhancethe early detection of Zika, when used with the appropriate high riskpopulation.Table 1. Main signs and symptoms of Zika syndrome definition*Signs and symptoms included in the Centers for Disease Control andPrevention (CDC)’s Zika clinical case definitio

Utility of Natural Language Processing for Clinical Quality Measures Reporting

ObjectiveTo explain the utility of using an automated syndromic surveillanceprogram with advanced natural language processing (NLP) to improveclinical quality measures reporting for influenza immunization.IntroductionClinical quality measures (CQMs) are tools that help measure andtrack the quality of health care services. Measuring and reportingCQMs helps to ensure that our health care system is deliveringeffective, safe, efficient, patient-centered, equitable, and timely care.The CQM for influenza immunization measures the percentage ofpatients aged 6 months and older seen for a visit between October1 and March 31 who received (or reports previous receipt of) aninfluenza immunization. Centers for Disease Control and Preventionrecommends that everyone 6 months of age and older receive aninfluenza immunization every season, which can reduce influenza-related morbidity and mortality and hospitalizations.MethodsPatients at a large academic medical center who had a visit toan affiliated outpatient clinic during June 1 - 8, 2016 were initiallyidentified using their electronic medical record (EMR). The 2,543patients who were selected did not have documentation of influenzaimmunization in a discrete field of the EMR. All free text notes forthese patients between August 1, 2015 and March 31, 2016 wereretrieved and analyzed using the sophisticated NLP built withinGeographic Utilization of Artificial Intelligence in Real-Timefor Disease Identification and Alert Notification (GUARDIAN)– a syndromic surveillance program – to identify any mention ofinfluenza immunization. The goal was to identify additional cases thatmet the CQM measure for influenza immunization and to distinguishdocumented exceptions. The patients with influenza immunizationmentioned were further categorized by GUARDIAN NLP intoReceived, Recommended, Refused, Allergic, and Unavailable.If more than one category was applicable for a patient, they wereindependently counted in their respective categories. A descriptiveanalysis was conducted, along with manual review of a sample ofcases per each category.ResultsFor the 2,543 patients who did not have influenza immunizationdocumentation in a discrete field of the EMR, a total of 78,642 freetext notes were processed using GUARDIAN. Four hundred fiftythree (17.8%) patients had some mention of influenza immunizationwithin the notes, which could potentially be utilized to meet the CQMinfluenza immunization requirement. Twenty two percent (n=101)of patients mentioned already having received the immunizationwhile 34.7% (n=157) patients refused it during the study time frame.There were 27 patients with the mention of influenza immunization,who could not be differentiated into a specific category. The numberof patients placed into a single category of influenza immunizationwas 351 (77.5%), while 75 (16.6%) were classified into more thanone category. See Table 1.ConclusionsUsing GUARDIAN’s NLP can identify additional patients whomay meet the CQM measure for influenza immunization or whomay be exempt. This tool can be used to improve CQM reportingand improve overall influenza immunization coverage by using it toalert providers. Next steps involve further refinement of influenzaimmunization categories, automating the process of using the NLPto identify and report additional cases, as well as using the NLP forother CQMs.Table 1. Categorization of influenza immunization documentation within freetext notes of 453 patients using NL

Utility of Natural Language Processing for Clinical Quality Measures Reporting

ObjectiveTo explain the utility of using an automated syndromic surveillanceprogram with advanced natural language processing (NLP) to improveclinical quality measures reporting for influenza immunization.IntroductionClinical quality measures (CQMs) are tools that help measure andtrack the quality of health care services. Measuring and reportingCQMs helps to ensure that our health care system is deliveringeffective, safe, efficient, patient-centered, equitable, and timely care.The CQM for influenza immunization measures the percentage ofpatients aged 6 months and older seen for a visit between October1 and March 31 who received (or reports previous receipt of) aninfluenza immunization. Centers for Disease Control and Preventionrecommends that everyone 6 months of age and older receive aninfluenza immunization every season, which can reduce influenza-related morbidity and mortality and hospitalizations.MethodsPatients at a large academic medical center who had a visit toan affiliated outpatient clinic during June 1 - 8, 2016 were initiallyidentified using their electronic medical record (EMR). The 2,543patients who were selected did not have documentation of influenzaimmunization in a discrete field of the EMR. All free text notes forthese patients between August 1, 2015 and March 31, 2016 wereretrieved and analyzed using the sophisticated NLP built withinGeographic Utilization of Artificial Intelligence in Real-Timefor Disease Identification and Alert Notification (GUARDIAN)– a syndromic surveillance program – to identify any mention ofinfluenza immunization. The goal was to identify additional cases thatmet the CQM measure for influenza immunization and to distinguishdocumented exceptions. The patients with influenza immunizationmentioned were further categorized by GUARDIAN NLP intoReceived, Recommended, Refused, Allergic, and Unavailable.If more than one category was applicable for a patient, they wereindependently counted in their respective categories. A descriptiveanalysis was conducted, along with manual review of a sample ofcases per each category.ResultsFor the 2,543 patients who did not have influenza immunizationdocumentation in a discrete field of the EMR, a total of 78,642 freetext notes were processed using GUARDIAN. Four hundred fiftythree (17.8%) patients had some mention of influenza immunizationwithin the notes, which could potentially be utilized to meet the CQMinfluenza immunization requirement. Twenty two percent (n=101)of patients mentioned already having received the immunizationwhile 34.7% (n=157) patients refused it during the study time frame.There were 27 patients with the mention of influenza immunization,who could not be differentiated into a specific category. The numberof patients placed into a single category of influenza immunizationwas 351 (77.5%), while 75 (16.6%) were classified into more thanone category. See Table 1.ConclusionsUsing GUARDIAN’s NLP can identify additional patients whomay meet the CQM measure for influenza immunization or whomay be exempt. This tool can be used to improve CQM reportingand improve overall influenza immunization coverage by using it toalert providers. Next steps involve further refinement of influenzaimmunization categories, automating the process of using the NLPto identify and report additional cases, as well as using the NLP forother CQMs.Table 1. Categorization of influenza immunization documentation within freetext notes of 453 patients using NL

A Novel Syndrome Definition Validation Approach for Rarely Occurring Diseases

OBJECTIVE: To develop and test a novel syndrome definition validation approach for rarely occurring diseases. INTRODUCTION: Early detection of rarely occurring but potentially harmful diseases such as bio-threat agents (e.g., anthrax), chemical agents (e.g., sarin), and naturally occurring diseases (e.g., meningitis) is critical for rapid initiation of treatment, infection control measures, and emergency response plans. To facilitate clinicians’ ability to detect these diseases, various syndrome definitions have been developed. Due to the rarity of these diseases, standard statistical methodologies for validating syndrome definitions are not applicable. METHODS: Syndrome definitions were developed by researchers for the Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification (GUARDIAN) surveillance system (1 1. Partition of literature articles: Literature articles that described positive cases were randomly divided to generate detection (75% of articles) and testing (25% of articles) syndrome definitions. 2. Synthetic case generation: Syndrome definitions and associated statistical measures were reverse engineered using probability of occurrence and inverse Gaussian function to generate potentially infinite positive artificial cases. 3. Clinical filter application: To avoid clinically incompatible combinations of newly generated symptoms, rules based on clinically guided knowledge from emergency department (ED) physicians were applied. Steps 2 and 3 were repeated for both detection and testing syndrome definitions. 4. a. ED negative case sample: Detection syndrome definitions were tested using a random sample of negative ED cases. Knowledge gained through false positive cases was utilized to modify the surveillance algorithms and system thresholds. b. 10-fold cross-validation: Standard 10-fold cross-validation on detection articles of positive cases and ED negative cases was utilized to generate performance metrics. Suspected cases were reviewed by ED clinicians for threshold enhancement. c. Literature articles (n=1): The ability of syndrome definitions to correctly flag literature articles with n=1 case was documented. 5. a. Testing sample: Synthetic positive cases generated from the testing articles along with another set of ED negative cases were evaluated by the respective syndrome definition. Suspected cases were clinically evaluated. b. Literature articles (n=1): Similar to detection step 4c, articles with n=1 were tested using syndrome definitions. c. True positive samples: When available, true positive cases from an ED were identified and sent through the GUARDIAN system. 6. Multi-syndrome validation: A combined sample of positive cases of multiple syndromes and ED negative cases were evaluated for detection of individual syndromes among other similar syndromes. RESULTS: To demonstrate the validation approach, the anthrax syndrome definition was utilized. This syndrome definition was developed with 25 articles containing positive anthrax cases used for detection, and the remaining 11 articles used for testing. With a 10-fold cross validation of the detection phase, the initial results showed accuracy was 99.4% (false positive rate of 0.65% and false negative rate of 0.00%). The testing phase initial validation revealed 99.2% accuracy for the anthrax syndrome definition. CONCLUSIONS: Syndrome specific synthetic samples that are validated through clinical filters allowed the generation of an unlimited number of positive cases. Correct identification by GUARDIAN of these cases indicates robust and reliable syndrome definitions. Utilization of these cases, in conjunction with adherence to a methodological process, was the cornerstone of the GUARDIAN syndrome definition validation approach. The validation approach was successfully demonstrated on anthrax and can be applied to other bio-threat agents, chemical agents, and naturally occurring diseases

A Novel Syndrome Definition Validation Approach for Rarely Occurring Diseases

Syndrome definitions for rarely occuring but potentially harmful diseases were developed by researchers for the Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification (GUARDIAN) surveillance system. The main steps for validation of the syndrome definitions are described

Adaptation of GUARDIAN for Syndromic Surveillance During the NATO Summit

During the NATO summit, the local health department was charged with collecting and analyzing syndromic surveillance data from emergency department (ED) visits that may indicate a man-made or naturally occurring infectious disease threat. GUARDIAN, an automated surveillance system, was programmed to conduct ED syndromic surveillance during the NATO summit. The automated GUARDIAN surveillance reports not only provided timely counts of potentially positive cases for each syndrome but also provided trend analysis with baseline measures

Natural Language Processing and Technical Challenges of Influenza-Like Illness Surveillance

Processing free-text clinical information in an electronic medical record may enhance surveillance systems for early identification of influenza-like illness outbreaks. However, processing clinical text using natural language processing (NLP) poses a challenge in preserving the semantics of the original information recorded. In this study, we discuss several NLP and technical issues as well as potential solutions for implementation in syndromic surveillance systems

Natural Language Processing and Technical Challenges of Influenza-Like Illness Surveillance

Processing free-text clinical information in an electronic medical record may enhance surveillance systems for early identification of influenza-like illness outbreaks. However, processing clinical text using natural language processing (NLP) poses a challenge in preserving the semantics of the original information recorded. In this study, we discuss several NLP and technical issues as well as potential solutions for implementation in syndromic surveillance systems