Background: Many healthcare providers are adopting clinical decision support (CDS) systems to improve patient safety and meet meaningful use requirements. Computerized alerts that prompt clinicians about drug-allergy, drug-drug, and drugdisease warnings or provide dosing guidance are most commonly implemented. Alert overrides, which occur when clinicians do not follow the guidance presented by the alert, can hinder improved patient outcomes. Methods: We present a review of CDS alerts and describe a proposal to develop novel methods for evaluating and improving CDS alerts that builds upon traditional informatics approaches. Our proposal incorporates previously described models for predicting alert overrides that utilize retrospective chart review to determine which alerts are clinically relevant and which overrides are justifiable. Results: Despite increasing implementations of CDS alerts, detailed evaluations rarely occur because of the extensive labor involved in manual chart reviews to determine alert and response appropriateness. Further, most studies have solely evaluated alert overrides that are appropriate or justifiable. Our proposal expands the use of web-based monitoring tools with an interactive dashboard for evaluating CDS alert and response appropriateness that incorporates the predictive models. The dashboard provides 2 views, an alert detail view and a patient detail view, to provide a full history of alerts and help put the patient's events in context. Conclusion: The proposed research introduces several innovations to address the challenges and gaps in alert evaluations. This research can transform alert evaluation processes across healthcare settings, leading to improved CDS, reduced alert fatigue, and increased patient safety