Evaluating Artificial Intelligence in Clinical Settings-Let Us Not Reinvent the Wheel

Given the requirement to minimize the risks and maximize the benefits of technology applications in health care provision, there is an urgent need to incorporate theory-informed health IT (HIT) evaluation frameworks into existing and emerging guidelines for the evaluation of artificial intelligence (AI). Such frameworks can help developers, implementers, and strategic decision makers to build on experience and the existing empirical evidence base. We provide a pragmatic conceptual overview of selected concrete examples of how existing theory-informed HIT evaluation frameworks may be used to inform the safe development and implementation of AI in health care settings. The list is not exhaustive and is intended to illustrate applications in line with various stakeholder requirements. Existing HIT evaluation frameworks can help to inform AI-based development and implementation by supporting developers and strategic decision makers in considering relevant technology, user, and organizational dimensions. This can facilitate the design of technologies, their implementation in user and organizational settings, and the sustainability and scalability of technologies.</p

Ligand-Mediated Receptor Assembly as an End Point for High-Throughput Chemical Toxicity Screening

The high throughput screening of chemicals for interaction with intracellular targets is gaining prominence in the toxicity evaluation of environmental chemicals. We describe ligand-mediated receptor assembly as an early event in receptor signaling and its application to the screening of chemicals for interaction with targeted receptors. We utilized bioluminescence resonance energy transfer (BRET) to detect and quantify assembly of the methyl farnesoate receptor (MfR) in response to various high-production volume and other chemicals. The hormone methyl farnesoate binds to the MfR to regulate various aspects of reproduction and development in crustaceans. The MfR protein subunits Met and SRC, cloned from <i>Daphnia pulex</i>, were fused to the fluorophore, mAmetrine and the photon generator, Rluc2, respectively. Ligand-mediated receptor assembly was measured by photon transfer from the photon donor to the fluorophore resulting in fluorescence emission. Overall, the BRET assay had comparable or greater sensitivity as compared to a traditional reporter gene assay. Further, chemicals that screened positive in the BRET assay also stimulated phenotypic outcomes in daphnids that result from MfR signaling. We concluded the BRET assay is an accurate, sensitive, and cost/time efficient alternative to traditional screening assays

Effectiveness and usage of a decision support system to improve stroke prevention in general practice: A cluster randomized controlled trial

Adherence to guidelines pertaining to stroke prevention in patients with atrial fibrillation is poor. Decision support systems have shown promise in increasing guideline adherence. To improve guideline adherence with a non-obtrusive clinical decision support system integrated in the workflow. Secondly, we seek to capture reasons for guideline non-adherence. A cluster randomized controlled trial in Dutch general practices. A decision support system was developed that implemented properties positively associated with effectiveness: real-time, non-interruptive and based on data from electronic health records. Recommendations were based on the Dutch general practitioners guideline for atrial fibrillation that uses the CHA2DS2-VAsc for stroke risk stratification. Usage data and responses to the recommendations were logged. Effectiveness was measured as adherence to the guideline. We used a chi square to test for group differences and a mixed effects model to correct for clustering and baseline adherence. Our analyses included 781 patients. Usage of the system was low (5%) and declined over time. In total, 76 notifications received a response: 58% dismissal and 42% acceptance. At the end of the study, both groups had improved, by 8% and 5% respectively. There was no statistically significant difference between groups (Control: 50%, Intervention: 55% P = 0.23). Clustered analysis revealed similar results. Only one usable reasons for non-adherence was captured. Our study could not demonstrate the effectiveness of a decision support system in general practice, which was likely due to lack of use. Our findings should be used to develop next generation decision support systems that are effective in the challenging setting of general practic

Acceptance and barriers pertaining to a general practice decision support system for multiple clinical conditions: a mixed methods evaluation

Background Many studies have investigated the use of clinical decision support systems as a means to improve care, but have thus far failed to show significant effects on patient-related outcomes. We developed a clinical decision support system that attempted to address issues that were identified in these studies. The system was implemented in Dutch general practice and was designed to be both unobtrusive and to respond in real time. Despite our efforts, usage of the system was low. In the current study we perform a mixed methods evaluation to identify remediable barriers which led to disappointing usage rates for our system. Methods A mixed methods evaluation employing an online questionnaire and focus group. The focus group was organized to clarify free text comments and receive more detailed feedback from general practitioners. Topics consisted of items based on results from the survey and additional open questions. Results The response rate for the questionnaire was 94%. Results from the questionnaire and focus group can be summarized as follows: The system was perceived as interruptive, despite its design. Participants felt that there were too many recommendations and that the relevance of the recommendations varied. Demographic based recommendations (e.g. age) were often irrelevant, while specific risk-based recommendations (e.g. diagnosis) were more relevant. The other main barrier to use was lack of time during the patient visit. Conclusion These results are likely to be useful to other researchers who are attempting to address the problems of interruption and alert fatigue in decision support.</p

Usage of the system during the 9 months.

<p>Usage of the system during the 9 months.</p

User activity

<p>Average usage (responses / notification) per general practitioner.</p

The CONSORT flow diagram.

<p>The CONSORT flow diagram.</p

Usage of the CDDS plugin over time.

<p>Blue dots indicate clicked notifications, red dots indicate responses to recommendations (clicking accept or decline), orange dots indicate the number of times the GP hovered over the notification window with their mouse regardless of whether the notification was opened.</p

The popup after clicking on a notification, containing background information, an actionable recommendation and response buttons that allowed the GP to either accept (1) or decline (2) the advice.

<p>The popup after clicking on a notification, containing background information, an actionable recommendation and response buttons that allowed the GP to either accept (1) or decline (2) the advice.</p