Model risk: illuminating the black box

This paper presents latest thinking from the Institute and Faculty of Actuaries’ Model Risk Working Party and follows on from their Phase I work, Model Risk: Daring to Open the Black Box. This is a more practical paper and presents the contributors’ experiences of model risk gained from a wide range of financial and non-financial organisations with suggestions for good practice and proven methods to reduce model risk. After a recap of the Phase I work, examples of model risk communication are given covering communication: to the Board; to the regulator; and to external stakeholders. We present a practical framework for model risk management and quantification with examples of the key actors, processes and cultural challenge. Lessons learned are then presented from other industries that make extensive use of models and include the weather forecasting, software and aerospace industries. Finally, a series of case studies in practical model risk management and mitigation are presented from the contributors’ own experiences covering primarily financial services

Model risk – daring to open up the black box

With the increasing use of complex quantitative models in applications throughout the financial world, model risk has become a major concern. Such risk is generated by the potential inaccuracy and inappropriate use of models in business applications, which can lead to substantial financial losses and reputational damage. In this paper we deal with the management and measurement of model risk. First, a model risk framework is developed, adapting concepts such as risk appetite, monitoring, and mitigation to the particular case of model risk. The usefulness of such a framework for preventing losses associated with model risk is demonstrated through case studies. Second, we investigate the ways in which different ways of using and perceiving models within an organisation both lead to different model risks. We identify four distinct model cultures and argue that in conditions of deep model uncertainty, each of those cultures makes a valuable contribution to model risk governance. Thus the space of legitimate challenges to models is expanded, such that, in addition to a technical critique, operational and commercial concerns are also addressed. Third, we discuss through the examples of proxy modelling, longevity risk and investment advice, common methods and challenges for quantifying model risk. Difficulties arise in mapping model errors to actual financial impact. In the case of irreducible model uncertainty, it is necessary to employ a variety of measurement approaches, based on statistical inference, fitting multiple models, and stress and scenario analysis

Evaluation of the anti-inflammatory effects of synthesised tanshinone I and isotanshinone I analogues in zebrafish

During inflammation, dysregulated neutrophil behaviour can play a major role in a range of chronic inflammatory diseases, for many of which current treatments are generally ineffective. Recently, specific naturally occurring tanshinones have shown promising anti-inflammatory effects by targeting neutrophils in vivo, yet such tanshinones, and moreover, their isomeric isotanshinone counterparts, are still a largely underexplored class of compounds, both in terms of synthesis and biological effects. To explore the anti-inflammatory effects of isotanshinones, and the tanshinones more generally, a series of substituted tanshinone and isotanshinone analogues was synthesised, alongside other structurally similar molecules. Evaluation of these using a transgenic zebrafish model of neutrophilic inflammation revealed differential anti-inflammatory profiles in vivo, with a number of compounds exhibiting promising effects. Several compounds reduce initial neutrophil recruitment and/or promote resolution of neutrophilic inflammation, of which two also result in increased apoptosis of human neutrophils. In particular, the methoxy-substituted tanshinone 39 specifically accelerates resolution of inflammation without affecting the recruitment of neutrophils to inflammatory sites, making this a particularly attractive candidate for potential pro-resolution therapeutics, as well as a possible lead for future development of functionalised tanshinones as molecular tools and/or chemical probes. The structurally related β-lapachones promote neutrophil recruitment but do not affect resolution. We also observed notable differences in toxicity profiles between compound classes. Overall, we provide new insights into the in vivo anti-inflammatory activities of several novel tanshinones, isotanshinones, and structurally related compounds

Grand Challenges in global eye health: a global prioritisation process using Delphi method

Background We undertook a Grand Challenges in Global Eye Health prioritisation exercise to identify the key issues that must be addressed to improve eye health in the context of an ageing population, to eliminate persistent inequities in health-care access, and to mitigate widespread resource limitations. Methods Drawing on methods used in previous Grand Challenges studies, we used a multi-step recruitment strategy to assemble a diverse panel of individuals from a range of disciplines relevant to global eye health from all regions globally to participate in a three-round, online, Delphi-like, prioritisation process to nominate and rank challenges in global eye health. Through this process, we developed both global and regional priority lists. Findings Between Sept 1 and Dec 12, 2019, 470 individuals complete round 1 of the process, of whom 336 completed all three rounds (round 2 between Feb 26 and March 18, 2020, and round 3 between April 2 and April 25, 2020) 156 (46%) of 336 were women, 180 (54%) were men. The proportion of participants who worked in each region ranged from 104 (31%) in sub-Saharan Africa to 21 (6%) in central Europe, eastern Europe, and in central Asia. Of 85 unique challenges identified after round 1, 16 challenges were prioritised at the global level; six focused on detection and treatment of conditions (cataract, refractive error, glaucoma, diabetic retinopathy, services for children and screening for early detection), two focused on addressing shortages in human resource capacity, five on other health service and policy factors (including strengthening policies, integration, health information systems, and budget allocation), and three on improving access to care and promoting equity. Interpretation This list of Grand Challenges serves as a starting point for immediate action by funders to guide investment in research and innovation in eye health. It challenges researchers, clinicians, and policy makers to build collaborations to address specific challenge

Blood eosinophil levels as a biomarker in COPD

Chronic obstructive pulmonary disease (COPD) is a heterogeneous disorder and patients respond differently to treatment. Blood eosinophils are a potential biomarker to stratify patient subsets for COPD therapy. We reviewed the value of blood eosinophils in predicting exacerbation risk and response to corticosteroid treatment in the available literature (PubMed articles in English; keywords: "COPD" and "eosinophil"; published prior to May 2017). Overall, clinical data suggest that in patients with a history of COPD exacerbations, a higher blood eosinophil count predicts an increased risk of future exacerbations and is associated with improved response to treatment with inhaled corticosteroids (in combination with long-acting bronchodilator[s]). Blood eosinophils are therefore a promising biomarker for phenotyping patients with COPD, although prospective studies are needed to assess blood eosinophils as a biomarker of corticosteroid response for this

Examining the Threat of ChatGPT to the Validity of Short Answer Assessments in an Undergraduate Medical Program

OBJECTIVES ChatGPT is an artificial intelligence model that can interpret free-text prompts and return detailed, human-like responses across a wide domain of subjects. This study evaluated the extent of the threat posed by ChatGPT to the validity of short-answer assessment problems used to examine pre-clerkship medical students in our undergraduate medical education program. METHODS Forty problems used in prior student assessments were retrieved and stratified by levels of Bloom's Taxonomy. Thirty of these problems were submitted to ChatGPT-3.5. For the remaining 10 problems, we retrieved past minimally passing student responses. Six tutors graded each of the 40 responses. Comparison of performance between student-generated and ChatGPT-generated answers aggregated as a whole and grouped by Bloom's levels of cognitive reasoning, was done using t-tests, ANOVA, Cronbach's alpha, and Cohen's d. Scores for ChatGPT-generated responses were also compared to historical class average performance. RESULTS ChatGPT-generated responses received a mean score of 3.29 out of 5 (n = 30, 95% CI 2.93-3.65) compared to 2.38 for a group of students meeting minimum passing marks (n = 10, 95% CI 1.94-2.82), representing higher performance ( P  = .008, η 2  = 0.169), but was outperformed by historical class average scores on the same 30 problems (mean 3.67, P  = .018) when including all past responses regardless of student performance level. There was no statistically significant trend in performance across domains of Bloom's Taxonomy. CONCLUSION While ChatGPT was able to pass short answer assessment problems spanning the pre-clerkship curriculum, it outperformed only underperforming students. We remark that tutors in several cases were convinced that ChatGPT-produced responses were produced by students. Risks to assessment validity include uncertainty in identifying struggling students and inability to intervene in a timely manner. The performance of ChatGPT on problems requiring increasing demands of cognitive reasoning warrants further research