Judging Risk

Risk assessment plays an increasingly pervasive role in criminal justice in the United States at all stages of the process, from policing, to pre-trial, sentencing, corrections, and during parole. As efforts to reduce incarceration have led to adoption of risk-assessment tools, critics have begun to ask whether various instruments in use are valid and whether they might reinforce rather than reduce bias in criminal justice outcomes. Such work has neglected how decisionmakers use risk-assessment in practice. In this Article, we examine in detail the judging of risk assessment and we study why decisionmakers so often fail to consistently use such quantitative information

Expert Elicitation for Reliable System Design

This paper reviews the role of expert judgement to support reliability assessments within the systems engineering design process. Generic design processes are described to give the context and a discussion is given about the nature of the reliability assessments required in the different systems engineering phases. It is argued that, as far as meeting reliability requirements is concerned, the whole design process is more akin to a statistical control process than to a straightforward statistical problem of assessing an unknown distribution. This leads to features of the expert judgement problem in the design context which are substantially different from those seen, for example, in risk assessment. In particular, the role of experts in problem structuring and in developing failure mitigation options is much more prominent, and there is a need to take into account the reliability potential for future mitigation measures downstream in the system life cycle. An overview is given of the stakeholders typically involved in large scale systems engineering design projects, and this is used to argue the need for methods that expose potential judgemental biases in order to generate analyses that can be said to provide rational consensus about uncertainties. Finally, a number of key points are developed with the aim of moving toward a framework that provides a holistic method for tracking reliability assessment through the design process.Comment: This paper commented in: [arXiv:0708.0285], [arXiv:0708.0287], [arXiv:0708.0288]. Rejoinder in [arXiv:0708.0293]. Published at http://dx.doi.org/10.1214/088342306000000510 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Multiple Criteria Decision Analysis (MCDA) for evaluating new medicines in Health Technology Assessment and beyond: the Advance Value Framework

Escalating drug prices have catalysed the generation of numerous “value frameworks” with the aim of informing payers, clinicians and patients on the assessment process of new medicines for the purpose of coverage and treatment selection decisions. Although this is an important step towards a more inclusive Value Based Assessment (VBA) approach, aspects of these frameworks are based on weak methodologies and could potentially result in misleading recommendations or decisions. A Multiple Criteria Decision Analysis (MCDA) methodological process based on Multi Attribute Value Theory (MAVT) is adopted for building a multi-criteria evaluation model. A five-stage model-building process is followed, using a top-down “value-focused thinking” approach, involving literature reviews and expert consultations. A generic value tree is structured capturing decision-makers’ concerns for assessing the value of new medicines in the context of Health Technology Assessment (HTA) and in alignment with decision theory. The resulting value tree (Advance Value Tree) spans three levels of criteria (top level criteria clusters, mid-level criteria, bottom level sub-criteria or attributes) relating to five key domains that can be explicitly measured and assessed: (a) burden of disease, (b) therapeutic impact, (c) safety profile (d) innovation level, and (e) socioeconomic impact. A number of MAVT modelling techniques are introduced for operationalising (i.e. estimating) the model, for scoring the alternative options, assigning relative weights of importance to the criteria, and combining scores and weights. Overall, the combination of these MCDA modelling techniques for the elicitation and construction of value preferences across the generic value tree provides 3 a new value framework (Advance Value Framework) enabling the comprehensive measurement of value in a transparent and structured way. Given the flexibility to meet diverse requirements and become readily adaptable across different settings, it could be tested as a decision-support tool for decision-makers to aid coverage and reimbursement of new medicines

A survey of simulation techniques in commerce and defence

Despite the developments in Modelling and Simulation (M&S) tools and techniques over the past years, there has been a gap in the M&S research and practice in healthcare on developing a toolkit to assist the modellers and simulation practitioners with selecting an appropriate set of techniques. This study is a preliminary step towards this goal. This paper presents some results from a systematic literature survey on applications of M&S in the commerce and defence domains that could inspire some improvements in the healthcare. Interim results show that in the commercial sector Discrete-Event Simulation (DES) has been the most widely used technique with System Dynamics (SD) in second place. However in the defence sector, SD has gained relatively more attention. SD has been found quite useful for qualitative and soft factors analysis. From both the surveys it becomes clear that there is a growing trend towards using hybrid M&S approaches

Analyzing Supply Risks and Product Characteristics - A Systematic Literature Review

The environment in which companies operate is increasingly volatile and complex. This results in an increased exposure to disruptions. Past disruptions have especially affected procurement. Thus, companies need to prepare for disruptions. The preparedness for disruptions in the context of procurement is significantly influenced by the design of the procurement strategy. However, a high number of purchased articles and a variety of influencing factors lead to high complexity in procurement. The systematic design of the procurement strategy should therefore take into account the criticality of the purchased articles. This enables to focus on the purchased articles that have a high impact on the disruption preparedness. Existing approaches regarding the design of the procurement strategy in uncertain environments either lack practical applicability and objective evaluation or focus on the criticality of raw materials rather than of purchased articles. Therefore, a data-based approach for the systematic design of the procurement strategy in the context of the Internet of Production has been proposed. One central aspect of this approach is the identification of success-critical purchased articles. Thus, this paper proposes a framework for characterizing purchased articles regarding supply risks by combining two systematic analyses. First, a systematic literature review is performed to answer the question of what factors can be used to describe the supply risks of purchased articles. The results are analyzed regarding sources and impacts of risks and thus contribute to a structured characterization of supply risks. Second, existing criticality assessment approaches for raw materials are analyzed to identify categories and indicators that describe purchased articles. The results of both reviews provide the basis for linking product characteristics with supply risks and assessing product criticality which will be integrated into an app prototype

Provision and Collection of Safety Evidence: A Systematic Literature Review

Safety-Critical Systems (SCS) are becoming more and more present in modern societies’ daily lives, increasing people’s dependence on them. Current SCS are firmly based on computational technology; possible failures in the operation of these systems can lead to accidents and endanger human life, as well as damage the environment and property. SCS are present in many areas such as avionics, automotive systems, industrial plants (chemical, oil & gas, and nuclear), medical devices, railroad control, defense, and aerospace systems. Companies that develop SCS must present evidence of their safety to obtain certification and authorization. This paper presents a Systematic Literature Review (SLR) to investigate processes, tools, and techniques for collecting and managing safety evidence in SCS. The authors conducted this SLR according to the guidelines proposed by Kitchenham and Charters. The SLR comprises seven (7) research questions that investigate essential aspects of collecting and managing safety evidence. The primary studies analyzed in this SLR were selected based on a search string applied into four data sources: ACM, IEEE Xplore, SpringerLink, and ScienceDirect. Data extraction considered (fifty-one) 51 primary studies. The authors identified eleven (11) different approaches covering processes, tools, and techniques for collecting and managing safety evidence. Despite other SLR works conducted about safety evidence, none of them focused on the details related to safety evidence collection. We found that very few approaches focused specifically on the process of collecting safety evidence

A PRISMA-driven systematic mapping study on system assurance weakeners

Context: An assurance case is a structured hierarchy of claims aiming at demonstrating that a given mission-critical system supports specific requirements (e.g., safety, security, privacy). The presence of assurance weakeners (i.e., assurance deficits, logical fallacies) in assurance cases reflects insufficient evidence, knowledge, or gaps in reasoning. These weakeners can undermine confidence in assurance arguments, potentially hindering the verification of mission-critical system capabilities. Objectives: As a stepping stone for future research on assurance weakeners, we aim to initiate the first comprehensive systematic mapping study on this subject. Methods: We followed the well-established PRISMA 2020 and SEGRESS guidelines to conduct our systematic mapping study. We searched for primary studies in five digital libraries and focused on the 2012-2023 publication year range. Our selection criteria focused on studies addressing assurance weakeners at the modeling level, resulting in the inclusion of 39 primary studies in our systematic review. Results: Our systematic mapping study reports a taxonomy (map) that provides a uniform categorization of assurance weakeners and approaches proposed to manage them at the modeling level. Conclusion: Our study findings suggest that the SACM (Structured Assurance Case Metamodel) -- a standard specified by the OMG (Object Management Group) -- may be the best specification to capture structured arguments and reason about their potential assurance weakeners