Master your Metrics with Calibration

Machine learning models deployed in real-world applications are often evaluated with precision-based metrics such as F1-score or AUC-PR (Area Under the Curve of Precision Recall). Heavily dependent on the class prior, such metrics make it difficult to interpret the variation of a model's performance over different subpopulations/subperiods in a dataset. In this paper, we propose a way to calibrate the metrics so that they can be made invariant to the prior. We conduct a large number of experiments on balanced and imbalanced data to assess the behavior of calibrated metrics and show that they improve interpretability and provide a better control over what is really measured. We describe specific real-world use-cases where calibration is beneficial such as, for instance, model monitoring in production, reporting, or fairness evaluation.Comment: Presented at IDA202

Analysis of Serological Surveillance Data: Patterns of Blood Borne Infection and Heterogeneity in People who Inject Drugs

A large proportion of blood-borne viruses (BBV) are transmitted via injecting drug use. Understanding patterns of risk and monitoring trends over time in people who inject drugs (PWID) is therefore a crucial part of developing public health policy. Risks of infection with hepatitis C and B (HCV, HBV) and HIV in PWID are investigated using serial cross-sectional surveillance data, in particular via force of infection (FOI) models. Standard models are extended to include a wealth of covariate data. Individual variability is considered via bivariate shared frailty models and correlations between infections. Gamma, inverse Gaussian and time-varying frailty models are fitted to investigate how variability in risk evolves throughout injecting career. Finally, models are extended to the trivariate case and different forms of component frailty models are proposed. Recent initiates were found to be at high risk of infection, in particular in London and the North West. Subsequently the FOI is broadly constant, and similar across different regions. Frailty models indicate that there is substantial individual variability in risk, although this declines over the course of injecting career. Females have higher overall risks of HCV and HBV, but are less heterogeneous than males. Including covariate data on demographics and a number of risk factors only resulted in modest reductions in frailty variance. Correlations between HCV-HBV and HBV-HIV associations were stronger than for HCV-HIV; trivariate models including additional pairwise components for HCV-HBV and HBV-HIV provided an improvement in model fit compared to a shared frailty model. Many of the results in this thesis point towards greater variation in risk at initiation, potentially due to the varied circumstances in which individuals start injecting, followed by more comparable risks in those with established injecting behaviour. The relative importance of injecting and sexual risk may explain patterns of risk and correlation in the three infections

Composite load spectra for select space propulsion structural components

The objective of this program is to develop generic load models with multiple levels of progressive sophistication to simulate the composite (combined) load spectra that are induced in space propulsion system components, representative of Space Shuttle Main Engines (SSME), such as transfer ducts, turbine blades, and liquid oxygen posts and system ducting. The first approach will consist of using state of the art probabilistic methods to describe the individual loading conditions and combinations of these loading conditions to synthesize the composite load spectra simulation. The second approach will consist of developing coupled models for composite load spectra simulation which combine the deterministic models for composite load dynamic, acoustic, high pressure, and high rotational speed, etc., load simulation using statistically varying coefficients. These coefficients will then be determined using advanced probabilistic simulation methods with and without strategically selected experimental data

Event-by-event Hydrodynamic Simulations for Relativistic Heavy-ion Collisions

In this thesis, I show my Ph.D. work on event-by-event hydrodynamic simulations for relativistic heavy-ion collision. I show that event-by-event hydrodynamic simulations have become an indispensable tool for studying relativistic heavy-ion collisions and how it can be used to explain many phenomena. Different chapters focus on different topics; it mainly includes: Chap 2: comparison between single-shot hydrodynamics event-by-event hydrodynamic simulations. Chap 3: using the elliptic and triangular flow data measured by the ALICE collaboration at the LHC to constrain initial condition models. Chap 4: study on correlations between event-plane angles. Chap 5: how resonance decay calculation can be speed up by a factor of 10. Chap 6: study on fluctuations of event planes angle {\Psi}n(pT) and their theoretical and experimental consequences. Chap 7: sampling particles according to the Cooper-Frye formula.Comment: Zhi Qiu's Ph.D. thesi

Markov chains and schrodinger bridges

The study of sequences of dependent random variables arose at the beginning of the twentieth century. In 1906 the Russian mathematician Andrei Andreyevich Markov (1856-1922), a Chebyshev’s pupil, introduced some mathematical models to this end. His focus was where the present is a sufficient statistis of the past to predict the future. These sequence have been named Markov chains. Even if it was a significant step in probability theory history, these models were not immediately considered by the scientific community. They were really appreciated only a few years later. Indeed, the study of Markov chains hugely spread only from the 1950s on. Nowadays, on the other hand, they are utilized in a variety of applications ranging from biology to psychology, from genetics to electrical engineering. It is interesting to study how such models evolve over time and if they converge to a stationary situation, namely there is a limiting probability distribution. The convergence of a Markov chain, however is not always guaranteed, and it is not known a priori how much time takes to converge. These facts make the use of a Markov chain model more complex than its relatively simple theory. Suppose we only deal with Markov chains whose convergence is guaranteed. In many applications, it is desirable to control the Markov chain by changing its transition mechanisms so as to achieve minimum cost, minimum queue length, etc. Another goal may be to drive the chain to a desired distribution at a given final time. This is achieve by the theory of Schrödinger bridges. The purpose of this thesis is to describe the recently developed theory of Schrödinger bridges for Markov chains, and to investigate its effectiveness by simulation on various examples. Of particular interest to us are chains that converges slowly to the equilibrium distribution such as those that arise from random geometric graphs. Schrödinger bridges allow in principle the possibility of controlling a chain to its invariant distribution in finite time. The outline of this thesis is as follows. In Chapters 1-3, we collect some basics material on probability, combinatorics and random variables; In Chapters 4-7, we introduce Markov chains, their properties and classificate them. We then give some examples distinguishing a Markov chain according its state space, namely finite or countable. Finally, we analyze the most important examples previously given. In Chapters 8-9, first we deal with general maximum entropy problems, then we focus on the theory of Schrödinger bridges. In Chapters 10-11, after introducing the average consensus problem, we discuss the importance of random geoemtric graphs. Finally we explain the algorithm of simulation for Schrödinger bridges giving a time analysis of its execution

Conditional Density Models Integrating Fuzzy and Probabilistic Representations of Uncertainty

__Abstract__ Conditional density estimation is an important problem in a variety of areas such as system identification, machine learning, artificial intelligence, empirical economics, macroeconomic analysis, quantitative finance and risk management. This work considers the general problem of conditional density estimation, i.e., estimating and predicting the density of a response variable as a function of covariates. The semi-parametric models proposed and developed in this work combine fuzzy and probabilistic representations of uncertainty, while making very few assumptions regarding the functional form of the response variable's density or changes of the functional form across the space of covariates. These models possess sufficient generalization power to approximate a non-standard density and the ability to describe the underlying process using simple linguistic descriptors despite the complexity and possible non-linearity of this process. These novel models are applied to real world quantitative finance and risk management problems by analyzing financial time-series data containing non-trivial statistical properties, such as fat tails, asymmetric distributions and changing variation over time

Insurance loss coverage under restricted risk classification

Insurers hope to make profit through pooling policies from a large number of individuals. Unless the risk in question is similar for all potential customers, an insurer is exposed to the possibility of adverse selection by attracting only high-risk individuals. To counter this, insurers have traditionally employed underwriting principles, identifying suitable risk factors to subdivide their potential customers into homogeneous risk groups, based on which risk-related premiums can be charged. In reality, however, insurers may not have all the information reflecting individuals' risks due to information asymmetry or restrictions on using certain risk factors by regulators. In either case, conventional wisdom suggests that the absence of risk classification in an insurance market is likely to lead to a vicious circle: increasing premium with the prime aim to recover losses from over-subscription by high risks would lead to more low risks dropping out of the market; and eventually leading to a collapse of the whole insurance system, i.e. an adverse selection spiral. However, this concept is difficult to reconcile with the successful operation of many insurance markets, even in the presence of some restrictions on risk classification by regulators. Theoretical analysis of polices under asymmetric information began in the 1960s and 1970s (Arrow (1963), Pauly (1974), Rothschild & Stiglitz (1976)), by which time the adverse consequences of information asymmetry had already been widely accepted. However, empirical test results of its presence are mixed and varied by markets. Arguably from society's viewpoint, the high risks are those who most need insurance. That is, if the social purpose of insurance is to compensate the population's losses, then insuring high risks contributes more to this purpose than insuring low risks. In this case, restriction on risk classification may be reasonable, otherwise premium for high risks would be too high to be affordable. Thus, the traditional insurers' risk classification practices might be considered as contrary to this social purpose. To highlight this issue, ''loss coverage'' was introduced in Thomas (2008) as the expected population losses compensated by insurance. A higher loss coverage indicates that a higher proportion of the population's expected losses can be compensated by insurance. This might be a good result for the population as a whole. A corollary of this concept is that, from a public policy perspective, adverse selection might not always be a bad thing. The author argued that a moderate degree of adverse selection could be negated by the positive influence of loss coverage. Therefore, when analysing the impact of restricting insurance risk classification, loss coverage might be a better measure than adverse selection. In this thesis, we model the outcome in an insurance market where a pooled premium is charged as a result of an absence of risk-classification. The outcome is characterised by four quantities: equilibrium premium, adverse selection, loss coverage and social welfare. Social welfare is defined as the total expected utility of individuals (including those who buy insurance and those who do not buy insurance) at a given premium. Using a general family of demand functions (of which iso-elastic demand and negative-exponential demand are examples) with a non-decreasing demand elasticity function with respect to premium, we first analyse the case when low and high risk-groups have the same constant demand elasticity. Then we generalise the results to the case where demand elasticities could be different. In general, equilibrium premium and adverse selection increase monotonically with demand elasticity, but loss coverage first increases and then decreases. The results are consistent with the ideas proposed by Thomas (2008, 2009) that loss coverage will be increased if a moderate degree of adverse selection is tolerated. Furthermore, we are able to show that, for iso-elastic demand with equal demand elasticities for high and low risks, social welfare moves in the same direction as loss coverage, i.e. social welfare at pooled premium is higher than at risk-differentiated premiums, when demand elasticity is less than 1. Therefore, we argue that loss coverage may be a better measure than adverse selection to quantify the impact of risk classification scheme being restricted. Moreover, (observable) loss coverage could also be a useful proxy for social welfare, which depends on unobservable utility functions. Therefore, adverse election is not always a bad thing, if demand elasticity is sufficiently low. The research findings should add to the wider public policy debate on these issues and provide necessary research insights for informed decision making by actuaries, regulators, policyholders, insurers, policy-makers, capital providers and other stakeholders

Évolution de la coopération au sein d'une population dans un environnement spatial

Mémoire numérisé par la Direction des bibliothèques de l'Université de Montréal