Sales Channel Optimization via Simulations Based on Observational Data with Delayed Rewards: A Case Study at LinkedIn

Training models on data obtained from randomized experiments is ideal for making good decisions. However, randomized experiments are often time-consuming, expensive, risky, infeasible or unethical to perform, leaving decision makers little choice but to rely on observational data collected under historical policies when training models. This opens questions regarding not only which decision-making policies would perform best in practice, but also regarding the impact of different data collection protocols on the performance of various policies trained on the data, or the robustness of policy performance with respect to changes in problem characteristics such as action- or reward- specific delays in observing outcomes. We aim to answer such questions for the problem of optimizing sales channel allocations at LinkedIn, where sales accounts (leads) need to be allocated to one of three channels, with the goal of maximizing the number of successful conversions over a period of time. A key problem feature constitutes the presence of stochastic delays in observing allocation outcomes, whose distribution is both channel- and outcome- dependent. We built a discrete-time simulation that can handle our problem features and used it to evaluate: a) a historical rule-based policy; b) a supervised machine learning policy (XGBoost); and c) multi-armed bandit (MAB) policies, under different scenarios involving: i) data collection used for training (observational vs randomized); ii) lead conversion scenarios; iii) delay distributions. Our simulation results indicate that LinUCB, a simple MAB policy, consistently outperforms the other policies, achieving a 18-47% lift relative to a rule-based policyComment: Accepted at REVEAL'22 Workshop (16th ACM Conference on Recommender Systems - RecSys 2022

Epsilon*: Privacy Metric for Machine Learning Models

We introduce Epsilon*, a new privacy metric for measuring the privacy risk of a single model instance prior to, during, or after deployment of privacy mitigation strategies. The metric does not require access to the training data sampling or model training algorithm. Epsilon* is a function of true positive and false positive rates in a hypothesis test used by an adversary in a membership inference attack. We distinguish between quantifying the privacy loss of a trained model instance and quantifying the privacy loss of the training mechanism which produces this model instance. Existing approaches in the privacy auditing literature provide lower bounds for the latter, while our metric provides a lower bound for the former by relying on an (${\epsilon}$,${\delta}$)-type of quantification of the privacy of the trained model instance. We establish a relationship between these lower bounds and show how to implement Epsilon* to avoid numerical and noise amplification instability. We further show in experiments on benchmark public data sets that Epsilon* is sensitive to privacy risk mitigation by training with differential privacy (DP), where the value of Epsilon* is reduced by up to 800% compared to the Epsilon* values of non-DP trained baseline models. This metric allows privacy auditors to be independent of model owners, and enables all decision-makers to visualize the privacy-utility landscape to make informed decisions regarding the trade-offs between model privacy and utility

Bayesian optimization for materials design

We introduce Bayesian optimization, a technique developed for optimizing time-consuming engineering simulations and for fitting machine learning models on large datasets. Bayesian optimization guides the choice of experiments during materials design and discovery to find good material designs in as few experiments as possible. We focus on the case when materials designs are parameterized by a low-dimensional vector. Bayesian optimization is built on a statistical technique called Gaussian process regression, which allows predicting the performance of a new design based on previously tested designs. After providing a detailed introduction to Gaussian process regression, we introduce two Bayesian optimization methods: expected improvement, for design problems with noise-free evaluations; and the knowledge-gradient method, which generalizes expected improvement and may be used in design problems with noisy evaluations. Both methods are derived using a value-of-information analysis, and enjoy one-step Bayes-optimality

Screening and Rapid Molecular Diagnosis of Tuberculosis in Prisons in Russia and Eastern Europe: A Cost-Effectiveness Analysis

<div><h3>Background</h3><p>Prisons of the former Soviet Union (FSU) have high rates of multidrug-resistant tuberculosis (MDR-TB) and are thought to drive general population tuberculosis (TB) epidemics. Effective prison case detection, though employing more expensive technologies, may reduce long-term treatment costs and slow MDR-TB transmission.</p> <h3>Methods and Findings</h3><p>We developed a dynamic transmission model of TB and drug resistance matched to the epidemiology and costs in FSU prisons. We evaluated eight strategies for TB screening and diagnosis involving, alone or in combination, self-referral, symptom screening, mass miniature radiography (MMR), and sputum PCR with probes for rifampin resistance (Xpert MTB/RIF). Over a 10-y horizon, we projected costs, quality-adjusted life years (QALYs), and TB and MDR-TB prevalence. Using sputum PCR as an annual primary screening tool among the general prison population most effectively reduced overall TB prevalence (from 2.78% to 2.31%) and MDR-TB prevalence (from 0.74% to 0.63%), and cost US$543/QALY for additional QALYs gained compared to MMR screening with sputum PCR reserved for rapid detection of MDR-TB. Adding sputum PCR to the currently used strategy of annual MMR screening was cost-saving over 10 y compared to MMR screening alone, but produced only a modest reduction in MDR-TB prevalence (from 0.74% to 0.69%) and had minimal effect on overall TB prevalence (from 2.78% to 2.74%). Strategies based on symptom screening alone were less effective and more expensive than MMR-based strategies. Study limitations included scarce primary TB time-series data in FSU prisons and uncertainties regarding screening test characteristics.</p> <h3>Conclusions</h3><p>In prisons of the FSU, annual screening of the general inmate population with sputum PCR most effectively reduces TB and MDR-TB prevalence, doing so cost-effectively. If this approach is not feasible, the current strategy of annual MMR is both more effective and less expensive than strategies using self-referral or symptom screening alone, and the addition of sputum PCR for rapid MDR-TB detection may be cost-saving over time.</p> <p> <em>Please see later in the article for the Editors' Summary</em></p> </div

The effects of alternative screening and diagnostic strategies on TB and MDR-TB prevalence.

<p>(A) Prevalence of TB (both non-MDR-TB and MDR-TB) among prison population over 10-y time horizon. (B) Prevalence of MDR-TB among prison population over 10-y time horizon. Strategy 1 (S1), self-referral only (no screening), is not shown.</p

Cost estimates.

<p>Costs given in US dollars. Values are from primary cost analysis except for sputum PCR, which was adjusted from <a href="http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.1001348#pmed.1001348-Vassall1" target="_blank">[14]</a>. Screening costs are applied to all individuals not currently being treated for active TB. Diagnostic costs are applied to those individuals who test positive and include only those additional tests and clinical evaluations not part of a given screening strategy's screening test. Further work-up costs to determine appropriate treatment (e.g., drug sensitivity testing) are included in treatment costs if they are not part of an earlier screening strategy. See <a href="http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.1001348#pmed.1001348.s011" target="_blank">Table S7</a> for further details.</p

Outcomes for country-specific analysis.

<p>Overall TB prevalence rates (A–C), MDR-TB prevalence rates (D–F), and cost-effectiveness frontiers (G–I) over 10 y within prisons in three countries. These outcomes reflect model prisons in Tajikistan (A, D, and G), the Russian Federation (B, E, and H), and Latvia (C, F, and I). Strategy 1 (S1), self-referral only (no screening), is not shown in tracings of overall and MDR-TB prevalence (A–F).</p

Natural history, diagnosis, and treatment of TB.

<p>Simplified diagram of the health states and transitions in our model. Screening and diagnostic alternatives affect the rate of transition from undetected to detected active disease, represented here by a dashed arrow. Death from all states is not shown. See Figures <a href="http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.1001348#pmed.1001348.s001" target="_blank">S1</a> and <a href="http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.1001348#pmed.1001348.s002" target="_blank">S2</a> for more detail regarding the model structure.</p