Supervised Machine Learning Under Test-Time Resource Constraints: A Trade-off Between Accuracy and Cost

The past decade has witnessed how the field of machine learning has established itself as a necessary component in several multi-billion-dollar industries. The real-world industrial setting introduces an interesting new problem to machine learning research: computational resources must be budgeted and cost must be strictly accounted for during test-time. A typical problem is that if an application consumes x additional units of cost during test-time, but will improve accuracy by y percent, should the additional x resources be allocated? The core of this problem is a trade-off between accuracy and cost. In this thesis, we examine components of test-time cost, and develop different strategies to manage this trade-off. We first investigate test-time cost and discover that it typically consists of two parts: feature extraction cost and classifier evaluation cost. The former reflects the computational efforts of transforming data instances to feature vectors, and could be highly variable when features are heterogeneous. The latter reflects the effort of evaluating a classifier, which could be substantial, in particular nonparametric algorithms. We then propose three strategies to explicitly trade-off accuracy and the two components of test-time cost during classifier training. To budget the feature extraction cost, we first introduce two algorithms: GreedyMiser and Anytime Representation Learning (AFR). GreedyMiser employs a strategy that incorporates the extraction cost information during classifier training to explicitly minimize the test-time cost. AFR extends GreedyMiser to learn a cost-sensitive feature representation rather than a classifier, and turns traditional Support Vector Machines (SVM) into test- time cost-sensitive anytime classifiers. GreedyMiser and AFR are evaluated on two real-world data sets from two different application domains, and both achieve record performance. We then introduce Cost Sensitive Tree of Classifiers (CSTC) and Cost Sensitive Cascade of Classifiers (CSCC), which share a common strategy that trades-off the accuracy and the amortized test-time cost. CSTC introduces a tree structure and directs test inputs along different tree traversal paths, each is optimized for a specific sub-partition of the input space, extracting different, specialized subsets of features. CSCC extends CSTC and builds a linear cascade, instead of a tree, to cope with class-imbalanced binary classification tasks. Since both CSTC and CSCC extract different features for different inputs, the amortized test-time cost is greatly reduced while maintaining high accuracy. Both approaches out-perform the current state-of-the-art on real-world data sets. To trade-off accuracy and high classifier evaluation cost of nonparametric classifiers, we propose a model compression strategy and develop Compressed Vector Machines (CVM). CVM focuses on the nonparametric kernel Support Vector Machines (SVM), whose test-time evaluation cost is typically substantial when learned from large training sets. CVM is a post-processing algorithm which compresses the learned SVM model by reducing and optimizing support vectors. On several benchmark data sets, CVM maintains high test accuracy while reducing the test-time evaluation cost by several orders of magnitude

Efficient techniques for cost-sensitive learning with multiple cost considerations

University of Technology, Sydney. Faculty of Engineering and Information Technology.Cost-sensitive learning is one of the active research topics in data mining and machine learning, designed for dealing with the non-uniform cost of misclassification errors. In the last ten to fifteen years, diverse learning methods and techniques were proposed to minimize the total cost of misclassification, test and other types. This thesis studies the up-to-date prevailing cost-sensitive learning methods and techniques, and proposes some new and efficient cost-sensitive learning methods and techniques in the following three areas: First, we focus on the data over-fitting issue. In an applied context of cost-sensitive learning, many existing data mining algorithms can generate good results on training data but normally do not produce an optimal model when applied to unseen data in real world applications. We deal with this issue by developing three simple and efficient strategies - feature selection, smoothing and threshold pruning to overcome data over-fitting in cost-sensitive learning. This work sets up a solid foundation for our further research and analysis in this thesis in the other areas of cost-sensitive learning. Second, we design and develop an innovative and practical objective-resource cost-sensitive learning framework for addressing a real world issue where multiple cost units are involved. A lazy cost-sensitive decision tree is built to minimize the objective cost subjecting to given budgets of other resource costs. Finally, we study semi-supervised learning approach in the context of cost-sensitive learning. Two new classification algorithms are proposed to learn cost-sensitive classifier from training datasets with a small amount of labelled data and plenty unlabelled data. We also analyse the impact of the different input parameters to the performance of our new algorithms

Colorectal cancer: Cost-effectiveness of screening and chemoprevention in average risk males

This study is an economic evaluation of currently recommended colorectal cancer (CRC) screening procedures, and strategies that incorporate chemopreventive options such as aspirin or a cycooxygenase-2 inhibitor. A decision analysis model was constructed to compare alternative CRC screening strategies. A Markov model was employed to simulate the natural history of CRC. Quality adjusted life years were used as the primary outcome measure. The base case analysis represents the overall cost and effectiveness associated with each screening strategy. Incremental cost-effectiveness ratios (ICERs) were calculated for each screening strategy. One-way sensitivity analyses were performed to assess the factors that have the greatest effect on the cost-effectiveness of screening. The most cost-effective screening strategy was Fecal Occult Blood Test (FOBT); followed by FOBT plus aspirin, colonoscopy, and colonoscopy plus aspirin. The ICER of FOBT was {dollar}13,014.85 compared to Natural History. The model was sensitive to the costs of FOBT, colonoscopy, and aspirin

Development of a rapid cost effective test for ovine Johne's Disease based on testing of pooled faeces : final report.

This project was undertaken to develop and evaluate a rapid, cost-effective, flock test for Mycobacterium paratuberculosis in pooled faecal samples, based on hybridisation-capture polymerase chain reaction (HC-PCR). However, a simpler direct technique (DPCR) was found to be more sensitive than HC-PCR. About 67% of culture positive pooled faecal samples were positive when tested using DPCR. In a blind trial, 83% of 12 farms identified by culture of pooled faecal samples were detected using DPCR. The cost of DPCR is no greater than that of other flock detection strategies. The test is suitable for use in the National Ovine Johne’s Disease Control and Evaluation Program. A constraint exists in that Veterinary Committee does not recognise the results of DNA-based tests for M. paratuberculosis as being definitive. The costs of follow-up testing to confirm infection are high. Recommendations are made to improve the test and reduce its cost

The clinical effectiveness and cost-effectiveness of screening for open angle glaucoma : a systematic review and economic evaluation

Objectives: To assess whether open angle glaucoma (OAG) screening meets the UK National Screening Committee criteria, to compare screening strategies with case finding, to estimate test parameters, to model estimates of cost and cost-effectiveness, and to identify areas for future research. Data sources: Major electronic databases were searched up to December 2005. Review methods: Screening strategies were developed by wide consultation. Markov submodels were developed to represent screening strategies. Parameter estimates were determined by systematic reviews of epidemiology, economic evaluations of screening, and effectiveness (test accuracy, screening and treatment). Tailored highly sensitive electronic searches were undertaken. Results: Most potential screening tests reviewed had an estimated specificity of 85% or higher. No test was clearly most accurate, with only a few, heterogeneous studies for each test. No randomised controlled trials (RCTs) of screening were identified. Based on two treatment RCTs, early treatment reduces the risk of progression. Extrapolating from this, and assuming accelerated progression with advancing disease severity, without treatment the mean time to blindness in at least one eye was approximately 23 years, compared to 35 years with treatment. Prevalence would have to be about 3–4% in 40 year olds with a screening interval of 10 years to approach costeffectiveness. It is predicted that screening might be cost-effective in a 50-year-old cohort at a prevalence of 4% with a 10-year screening interval. General population screening at any age, thus, appears not to be cost-effective. Selective screening of groups with higher prevalence (family history, black ethnicity) might be worthwhile, although this would only cover 6% of the population. Extension to include other at-risk cohorts (e.g. myopia and diabetes) would include 37% of the general population, but the prevalence is then too low for screening to be considered cost-effective. Screening using a test with initial automated classification followed by assessment by a specialised optometrist, for test positives, was more cost-effective than initial specialised optometric assessment. The cost-effectiveness of the screening programme was highly sensitive to the perspective on costs (NHS or societal). In the base-case model, the NHS costs of visual impairment were estimated as £669. If annual societal costs were £8800, then screening might be considered cost-effective for a 40-year-old cohort with 1% OAG prevalence assuming a willingness to pay of £30,000 per quality-adjusted life-year. Of lesser importance were changes to estimates of attendance for sight tests, incidence of OAG, rate of progression and utility values for each stage of OAG severity. Cost-effectiveness was not particularly sensitive to the accuracy of screening tests within the ranges observed. However, a highly specific test is required to reduce large numbers of false-positive referrals. The findings that population screening is unlikely to be cost-effective are based on an economic model whose parameter estimates have considerable uncertainty. In particular, if rate of progression and/or costs of visual impairment are higher than estimated then screening could be cost-effective. Conclusions: While population screening is not costeffective, the targeted screening of high-risk groups may be. Procedures for identifying those at risk, for quality assuring the programme, as well as adequate service provision for those screened positive would all be needed. Glaucoma detection can be improved by increasing attendance for eye examination, and improving the performance of current testing by either refining practice or adding in a technology-based first assessment, the latter being the more cost-effective option. This has implications for any future organisational changes in community eye-care services. Further research should aim to develop and provide quality data to populate the economic model, by conducting a feasibility study of interventions to improve detection, by obtaining further data on costs of blindness, risk of progression and health outcomes, and by conducting an RCT of interventions to improve the uptake of glaucoma testing.Peer reviewedPublisher PD

Cost-effectiveness of non-invasive methods for assessment and monitoring of liver fibrosis and cirrhosis in patients with chronic liver disease: systematic review and economic evaluation

BACKGROUND: Liver biopsy is the reference standard for diagnosing the extent of fibrosis in chronic liver disease; however, it is invasive, with the potential for serious complications. Alternatives to biopsy include non-invasive liver tests (NILTs); however, the cost-effectiveness of these needs to be established. OBJECTIVE: To assess the diagnostic accuracy and cost-effectiveness of NILTs in patients with chronic liver disease. DATA SOURCES: We searched various databases from 1998 to April 2012, recent conference proceedings and reference lists. METHODS: We included studies that assessed the diagnostic accuracy of NILTs using liver biopsy as the reference standard. Diagnostic studies were assessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. Meta-analysis was conducted using the bivariate random-effects model with correlation between sensitivity and specificity (whenever possible). Decision models were used to evaluate the cost-effectiveness of the NILTs. Expected costs were estimated using a NHS perspective and health outcomes were measured as quality-adjusted life-years (QALYs). Markov models were developed to estimate long-term costs and QALYs following testing, and antiviral treatment where indicated, for chronic hepatitis B (HBV) and chronic hepatitis C (HCV). NILTs were compared with each other, sequential testing strategies, biopsy and strategies including no testing. For alcoholic liver disease (ALD), we assessed the cost-effectiveness of NILTs in the context of potentially increasing abstinence from alcohol. Owing to a lack of data and treatments specifically for fibrosis in patients with non-alcoholic fatty liver disease (NAFLD), the analysis was limited to an incremental cost per correct diagnosis. An analysis of NILTs to identify patients with cirrhosis for increased monitoring was also conducted. RESULTS: Given a cost-effectiveness threshold of £20,000 per QALY, treating everyone with HCV without prior testing was cost-effective with an incremental cost-effectiveness ratio (ICER) of £9204. This was robust in most sensitivity analyses but sensitive to the extent of treatment benefit for patients with mild fibrosis. For HBV [hepatitis B e antigen (HBeAg)-negative)] this strategy had an ICER of £28,137, which was cost-effective only if the upper bound of the standard UK cost-effectiveness threshold range (£30,000) is acceptable. For HBeAg-positive disease, two NILTs applied sequentially (hyaluronic acid and magnetic resonance elastography) were cost-effective at a £20,000 threshold (ICER: £19,612); however, the results were highly uncertain, with several test strategies having similar expected outcomes and costs. For patients with ALD, liver biopsy was the cost-effective strategy, with an ICER of £822. LIMITATIONS: A substantial number of tests had only one study from which diagnostic accuracy was derived; therefore, there is a high risk of bias. Most NILTs did not have validated cut-offs for diagnosis of specific fibrosis stages. The findings of the ALD model were dependent on assuptions about abstinence rates assumptions and the modelling approach for NAFLD was hindered by the lack of evidence on clinically effective treatments. CONCLUSIONS: Treating everyone without NILTs is cost-effective for patients with HCV, but only for HBeAg-negative if the higher cost-effectiveness threshold is appropriate. For HBeAg-positive, two NILTs applied sequentially were cost-effective but highly uncertain. Further evidence for treatment effectiveness is required for ALD and NAFLD. STUDY REGISTRATION: This study is registered as PROSPERO CRD42011001561. FUNDING: The National Institute for Health Research Health Technology Assessment programme

To test or to treat? an analysis of influenza testing and Antiviral treatment strategies using economic computer modeling

Background: Due to the unpredictable burden of pandemic influenza, the best strategy to manage testing, such as rapid or polymerase chain reaction (PCR), and antiviral medications for patients who present with influenza-like illness (ILI) is unknown. Methodology/Principal Findings: We developed a set of computer simulation models to evaluate the potential economic value of seven strategies under seasonal and pandemic influenza conditions: (1) using clinical judgment alone to guide antiviral use, (2) using PCR to determine whether to initiate antivirals, (3) using a rapid (point-of-care) test to determine antiviral use, (4) using a combination of a point-of-care test and clinical judgment, (5) using clinical judgment and confirming the diagnosis with PCR testing, (6) treating all with antivirals, and (7) not treating anyone with antivirals. For healthy younger adults (<65 years old) presenting with ILI in a seasonal influenza scenario, strategies were only cost-effective from the societal perspective. Clinical judgment, followed by PCR and point-of-care testing, was found to be cost-effective given a high influenza probability. Doubling hospitalization risk and mortality (representing either higher risk individuals or more virulent strains) made using clinical judgment to guide antiviral decision-making cost-effective, as well as PCR testing, point-of-care testing, and point-of-care testing used in conjunction with clinical judgment. For older adults (≥65 years old), in both seasonal and pandemic influenza scenarios, employing PCR was the most cost-effective option, with the closest competitor being clinical judgment (when judgment accuracy ≥50%). Point-of-care testing plus clinical judgment was cost-effective with higher probabilities of influenza. Treating all symptomatic ILI patients with antivirals was cost-effective only in older adults. Conclusions/Significance: Our study delineated the conditions under which different testing and antiviral strategies may be cost-effective, showing the importance of accuracy, as seen with PCR or highly sensitive clinical judgment. © 2010 Lee et al

Development of a recombinase polymerase amplification lateral flow assay for the detection of active Trypanosoma evansi infections

Author summary Neglected tropical diseases (NTDs) affecting humans and/or domestic animals severely impair the socio-economic development of endemic areas. One of these diseases, animal trypanosomosis, affects livestock and is caused by the parasites of the Trypanosoma genus. The most widespread causative agent of animal trypanosomosis is T. evansi, which is found in large parts of the world (Africa, Asia, South America, Middle East, and the Mediterranean). Proper control and treatment of the disease requires the availability of reliable and sensitive diagnostic tools. DNA-based detection techniques are powerful and versatile in the sense that they can be tailored to achieve a high specificity and usually allow the reliable detection of low amounts of parasite genetic material. However, many DNA-based methodologies (such as PCR) require trained staff and well-equipped laboratories, which is why the research community has actively investigated in developing amplification strategies that are simple, fast, cost-effective and are suitable for use in minimally equipped laboratories and field settings. In this paper, we describe the development of a diagnostic test under a dipstick format for the specific detection of T. evansi, based on a DNA amplification principle (Recombinase Polymerase Amplification aka RPA) that meets the above-mentioned criteria. Background Animal trypanosomosis caused by Trypanosoma evansi is known as "surra" and is a widespread neglected tropical disease affecting wild and domestic animals mainly in South America, the Middle East, North Africa and Asia. An essential necessity for T. evansi infection control is the availability of reliable and sensitive diagnostic tools. While DNA-based PCR detection techniques meet these criteria, most of them require well-trained and experienced users as well as a laboratory environment allowing correct protocol execution. As an alternative, we developed a recombinase polymerase amplification (RPA) test for Type A T. evansi. The technology uses an isothermal nucleic acid amplification approach that is simple, fast, cost-effective and is suitable for use in minimally equipped laboratories and even field settings. Methodology/Principle findings An RPA assay targeting the T. evansi RoTat1.2 VSG gene was designed for the DNA-based detection of T. evansi. Comparing post-amplification visualization by agarose gel electrophoresis and a lateral flow (LF) format reveals that the latter displays a higher sensitivity. The RPA-LF assay is specific for RoTat1.2-expressing strains of T. evansi as it does not detect the genomic DNA of other trypanosomatids. Finally, experimental mouse infection trials demonstrate that the T. evansi specific RPA-LF can be employed as a test-of-cure tool

Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Overparametrized Deep Neural Networks (DNNs) often achieve astounding performances, but may potentially result in severe generalization error. Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization. Unfortunately, SAM s computational cost is roughly double that of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts SAM s efficiency at no cost to its generalization performance. ESAM includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. In the former, the sharpness measure is approximated by perturbing a stochastically chosen set of weights in each iteration; in the latter, the SAM loss is optimized using only a judiciously selected subset of data that is sensitive to the sharpness. We provide theoretical explanations as to why these strategies perform well. We also show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis base optimizers, while test accuracies are preserved or even improved