1,569 research outputs found
Robust Classification for Imprecise Environments
In real-world environments it usually is difficult to specify target
operating conditions precisely, for example, target misclassification costs.
This uncertainty makes building robust classification systems problematic. We
show that it is possible to build a hybrid classifier that will perform at
least as well as the best available classifier for any target conditions. In
some cases, the performance of the hybrid actually can surpass that of the best
known classifier. This robust performance extends across a wide variety of
comparison frameworks, including the optimization of metrics such as accuracy,
expected cost, lift, precision, recall, and workforce utilization. The hybrid
also is efficient to build, to store, and to update. The hybrid is based on a
method for the comparison of classifier performance that is robust to imprecise
class distributions and misclassification costs. The ROC convex hull (ROCCH)
method combines techniques from ROC analysis, decision analysis and
computational geometry, and adapts them to the particulars of analyzing learned
classifiers. The method is efficient and incremental, minimizes the management
of classifier performance data, and allows for clear visual comparisons and
sensitivity analyses. Finally, we point to empirical evidence that a robust
hybrid classifier indeed is needed for many real-world problems.Comment: 24 pages, 12 figures. To be published in Machine Learning Journal.
For related papers, see http://www.hpl.hp.com/personal/Tom_Fawcett/ROCCH
Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
In many fields of science, generalized likelihood ratio tests are established
tools for statistical inference. At the same time, it has become increasingly
common that a simulator (or generative model) is used to describe complex
processes that tie parameters of an underlying theory and measurement
apparatus to high-dimensional observations .
However, simulator often do not provide a way to evaluate the likelihood
function for a given observation , which motivates a new class of
likelihood-free inference algorithms. In this paper, we show that likelihood
ratios are invariant under a specific class of dimensionality reduction maps
. As a direct consequence, we show that
discriminative classifiers can be used to approximate the generalized
likelihood ratio statistic when only a generative model for the data is
available. This leads to a new machine learning-based approach to
likelihood-free inference that is complementary to Approximate Bayesian
Computation, and which does not require a prior on the model parameters.
Experimental results on artificial problems with known exact likelihoods
illustrate the potential of the proposed method.Comment: 35 pages, 5 figure
A Conversation with Seymour Geisser
Seymour Geisser received his bachelor's degree in Mathematics from the City
College of New York in 1950, and his M.A. and Ph.D. degrees in Mathematical
Statistics at the University of North Carolina in 1952 and 1955, respectively.
He then held positions at the National Bureau of Standards and the National
Institute of Mental Health until 1961. From 1961 until 1965, he was Chief of
the Biometry Section at the National Institute of Arthritis and Metabolic
Diseases, and also held the position of Professorial Lecturer at the George
Washington University from 1960 to 1965. From 1965 to 1970, he was the founding
Chair of the Department of Statistics at the State University of New York,
Buffalo, and in 1971, he became the founding Director of the School of
Statistics at the University of Minnesota, remaining in that position until
2001. He held visiting professorships at Iowa State University, 1960;
University of Wisconsin, 1964; University of Tel-Aviv (Israel), 1971;
University of Waterloo (Canada), 1972; Stanford University, 1976, 1977, 1988;
Carnegie Mellon University, 1976; University of the Orange Free State (South
Africa), 1978, 1993; Harvard University, 1981; University of Chicago, 1985;
University of Warwick (England), 1986; University of Modena (Italy), 1996; and
National Chiao Tung University (Taiwan), 1998. He was the Lady Davis Visiting
Professor, Hebrew University of Jerusalem, 1991, 1994, 1999, and the Schor
Scholar, Merck Research Laboratories, 2002-2003. He was a Fellow of the
Institute of Mathematical Statistics and the American Statistical Association.Comment: Published in at http://dx.doi.org/10.1214/088342307000000131 the
Statistical Science (http://www.imstat.org/sts/) by the Institute of
Mathematical Statistics (http://www.imstat.org
Bayesian Aspects of Classification Procedures
We consider several statistical approaches to binary classification and multiple hypothesis testing problems. Situations in which a binary choice must be made are common in science. Usually, there is uncertainty involved in making the choice and a great number of statistical techniques have been put forth to help researchers deal with this uncertainty in separating signal from noise in reasonable ways. For example, in genetic studies, one may want to identify genes that affect a certain biological process from among a larger set of genes. In such examples, costs are attached to making incorrect choices and many choices must be made at the same time. Reasonable ways of modeling the cost structure and choosing the appropriate criteria for evaluating the performance of statistical techniques are needed. The following three chapters have proposals of some Bayesian methods for these issues.
In the first chapter, we focus on an empirical Bayes approach to a popular binary classification problem formulation. In this framework, observations are treated as independent draws from a hierarchical model with a mixture prior distribution. The mixture prior combines prior distributions for the ``noise\u27\u27 and for the ``signal\u27\u27 observations. In the literature, parametric assumptions are usually made about the prior distribution from which the ``signal\u27\u27 observations come. We suggest a Bayes classification rule which minimizes the expectation of a flexible and easily interpretable mixture loss function which brings together constant penalties for false positive misclassifications and penalties for false negative misclassifications. Due in part to the form of the loss function, empirical Bayes techniques can then be used to construct the Bayes classification rule without specifying the ``signal\u27\u27 part of the mixture prior distribution. The proposed classification technique builds directly on the nonparametric mixture prior approach proposed by Raykar and Zhao (2010, 2011).
Many different criteria can be used to judge the success of a classification procedure. A very useful criterion called the False Discovery Rate (FDR) was introduced by Benjamini and Hochberg in a 1995 paper. For many applications, the FDR, which is defined as the expected proportion of false positive results among the observations declared to be ``signal\u27\u27, is a reasonable criterion to target. Bayesian versions of the false discovery rate, the so-called positive false discovery rate (pFDR) and local false discovery rate, were proposed by Storey (2002, 2003) and Efron and coauthors (2001), respectively. There is an interesting connection between the local false discovery rate and the nonparametric mixture prior approach for binary classification problems. The second part of the dissertation is focused on this link and provides a comparison of various approaches for estimating Bayesian false discovery rates.
The third chapter is an account of a connection between the celebrated Neyman-Pearson lemma and the area (AUC) under the receiver operating characteristic (ROC) curve when the observations that need to be classified come from a pair of normal distributions. Using this connection, it is possible to derive a classification rule which maximizes the AUC for binormal data
SCRIB: Set-classifier with Class-specific Risk Bounds for Blackbox Models
Despite deep learning (DL) success in classification problems, DL classifiers
do not provide a sound mechanism to decide when to refrain from predicting.
Recent works tried to control the overall prediction risk with classification
with rejection options. However, existing works overlook the different
significance of different classes. We introduce Set-classifier with
Class-specific RIsk Bounds (SCRIB) to tackle this problem, assigning multiple
labels to each example. Given the output of a black-box model on the validation
set, SCRIB constructs a set-classifier that controls the class-specific
prediction risks with a theoretical guarantee. The key idea is to reject when
the set classifier returns more than one label. We validated SCRIB on several
medical applications, including sleep staging on electroencephalogram (EEG)
data, X-ray COVID image classification, and atrial fibrillation detection based
on electrocardiogram (ECG) data. SCRIB obtained desirable class-specific risks,
which are 35\%-88\% closer to the target risks than baseline methods
Byzantine Attack and Defense in Cognitive Radio Networks: A Survey
The Byzantine attack in cooperative spectrum sensing (CSS), also known as the
spectrum sensing data falsification (SSDF) attack in the literature, is one of
the key adversaries to the success of cognitive radio networks (CRNs). In the
past couple of years, the research on the Byzantine attack and defense
strategies has gained worldwide increasing attention. In this paper, we provide
a comprehensive survey and tutorial on the recent advances in the Byzantine
attack and defense for CSS in CRNs. Specifically, we first briefly present the
preliminaries of CSS for general readers, including signal detection
techniques, hypothesis testing, and data fusion. Second, we analyze the spear
and shield relation between Byzantine attack and defense from three aspects:
the vulnerability of CSS to attack, the obstacles in CSS to defense, and the
games between attack and defense. Then, we propose a taxonomy of the existing
Byzantine attack behaviors and elaborate on the corresponding attack
parameters, which determine where, who, how, and when to launch attacks. Next,
from the perspectives of homogeneous or heterogeneous scenarios, we classify
the existing defense algorithms, and provide an in-depth tutorial on the
state-of-the-art Byzantine defense schemes, commonly known as robust or secure
CSS in the literature. Furthermore, we highlight the unsolved research
challenges and depict the future research directions.Comment: Accepted by IEEE Communications Surveys and Tutoiral
- …