43,835 research outputs found
Distribution of Mutual Information from Complete and Incomplete Data
Mutual information is widely used, in a descriptive way, to measure the
stochastic dependence of categorical random variables. In order to address
questions such as the reliability of the descriptive value, one must consider
sample-to-population inferential approaches. This paper deals with the
posterior distribution of mutual information, as obtained in a Bayesian
framework by a second-order Dirichlet prior distribution. The exact analytical
expression for the mean, and analytical approximations for the variance,
skewness and kurtosis are derived. These approximations have a guaranteed
accuracy level of the order O(1/n^3), where n is the sample size. Leading order
approximations for the mean and the variance are derived in the case of
incomplete samples. The derived analytical expressions allow the distribution
of mutual information to be approximated reliably and quickly. In fact, the
derived expressions can be computed with the same order of complexity needed
for descriptive mutual information. This makes the distribution of mutual
information become a concrete alternative to descriptive mutual information in
many applications which would benefit from moving to the inductive side. Some
of these prospective applications are discussed, and one of them, namely
feature selection, is shown to perform significantly better when inductive
mutual information is used.Comment: 26 pages, LaTeX, 5 figures, 4 table
Updating beliefs with incomplete observations
Currently, there is renewed interest in the problem, raised by Shafer in
1985, of updating probabilities when observations are incomplete. This is a
fundamental problem in general, and of particular interest for Bayesian
networks. Recently, Grunwald and Halpern have shown that commonly used updating
strategies fail in this case, except under very special assumptions. In this
paper we propose a new method for updating probabilities with incomplete
observations. Our approach is deliberately conservative: we make no assumptions
about the so-called incompleteness mechanism that associates complete with
incomplete observations. We model our ignorance about this mechanism by a
vacuous lower prevision, a tool from the theory of imprecise probabilities, and
we use only coherence arguments to turn prior into posterior probabilities. In
general, this new approach to updating produces lower and upper posterior
probabilities and expectations, as well as partially determinate decisions.
This is a logical consequence of the existing ignorance about the
incompleteness mechanism. We apply the new approach to the problem of
classification of new evidence in probabilistic expert systems, where it leads
to a new, so-called conservative updating rule. In the special case of Bayesian
networks constructed using expert knowledge, we provide an exact algorithm for
classification based on our updating rule, which has linear-time complexity for
a class of networks wider than polytrees. This result is then extended to the
more general framework of credal networks, where computations are often much
harder than with Bayesian nets. Using an example, we show that our rule appears
to provide a solid basis for reliable updating with incomplete observations,
when no strong assumptions about the incompleteness mechanism are justified.Comment: Replaced with extended versio
Robust Feature Selection by Mutual Information Distributions
Mutual information is widely used in artificial intelligence, in a
descriptive way, to measure the stochastic dependence of discrete random
variables. In order to address questions such as the reliability of the
empirical value, one must consider sample-to-population inferential approaches.
This paper deals with the distribution of mutual information, as obtained in a
Bayesian framework by a second-order Dirichlet prior distribution. The exact
analytical expression for the mean and an analytical approximation of the
variance are reported. Asymptotic approximations of the distribution are
proposed. The results are applied to the problem of selecting features for
incremental learning and classification of the naive Bayes classifier. A fast,
newly defined method is shown to outperform the traditional approach based on
empirical mutual information on a number of real data sets. Finally, a
theoretical development is reported that allows one to efficiently extend the
above methods to incomplete samples in an easy and effective way.Comment: 8 two-column page
Eight-Dimensional Mid-Infrared/Optical Bayesian Quasar Selection
We explore the multidimensional, multiwavelength selection of quasars from
mid-IR (MIR) plus optical data, specifically from Spitzer-IRAC and the Sloan
Digital Sky Survey (SDSS). We apply modern statistical techniques to combined
Spitzer MIR and SDSS optical data, allowing up to 8-D color selection of
quasars. Using a Bayesian selection method, we catalog 5546 quasar candidates
to an 8.0 um depth of 56 uJy over an area of ~24 sq. deg; ~70% of these
candidates are not identified by applying the same Bayesian algorithm to
4-color SDSS optical data alone. Our selection recovers 97.7% of known type 1
quasars in this area and greatly improves the effectiveness of identifying
3.5<z<5 quasars. Even using only the two shortest wavelength IRAC bandpasses,
it is possible to use our Bayesian techniques to select quasars with 97%
completeness and as little as 10% contamination. This sample has a photometric
redshift accuracy of 93.6% (Delta Z +/-0.3), remaining roughly constant when
the two reddest MIR bands are excluded. While our methods are designed to find
type 1 (unobscured) quasars, as many as 1200 of the objects are type 2
(obscured) quasar candidates. Coupling deep optical imaging data with deep
mid-IR data could enable selection of quasars in significant numbers past the
peak of the quasar luminosity function (QLF) to at least z~4. Such a sample
would constrain the shape of the QLF and enable quasar clustering studies over
the largest range of redshift and luminosity to date, yielding significant
gains in our understanding of quasars and the evolution of galaxies.Comment: 49 pages, 14 figures, 7 tables. AJ, accepte
Imputation Estimators Partially Correct for Model Misspecification
Inference problems with incomplete observations often aim at estimating
population properties of unobserved quantities. One simple way to accomplish
this estimation is to impute the unobserved quantities of interest at the
individual level and then take an empirical average of the imputed values. We
show that this simple imputation estimator can provide partial protection
against model misspecification. We illustrate imputation estimators' robustness
to model specification on three examples: mixture model-based clustering,
estimation of genotype frequencies in population genetics, and estimation of
Markovian evolutionary distances. In the final example, using a representative
model misspecification, we demonstrate that in non-degenerate cases, the
imputation estimator dominates the plug-in estimate asymptotically. We conclude
by outlining a Bayesian implementation of the imputation-based estimation.Comment: major rewrite, beta-binomial example removed, model based clustering
is added to the mixture model example, Bayesian approach is now illustrated
with the genetics exampl
Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness
In recent years, the notion of local robustness (or robustness for short) has
emerged as a desirable property of deep neural networks. Intuitively,
robustness means that small perturbations to an input do not cause the network
to perform misclassifications. In this paper, we present a novel algorithm for
verifying robustness properties of neural networks. Our method synergistically
combines gradient-based optimization methods for counterexample search with
abstraction-based proof search to obtain a sound and ({\delta}-)complete
decision procedure. Our method also employs a data-driven approach to learn a
verification policy that guides abstract interpretation during proof search. We
have implemented the proposed approach in a tool called Charon and
experimentally evaluated it on hundreds of benchmarks. Our experiments show
that the proposed approach significantly outperforms three state-of-the-art
tools, namely AI^2 , Reluplex, and Reluval
- …