6,026 research outputs found
Hybrid Models with Deep and Invertible Features
We propose a neural hybrid model consisting of a linear model defined on a
set of features computed by a deep, invertible transformation (i.e. a
normalizing flow). An attractive property of our model is that both
p(features), the density of the features, and p(targets | features), the
predictive distribution, can be computed exactly in a single feed-forward pass.
We show that our hybrid model, despite the invertibility constraints, achieves
similar accuracy to purely predictive models. Moreover the generative component
remains a good model of the input features despite the hybrid optimization
objective. This offers additional capabilities such as detection of
out-of-distribution inputs and enabling semi-supervised learning. The
availability of the exact joint density p(targets, features) also allows us to
compute many quantities readily, making our hybrid model a useful building
block for downstream applications of probabilistic deep learning.Comment: ICML 201
Toward Optimal Feature Selection in Naive Bayes for Text Categorization
Automated feature selection is important for text categorization to reduce
the feature size and to speed up the learning process of classifiers. In this
paper, we present a novel and efficient feature selection framework based on
the Information Theory, which aims to rank the features with their
discriminative capacity for classification. We first revisit two information
measures: Kullback-Leibler divergence and Jeffreys divergence for binary
hypothesis testing, and analyze their asymptotic properties relating to type I
and type II errors of a Bayesian classifier. We then introduce a new divergence
measure, called Jeffreys-Multi-Hypothesis (JMH) divergence, to measure
multi-distribution divergence for multi-class classification. Based on the
JMH-divergence, we develop two efficient feature selection methods, termed
maximum discrimination () and methods, for text categorization.
The promising results of extensive experiments demonstrate the effectiveness of
the proposed approaches.Comment: This paper has been submitted to the IEEE Trans. Knowledge and Data
Engineering. 14 pages, 5 figure
Discrete MDL Predicts in Total Variation
The Minimum Description Length (MDL) principle selects the model that has the
shortest code for data plus model. We show that for a countable class of
models, MDL predictions are close to the true distribution in a strong sense.
The result is completely general. No independence, ergodicity, stationarity,
identifiability, or other assumption on the model class need to be made. More
formally, we show that for any countable class of models, the distributions
selected by MDL (or MAP) asymptotically predict (merge with) the true measure
in the class in total variation distance. Implications for non-i.i.d. domains
like time-series forecasting, discriminative learning, and reinforcement
learning are discussed.Comment: 15 LaTeX page
- …