3 research outputs found
Variable Selection for Latent Dirichlet Allocation
In latent Dirichlet allocation (LDA), topics are multinomial distributions
over the entire vocabulary. However, the vocabulary usually contains many words
that are not relevant in forming the topics. We adopt a variable selection
method widely used in statistical modeling as a dimension reduction tool and
combine it with LDA. In this variable selection model for LDA (vsLDA), topics
are multinomial distributions over a subset of the vocabulary, and by excluding
words that are not informative for finding the latent topic structure of the
corpus, vsLDA finds topics that are more robust and discriminative. We compare
three models, vsLDA, LDA with symmetric priors, and LDA with asymmetric priors,
on heldout likelihood, MCMC chain consistency, and document classification. The
performance of vsLDA is better than symmetric LDA for likelihood and
classification, better than asymmetric LDA for consistency and classification,
and about the same in the other comparisons
Prediction Focused Topic Models for Electronic Health Records
Electronic Health Record (EHR) data can be represented as discrete counts
over a high dimensional set of possible procedures, diagnoses, and medications.
Supervised topic models present an attractive option for incorporating EHR data
as features into a prediction problem: given a patient's record, we estimate a
set of latent factors that are predictive of the response variable. However,
existing methods for supervised topic modeling struggle to balance prediction
quality and coherence of the latent factors. We introduce a novel approach, the
prediction-focused topic model, that uses the supervisory signal to retain only
features that improve, or do not hinder, prediction performance. By removing
features with irrelevant signal, the topic model is able to learn
task-relevant, interpretable topics. We demonstrate on a EHR dataset and a
movie review dataset that compared to existing approaches, prediction-focused
topic models are able to learn much more coherent topics while maintaining
competitive predictions.Comment: Machine Learning for Health (ML4H) at NeurIPS 2019 - Extended
Abstract. arXiv admin note: substantial text overlap with arXiv:1910.0549
Prediction Focused Topic Models via Feature Selection
Supervised topic models are often sought to balance prediction quality and
interpretability. However, when models are (inevitably) misspecified, standard
approaches rarely deliver on both. We introduce a novel approach, the
prediction-focused topic model, that uses the supervisory signal to retain only
vocabulary terms that improve, or at least do not hinder, prediction
performance. By removing terms with irrelevant signal, the topic model is able
to learn task-relevant, coherent topics. We demonstrate on several data sets
that compared to existing approaches, prediction-focused topic models learn
much more coherent topics while maintaining competitive predictions.Comment: AISTATS 2020. arXiv admin note: substantial text overlap with
arXiv:1911.0855