7,004 research outputs found
Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models
Mixture of Experts (MoE) are successful models for modeling heterogeneous
data in many statistical learning problems including regression, clustering and
classification. Generally fitted by maximum likelihood estimation via the
well-known EM algorithm, their application to high-dimensional problems is
still therefore challenging. We consider the problem of fitting and feature
selection in MoE models, and propose a regularized maximum likelihood
estimation approach that encourages sparse solutions for heterogeneous
regression data models with potentially high-dimensional predictors. Unlike
state-of-the art regularized MLE for MoE, the proposed modelings do not require
an approximate of the penalty function. We develop two hybrid EM algorithms: an
Expectation-Majorization-Maximization (EM/MM) algorithm, and an EM algorithm
with coordinate ascent algorithm. The proposed algorithms allow to
automatically obtaining sparse solutions without thresholding, and avoid matrix
inversion by allowing univariate parameter updates. An experimental study shows
the good performance of the algorithms in terms of recovering the actual sparse
solutions, parameter estimation, and clustering of heterogeneous regression
data
Effect fusion using model-based clustering
In social and economic studies many of the collected variables are measured
on a nominal scale, often with a large number of categories. The definition of
categories is usually not unambiguous and different classification schemes
using either a finer or a coarser grid are possible. Categorisation has an
impact when such a variable is included as covariate in a regression model: a
too fine grid will result in imprecise estimates of the corresponding effects,
whereas with a too coarse grid important effects will be missed, resulting in
biased effect estimates and poor predictive performance.
To achieve automatic grouping of levels with essentially the same effect, we
adopt a Bayesian approach and specify the prior on the level effects as a
location mixture of spiky normal components. Fusion of level effects is induced
by a prior on the mixture weights which encourages empty components.
Model-based clustering of the effects during MCMC sampling allows to
simultaneously detect categories which have essentially the same effect size
and identify variables with no effect at all. The properties of this approach
are investigated in simulation studies. Finally, the method is applied to
analyse effects of high-dimensional categorical predictors on income in
Austria
- …