21 research outputs found
Boosting as a Product of Experts
In this paper, we derive a novel probabilistic model of boosting as a Product
of Experts. We re-derive the boosting algorithm as a greedy incremental model
selection procedure which ensures that addition of new experts to the ensemble
does not decrease the likelihood of the data. These learning rules lead to a
generic boosting algorithm - POE- Boost which turns out to be similar to the
AdaBoost algorithm under certain assumptions on the expert probabilities. The
paper then extends the POEBoost algorithm to POEBoost.CS which handles
hypothesis that produce probabilistic predictions. This new algorithm is shown
to have better generalization performance compared to other state of the art
algorithms
Beyond Fano's Inequality: Bounds on the Optimal F-Score, BER, and Cost-Sensitive Risk and Their Implications
Bayesian locally weighted online learning
Locally weighted regression is a non-parametric technique of regression that is capable
of coping with non-stationarity of the input distribution. Online algorithms like
Receptive FieldWeighted Regression and Locally Weighted Projection Regression use
a sparse representation of the locally weighted model to approximate a target function,
resulting in an efficient learning algorithm. However, these algorithms are fairly sensitive
to parameter initializations and have multiple open learning parameters that are
usually set using some insights of the problem and local heuristics. In this thesis,
we attempt to alleviate these problems by using a probabilistic formulation of locally
weighted regression followed by a principled Bayesian inference of the parameters.
In the Randomly Varying Coefficient (RVC) model developed in this thesis, locally
weighted regression is set up as an ensemble of regression experts that provide
a local linear approximation to the target function. We train the individual experts independently
and then combine their predictions using a Product of Experts formalism.
Independent training of experts allows us to adapt the complexity of the regression
model dynamically while learning in an online fashion. The local experts themselves
are modeled using a hierarchical Bayesian probability distribution with Variational
Bayesian Expectation Maximization steps to learn the posterior distributions over the
parameters. The Bayesian modeling of the local experts leads to an inference procedure
that is fairly insensitive to parameter initializations and avoids problems like
overfitting. We further exploit the Bayesian inference procedure to derive efficient online
update rules for the parameters. Learning in the regression setting is also extended
to handle a classification task by making use of a logistic regression to model discrete
class labels.
The main contribution of the thesis is a spatially localised online learning algorithm
set up in a probabilistic framework with principled Bayesian inference rule for the
parameters of the model that learns local models completely independent of each other,
uses only local information and adapts the local model complexity in a data driven
fashion. This thesis, for the first time, brings together the computational efficiency
and the adaptability of ânon-competitiveâ locally weighted learning schemes and the
modelling guarantees of the Bayesian formulation
Kernel Carpentry for Online Regression using Randomly Varying Coefficient Model
We present a Bayesian formulation of locally weighted learning (LWL) using the novel concept of a randomly varying coefficient model. Based on thi
Bayesian locally weighted online learning
Locally weighted regression is a non-parametric technique of regression that is capable of coping with non-stationarity of the input distribution. Online algorithms like Receptive FieldWeighted Regression and Locally Weighted Projection Regression use a sparse representation of the locally weighted model to approximate a target function, resulting in an efficient learning algorithm. However, these algorithms are fairly sensitive to parameter initializations and have multiple open learning parameters that are usually set using some insights of the problem and local heuristics. In this thesis, we attempt to alleviate these problems by using a probabilistic formulation of locally weighted regression followed by a principled Bayesian inference of the parameters. In the Randomly Varying Coefficient (RVC) model developed in this thesis, locally weighted regression is set up as an ensemble of regression experts that provide a local linear approximation to the target function. We train the individual experts independently and then combine their predictions using a Product of Experts formalism. Independent training of experts allows us to adapt the complexity of the regression model dynamically while learning in an online fashion. The local experts themselves are modeled using a hierarchical Bayesian probability distribution with Variational Bayesian Expectation Maximization steps to learn the posterior distributions over the parameters. The Bayesian modeling of the local experts leads to an inference procedure that is fairly insensitive to parameter initializations and avoids problems like overfitting. We further exploit the Bayesian inference procedure to derive efficient online update rules for the parameters. Learning in the regression setting is also extended to handle a classification task by making use of a logistic regression to model discrete class labels. The main contribution of the thesis is a spatially localised online learning algorithm set up in a probabilistic framework with principled Bayesian inference rule for the parameters of the model that learns local models completely independent of each other, uses only local information and adapts the local model complexity in a data driven fashion. This thesis, for the first time, brings together the computational efficiency and the adaptability of ânon-competitiveâ locally weighted learning schemes and the modelling guarantees of the Bayesian formulation.EThOS - Electronic Theses Online ServiceGBUnited Kingdo