TAN Classifiers Based on Decomposable Distributions

The original publication is available at www.springerlink.comIn this paper we present several Bayesian algorithms for learning Tree Augmented Naive Bayes (TAN) models. We extend the results in Meila & Jaakkola (2000a) to TANs by proving that accepting a prior decomposable distribution over TAN's, we can compute the exact Bayesian model averaging over TAN structures and parameters in polynomial time. Furthermore, we prove that the k-maximum a posteriori (MAP) TAN structures can also be computed in polynomial time. We use these results to correct minor errors in Meila & Jaakkola (2000a) and to construct several TAN based classifiers provide consistently better predictions over Irvine datasets and artificially generated data than TAN based classifiers proposed in the literature.Peer reviewe

Marginal and simultaneous predictive classification using stratified graphical models

An inductive probabilistic classification rule must generally obey the principles of Bayesian predictive inference, such that all observed and unobserved stochastic quantities are jointly modeled and the parameter uncertainty is fully acknowledged through the posterior predictive distribution. Several such rules have been recently considered and their asymptotic behavior has been characterized under the assumption that the observed features or variables used for building a classifier are conditionally independent given a simultaneous labeling of both the training samples and those from an unknown origin. Here we extend the theoretical results to predictive classifiers acknowledging feature dependencies either through graphical models or sparser alternatives defined as stratified graphical models. We also show through experimentation with both synthetic and real data that the predictive classifiers based on stratified graphical models have consistently best accuracy compared with the predictive classifiers based on either conditionally independent features or on ordinary graphical models.Comment: 18 pages, 5 figure

Discriminative learning of Bayesian networks via factorized conditional log-likelihood

We propose an efficient and parameter-free scoring criterion, the factorized conditional log-likelihood (ˆfCLL), for learning Bayesian network classifiers. The proposed score is an approximation of the conditional log-likelihood criterion. The approximation is devised in order to guarantee decomposability over the network structure, as well as efficient estimation of the optimal parameters, achieving the same time and space complexity as the traditional log-likelihood scoring criterion. The resulting criterion has an information-theoretic interpretation based on interaction information, which exhibits its discriminative nature. To evaluate the performance of the proposed criterion, we present an empirical comparison with state-of-the-art classifiers. Results on a large suite of benchmark data sets from the UCI repository show that ˆfCLL-trained classifiers achieve at least as good accuracy as the best compared classifiers, using significantly less computational resources.Peer reviewe

Robust Bayesian Linear Classifier Ensembles

The original publication is available at http://www.springerlink.comEnsemble classifiers combine the classification results of several classifiers. Simple ensemble methods such as uniform averaging over a set of models usually provide an improvement over selecting the single best model. Usually probabilistic classifiers restrict the set of possible models that can be learnt in order to lower computational complexity costs. In these restricted spaces, where incorrect modelling assumptions are possibly made, uniform averaging sometimes performs even better than bayesian model averaging. Linear mixtures over sets of models provide an space that includes uniform averaging as a particular case. We develop two algorithms for learning maximum a posteriori weights for linear mixtures, based on expectation maximization and on constrained optimization. We provide a nontrivial example of the utility of these two algorithms by applying them for one dependence estimators.We develop the conjugate distribution for one dependence estimators and empirically show that uniform averaging is clearly superior to BMA for this family of models. After that we empirically show that the maximum a posteriori linear mixture weights improve accuracy significantly over uniform aggregation.Peer reviewe

Context-specific independence in graphical models

The theme of this thesis is context-speci c independence in graphical models. Considering a system of stochastic variables it is often the case that the variables are dependent of each other. This can, for instance, be seen by measuring the covariance between a pair of variables. Using graphical models, it is possible to visualize the dependence structure found in a set of stochastic variables. Using ordinary graphical models, such as Markov networks, Bayesian networks, and Gaussian graphical models, the type of dependencies that can be modeled is limited to marginal and conditional (in)dependencies. The models introduced in this thesis enable the graphical representation of context-speci c independencies, i.e. conditional independencies that hold only in a subset of the outcome space of the conditioning variables. In the articles included in this thesis, we introduce several types of graphical models that can represent context-speci c independencies. Models for both discrete variables and continuous variables are considered. A wide range of properties are examined for the introduced models, including identi ability, robustness, scoring, and optimization. In one article, a predictive classi er which utilizes context-speci c independence models is introduced. This classi er clearly demonstrates the potential bene ts of the introduced models. The purpose of the material included in the thesis prior to the articles is to provide the basic theory needed to understand the articles.Temat för avhandlingen är kontextspecifikt oberoende i grafiska modeller. Inom sannolikhetslära och statistik är en stokastisk variabel en variabel som påverkas av slumpen. Till skillnad från vanliga matematiska variabler antar en stokastisk variabel ett givet värde med en viss sannolikhet. För en mängd stokastiska variabler gäller det i regel att variablerna är beroende av varandra. Graden av beroende kan t.ex. mätas med kovariansen mellan två variabler. Med hjälp av grafiska modeller är det möjligt att visualisera beroendestrukturen för ett system av stokastiska variabler. Med hjälp av traditionella grafiska modeller såsom Markov nätverk, Bayesianska nätverk och Gaussiska grafiska modeller är det möjligt att visualisera marginellt och betingat oberoende. De modeller som introduceras i denna avhandling möjliggör en grafisk representation av kontextspecifikt oberoende, d.v.s. betingat oberoende som endast håller i en delmängd av de betingande variablernas utfallsrum. I artiklarna som inkluderats i avhandlingen introduceras flera typer av grafiska modeller som kan representera kontextspecifika oberoende. Både diskreta och kontinuerliga system behandlas. För dessa modeller undersöks många egenskaper inklusive identifierbarhet, stabilitet, modelljämförelse och optimering. I en artikel introduceras en prediktiv klassificerare som utnyttjar kontextspecifikt oberoende i grafiska modeller. Denna klassificerare visar tydligt hur användningen av kontextspecifika oberoende kan leda till förbättrade resultat i praktiska tillämpningar