Data-Driven Bayesian Network Learning: A Bi-Objective Approach to Address the Bias-Variance Decomposition

We present a novel bi-objective approach to address the data-driven learning problem of Bayesian networks. Both the log-likelihood and the complexity of each candidate Bayesian network are considered as objectives to be optimized by our proposed algorithm named Nondominated Sorting Genetic Algorithm for learning Bayesian networks (NS2BN) which is based on the well-known NSGA-II algorithm. The core idea is to reduce the implicit selection bias-variance decomposition while identifying a set of competitive models using both objectives. Numerical results suggest that, in stark contrast to the single-objective approach, our bi-objective approach is useful to find competitive Bayesian networks especially in the complexity. Furthermore, our approach presents the end user with a set of solutions by showing different Bayesian network and their respective MDL and classification accuracy results

How good is crude MDL for solving the bias-variance dilemma? An empirical investigation based on Bayesian networks.

The bias-variance dilemma is a well-known and important problem in Machine Learning. It basically relates the generalization capability (goodness of fit) of a learning method to its corresponding complexity. When we have enough data at hand, it is possible to use these data in such a way so as to minimize overfitting (the risk of selecting a complex model that generalizes poorly). Unfortunately, there are many situations where we simply do not have this required amount of data. Thus, we need to find methods capable of efficiently exploiting the available data while avoiding overfitting. Different metrics have been proposed to achieve this goal: the Minimum Description Length principle (MDL), Akaike's Information Criterion (AIC) and Bayesian Information Criterion (BIC), among others. In this paper, we focus on crude MDL and empirically evaluate its performance in selecting models with a good balance between goodness of fit and complexity: the so-called bias-variance dilemma, decomposition or tradeoff. Although the graphical interaction between these dimensions (bias and variance) is ubiquitous in the Machine Learning literature, few works present experimental evidence to recover such interaction. In our experiments, we argue that the resulting graphs allow us to gain insights that are difficult to unveil otherwise: that crude MDL naturally selects balanced models in terms of bias-variance, which not necessarily need be the gold-standard ones. We carry out these experiments using a specific model: a Bayesian network. In spite of these motivating results, we also should not overlook three other components that may significantly affect the final model selection: the search procedure, the noise rate and the sample size

LaDonna (House) Moore poses with others on couch

LaDonna (House) Moore (top left) poses with others on couch, c.1980\u27shttps://digitalcommons.georgefox.edu/gfu_photos_1980_1984/1103/thumbnail.jp

Exhaustive evaluation of BIC (low-entropy values).

<p>Exhaustive evaluation of BIC (low-entropy values).</p

Santa Clara

An archive of the Santa Clara University student newspaper from Santa Clara University in Californi

Graph with best value (AIC, MDL, BIC - random distribution).

<p>Graph with best value (AIC, MDL, BIC - random distribution).</p

Gold-standard Network.

<p>Gold-standard Network.</p

Graph with best MDL and BIC value.

<p>Graph with best MDL and BIC value.</p

Algorithm for randomly generating raw sample data.

<p>Algorithm for randomly generating raw sample data.</p

Minimum MDL2 values (random distribution).

<p>The red dot indicates the BN structure of <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0092866#pone-0092866-g022" target="_blank">Figure 22</a> whereas the green dot indicates the MDL2 value of the gold-standard network (<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0092866#pone-0092866-g009" target="_blank">Figure 9</a>). The distance between these two networks = 0.00018701910455 (computed as the log₂ of the ratio of gold-standard network/minimum network). A value bigger than 0 means that the minimum network has better MDL2 than the gold-standard.</p