5,426 research outputs found
Hyper-heuristic decision tree induction
A hyper-heuristic is any algorithm that searches or operates in the space of
heuristics as opposed to the space of solutions. Hyper-heuristics are
increasingly used in function and combinatorial optimization. Rather than
attempt to solve a problem using a fixed heuristic, a hyper-heuristic
approach attempts to find a combination of heuristics that solve a problem
(and in turn may be directly suitable for a class of problem instances).
Hyper-heuristics have been little explored in data mining. This work presents
novel hyper-heuristic approaches to data mining, by searching a space of
attribute selection criteria for decision tree building algorithm. The search is
conducted by a genetic algorithm. The result of the hyper-heuristic search in
this case is a strategy for selecting attributes while building decision trees.
Most hyper-heuristics work by trying to adapt the heuristic to the state of
the problem being solved. Our hyper-heuristic is no different. It employs a
strategy for adapting the heuristic used to build decision tree nodes
according to some set of features of the training set it is working on. We
introduce, explore and evaluate five different ways in which this problem
state can be represented for a hyper-heuristic that operates within a decisiontree
building algorithm. In each case, the hyper-heuristic is guided by a rule
set that tries to map features of the data set to be split by the decision tree
building algorithm to a heuristic to be used for splitting the same data set.
We also explore and evaluate three different sets of low-level heuristics that
could be employed by such a hyper-heuristic.
This work also makes a distinction between specialist hyper-heuristics and
generalist hyper-heuristics. The main difference between these two hyperheuristcs
is the number of training sets used by the hyper-heuristic genetic
algorithm. Specialist hyper-heuristics are created using a single data set from
a particular domain for evolving the hyper-heurisic rule set. Such algorithms
are expected to outperform standard algorithms on the kind of data set used
by the hyper-heuristic genetic algorithm. Generalist hyper-heuristics are
trained on multiple data sets from different domains and are expected to
deliver a robust and competitive performance over these data sets when
compared to standard algorithms.
We evaluate both approaches for each kind of hyper-heuristic presented in
this thesis. We use both real data sets as well as synthetic data sets. Our
results suggest that none of the hyper-heuristics presented in this work are
suited for specialization – in most cases, the hyper-heuristic’s performance on
the data set it was specialized for was not significantly better than that of
the best performing standard algorithm. On the other hand, the generalist
hyper-heuristics delivered results that were very competitive to the best
standard methods. In some cases we even achieved a significantly better
overall performance than all of the standard methods
Evolutionary design of decision-tree algorithms tailored to microarray gene expression data sets
Decision-tree induction algorithms are widely used in machine learning applications in which the goal is to extract knowledge from data and present it in a graphically intuitive way. The most successful strategy for inducing decision trees is the greedy top-down recursive approach, which has been continuously improved by researchers over the past 40 years. In this paper, we propose a paradigm shift in the research of decision trees: instead of proposing a new manually designed method for inducing decision trees, we propose automatically designing decision-tree induction algorithms tailored to a specific type of classification data set (or application domain). Following recent breakthroughs in the automatic design of machine learning algorithms, we propose a hyper-heuristic evolutionary algorithm called hyper-heuristic evolutionary algorithm for designing decision-tree algorithms (HEAD-DT) that evolves design components of top-down decision-tree induction algorithms. By the end of the evolution, we expect HEAD-DT to generate a new and possibly better decision-tree algorithm for a given application domain. We perform extensive experiments in 35 real-world microarray gene expression data sets to assess the performance of HEAD-DT, and compare it with very well known decision-tree algorithms such as C4.5, CART, and REPTree. Results show that HEAD-DT is capable of generating algorithms that significantly outperform the baseline manually designed decision-tree algorithms regarding predictive accuracy and F-measure
GENESIM : genetic extraction of a single, interpretable model
Models obtained by decision tree induction techniques excel in being
interpretable.However, they can be prone to overfitting, which results in a low
predictive performance. Ensemble techniques are able to achieve a higher
accuracy. However, this comes at a cost of losing interpretability of the
resulting model. This makes ensemble techniques impractical in applications
where decision support, instead of decision making, is crucial.
To bridge this gap, we present the GENESIM algorithm that transforms an
ensemble of decision trees to a single decision tree with an enhanced
predictive performance by using a genetic algorithm. We compared GENESIM to
prevalent decision tree induction and ensemble techniques using twelve publicly
available data sets. The results show that GENESIM achieves a better predictive
performance on most of these data sets than decision tree induction techniques
and a predictive performance in the same order of magnitude as the ensemble
techniques. Moreover, the resulting model of GENESIM has a very low complexity,
making it very interpretable, in contrast to ensemble techniques.Comment: Presented at NIPS 2016 Workshop on Interpretable Machine Learning in
Complex System
Structured Prediction of Sequences and Trees using Infinite Contexts
Linguistic structures exhibit a rich array of global phenomena, however
commonly used Markov models are unable to adequately describe these phenomena
due to their strong locality assumptions. We propose a novel hierarchical model
for structured prediction over sequences and trees which exploits global
context by conditioning each generation decision on an unbounded context of
prior decisions. This builds on the success of Markov models but without
imposing a fixed bound in order to better represent global phenomena. To
facilitate learning of this large and unbounded model, we use a hierarchical
Pitman-Yor process prior which provides a recursive form of smoothing. We
propose prediction algorithms based on A* and Markov Chain Monte Carlo
sampling. Empirical results demonstrate the potential of our model compared to
baseline finite-context Markov models on part-of-speech tagging and syntactic
parsing
- …