7,516 research outputs found
On PAC-Bayesian Bounds for Random Forests
Existing guarantees in terms of rigorous upper bounds on the generalization
error for the original random forest algorithm, one of the most frequently used
machine learning methods, are unsatisfying. We discuss and evaluate various
PAC-Bayesian approaches to derive such bounds. The bounds do not require
additional hold-out data, because the out-of-bag samples from the bagging in
the training process can be exploited. A random forest predicts by taking a
majority vote of an ensemble of decision trees. The first approach is to bound
the error of the vote by twice the error of the corresponding Gibbs classifier
(classifying with a single member of the ensemble selected at random). However,
this approach does not take into account the effect of averaging out of errors
of individual classifiers when taking the majority vote. This effect provides a
significant boost in performance when the errors are independent or negatively
correlated, but when the correlations are strong the advantage from taking the
majority vote is small. The second approach based on PAC-Bayesian C-bounds
takes dependencies between ensemble members into account, but it requires
estimating correlations between the errors of the individual classifiers. When
the correlations are high or the estimation is poor, the bounds degrade. In our
experiments, we compute generalization bounds for random forests on various
benchmark data sets. Because the individual decision trees already perform
well, their predictions are highly correlated and the C-bounds do not lead to
satisfactory results. For the same reason, the bounds based on the analysis of
Gibbs classifiers are typically superior and often reasonably tight. Bounds
based on a validation set coming at the cost of a smaller training set gave
better performance guarantees, but worse performance in most experiments
Moment-Based Spectral Analysis of Random Graphs with Given Expected Degrees
In this paper, we analyze the limiting spectral distribution of the adjacency
matrix of a random graph ensemble, proposed by Chung and Lu, in which a given
expected degree sequence
is prescribed on the ensemble. Let if there is an edge
between the nodes and zero otherwise, and consider the normalized
random adjacency matrix of the graph ensemble: . The empirical spectral distribution
of denoted by is the empirical
measure putting a mass at each of the real eigenvalues of the
symmetric matrix . Under some technical conditions on the
expected degree sequence, we show that with probability one,
converges weakly to a deterministic
distribution . Furthermore, we fully characterize this
distribution by providing explicit expressions for the moments of
. We apply our results to well-known degree distributions,
such as power-law and exponential. The asymptotic expressions of the spectral
moments in each case provide significant insights about the bulk behavior of
the eigenvalue spectrum
Next nearest neighbour Ising models on random graphs
This paper develops results for the next nearest neighbour Ising model on
random graphs. Besides being an essential ingredient in classic models for
frustrated systems, second neighbour interactions interactions arise naturally
in several applications such as the colour diversity problem and graphical
games. We demonstrate ensembles of random graphs, including regular
connectivity graphs, that have a periodic variation of free energy, with either
the ratio of nearest to next nearest couplings, or the mean number of nearest
neighbours. When the coupling ratio is integer paramagnetic phases can be found
at zero temperature. This is shown to be related to the locked or unlocked
nature of the interactions. For anti-ferromagnetic couplings, spin glass phases
are demonstrated at low temperature. The interaction structure is formulated as
a factor graph, the solution on a tree is developed. The replica symmetric and
energetic one-step replica symmetry breaking solution is developed using the
cavity method. We calculate within these frameworks the phase diagram and
demonstrate the existence of dynamical transitions at zero temperature for
cases of anti-ferromagnetic coupling on regular and inhomogeneous random
graphs.Comment: 55 pages, 15 figures, version 2 with minor revisions, to be published
J. Stat. Mec
Machine learning techniques in joint default assessment
This paper studies the consequences of capturing non-linear dependence among
the covariates that drive the default of different obligors and the overall
riskiness of their credit portfolio. Joint default modeling is, without loss of
generality, the classical Bernoulli mixture model. Using an application to a
credit card dataset we show that, even when Machine Learning techniques perform
only slightly better than Logistic Regression in classifying individual
defaults as a function of the covariates, they do outperform it at the
portfolio level. This happens because they capture linear and non-linear
dependence among the covariates, whereas Logistic Regression only captures
linear dependence. The ability of Machine Learning methods to capture
non-linear dependence among the covariates produces higher default correlation
compared with Logistic Regression. As a consequence, on our data, Logistic
Regression underestimates the riskiness of the credit portfolio
Smoothed Complexity Theory
Smoothed analysis is a new way of analyzing algorithms introduced by Spielman
and Teng (J. ACM, 2004). Classical methods like worst-case or average-case
analysis have accompanying complexity classes, like P and AvgP, respectively.
While worst-case or average-case analysis give us a means to talk about the
running time of a particular algorithm, complexity classes allows us to talk
about the inherent difficulty of problems.
Smoothed analysis is a hybrid of worst-case and average-case analysis and
compensates some of their drawbacks. Despite its success for the analysis of
single algorithms and problems, there is no embedding of smoothed analysis into
computational complexity theory, which is necessary to classify problems
according to their intrinsic difficulty.
We propose a framework for smoothed complexity theory, define the relevant
classes, and prove some first hardness results (of bounded halting and tiling)
and tractability results (binary optimization problems, graph coloring,
satisfiability). Furthermore, we discuss extensions and shortcomings of our
model and relate it to semi-random models.Comment: to be presented at MFCS 201
- …