Optimization by gradient boosting

Gradient boosting is a state-of-the-art prediction technique that sequentially produces a model in the form of linear combinations of simple predictors---typically decision trees---by solving an infinite-dimensional convex optimization problem. We provide in the present paper a thorough analysis of two widespread versions of gradient boosting, and introduce a general framework for studying these algorithms from the point of view of functional optimization. We prove their convergence as the number of iterations tends to infinity and highlight the importance of having a strongly convex risk functional to minimize. We also present a reasonable statistical context ensuring consistency properties of the boosting predictors as the sample size grows. In our approach, the optimization procedures are run forever (that is, without resorting to an early stopping strategy), and statistical regularization is basically achieved via an appropriate $L^2$ penalization of the loss and strong convexity arguments

Statistical analysis of kk-nearest neighbor collaborative recommendation

Collaborative recommendation is an information-filtering technique that attempts to present information items that are likely of interest to an Internet user. Traditionally, collaborative systems deal with situations with two types of variables, users and items. In its most common form, the problem is framed as trying to estimate ratings for items that have not yet been consumed by a user. Despite wide-ranging literature, little is known about the statistical properties of recommendation systems. In fact, no clear probabilistic model even exists which would allow us to precisely describe the mathematical forces driving collaborative filtering. To provide an initial contribution to this, we propose to set out a general sequential stochastic model for collaborative recommendation. We offer an in-depth analysis of the so-called cosine-type nearest neighbor collaborative method, which is one of the most widely used algorithms in collaborative filtering, and analyze its asymptotic performance as the number of users grows. We establish consistency of the procedure under mild assumptions on the model. Rates of convergence and examples are also provided.Comment: Published in at http://dx.doi.org/10.1214/09-AOS759 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Cox process functional learning

International audienceThis article addresses the problem of supervised classification of Cox process trajectories, whose random intensity is driven by some exogenous random covariable. The classification task is achieved through a regularized convex empirical risk minimization procedure, and a nonasymptotic oracle inequality is derived. We show that the algorithm provides a Bayes-risk consistent classifier. Furthermore, it is proved that the classifier converges at a rate which adapts to the unknown regularity of the intensity process. Our results are obtained by taking advantage of martingale and stochastic calculus arguments, which are natural in this context and fully exploit the functional nature of the problem

On the kernel rule for function classification

International audienceLet X be a random variable taking values in a function space F, and let Y be a discrete random label with values 0 and 1. We investigate asymptotic properties of the moving window classification rule based on independent copies of the pair (X, Y ). Contrary to the finite dimensional case, it is shown that the moving window classifier is not universally consistent in the sense that its probability of error may not converge to the Bayes risk for some distributions of (X, Y ). Sufficient conditions both on the space F and the distribution of X are then given to ensure consistency

Sur l'estimation du support d'une densité

International audienceEtant donnée une densité de probabilité multivariée inconnue $f$ à support compact et un $n$-échantillon i.i.d. issu de $f$, nous étudions l'estimateur du support de $f$ défini par l'union des boules de rayon $r_n$ centrées sur les observations. Afin de mesurer la qualité de l'estimation, nous utilisons un critère général fondé sur le volume de la différence symétrique. Sous quelques hypothèses peu restrictives, et en utilisant des outils de la géométrie riemannienne, nous établissons les vitesses de convergence exactes de l'estimateur du support tout en examinant les conséquences statistiques de ces résultats

Un modèle stochastique pour les systèmes de recommandation

International audienceLes systèmes de recommandation établissent des suggestions personnalisées à des individus concernant des objets (livres, films, musique) susceptibles de les intéresser. Les recommandations sont généralement basées sur l'estimation de notes relatives à des objets que l'utilisateur n'a pas consommés. En dépit d'une littérature abondante, les propriétés statistiques des systèmes de recommandation ne sont pas encore clairement établies. Dans ce travail, nous proposons un modèle stochastique pour les systèmes de recommandation et nous analysons ses propriétés asymptotiques lorsque le nombre d'utilisateurs augmente. Nous établissons la convergence de la procédure sous de faibles hypothèses concernant le modèle. Les vitesses de convergence sont également présentées

Autochtones et langue française dans les départements et territoires d'Outre-Mer

L'importance de la population allochtone et de la pratique écrite de la langue française, fait apparaître de grandes différences entre les départements et territoires d'outre-mer, mais aussi à l'intérieur de chacun d'eux. (Résumé d'auteur

Asymptotic Normality in Density Support Estimation

http://www.math.washington.edu/~ejpecp/index.phpInternational audienceLet $X_1,\dots,X_n$ be $n$ independent observations drawn from a multivariate probability density $f$ with compact support $S_f$. This paper is devoted to the study of the estimator $\hat{S}_n$ of $S_f$ defined as unions of balls centered at the $X_i$ and of common radius $r_n$. Using tools from Riemannian geometry, and under mild assumptions on $f$ and the sequence $(r_n)$, we prove a central limit theorem for $\lambda (S_n \Delta S_f)$, where $\lambda$ denotes the Lebesgue measure on $\mathbb R^d$ and $\Delta$ the symmetric difference operatio

Strongly consistent model selections for densities

International audienceLet f be an unknown multivariate density belonging to a set of densities Fk! of finite associated Vapnik-Chervonenkis dimension, where the complexity k! is unknown, and Fk ! Fk+1 for all k. Given an i.i.d. sample of size n drawn from f, this article presents a density estimate ˆ fKn yielding almost sure convergence of the estimated complexity Kn to the true but unknown k!, and with the property E{ ! | ˆ fKn − f|} = O(1/#n). The methodology is inspired by the combinatorial tools developed in Devroye and Lugosi [8] and it includes a wide range of density models, such as mixture models and exponential families

On the Data Complexity of Statistical Attacks Against Block Ciphers (full version)

Many attacks on iterated block ciphers rely on statistical considerations using plaintext/ciphertext pairs to distinguish some part of the cipher from a random permutation. We provide here a simple formula for estimating the amount of plaintext/ciphertext pairs which is needed for such distinguishers and which applies to a lot of different scenarios (linear cryptanalysis, differential-linear cryptanalysis, differential/truncated differential/impossible differential cryptanalysis). The asymptotic data complexities of all these attacks are then derived. Moreover, we give an efficient algorithm for computing the data complexity accurately