Predictive PAC Learning and Process Decompositions

We informally call a stochastic process learnable if it admits a generalization error approaching zero in probability for any concept class with finite VC-dimension (IID processes are the simplest example). A mixture of learnable processes need not be learnable itself, and certainly its generalization error need not decay at the same rate. In this paper, we argue that it is natural in predictive PAC to condition not on the past observations but on the mixture component of the sample path. This definition not only matches what a realistic learner might demand, but also allows us to sidestep several otherwise grave problems in learning from dependent data. In particular, we give a novel PAC generalization bound for mixtures of learnable processes with a generalization error that is not worse than that of each mixture component. We also provide a characterization of mixtures of absolutely regular ($\beta$-mixing) processes, of independent probability-theoretic interest.Comment: 9 pages, accepted in NIPS 201

Exchangeable Variable Models

A sequence of random variables is exchangeable if its joint distribution is invariant under variable permutations. We introduce exchangeable variable models (EVMs) as a novel class of probabilistic models whose basic building blocks are partially exchangeable sequences, a generalization of exchangeable sequences. We prove that a family of tractable EVMs is optimal under zero-one loss for a large class of functions, including parity and threshold functions, and strictly subsumes existing tractable independence-based model families. Extensive experiments show that EVMs outperform state of the art classifiers such as SVMs and probabilistic models which are solely based on independence assumptions.Comment: ICML 201

Berry Esseen bounds for combinatorial central limit theorems and pattern occurrences, using zero and size biasing

Berry Esseen type bounds to the normal, based on zero- and size-bias couplings, are derived using Stein's method. The zero biasing bounds are illustrated with an application to combinatorial central limit theorems where the random permutation has either the uniform distribution or one which is constant over permutations with the same cycle type and having no fixed points. The size biasing bounds are applied to the occurrences of fixed relatively ordered sub-sequences (such as rising sequences) in a random permutation, and to the occurrences of patterns, extreme values, and subgraphs on finite graphs.Comment: 23 page

Limit theorems for a class of identically distributed random variables

A new type of stochastic dependence for a sequence of random variables is introduced and studied. Precisely, (X_n)_{n\geq 1} is said to be conditionally identically distributed (c.i.d.), with respect to a filtration (G_n)_{n\geq 0}, if it is adapted to (G_n)_{n\geq 0} and, for each n\geq 0, (X_k)_{k>n} is identically distributed given the past G_n. In case G_0={\varnothing,\Omega} and G_n=\sigma(X_1,...,X_n), a result of Kallenberg implies that (X_n)_{n\geq 1} is exchangeable if and only if it is stationary and c.i.d. After giving some natural examples of nonexchangeable c.i.d. sequences, it is shown that (X_n)_{n\geq 1} is exchangeable if and only if (X_{\tau(n)})_{n\geq 1} is c.i.d. for any finite permutation \tau of {1,2,...}, and that the distribution of a c.i.d. sequence agrees with an exchangeable law on a certain sub-\sigma-field. Moreover, (1/n)\sum_{k=1}^nX_k converges a.s. and in L^1 whenever (X_n)_{n\geq 1} is (real-valued) c.i.d. and E[| X_1| ]<\infty. As to the CLT, three types of random centering are considered. One such centering, significant in Bayesian prediction and discrete time filtering, is E[X_{n+1}| G_n]. For each centering, convergence in distribution of the corresponding empirical process is analyzed under uniform distance.Comment: Published by the Institute of Mathematical Statistics (http://www.imstat.org) in the Annals of Probability (http://www.imstat.org/aop/) at http://dx.doi.org/10.1214/00911790400000067

On q-Gaussians and Exchangeability

The q-Gaussians are discussed from the point of view of variance mixtures of normals and exchangeability. For each q< 3, there is a q-Gaussian distribution that maximizes the Tsallis entropy under suitable constraints. This paper shows that q-Gaussian random variables can be represented as variance mixtures of normals. These variance mixtures of normals are the attractors in central limit theorems for sequences of exchangeable random variables; thereby, providing a possible model that has been extensively studied in probability theory. The formulation provided has the additional advantage of yielding process versions which are naturally q-Brownian motions. Explicit mixing distributions for q-Gaussians should facilitate applications to areas such as option pricing. The model might provide insight into the study of superstatistics.Comment: 14 page

Sharp Total Variation Bounds for Finitely Exchangeable Arrays

In this article we demonstrate the relationship between finitely exchangeable arrays and finitely exchangeable sequences. We then derive sharp bounds on the total variation distance between distributions of finitely and infinitely exchangeable arrays

Anomalous scaling due to correlations: Limit theorems and self-similar processes

We derive theorems which outline explicit mechanisms by which anomalous scaling for the probability density function of the sum of many correlated random variables asymptotically prevails. The results characterize general anomalous scaling forms, justify their universal character, and specify universality domains in the spaces of joint probability density functions of the summand variables. These density functions are assumed to be invariant under arbitrary permutations of their arguments. Examples from the theory of critical phenomena are discussed. The novel notion of stability implied by the limit theorems also allows us to define sequences of random variables whose sum satisfies anomalous scaling for any finite number of summands. If regarded as developing in time, the stochastic processes described by these variables are non-Markovian generalizations of Gaussian processes with uncorrelated increments, and provide, e.g., explicit realizations of a recently proposed model of index evolution in finance.Comment: Through text revision. 15 pages, 3 figure