No Free Lunch versus Occam's Razor in Supervised Learning

The No Free Lunch theorems are often used to argue that domain specific knowledge is required to design successful algorithms. We use algorithmic information theory to argue the case for a universal bias allowing an algorithm to succeed in all interesting problem domains. Additionally, we give a new algorithm for off-line classification, inspired by Solomonoff induction, with good performance on all structured problems under reasonable assumptions. This includes a proof of the efficacy of the well-known heuristic of randomly selecting training data in the hope of reducing misclassification rates.Comment: 16 LaTeX pages, 1 figur

Transition asymptotics for reaction-diffusion in random media

We describe a universal transition mechanism characterizing the passage to an annealed behavior and to a regime where the fluctuations about this behavior are Gaussian, for the long time asymptotics of the empirical average of the expected value of the number of random walks which branch and annihilate on ${\mathbb Z}^d$, with stationary random rates. The random walks are independent, continuous time rate $2d\kappa$, simple, symmetric, with $\kappa \ge 0$. A random walk at $x\in{\mathbb Z}^d$, binary branches at rate $v_+(x)$, and annihilates at rate $v_-(x)$. The random environment $w$ has coordinates $w(x)=(v_-(x),v_+(x))$ which are i.i.d. We identify a natural way to describe the annealed-Gaussian transition mechanism under mild conditions on the rates. Indeed, we introduce the exponents $F_\theta(t):=\frac{H_1((1+\theta)t)-(1+\theta)H_1(t)}{\theta}$, and assume that $\frac{F_{2\theta}(t)-F_\theta(t)}{\theta\log(\kappa t+e)}\to\infty$ for $|\theta|>0$ small enough, where $H_1(t):=\log$ and $$ denotes the average of the expected value of the number of particles $m(0,t,w)$ at time $t$ and an environment of rates $w$, given that initially there was only one particle at 0. Then the empirical average of $m(x,t,w)$ over a box of side $L(t)$ has different behaviors: if $L(t)\ge e^{\frac{1}{d} F_\epsilon(t)}$ for some $\epsilon >0$ and large enough $t$, a law of large numbers is satisfied; if $L(t)\ge e^{\frac{1}{d} F_\epsilon (2t)}$ for some $\epsilon>0$ and large enough $t$, a CLT is satisfied. These statements are violated if the reversed inequalities are satisfied for some negative $\epsilon$. Applications to potentials with Weibull, Frechet and double exponential tails are given.Comment: To appear in: Probability and Mathematical Physics: A Volume in Honor of Stanislav Molchanov, Editors - AMS | CRM, (2007