Gretton, A

Poczos, B

Sriperumbudur, B

Szabo, Z

UCL Discovery

Distribution Regression – Make It Simple and Consistent∗Zoltán Szabó1, Bharath K. Sriperumbudur2, Barnabás Póczos3, Arthur Gretton11Gatsby Unit, UCL 2Department of Statistics, PSU 3Machine Learning Department, CMUProblem•Distribution regression:• Input = distribution, output ∈ R/Rd/separable Hilbert space.• Challenge: sampled input distributions.•Examples:•multiple instance learning (MIL),• point estimates of statistics (entropy/hyperparameter/. . . ).•Existing methods: heuristics, or require densityestimation (which typically scale poorly in dimension).Distribution Regression•D(X) distributions on domain (X, k).• z = {(xi, yi)}li=1 i.i.d.∼ M: (xi, yi) ∈ D(X)× Y .•Given: zˆ = {({xi,n}Nn=1, yi)}li=1, where {xi,n}Nn=1 i.i.d.∼ xi.•Goal: learn the relation between (x, y) given zˆ.• Idea: D(X) µ−→ X ⊆ H(k) f∈H(K)−−−−→ Y .•Mean embedding: µx =∫X k(·, u)dx(u).Objective Function, Algorithm•Cost function (of MERR):fλzˆ = arg minf∈H1ll∑i=1‖f (µxˆi)− yi‖2Y + λ ‖f‖2H (λ > 0).•Analytical solution: prediction on a new distribution t(fλzˆ ◦ µ)(t) = k(K + lλIl)−1[y1; . . . ; yl],K = [K(µxˆi, µxˆj)] ∈ L(Y )l×l,k = [K(µxˆ1, µt); . . . ;K(µxˆl, µt)] ∈ L(Y )1×l.•Example: If Y = Rd, then L(Y ) = Rd×d.Goal in Details•Regression function: fρ(µa) =∫Y ydρ(y|µa).•Contribution: analysis of the excess riskE˜(fλzˆ , fρ) = E [fλzˆ ]− E [fρ] ≤ g(l, N, λ)→ 0 and rates,E [f ] = E(x,y) ‖f (µx)− y‖2Y (expected risk).Blanket Assumptions•X: separable, topological domain.• k: bounded, continuous.• Y : separable Hilbert space.•K: bounded, Hölder continuous (h ∈ (0, 1]: exponent).•X = µ (M+1 (D(X))) ∈ B(H).• y: bounded.Example: IfK(µa, µb) = 〈µa, µb〉H⇒ we get the set kernelK(µxˆi, µxˆj) =1N 2N∑n,m=1k(xi,n, xj,m).Performance Guarantees•Well-specified case (fρ ∈ H): fρ is ’c-smooth’ with’b-decaying covariance operator’ and l ≥ λ−1b−1, thenE˜(fλzˆ , fρ) ≤logh(l)Nhλ3+ λc + 1l2λ+ 1lλ1b.•Misspecified case (fρ ∈ L2ρX\H): fρ is ’s-smooth’, L2ρXis separable, and 1λ2 ≤ l, thenE˜(fλzˆ , fρ) ≤logh2(l)Nh2λ32+ 1√lλ+√λmin(1,s)λ√l+ λmin(1,s).ApplicationsSupervised entropy learning:• Label = entropy of the distribution represented by a bag.0 1 2 3−2−1012RMSE Values: MERR=0.11, DFDR=0.285rotation angle (α)Entropy  trueMERRDFDR(a) Entropy of Gaussian0.20.40.60.81MERR DFDRRMSE(b) Boxplot of RMSEAerosol prediction:•Bag = multispectral satellite image ’pixels’ over an area.• Label = aerosol value (highly accurate, expensiveground-based instrument).•Performance:Method 100×RMSE ±stdBaseline [mixture model (EM)] 7.5− 8.5 ±0.1− 0.6MERR: linear K, single 7.91 ±1.61MERR: linear K, ensemble 7.86 ±1.71MERR: nonlinear K, single 7.90 ±1.63MERR: nonlinear K, ensemble 7.81 ±1.64Code: in ITE (https://bitbucket.org/szzoli/ite/).AcknowledgementsThis work was supported by the Gatsby Charitable Foundation, and byNSF grants IIS1247658 and IIS1250350. The work was carried out whileBharath K. Sriperumbudur was a research fellow in the Statistical Lab-oratory, Department of Pure Mathematics and Mathematical Statisticsat the University of Cambridge, UK.∗Data, Learning and Inference workshop (DALI), La Palma (Canaries, Spain), Apr. 10-12, 2015.

Distribution Regression - Make It Simple and Consistent

http://discovery.ucl.ac.uk/1468773/1/szabo15distribution_poster.pdf

Distribution Regression - Make It Simple and Consistent

Abstract

Similar works

Full text

Available Versions

UCL Discovery