We present a neural process that models exchangeable sequences of high-dimensional complex observations conditionally on a set of labels or tags. Our model combines the expressiveness of deep neural networks with the data-efficiency of Gaussian processes, resulting in a probabilistic model for which the posterior distribution is easy to evaluate and sample from, and the computational complexity scales linearly with the number of observations. The advantages of the proposed architecture are demonstrated on a challenging few-shot view reconstruction task which requires generalisation from short sequences of viewpoints

Dambre, J

Gal, Y

Gretton, A

Korshunova, I

English

UCL Discovery

Conditional BRUNO: A Deep Recurrent Process forExchangeable Labelled DataIryna Korshunova1 Yarin Gal2 Joni Dambre1∗ Arthur Gretton3∗1Ghent University 2University of Oxford 3Gatsby Unit, UCLiryna.korshunova@ugent.be1 IntroductionExchangeability is often an implicit assumption underlying many machine learning algorithms. Itimplies that any re-ordering of a finite sequence of observations is equally likely. Thus, it allows toreason about the future observations based on the behaviour of the previous ones. Some problemscan be explicitly formulated in terms of modelling exchangeable sequences. For instance, few-shotconcept learning can be seen as learning to complete short exchangeable sequences [13]. BRUNO [12]follows this approach by modelling autoregressive distributions p(xn|x1:n−1) of an exchangeableprocess. In this work, we extend the idea of BRUNO to the conditional case, where we wish to modelp(xn|hn, x1:n−1, h1:n−1) with hi’s being labels or tags associated with images xi’s.Formally, a stochastic process x1, x2, x3, . . . is said to be exchangeable if for all n and all permuta-tions πp(x1, . . . , xn) = p(xπ(1), . . . , xπ(n)), (1)i. e. the joint probability remains the same under any permutation of the sequence.The concept of exchangeability is intimately related to Bayesian statistics via de Finetti’s theorem,which states that every exchangeable process is a mixture of i. i. d. processes:p(x1, . . . , xn) =∫p(θ)n∏i=1p(xi|θ)dθ, (2)where θ is a parameter (finite or infinite-dimensional) conditioned on which, xi’s are i. i. d. [1].This theorem gives two ways of defining models of exchangeable sequences. One is via explicitBayesian modelling: define a prior p(θ), a likelihood p(xi|θ) and calculate the posterior in Eq. 2directly. Here, the difficulty is the intractability of the posterior as it requires an integration over theparameter θ. A common solution is to use a variational approximation. The neural statistician [5]implements this approach by building upon a variational autoencoder model (VAE) [11].The second way is to construct an exchangeable process while modelling its autoregressive distri-butions p(xn|x1:n−1) directly without referring to the underlying Bayesian model. BRUNO [12]proposes a design for doing so. It consists of two components: (a) a bijective mapping that transformsan intricate input space X into a Gaussian latent space Z , and (b) exchangeable Gaussian processes(GPs) defined in the latent space Z . Using deep neural networks to implement the bijection allows tomodel complex and high-dimensional inputs, while the whole construction of BRUNO guaranteesthat the process in X is exchangeable.A natural extension when building exchangeable models would be to have a conditional process withtwo associated sequences: x1, x2, x3, . . . and h1, h2, h3, . . . . For instance, xi could be an imageand hi a vector of descriptive labels or tags. By analogy with Eq. 1, the exchangeability propertybecomes:p(x1, . . . , xn|h1, . . . , hn) = p(xπ(1), . . . , xπ(n)|hπ(1), . . . , hπ(n)). (3)∗Equal contributionThird workshop on Bayesian Deep Learning (NeurIPS 2018), Montréal, Canada.To have a valid stochastic process, we also need a consistency property as imposed by the Kolmogorovextension theorem [14]:p(x1:m|h1:m) =∫p (x1:n|h1:n) dxm+1:n for 1 ≤ m < n. (4)To our best knowledge, Bayesian theory does not have an established proof of de Finetti’s theoremfor conditional probabilities. Namely, that the two conditions above ensure that one can represent theprocess as a mixture of conditionally i.i.d. models as given in Eq. 5. For the processes where xi andhi take values from a finite set, this theorem is proven in the field of quantum physics [2]. However,it is yet unclear how to extend their results to continuum variables.p(x1:n|h1:n) =∫p(θ)n∏i=1p(xi|hi, θ)dθ. (5)Relying on the conditional version of de Finetti’s theorem, neural processes [7] take an approachthat is similar to the neural statistician’s. Namely, by extending the VAE model to handle collectionsof (xi, hi) input pairs and dealing with a lower bound on p(xn|hn, x1:n−1, h1:n−1). Versa [8] alsofollows the idea of approximating the posterior predictive distribution, though it uses a trainingprocedure that differs from the standard variational inference. Both models achieve permutationinvariance of p(θ|x1:n, h1:n) with respect to the conditioning inputs by using instance-poolingoperations, e.g. the mean over representations of (xi, hi) pairs.Another option is to use the idea of BRUNO and construct a process that satisfies Eq. 3 and 4.In the next section, we show this can be done by modifying the architecture of BRUNO. Namely,by conditioning the bijective transformation f : X 7→ Z on the tags, such that zi = fhi(xi). Aschematic of our model is given in Fig. 1. While the change to the model is incremental, the fact thatBRUNO can be so easily extended to the conditional case is interesting by itself. Moreover, it addsan important class into the collection of meta-learning problems that BRUNO is capable of solving.z11 z21x11 x21h1z12 z22x12 x22h2z1 z2x1 x2hGP GPp(x2|x1,h1:2) = p(z12|z11)p(z22|z12)∣∣ det ∂z2∂x2∣∣p(x1|h1) = p(z11)p(z21)∣∣ det ∂z1∂x1∣∣sampleRealNVPRealNVPRealNVP-1Figure 1: A schematic of the conditional BRUNO model.2 Conditional BRUNOThe bijective transformation part of BRUNO is carried out by a Real NVP [4] – a deep, stablyinvertible and learnable neural network architecture that transforms some density p(x) into a desiredprobability distribution p(z). It is implemented as a sequence of alternating coupling layers, withevery layer transforming a half of its input dimensions while copying the other half directly to theoutput. In case of modelling a conditional distribution p(x|h), we can make the transformationdependendent on h, so the outputs of the coupling layer become:{x1:dout = x1:dinxd+1:Dout = xd+1:Din  exp(s(x1:din ,h)) + t(x1:din ,h),(6)where  is an elementwise product, and functions s (scale) and t (translation) are usually deep neuralnetworks. We achieved the conditioning on h by adding a bias computed from the features of h toevery layer inside the s and t networks.As in case of the original Real NVP model, we can assume a fixed distribution for the latents zdue to the fact that dependence of x on h is introduced via the Jacobian of the transformation. The2latter is used in the change of variables formula: p(x|h) = p(z) |detJh| . For the same reasons, theconditional BRUNO can use the identical assumptions as its unconditional counterpart:A1: dimensions {zd}d=1,...,D are independent, so p(z) =∏Dd=1 p(zd)A2: for every dimension d, we assume that (zd1 , . . . zdn) ∼ MVNn(0,Σd), where Σd is a n × ncovariance matrix with Σdii = vd and Σdij,i 6=j = ρd, 0 ≤ ρd < vd.3 ExperimentsWe consider a task of few-shot image reconstruction, where the model is required to infer howan object looks from various angles based on a small set of observed views [8]. In our model,this problem can be framed as generating samples from a predictive conditional distributionp(xn|hn,x1:n−1,h1:n−1), where hn is a desired angle and x1:n−1 is a set of observed viewsassociated with angles h1:n−1. We use airplanes and chairs from the ShapeNetCore v2 [3] dataset asconstructed by Gordon et al. [8], and train the conditional BRUNO on one-shot tasks. We give a singlerandom view x1 and its angle h1 and the goal is to predict N views of the same object under anglesh1, . . . ,hN . Namely, the objective in a single task is to maximise L =∑Nn=1 log p(xn|hn,x1,h1)with respect to the Real NVP parameters and variance-covariance parameters of the latent GPs. Note,that unlike in Gordon et al. [8], we train a single model on a combined set of chairs and airplanes.In Figure 2, we show samples from a conditional BRUNO when the model was given a singleviewpoint from an object not seen during training. Note that conditioned on a chair, it never samplesan airplane or vice versa. Moreover, from Figure 3 one can see better how the model’s uncertaintyabout the object is reflected in the samples. Specifically, in the case when a single shot it is conditionedupon gives insufficient information about the object, conditional BRUNO generates diverse objectswhich are quite consistent with the given shot. Also, our samples always have a correct orientationand their quality is superior to the one from Versa [8] as our samples are sharp.Figure 2: One-shot BRUNO samples for the unseen test objects. Here, the model is given a singleview (x1,h1) of a chair or an airplane. This input shot is marked in red. On the top row is the groundtruth, whereas the three rows underneath contain samples from the model conditioned on the inputshot and a desired angle.Figure 3: BRUNO samples for the test objects conditioned on 1 or 4 shots and an angle. From asingle shot, it is difficult to infer the exact appearance, e.g. the front of the chair or the aiplane type.Thus, the model increases the variability of the samples along these dimensions. With more inputshots, samples become more consistent. This is most noticeable for the shape of the airplane’s wings.A more complex version of the few-shot image reconstruction, is learning to render scenes as done byGenerative Query Networks [6], which are similar to the aforementioned neural processes [7] in their3core idea. We suppose that conditional BRUNO, when scaled to complex datasets, might become aviable alternative to the VAE-based types of models.4 ConclusionWe showed that BRUNO [12] can be easily extended to the conditional case while maintaining itsappealing properties such as provable exchangeability, exact posterior computation, fast samplingand recurrent formulation of the Bayesian updates. These features make BRUNO a simple yet aneffective and flexible model for meta-learning.BRUNO combines the data-efficiency of GPs with a power of deep learning to model complex datatypes, and while the former is unlikely to be improved, we expect BRUNO to greatly benefit from therecent advances in building normalising flows, which is currently an active area of research [9, 10].AcknowledgementsWe would like to thank Jonas Degrave for insightful discussions and Figure 1, Thu Nguyen Phuoc foran overview of ShapeNet papers, Jonathan Gordon and John Bronskill for answering questions aboutVersa and their code on github, and Ferenc Huszár for the whole idea of exchangeability via RNNs.References[1] Aldous, D., Hennequin, P., Ibragimov, I., and Jacod, J. (1985). Ecole d’Ete de Probabilites de Saint-FlourXIII, 1983. Lecture Notes in Mathematics. Springer Berlin Heidelberg.[2] Barrett, J. and Leifer, M. (2009). The de Finetti theorem for test spaces. New Journal of Physics, 11(3).[3] Chang, A. X., Funkhouser, T. A., Guibas, L. J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M.,Song, S., Su, H., Xiao, J., Yi, L., and Yu, F. (2015). ShapeNet: An information-rich 3D model repository.arXiv preprint arXiv:1512.03012.[4] Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2017). Density estimation using Real NVP. In Proceedings ofthe 5th International Conference on Learning Representations.[5] Edwards, H. and Storkey, A. (2017). Towards a neural statistician. In Proceedings of the 5th InternationalConference on Learning Representations.[6] Eslami, S. M. A., Jimenez Rezende, D., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A.,Rusu, A. A., Danihelka, I., Gregor, K., Reichert, D. P., Buesing, L., Weber, T., Vinyals, O., Rosenbaum, D.,Rabinowitz, N., King, H., Hillier, C., Botvinick, M., Wierstra, D., Kavukcuoglu, K., and Hassabis, D. (2018).Neural scene representation and rendering. Science, 360(6394):1204–1210.[7] Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S. M. A., and Teh, Y. W. (2018).Neural processes. Theoretical Foundations and Applications of Deep Generative Models, ICML workshop.[8] Gordon, J., Bronskill, J., Bauer, M., Nowozin, S., and Turner, R. E. (2018). Decision-theoretic meta-learning:Versatile and efficient amortization of few-shot learning. arXiv preprint arXiv:1805.09921.[9] Grathwohl, W., Chen, R. T. Q., Bettencourt, J., Sutskever, I., and Duvenaud, D. (2018). Ffjord: Free-formcontinuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367.[10] Kingma, D. P. and Dhariwal, P. (2018). Glow: Generative flow with invertible 1x1 convolutions. arXivpreprint arXiv:1807.03039.[11] Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In Proceedings of the 2ndInternational Conference on Learning Representations.[12] Korshunova, I., Degrave, J., Huszár, F., Gal, Y., Gretton, A., and Dambre, J. (2018). BRUNO: A deeprecurrent model for exchangeable data. In Proceedings of the 32th International Conference on NeuralInformation Processing Systems.[13] Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. (2015). Human-level concept learning throughprobabilistic program induction. Science.[14] Øksendal, B. (2003). Stochastic Differential Equations: An Introduction with Applications. Hochschultext/ Universitext. Springer.4

Conditional BRUNO: A neural process for exchangeable labelled data

https://discovery.ucl.ac.uk/id/eprint/10084670/1/BayesianBRUNO.pdf

Conditional BRUNO: A neural process for exchangeable labelled data

Abstract

Similar works

Full text

Available Versions

UCL Discovery