International audienceInterpretable progression models for longitudinal neuroimaging data are crucial to understanding neurodegenerative diseases. Well validated geometric progression models for biomarkers do not scale for such high dimensional data. In this work, we analyse a recent approach that combines a Variational Autoencoder with a latent linear mixed-effects model, and demonstrate that imposing a Euclidean prior on the latent space allows the network to learn the geometry of the observation manifold, and model non linear dynamics

Durrleman​, Stanley

Sauty, Benoît

INRIA a CCSD electronic archive server

HAL Id: hal-03800499https://hal.inria.fr/hal-03800499Submitted on 6 Oct 2022HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Longitudinal Variational Autoencoders learn aRiemannian progression model for imaging dataBenoît Sauty, Stanley Durrleman To cite this version:Benoît Sauty, Stanley Durrleman . Longitudinal Variational Autoencoders learn a Riemannian pro-gression model for imaging data. GeoMedIA Workshop 2022 - Geometric Deep Learning in MedicalImage Analysis, Nov 2022, Amsterdam, Netherlands. ￿hal-03800499￿Proceedings of Machine Learning Research 1:1–4, 2022 Submission for GeoMedIA WorkshopLongitudinal Variational Autoencoders learn a Riemannianprogression model for imaging data.Benôıt SautyStanley DurrlemanAbstractInterpretable progression models for longitudinal neuroimaging data are crucial to un-derstanding neurodegenerative diseases. Well validated geometric progression models forbiomarkers do not scale for such high dimensional data. In this work, we analyse a re-cent approach that combines a Variational Autoencoder with a latent linear mixed-effectsmodel, and demonstrate that imposing a Euclidean prior on the latent space allows thenetwork to learn the geometry of the observation manifold, and model non linear dynamics.Keywords: Longitudinal progression - Riemannian manifolds - Variational Autoencoders1. IntroductionMany flavours of geometric disease progression models for biomarkers assume that the datalie on a manifold and evolve according to non-linear geodesics across time. Generalization ofsuch models to high dimensional neuroimaging data has called for the use of deep learningdimension reduction techniques. On the other hand, much attention has been put on findinggeometric interpretations of deep learning generative models, in order to refine the learneddistribution. Through the geometric interpretation of a recently introduced LongitudinalVariational Autoencoder model, we will bridge the gap between the well validated diseaseprogression models for biomarkers and the novel deep-learning-based methods.1.1. Geometric disease progression modelsMixed-effects models provide one of the most popular disease progression framework forlongitudinal data. Individual trajectories are modeled as small variations around thepopulation-average trajectory. Early models used linear modeling while non linearitieswere later added with polynomial, logistic and exponential regressions. One less restrictiveassumption is to consider that the observed biomarkers follow continuous trajectories inthe space of observations that is assumed to be a Riemannian manifold (Schiratti et al.,2015). This approach provides spatio-temporal models that describe the average trajectoryas a geodesic and the inter-patients variability as the parallel transport of this trajectoryon the manifold. Particular cases have been derived with a metric that is set a prioriin order to yield closed-form trajectories (Koval et al., 2017; Schiratti et al., 2017), butless restrictive assumptions have been made in order to learn the appropriate metric fromthe data (Gruffaz et al., 2021; Sauty and Durrleman, 2022b), at the expense of an addedcomputational burden for the computation of the exponentials as Hamiltonian flows.Those models however do not scale with high dimensional data and are mostly limitedto the study of biomarkers. For neurodegenerative diseases, studying the alterations ofthe brain is a crucial task in order to understand the early stages of the disease. However,progression models for neuroimaging data are still understudied, and the use of deep learningas a ”black-box” constitutes a barrier to widespread application by clinicians.© 2022 B. Sauty & S. Durrleman.Sauty Durrleman1.2. Geometry-aware deep generative modelsDeep generative models aim to provide a neural network generator pθ : z ∈ Z 7→ x ∈ X ,parametrized by θ, from a latent space Z ⊂ Rn to a target space X ⊂ RN (usually n << N)such that pθ maps a simple distribution in Z to the complex data distribution in X . TheJacobian of the generator J = ∂pθ∂z provides a linear mapping from tangent vectors of Zto tangent vectors of X and M = JTJ thus defines a smoothly varying inner product onthe tangent bundle TZ, and can be seen as a Riemannian metric on Z. For VAEs, thegenerator network – the decoder – is coupled with an encoder network qϕ, parametrizedby ϕ, that provides a variational estimate of the posterior distribution qϕ(z|x) for x ∈ X .When adding a prior on the latent space, we obtain a lower bound for the log likelihoodthat can be optimized through backpropagation (Kingma and Welling, 2013).In Arvanitidis et al. (2017); Chen et al. (2018), the Riemannian metric M is used todescribe the geometry of the latent space of a VAE, significantly improving interpolantsand distance computation, and Shao et al. (2018) experimentally note that the curvatureof the learned latent space is almost flat. Falorsi et al. (2018), on the other hand, proposedto introduce Riemannian geometry tools in the input space of the VAE itself, in order toallow the generation of manifold valued data.1.3. Longitudinal Variational AutoencodersIn recent work, Sauty and Durrleman (2022a) introduced a progression model for imagingdata that embeds a linear mixed-effect model in the latent space of a VAE. This modelcan then be calibrated on longitudinal databases of neuroimaging data such as t1-MRI orFDG-PET scans in order to provide a simple parametrization of the individual trajectoriesin Z, that translate into non-linear trajectories in the image domain. An iterative MCMCoptimization scheme allows to ensure that the VAE reconstructs images correctly whileallowing to match the simple linear mixed-effect description of latent encodings to thecomplex trajectories in the observation space. Such an approach allows sampling individualtrajectories at any timepoint – future or past – for prediction and missing data imputation.Contributions The geometric nature of the VAE has been recognised and extensivelystudied. The most widespread paradigm is to embed the latent space with a Riemannianmanifold structure, equipped with the pull-back of the Euclidean metric of the observationspace trough the decoder. On the other hand, it is standard to consider that medicaldata lie on a non-Euclidean manifold, on which trajectories are parametrized using mixed-effects models, to provide interpretable non-linear progression models. For high dimensionalneuroimaging data, the use of dimension reduction techniques such as VAEs is required.In this work we provide a geometric interpretation of the LVAE progression model. Wedemonstrate that imposing a Euclidean prior on the latent space allows to characterise theRiemannian metric of the observation manifold as the push-forward of the Euclidean metricthrough the decoder, providing a simple way to compute exponentials and geodesics in theimage domain. This ”reversed” conception of the geometry of the VAEs, first introducedin Louis et al. (2019), bridges the gap between well-validated geometric progression modelsfor biomakers and recent approaches that integrate deep learning tools in order to modellongitudinal neuroimaging data.2Geometry in LVAE2. Geometry of the observation manifoldWith the former notations, pθ(Z) is a n-dimensional immersed submanifold of RN if theactivations functions are smooth and monotonic and if the weight matrix of each layer hasmaximal rank (Shao et al., 2018). The first condition is a design choice and the secondcondition can be checked after training. If we notate g the Euclidean metric on Z, thepush-forward of g on pθ(Z) is defined, for any smooth vector fields U, V on pθ(Z), asp∗θ(g)(U, V ) = g((pθ)∗(U), (pθ)∗(V ))where (pθ)∗(U) and (pθ)∗(V ) are the pull-back of U and V on Z, which are defined by(pθ)∗(U) : f 7→ U(f ◦ p−1θ ) for smooth functions f : Z 7→ R. Any geodesic γ : [0, 1] → Zon (Z, g) thus translates to a geodesic pθ : [0, 1] → X on (pθ(Z), p∗θ(g)). The decodernetwork is an isometry between (Z, g) and (pθ(Z), p∗θ(g)) so computation of Riemannianexponential or parallel transport can be done in Z inexpensively, before being push-forwardto the observation space. In the context of LVAE, the linear trajectories of the latentspace and inter-patients translations thus translate to geodesics and parallel transport inthe corresponding submanifold of the observation space.Figure 1: Schematic representation of the LVAE model. A linear trajectory is fitted onthe latent representations (dots in Z) of successive images, which then decodesinto a geodesic of the observation space. Orange line is the population-averagetrajectory, travelled across time. Blue patients has 4 visits and violet patient has3, displaying a progression of brain alterations.3. ConclusionWe provided a geometric interpretation the LVAE model that illustrates how the neuralnetwork can be understood as a way to parametrize the Riemannian metric of the obser-vation manifold. The LVAE thus provides an extension of existing well-validated geometricmodels and provides a cheap way to compute nonlinear dynamics for high dimensionalneuroimaging data.3Sauty DurrlemanReferencesGeorgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on thecurvature of deep generative models. arXiv preprint arXiv:1710.11379, 2017.Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and PatrickSmagt. Metrics for deep generative models. In International Conference on ArtificialIntelligence and Statistics, pages 1540–1550. PMLR, 2018.Luca Falorsi, Pim De Haan, Tim R Davidson, Nicola De Cao, Maurice Weiler, PatrickForré, and Taco S Cohen. Explorations in homeomorphic variational auto-encoding.arXiv preprint arXiv:1807.04689, 2018.Samuel Gruffaz, Pierre-Emmanuel Poulet, Etienne Maheux, Bruno Jedynak, and StanleyDurrleman. Learning riemannian metric for disease progression modeling. Advances inNeural Information Processing Systems, 34, 2021.Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprintarXiv:1312.6114, 2013.I. Koval, J-B Schiratti, A. Routier, M. Bacci, O. Colliot, S. Allassonnière, and S. Durrle-man. Statistical learning of spatiotemporal patterns from longitudinal manifold-valuednetworks. In International Conference on MICCAI, pages 451–459. Springer, 2017.Maxime Louis, Raphaël Couronné, Igor Koval, Benjamin Charlier, and Stanley Durrleman.Riemannian geometry learning for disease progression modelling. In International Con-ference on Information Processing in Medical Imaging, pages 542–553. Springer, 2019.Benôıt Sauty and Stanley Durrleman. Progression models for imaging data with longitudinalvariational auto encoders. In MICCAI 2022, International Conference on Medical ImageComputing and Computer Assisted Intervention, 2022a.Benôıt Sauty and Stanley Durrleman. Riemannian metric learning for progression modelingof longitudinal datasets. In 2022 IEEE 19th International Symposium on BiomedicalImaging (ISBI), pages 1–5. IEEE, 2022b.J-B Schiratti, S. Allassonniere, O. Colliot, and S. Durrleman. Learning spatiotemporaltrajectories from manifold-valued longitudinal data. In Neural Information ProcessingSystems, number 28, 2015.J-B Schiratti, S. Allassonnière, O. Colliot, and S. Durrleman. A bayesian mixed-effectsmodel to learn trajectories of changes from repeated manifold-valued observations. TheJournal of Machine Learning Research, 18(1):4840–4872, 2017.Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deepgenerative models. In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition Workshops, pages 315–323, 2018.4

Longitudinal Variational Autoencoders learn a Riemannian progression model for imaging data

https://hal.inria.fr/hal-03800499v1/file/GeoMedia_2022___Geometric_insights%20%281%29.pdf

Longitudinal Variational Autoencoders learn a Riemannian progression model for imaging data

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server