Manifold learning flows are a class of generative modelling techniques that
assume a low-dimensional manifold description of the data. The embedding of
such a manifold into the high-dimensional space of the data is achieved via
learnable invertible transformations. Therefore, once the manifold is properly
aligned via a reconstruction loss, the probability density is tractable on the
manifold and maximum likelihood can be used to optimize the network parameters.
Naturally, the lower-dimensional representation of the data requires an
injective-mapping. Recent approaches were able to enforce that the density
aligns with the modelled manifold, while efficiently calculating the density
volume-change term when embedding to the higher-dimensional space. However,
unless the injective-mapping is analytically predefined, the learned manifold
is not necessarily an efficient representation of the data. Namely, the latent
dimensions of such models frequently learn an entangled intrinsic basis, with
degenerate information being stored in each dimension. Alternatively, if a
locally orthogonal and/or sparse basis is to be learned, here coined canonical
intrinsic basis, it can serve in learning a more compact latent space
representation. Toward this end, we propose a canonical manifold learning flow
method, where a novel optimization objective enforces the transformation matrix
to have few prominent and non-degenerate basis functions. We demonstrate that
by minimizing the off-diagonal manifold metric elements β1β-norm, we can
achieve such a basis, which is simultaneously sparse and/or orthogonal.
Canonical manifold flow yields a more efficient use of the latent space,
automatically generating fewer prominent and distinct dimensions to represent
data, and a better approximation of target distributions than other manifold
flow methods in most experiments we conducted, resulting in lower FID scores.Comment: NeurIPS 202