1,626 research outputs found
Understanding VAEs in Fisher-Shannon Plane
In information theory, Fisher information and Shannon information (entropy)
are respectively used to quantify the uncertainty associated with the
distribution modeling and the uncertainty in specifying the outcome of given
variables. These two quantities are complementary and are jointly applied to
information behavior analysis in most cases. The uncertainty property in
information asserts a fundamental trade-off between Fisher information and
Shannon information, which enlightens us the relationship between the encoder
and the decoder in variational auto-encoders (VAEs). In this paper, we
investigate VAEs in the Fisher-Shannon plane and demonstrate that the
representation learning and the log-likelihood estimation are intrinsically
related to these two information quantities. Through extensive qualitative and
quantitative experiments, we provide with a better comprehension of VAEs in
tasks such as high-resolution reconstruction, and representation learning in
the perspective of Fisher information and Shannon information. We further
propose a variant of VAEs, termed as Fisher auto-encoder (FAE), for practical
needs to balance Fisher information and Shannon information. Our experimental
results have demonstrated its promise in improving the reconstruction accuracy
and avoiding the non-informative latent code as occurred in previous works
- …