3 research outputs found
Feature Learning in Image Hierarchies using Functional Maximal Correlation
This paper proposes the Hierarchical Functional Maximal Correlation Algorithm
(HFMCA), a hierarchical methodology that characterizes dependencies across two
hierarchical levels in multiview systems. By framing view similarities as
dependencies and ensuring contrastivity by imposing orthonormality, HFMCA
achieves faster convergence and increased stability in self-supervised
learning. HFMCA defines and measures dependencies within image hierarchies,
from pixels and patches to full images. We find that the network topology for
approximating orthonormal basis functions aligns with a vanilla CNN, enabling
the decomposition of density ratios between neighboring layers of feature maps.
This approach provides powerful interpretability, revealing the resemblance
between supervision and self-supervision through the lens of internal
representations
The Cross Density Kernel Function: A Novel Framework to Quantify Statistical Dependence for Random Processes
This paper proposes a novel multivariate definition of statistical dependence
using a functional methodology inspired by Alfred R\'enyi. We define a new
symmetric and self-adjoint cross density kernel through a recursive
bidirectional statistical mapping between conditional densities of continuous
random processes, which estimates their statistical dependence. Therefore, the
kernel eigenspectrum is proposed as a new multivariate statistical dependence
measure, and the formulation requires fewer assumptions about the data
generation model than current methods. The measure can also be estimated from
realizations. The proposed functional maximum correlation algorithm (FMCA) is
applied to a learning architecture with two multivariate neural networks. The
FMCA optimal solution is an equilibrium point that estimates the eigenspectrum
of the cross density kernel. Preliminary results with synthetic data and medium
size image datasets corroborate the theory. Four different strategies of
applying the cross density kernel are thoroughly discussed and implemented to
show the versatility and stability of the methodology, and it transcends
supervised learning. When two random processes are high-dimensional real-world
images and white uniform noise, respectively, the algorithm learns a factorial
code i.e., the occurrence of a code guarantees that a certain input in the
training set was present, which is quite important for feature learning