1 research outputs found
Mode Combinability: Exploring Convex Combinations of Permutation Aligned Models
We explore element-wise convex combinations of two permutation-aligned neural
network parameter vectors and of size . We conduct
extensive experiments by examining various distributions of such model
combinations parametrized by elements of the hypercube and its
vicinity. Our findings reveal that broad regions of the hypercube form surfaces
of low loss values, indicating that the notion of linear mode connectivity
extends to a more general phenomenon which we call mode combinability. We also
make several novel observations regarding linear mode connectivity and model
re-basin. We demonstrate a transitivity property: two models re-based to a
common third model are also linear mode connected, and a robustness property:
even with significant perturbations of the neuron matchings the resulting
combinations continue to form a working model. Moreover, we analyze the
functional and weight similarity of model combinations and show that such
combinations are non-vacuous in the sense that there are significant functional
differences between the resulting models