In this paper, we study linear regression applied to data structured on a
manifold. We assume that the data manifold is smooth and is embedded in a
Euclidean space, and our objective is to reveal the impact of the data
manifold's extrinsic geometry on the regression. Specifically, we analyze the
impact of the manifold's curvatures (or higher order nonlinearity in the
parameterization when the curvatures are locally zero) on the uniqueness of the
regression solution. Our findings suggest that the corresponding linear
regression does not have a unique solution when the embedded submanifold is
flat in some dimensions. Otherwise, the manifold's curvature (or higher order
nonlinearity in the embedding) may contribute significantly, particularly in
the solution associated with the normal directions of the manifold. Our
findings thus reveal the role of data manifold geometry in ensuring the
stability of regression models for out-of-distribution inferences.Comment: 13 pages, 6 figures, accepted to TAGML23 workshop of ICML2023, to be
published in PML