A family of probability distributions parametrized by an open domain
Λ in Rn defines the Fisher information matrix on this domain which
is positive semi-definite. In information geometry the standard assumption has
been that the Fisher information matrix tensor is positive definite defining in
this way a Riemannian metric on Λ. If we replace the "positive
definite" assumption by the existence of a suitable torsion-free connection, a
foliation with a transversely Hessian structure appears naturally. In the paper
we develop the study of transversely Hessian foliations in view of applications
in information geometry