We study the critical behavior of a random field O(N) spin model with a
second-rank random anisotropy term in spatial dimensions 4<d<6, by means of
the replica method and the 1/N expansion. We obtain a replica-symmetric
solution of the saddle-point equation, and we find the phase transition obeying
dimensional reduction. We study the stability of the replica-symmetric saddle
point against the fluctuation induced by the second-rank random anisotropy. We
show that the eigenvalue of the Hessian at the replica-symmetric saddle point
is strictly positive. Therefore, this saddle point is stable and the
dimensional reduction holds in the 1/N expansion. To check the consistency with
the functional renormalization group method, we obtain all fixed points of the
renormalization group in the large N limit and discuss their stability. We
find that the analytic fixed point yielding the dimensional reduction is
practically singly unstable in a coupling constant space of the given model
with large N. Thus, we conclude that the dimensional reduction holds for
sufficiently large N.Comment: 19 pages, 5 figure