We allow for nonlinear effects in the likelihood analysis of galaxy peculiar
velocities, and obtain ~35%-lower values for the cosmological density parameter
Om and the amplitude of mass-density fluctuations. The power spectrum in the
linear regime is assumed to be a flat LCDM model (h=0.65, n=1, COBE) with only
Om as a free parameter. Since the likelihood is driven by the nonlinear regime,
we "break" the power spectrum at k_b=0.2 h/Mpc and fit a power law at k>k_b.
This allows for independent matching of the nonlinear behavior and an unbiased
fit in the linear regime. The analysis assumes Gaussian fluctuations and
errors, and a linear relation between velocity and density. Tests using proper
mock catalogs demonstrate a reduced bias and a better fit. We find for the
Mark3 and SFI data Om_m=0.32+-0.06 and 0.37+-0.09 respectively, with
sigma_8*Om^0.6 = 0.49+-0.06 and 0.63+-0.08, in agreement with constraints from
other data. The quoted 90% errors include cosmic variance. The improvement in
likelihood due to the nonlinear correction is very significant for Mark3 and
moderately so for SFI. When allowing deviations from LCDM, we find an
indication for a wiggle in the power spectrum: an excess near k=0.05 and a
deficiency at k=0.1 (cold flow). This may be related to the wiggle seen in the
power spectrum from redshift surveys and the second peak in the CMB anisotropy.
A chi^2 test applied to modes of a Principal Component Analysis (PCA) shows
that the nonlinear procedure improves the goodness of fit and reduces a spatial
gradient of concern in the linear analysis. The PCA allows addressing spatial
features of the data and fine-tuning the theoretical and error models. It shows
that the models used are appropriate for the cosmological parameter estimation
performed. We address the potential for optimal data compression using PCA.Comment: 18 pages, LaTex, uses emulateapj.sty, ApJ in press (August 10, 2001),
improvements to text and figures, updated reference