Regularized Covariance Matrix Estimation in Complex Elliptically Symmetric Distributions Using the Expected Likelihood Approach - Part 1: The Over-Sampled Case

In \cite{Abramovich04}, it was demonstrated that the likelihood ratio (LR) for multivariate complex Gaussian distribution has the invariance property that can be exploited in many applications. Specifically, the probability density function (p.d.f.) of this LR for the (unknown) actual covariance matrix $\R_{0}$ does not depend on this matrix and is fully specified by the matrix dimension $M$ and the number of independent training samples $T$. Since this p.d.f. could therefore be pre-calculated for any a priori known $(M,T)$, one gets a possibility to compare the LR of any derived covariance matrix estimate against this p.d.f., and eventually get an estimate that is statistically ``as likely'' as the a priori unknown actual covariance matrix. This ``expected likelihood'' (EL) quality assessment allows for significant improvement of MUSIC DOA estimation performance in the so-called ``threshold area'' \cite{Abramovich04,Abramovich07d}, and for diagonal loading and TVAR model order selection in adaptive detectors \cite{Abramovich07,Abramovich07b}. Recently, a broad class of the so-called complex elliptically symmetric (CES) distributions has been introduced for description of highly in-homogeneous clutter returns. The aim of this series of two papers is to extend the EL approach to this class of CES distributions as well as to a particularly important derivative of CES, namely the complex angular central distribution (ACG). For both cases, we demonstrate a similar invariance property for the LR associated with the true scatter matrix \mSigma_{0}. Furthermore, we derive fixed point regularized covariance matrix estimates using the generalized expected likelihood methodology. This first part is devoted to the conventional scenario ($T \geq M$) while Part 2 deals with the under-sampled scenario ($T \leq M$)

Regularized Covariance Matrix Estimation in Complex Elliptically Symmetric Distributions Using the Expected Likelihood Approach - Part 2: The Under-Sampled Case

In the first part of this series of two papers, we extended the expected likelihood approach originally developed in the Gaussian case, to the broader class of complex elliptically symmetric (CES) distributions and complex angular central Gaussian (ACG) distributions. More precisely, we demonstrated that the probability density function (p.d.f.) of the likelihood ratio (LR) for the (unknown) actual scatter matrix \mSigma_{0} does not depend on the latter: it only depends on the density generator for the CES distribution and is distribution-free in the case of ACG distributed data, i.e., it only depends on the matrix dimension $M$ and the number of independent training samples $T$, assuming that $T \geq M$. Additionally, regularized scatter matrix estimates based on the EL methodology were derived. In this second part, we consider the under-sampled scenario ($T \leq M$) which deserves a specific treatment since conventional maximum likelihood estimates do not exist. Indeed, inference about the scatter matrix can only be made in the $T$-dimensional subspace spanned by the columns of the data matrix. We extend the results derived under the Gaussian assumption to the CES and ACG class of distributions. Invariance properties of the under-sampled likelihood ratio evaluated at \mSigma_{0} are presented. Remarkably enough, in the ACG case, the p.d.f. of this LR can be written in a rather simple form as a product of beta distributed random variables. The regularized schemes derived in the first part, based on the EL principle, are extended to the under-sampled scenario and assessed through numerical simulations

Non-Gaussianity in the Weak Lensing Correlation Function Likelihood -- Implications for Cosmological Parameter Biases

We study the significance of non-Gaussianity in the likelihood of weak lensing shear two-point correlation functions, detecting significantly non-zero skewness and kurtosis in one-dimensional marginal distributions of shear two-point correlation functions in simulated weak lensing data. We examine the implications in the context of future surveys, in particular LSST, with derivations of how the non-Gaussianity scales with survey area. We show that there is no significant bias in one-dimensional posteriors of $\Omega_{\rm m}$ and $\sigma_{\rm 8}$ due to the non-Gaussian likelihood distributions of shear correlations functions using the mock data ($100$ deg$^{2}$). We also present a systematic approach to constructing approximate multivariate likelihoods with one-dimensional parametric functions by assuming independence or more flexible non-parametric multivariate methods after decorrelating the data points using principal component analysis (PCA). While the use of PCA does not modify the non-Gaussianity of the multivariate likelihood, we find empirically that the one-dimensional marginal sampling distributions of the PCA components exhibit less skewness and kurtosis than the original shear correlation functions.Modeling the likelihood with marginal parametric functions based on the assumption of independence between PCA components thus gives a lower limit for the biases. We further demonstrate that the difference in cosmological parameter constraints between the multivariate Gaussian likelihood model and more complex non-Gaussian likelihood models would be even smaller for an LSST-like survey. In addition, the PCA approach automatically serves as a data compression method, enabling the retention of the majority of the cosmological information while reducing the dimensionality of the data vector by a factor of $\sim$5.Comment: 16 pages, 10 figures, published MNRA

The non-Gaussianity of the cosmic shear likelihood - or: How odd is the Chandra Deep Field South?

(abridged) We study the validity of the approximation of a Gaussian cosmic shear likelihood. We estimate the true likelihood for a fiducial cosmological model from a large set of ray-tracing simulations and investigate the impact of non-Gaussianity on cosmological parameter estimation. We investigate how odd the recently reported very low value of $\sigma_8$ really is as derived from the \textit{Chandra} Deep Field South (CDFS) using cosmic shear by taking the non-Gaussianity of the likelihood into account as well as the possibility of biases coming from the way the CDFS was selected. We find that the cosmic shear likelihood is significantly non-Gaussian. This leads to both a shift of the maximum of the posterior distribution and a significantly smaller credible region compared to the Gaussian case. We re-analyse the CDFS cosmic shear data using the non-Gaussian likelihood. Assuming that the CDFS is a random pointing, we find $\sigma_8=0.68_{-0.16}^{+0.09}$ for fixed $\Omega_{\rm m}=0.25$. In a WMAP5-like cosmology, a value equal to or lower than this would be expected in $\approx 5$ of the times. Taking biases into account arising from the way the CDFS was selected, which we model as being dependent on the number of haloes in the CDFS, we obtain $\sigma_8 = 0.71^{+0.10}_{-0.15}$. Combining the CDFS data with the parameter constraints from WMAP5 yields $\Omega_{\rm m} = 0.26^{+0.03}_{-0.02}$ and $\sigma_8 = 0.79^{+0.04}_{-0.03}$ for a flat universe.Comment: 18 pages, 16 figures, accepted for publication in A&A; New Bayesian treatment of field selection bia

KL Estimation of the Power Spectrum Parameters from the Angular Distribution of Galaxies in Early SDSS Data

We present measurements of parameters of the 3-dimensional power spectrum of galaxy clustering from 222 square degrees of early imaging data in the Sloan Digital Sky Survey. The projected galaxy distribution on the sky is expanded over a set of Karhunen-Loeve eigenfunctions, which optimize the signal-to-noise ratio in our analysis. A maximum likelihood analysis is used to estimate parameters that set the shape and amplitude of the 3-dimensional power spectrum. Our best estimates are Gamma=0.188 +/- 0.04 and sigma_8L = 0.915 +/- 0.06 (statistical errors only), for a flat Universe with a cosmological constant. We demonstrate that our measurements contain signal from scales at or beyond the peak of the 3D power spectrum. We discuss how the results scale with systematic uncertainties, like the radial selection function. We find that the central values satisfy the analytically estimated scaling relation. We have also explored the effects of evolutionary corrections, various truncations of the KL basis, seeing, sample size and limiting magnitude. We find that the impact of most of these uncertainties stay within the 2-sigma uncertainties of our fiducial result.Comment: Fig 1 postscript problem correcte

A re-analysis of the three-year WMAP temperature power spectrum and likelihood

We analyze the three-year WMAP temperature anisotropy data seeking to confirm the power spectrum and likelihoods published by the WMAP team. We apply five independent implementations of four algorithms to the power spectrum estimation and two implementations to the parameter estimation. Our single most important result is that we broadly confirm the WMAP power spectrum and analysis. Still, we do find two small but potentially important discrepancies: On large angular scales there is a small power excess in the WMAP spectrum (5-10% at l<~30) primarily due to likelihood approximation issues between 13 <= l <~30. On small angular scales there is a systematic difference between the V- and W-band spectra (few percent at l>~300). Recently, the latter discrepancy was explained by Huffenberger et al. (2006) in terms of over-subtraction of unresolved point sources. As far as the low-l bias is concerned, most parameters are affected by a few tenths of a sigma. The most important effect is seen in n_s. For the combination of WMAP, Acbar and BOOMERanG, the significance of n_s =/ 1 drops from ~2.7 sigma to ~2.3 sigma when correcting for this bias. We propose a few simple improvements to the low-l WMAP likelihood code, and introduce two important extensions to the Gibbs sampling method that allows for proper sampling of the low signal-to-noise regime. Finally, we make the products from the Gibbs sampling analysis publically available, thereby providing a fast and simple route to the exact likelihood without the need of expensive matrix inversions.Comment: 14 pages, 7 figures. Accepted for publication in ApJ. Numerical results unchanged, but interpretation sharpened: Likelihood approximation issues at l=13-30 far more important than potential foreground issues at l <= 12. Gibbs products (spectrum and sky samples, and "easy-to-use" likelihood module) available from http://www.astro.uio.no/~hke/ under "Research

Estimating the large-scale angular power spectrum in the presence of systematics: a case study of Sloan Digital Sky Survey quasars

The angular power spectrum is a powerful statistic for analysing cosmological signals imprinted in the clustering of matter. However, current galaxy and quasar surveys cover limited portions of the sky, and are contaminated by systematics that can mimic cosmological signatures and jeopardise the interpretation of the measured power spectra. We provide a framework for obtaining unbiased estimates of the angular power spectra of large-scale structure surveys at the largest scales using quadratic estimators. The method is tested by analysing the 600 CMASS mock catalogues constructed by Manera et al. (2013) for the Baryon Oscillation Spectroscopic Survey (BOSS). We then consider the Richards et al. (2009) catalogue of photometric quasars from the Sixth Data Release (DR6) of the Sloan Digital Sky Survey (SDSS), which is known to include significant stellar contamination and systematic uncertainties. Focusing on the sample of ultraviolet-excess (UVX) sources, we show that the excess clustering power present on the largest-scales can be largely mitigated by making use of improved sky masks and projecting out the modes corresponding to the principal systematics. In particular, we find that the sample of objects with photometric redshift $1.3 < z_p < 2.2$ exhibits no evidence of contamination when using our most conservative mask and mode projection. This indicates that any residual systematics are well within the statistical uncertainties. We conclude that, using our approach, this sample can be used for cosmological studies.Comment: 18 pages, 18 figures. Version accepted by MNRA

Hierarchical Cosmic Shear Power Spectrum Inference

We develop a Bayesian hierarchical modelling approach for cosmic shear power spectrum inference, jointly sampling from the posterior distribution of the cosmic shear field and its (tomographic) power spectra. Inference of the shear power spectrum is a powerful intermediate product for a cosmic shear analysis, since it requires very few model assumptions and can be used to perform inference on a wide range of cosmological models \emph{a posteriori} without loss of information. We show that joint posterior for the shear map and power spectrum can be sampled effectively by Gibbs sampling, iteratively drawing samples from the map and power spectrum, each conditional on the other. This approach neatly circumvents difficulties associated with complicated survey geometry and masks that plague frequentist power spectrum estimators, since the power spectrum inference provides prior information about the field in masked regions at every sampling step. We demonstrate this approach for inference of tomographic shear $E$-mode, $B$-mode and $EB$-cross power spectra from a simulated galaxy shear catalogue with a number of important features; galaxies distributed on the sky and in redshift with photometric redshift uncertainties, realistic random ellipticity noise for every galaxy and a complicated survey mask. The obtained posterior distributions for the tomographic power spectrum coefficients recover the underlying simulated power spectra for both $E$- and $B$-modes.Comment: 16 pages, 8 figures, accepted by MNRA

A Measurement of the Angular Power Spectrum of the CMB Temperature Anisotropy from the 2003 Flight of Boomerang

We report on observations of the Cosmic Microwave Background (CMB) obtained during the January 2003 flight of Boomerang . These results are derived from 195 hours of observation with four 145 GHz Polarization Sensitive Bolometer (PSB) pairs, identical in design to the four 143 GHz Planck HFI polarized pixels. The data include 75 hours of observations distributed over 1.84% of the sky with an additional 120 hours concentrated on the central portion of the field, itself representing 0.22% of the full sky. From these data we derive an estimate of the angular power spectrum of temperature fluctuations of the CMB in 24 bands over the multipole range (50 < l < 1500). A series of features, consistent with those expected from acoustic oscillations in the primordial photon-baryon fluid, are clearly evident in the power spectrum, as is the exponential damping of power on scales smaller than the photon mean free path at the epoch of last scattering (l > 900). As a consistency check, the collaboration has performed two fully independent analyses of the time ordered data, which are found to be in excellent agreement.Comment: 11 pages, 7 figures, 3 tables. High resolution figures and data are available at http://cmb.phys.cwru.edu/boomerang/ and http://oberon.roma1.infn.it/boomerang/b2