Collaborative Nested Sampling: Big Data vs. complex physical models

The data torrent unleashed by current and upcoming astronomical surveys demands scalable analysis methods. Many machine learning approaches scale well, but separating the instrument measurement from the physical effects of interest, dealing with variable errors, and deriving parameter uncertainties is often an after-thought. Classic forward-folding analyses with Markov Chain Monte Carlo or Nested Sampling enable parameter estimation and model comparison, even for complex and slow-to-evaluate physical models. However, these approaches require independent runs for each data set, implying an unfeasible number of model evaluations in the Big Data regime. Here I present a new algorithm, collaborative nested sampling, for deriving parameter probability distributions for each observation. Importantly, the number of physical model evaluations scales sub-linearly with the number of data sets, and no assumptions about homogeneous errors, Gaussianity, the form of the model or heterogeneity/completeness of the observations need to be made. Collaborative nested sampling has immediate application in speeding up analyses of large surveys, integral-field-unit observations, and Monte Carlo simulations.Comment: Resubmitted to PASP Focus on Machine Intelligence in Astronomy and Astrophysics after first referee report. Figure 6 demonstrates the scaling for Collaborative MultiNest, PolyChord and RadFriends implementations. Figure 10 application to MUSE IFU data. Implementation at https://github.com/JohannesBuchner/massivedatan

A statistical test for Nested Sampling algorithms

Nested sampling is an iterative integration procedure that shrinks the prior volume towards higher likelihoods by removing a "live" point at a time. A replacement point is drawn uniformly from the prior above an ever-increasing likelihood threshold. Thus, the problem of drawing from a space above a certain likelihood value arises naturally in nested sampling, making algorithms that solve this problem a key ingredient to the nested sampling framework. If the drawn points are distributed uniformly, the removal of a point shrinks the volume in a well-understood way, and the integration of nested sampling is unbiased. In this work, I develop a statistical test to check whether this is the case. This "Shrinkage Test" is useful to verify nested sampling algorithms in a controlled environment. I apply the shrinkage test to a test-problem, and show that some existing algorithms fail to pass it due to over-optimisation. I then demonstrate that a simple algorithm can be constructed which is robust against this type of problem. This RADFRIENDS algorithm is, however, inefficient in comparison to MULTINEST.Comment: 11 pages, 7 figures. Published in Statistics and Computing, Springer, September 201

Relativistic reflection from accretion disks in the population of Active Galactic Nuclei at z=0.5-4

We report the detection of relativistically broadened iron K alpha emission in the X-ray spectra of AGN detected in the 4Ms CDF-S. Using the Bayesian X-ray analysis (BXA) package, we fit 199 hard band (2-7 keV) selected sources in the redshift range z=0.5--4 with three models: (i) an absorbed power-law, (ii) the first model plus a narrow reflection component, and (iii) the second model with an additional relativistic broadened reflection. The Bayesian evidence for the full sample of sources selects the model with the additional broad component as being 10^5 times more probable to describe the data better than the second model. For the two brightest sources in our sample, CID 190 (z=0.734) and CID 104 (z=0.543), BXA reveals the relativistic signatures in the individual spectra. We estimate the fraction of sources containing a broad component to be 54^{+35}_{-37}% (107/199 sources). Considering that the low signal-to-noise ratio of some spectra prevents the detection of the broad iron K alpha line, we infer an intrinsic fraction with broad emission of around two thirds. The detection of relativistic signatures in the X-ray spectra of these sources suggests that they are powered by a radiatively efficient accretion disk. Preliminary evidence is found that the spin of the black hole is high, with a maximally spinning Kerr BH model (a=1) providing a significantly better fit than a Schwarzschild model (a=0). Our analysis demonstrate the potential of X-ray spectroscopy to measure this key parameter in typical SMBH systems at the peak of BH growth.Comment: 10 pages, 5 figures, accepted for publication in MNRA

Nested Sampling Methods

Nested sampling (NS) computes parameter posterior distributions and makes Bayesian model comparison computationally feasible. Its strengths are the unsupervised navigation of complex, potentially multi-modal posteriors until a well-defined termination point. A systematic literature review of nested sampling algorithms and variants is presented. We focus on complete algorithms, including solutions to likelihood-restricted prior sampling, parallelisation, termination and diagnostics. The relation between number of live points, dimensionality and computational cost is studied for two complete algorithms. A new formulation of NS is presented, which casts the parameter space exploration as a search on a tree. Previously published ways of obtaining robust error estimates and dynamic variations of the number of live points are presented as special cases of this formulation. A new on-line diagnostic test is presented based on previous insertion rank order work. The survey of nested sampling methods concludes with outlooks for future research.Comment: Updated version incorporating constructive input from four(!) positive reports (two referees, assistant editor and editor). The open-source UltraNest package and astrostatistics tutorials can be found at https://johannesbuchner.github.io/UltraNest

UltraNest -- a robust, general purpose Bayesian inference engine

UltraNest is a general-purpose Bayesian inference package for parameter estimation and model comparison. It allows fitting arbitrary models specified as likelihood functions written in Python, C, C++, Fortran, Julia or R. With a focus on correctness and speed (in that order), UltraNest is especially useful for multi-modal or non-Gaussian parameter spaces, computational expensive models, in robust pipelines. Parallelisation to computing clusters and resuming incomplete runs is available.Comment: Longer version of the paper published in JOSS. UltraNest can be found at https://johannesbuchner.github.io/UltraNest

AGN sub-populations important for black hole mass growth: a rule of thumb

For building the super-massive black hole population within a Hubble time, only sub-populations with more than 10^52 erg/s / L objects on the sky are relevant, where L is the sample-averaged luminosity.Comment: Comments are welcome. Intended for publication as a research note in RNAA

Closing In on the Hsp90 Chaperone-Client Relationship

The molecular chaperone Hsp90 regulates the activity and stability of a set of client proteins. Despite progress in understanding its mechanism, the interaction of Hsp90 with clients has remained enigmatic. Now, in a recent issue of Molecular Cell, Street and coworkers present results that integrate the client in the Hsp90 chaperone cycle

XZ: Deriving redshifts from X-ray spectra of obscured AGN

Context: Redshifts are fundamental for our understanding of extragalactic X-ray sources. Ambiguous counterpart associations, expensive optical spectroscopy and/or multimission multiwavelength coverage to resolve degeneracies make estimation often difficult in practice. Aims: We attempt to constrain redshifts of obscured Active Galactic Nuclei (AGN) using only low-resolution X-ray spectra. Methods: Our XZ method fits AGN X-ray spectra with a moderately complex spectral model incorporating a corona, torus obscurer and warm mirror. Using the Bayesian X-ray Astronomy (BXA) package, we constrain redshift, column density, photon index and luminosity simultaneously. The redshift information primarily comes from absorption edges in Compton-thin AGN, and from the Fe K$\alpha$ fluorescent line in heavily obscured AGN. A new generic background fitting method allows us to extract more information from limited numbers of source counts. Results: We derive redshift constraints for 74/321 hard-band detected sources in the Chandra deep field South. Comparing with spectroscopic redshifts, we find an outlier fraction of 8%, indicating that our model assumptions are valid. For three Chandra deep fields, we release our XZ redshift estimates. Conclusions: The independent XZ estimate is easy to apply and effective for a large fraction of obscured AGN in todays deep surveys without the need for any additional data. Comparing to different redshift estimation methods, XZ can resolve degeneracies in photometric redshifts, help to detect potential association problems and confirm uncertain single-line spectroscopic redshifts. With high spectral resolution and large collecting area, this technique will be highly effective for Athena/WFI observations.Comment: 20 pages, 16 figures in paper, 14 in appendice

Relativistic accretion disk reflection in AGN X-ray spectra at z=0.5--4: a study of four \textit{Chandra} deep fields

We confirm that the spectra are best fit by a model containing two Compton reflection components, one from distant material, and the other displaying relativistic broadening, most likely from the inner accretion disk. The degree of relativistic broadening indicates a preference for high black hole spin, but the reflection is weaker than that expected for a flat disk illuminated by a point source. We investigate the Compton reflection signatures as a function of luminosity, redshift and obscuration, confirming an X-ray Baldwin effect for both the narrow and broad components of the iron line. Anti-correlations are also seen with redshift and obscuring column density, but are difficult to disentangle from the Baldwin effect. Our methodology is able to extract information from multiple spectra with low signal-to-noise ratio, and can be applied to future data sets such as eROSITA. We show using simulations, however, that it is necessary to apply an appropriate signal-to-noise ratio cut to the samples to ensure the spectra add useful information.Comment: 16 pages, 10 figures, submitted to MNRA