Monitoring Networked Applications With Incremental Quantile Estimation

Networked applications have software components that reside on different computers. Email, for example, has database, processing, and user interface components that can be distributed across a network and shared by users in different locations or work groups. End-to-end performance and reliability metrics describe the software quality experienced by these groups of users, taking into account all the software components in the pipeline. Each user produces only some of the data needed to understand the quality of the application for the group, so group performance metrics are obtained by combining summary statistics that each end computer periodically (and automatically) sends to a central server. The group quality metrics usually focus on medians and tail quantiles rather than on averages. Distributed quantile estimation is challenging, though, especially when passing large amounts of data around the network solely to compute quality metrics is undesirable. This paper describes an Incremental Quantile (IQ) estimation method that is designed for performance monitoring at arbitrary levels of network aggregation and time resolution when only a limited amount of data can be transferred. Applications to both real and simulated data are provided.Comment: This paper commented in: [arXiv:0708.0317], [arXiv:0708.0336], [arXiv:0708.0338]. Rejoinder in [arXiv:0708.0339]. Published at http://dx.doi.org/10.1214/088342306000000583 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Optimal Moments for the Analysis of Peculiar Velocity Surveys II: Testing

Analyses of peculiar velocity surveys face several challenges, including low signal--to--noise in individual velocity measurements and the presence of small--scale, nonlinear flows. This is the second in a series of papers in which we describe a new method of overcoming these problems by using data compression as a filter with which to separate large--scale, linear flows from small--scale noise that can bias results. We demonstrate the effectiveness of our method using realistic catalogs of galaxy velocities drawn from N--body simulations. Our tests show that a likelihood analysis of simulated catalogs that uses all of the information contained in the peculiar velocities results in a bias in the estimation of the power spectrum shape parameter $\Gamma$ and amplitude $\beta$, and that our method of analysis effectively removes this bias. We expect that this new method will cause peculiar velocity surveys to re--emerge as a useful tool to determine cosmological parameters.Comment: 28 pages, 9 figure

Rejoinder: Monitoring Networked Applications With Incremental Quantile Estimation

Rejoinder: Monitoring Networked Applications With Incremental Quantile Estimation [arXiv:0708.0302]Comment: Published at http://dx.doi.org/10.1214/088342306000000592 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Optimal Moments for the Analysis of Peculiar Velocity Surveys

We present a new method for the analysis of peculiar velocity surveys which removes contributions to velocities from small scale, nonlinear velocity modes while retaining information about large scale motions. Our method utilizes Karhunen--Lo\`eve methods of data compression to construct a set of moments out of the velocities which are minimally sensitive to small scale power. The set of moments are then used in a likelihood analysis. We develop criteria for the selection of moments, as well as a statistic to quantify the overall sensitivity of a set of moments to small scale power. Although we discuss our method in the context of peculiar velocity surveys, it may also prove useful in other situations where data filtering is required.Comment: 25 Pages, 3 figures. Submitted to Ap