Adapting SAM for CDF

The CDF and D0 experiments probe the high-energy frontier and as they do so have accumulated hundreds of Terabytes of data on the way to petabytes of data over the next two years. The experiments have made a commitment to use the developing Grid based on the SAM system to handle these data. The D0 SAM has been extended for use in CDF as common patterns of design emerged to meet the similar requirements of these experiments. The process by which the merger was achieved is explained with particular emphasis on lessons learned concerning the database design patterns plus realization of the use cases.Comment: Talk from the 2003 Computing in High Energy and Nuclear Physics (CHEP03), La Jolla, Ca, USA, March 2003, 4 pages, pdf format, TUAT00

Spectra: Robust Estimation of Distribution Functions in Networks

Distributed aggregation allows the derivation of a given global aggregate property from many individual local values in nodes of an interconnected network system. Simple aggregates such as minima/maxima, counts, sums and averages have been thoroughly studied in the past and are important tools for distributed algorithms and network coordination. Nonetheless, this kind of aggregates may not be comprehensive enough to characterize biased data distributions or when in presence of outliers, making the case for richer estimates of the values on the network. This work presents Spectra, a distributed algorithm for the estimation of distribution functions over large scale networks. The estimate is available at all nodes and the technique depicts important properties, namely: robust when exposed to high levels of message loss, fast convergence speed and fine precision in the estimate. It can also dynamically cope with changes of the sampled local property, not requiring algorithm restarts, and is highly resilient to node churn. The proposed approach is experimentally evaluated and contrasted to a competing state of the art distribution aggregation technique.Comment: Full version of the paper published at 12th IFIP International Conference on Distributed Applications and Interoperable Systems (DAIS), Stockholm (Sweden), June 201

The Starburst Nature of Lyman-Break Galaxies: Testing UV Extinction with X-rays

We derive the bolometric to X-ray correlation for a local sample of normal and starburst galaxies and use it, in combination with several UV reddening schemes, to predict the 2--8 keV X-ray luminosity for a sample of 24 Lyman-break galaxies in the HDF/CDF-N. We find that the mean X-ray luminosity, as predicted from the Meurer UV reddening relation for starburst galaxies, agrees extremely well with the Brandt stacking analysis. This provides additional evidence that Lyman-break galaxies can be considered as scaled-up local starbursts and that the locally derived starburst UV reddening relation may be a reasonable tool for estimating the UV extinction at high redshift. Our analysis shows that the Lyman-break sample can not have far-IR to far-UV flux ratios similar to nearby ULIGs, as this would predict a mean X-ray luminosity 100 times larger than observed, as well as far-IR luminosities large enough to be detected in the sub-mm. We calculate the UV reddening expected from the Calzetti effective starburst attenuation curve and the radiative transfer models of Witt & Gordon for low metallicity dust in a shell geometry with homogeneous or clumpy dust distributions and find that all are consistent with the observed X-ray emission. Finally, we show that the mean X-ray luminosity of the sample would be under predicted by a factor of 6 if the the far-UV is unattenuated by dust.Comment: 7 pages, 3 figures. Accepted for publication in A

Event-triggered Learning

The efficient exchange of information is an essential aspect of intelligent collective behavior. Event-triggered control and estimation achieve some efficiency by replacing continuous data exchange between agents with intermittent, or event-triggered communication. Typically, model-based predictions are used at times of no data transmission, and updates are sent only when the prediction error grows too large. The effectiveness in reducing communication thus strongly depends on the quality of the prediction model. In this article, we propose event-triggered learning as a novel concept to reduce communication even further and to also adapt to changing dynamics. By monitoring the actual communication rate and comparing it to the one that is induced by the model, we detect a mismatch between model and reality and trigger model learning when needed. Specifically, for linear Gaussian dynamics, we derive different classes of learning triggers solely based on a statistical analysis of inter-communication times and formally prove their effectiveness with the aid of concentration inequalities

Cakewalk Sampling

We study the task of finding good local optima in combinatorial optimization problems. Although combinatorial optimization is NP-hard in general, locally optimal solutions are frequently used in practice. Local search methods however typically converge to a limited set of optima that depend on their initialization. Sampling methods on the other hand can access any valid solution, and thus can be used either directly or alongside methods of the former type as a way for finding good local optima. Since the effectiveness of this strategy depends on the sampling distribution, we derive a robust learning algorithm that adapts sampling distributions towards good local optima of arbitrary objective functions. As a first use case, we empirically study the efficiency in which sampling methods can recover locally maximal cliques in undirected graphs. Not only do we show how our adaptive sampler outperforms related methods, we also show how it can even approach the performance of established clique algorithms. As a second use case, we consider how greedy algorithms can be combined with our adaptive sampler, and we demonstrate how this leads to superior performance in k-medoid clustering. Together, these findings suggest that our adaptive sampler can provide an effective strategy to combinatorial optimization problems that arise in practice.Comment: Accepted as a conference paper by AAAI-2020 (oral presentation