Distributed Computing Grid Experiences in CMS

The CMS experiment is currently developing a computing system capable of serving, processing and archiving the large number of events that will be generated when the CMS detector starts taking data. During 2004 CMS undertook a large scale data challenge to demonstrate the ability of the CMS computing system to cope with a sustained data-taking rate equivalent to 25% of startup rate. Its goals were: to run CMS event reconstruction at CERN for a sustained period at 25 Hz input rate; to distribute the data to several regional centers; and enable data access at those centers for analysis. Grid middleware was utilized to help complete all aspects of the challenge. To continue to provide scalable access from anywhere in the world to the data, CMS is developing a layer of software that uses Grid tools to gain access to data and resources, and that aims to provide physicists with a user friendly interface for submitting their analysis jobs. This paper describes the data challenge experience with Grid infrastructure and the current development of the CMS analysis system

Multi-core job submission and grid resource scheduling for ATLAS AthenaMP

AthenaMP is the multi-core implementation of the ATLAS software framework and allows the efficient sharing of memory pages between multiple threads of execution. This has now been validated for production and delivers a significant reduction on the overall application memory footprint with negligible CPU overhead. Before AthenaMP can be routinely run on the LHC Computing Grid it must be determined how the computing resources available to ATLAS can best exploit the notable improvements delivered by switching to this multi-process model. A study into the effectiveness and scalability of AthenaMP in a production environment will be presented. Best practices for configuring the main LRMS implementations currently used by grid sites will be identified in the context of multi-core scheduling optimisation

A Big Data Analyzer for Large Trace Logs

Current generation of Internet-based services are typically hosted on large data centers that take the form of warehouse-size structures housing tens of thousands of servers. Continued availability of a modern data center is the result of a complex orchestration among many internal and external actors including computing hardware, multiple layers of intricate software, networking and storage devices, electrical power and cooling plants. During the course of their operation, many of these components produce large amounts of data in the form of event and error logs that are essential not only for identifying and resolving problems but also for improving data center efficiency and management. Most of these activities would benefit significantly from data analytics techniques to exploit hidden statistical patterns and correlations that may be present in the data. The sheer volume of data to be analyzed makes uncovering these correlations and patterns a challenging task. This paper presents BiDAl, a prototype Java tool for log-data analysis that incorporates several Big Data technologies in order to simplify the task of extracting information from data traces produced by large clusters and server farms. BiDAl provides the user with several analysis languages (SQL, R and Hadoop MapReduce) and storage backends (HDFS and SQLite) that can be freely mixed and matched so that a custom tool for a specific task can be easily constructed. BiDAl has a modular architecture so that it can be extended with other backends and analysis languages in the future. In this paper we present the design of BiDAl and describe our experience using it to analyze publicly-available traces from Google data clusters, with the goal of building a realistic model of a complex data center.Comment: 26 pages, 10 figure

Data processing activities at the MAGIC site

open11siMAGIC is a system of two imaging atmospheric Cherenkov telescopes located on the Canary Is- land of La Palma. The fast processing of the data at the observation site plays an essential part in the operation of the telescopes and has continuously improved since the beginning of the exper- iment. The on-site computing can be divided into three major contributions: the MAGIC online analysis (MOLA), providing preliminary real time analysis results; the on-site analysis (OSA), providing final data products at the end of each observation night; and the Data Check (DC), a daily check on the performance of the telescope’s subsystems and the quality control of the data observed during the previous night. We present the status of the system, including the latest upgrades and details on its performance.openFidalgo, David; Nievas-Rosillo, Miguel; Babic, Ana; Contreras, Jose-Luis; Doro, Michele; Godinovic, Nikola; Hrupec, Dario; Lorca, Alejandro; Moralejo, Abelardo; Satalecka, Konstancja; Will, MartinFidalgo, David; Nievas Rosillo, Miguel; Babic, Ana; Contreras, Jose Luis; Doro, Michele; Godinovic, Nikola; Hrupec, Dario; Lorca, Alejandro; Moralejo, Abelardo; Satalecka, Konstancja; Will, Marti

Logging and bookkeeping, Administrator's guide

Logging and Bookkeeping (LB for short) is a Grid service that keeps a short-term trace of Grid jobs as they are processed by individual Grid component