XkitS：A computational storage framework for high energy physics based on EOS storage system

Large-scale high-energy physics experiments generate scientific data at the scale of petabytes or even exabytes, requiring high-performance data IO for processing. However, in large computing centers, computing and storage devices are typically separated. Large-scale data transfer has become a bottleneck for some data-intensive computing tasks, such as data encoding and decoding, compression, sorting, etc. The time spent on data transfer can account for 50% of the entire computing task. The larger the amount of data accessed, the more significant this cost becomes. One attractive solution to address this problem is to offload a portion of data processing to the storage layer. However, modifying traditional storage systems to support computation offloading is often cumbersome and requires a broad understanding of their internal principles. Therefore, we have designed a flexible software framework called XkitS, which builds a computable storage system by extending the existing storage system EOS. This framework is deployed on the EOS FTS storage server and offloads computational tasks by invoking the computing capabilities (CPU, FPGA, etc.) on FTS. Currently, it has been tested and applied in the data processing of the Large High Altitude Air Shower Observatory (LHAASO), and the results show that the time spent on data decoding using the computable storage technology is half of that using the original method

ProAgent: Building Proactive Cooperative Agents with Large Language Models

Building agents with adaptive behavior in cooperative tasks stands as a paramount goal in the realm of multi-agent systems. Current approaches to developing cooperative agents rely primarily on learning-based methods, whose policy generalization depends heavily on the diversity of teammates they interact with during the training phase. Such reliance, however, constrains the agents' capacity for strategic adaptation when cooperating with unfamiliar teammates, which becomes a significant challenge in zero-shot coordination scenarios. To address this challenge, we propose ProAgent, a novel framework that harnesses large language models (LLMs) to create proactive agents capable of dynamically adapting their behavior to enhance cooperation with teammates. ProAgent can analyze the present state, and infer the intentions of teammates from observations. It then updates its beliefs in alignment with the teammates' subsequent actual behaviors. Moreover, ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various of coordination scenarios. Experimental evaluations conducted within the Overcooked-AI environment unveil the remarkable performance superiority of ProAgent, outperforming five methods based on self-play and population-based training when cooperating with AI agents. Furthermore, in partnered with human proxy models, its performance exhibits an average improvement exceeding 10% compared to the current state-of-the-art method. For more information about our project, please visit~\url{https://pku-proagent.github.io}.Comment: v3 is the AAAI'24 camera ready version, which polished abstract and introduction based on the reviewers' comments, and enriched related works. 7 pages of main content, 2 pages of references, 2 figures and 1 tabl

A High-Speed Asynchronous Data I/O Method for HEPS

The High Energy Photon Source (HEPS) is expected to produce a substantial volume of data, lead to immense data I/O pressure during computing. Inefficient data I/O can significantly impact computing performance. To address this challenge, firstly, we have developed a data I/O framework for HEPS. This framework consists of three layers: data channel layer, distributed memory management layer, and I/O interface layer. It mask the underlying data differences in formats and sources, while implementing efficient I/O methods. Additionally, it supports both stream computing and batch computing. Secondly, we have designed a data processing pipeline scheme aimed at reducing I/O latency and optimizing I/O bandwidth utilization during the processing of high-throughput data. This involves breaking down the computing task into several stages, including data loading, data pre-processing, data processing, and data writing, which are executed asynchronously and in parallel. Finally, we introduce the design of stream data I/O process. The primary objective of stream data I/O is to enable real-time online processing of raw data, avoiding I/O bottlenecks caused by disk storage. This approach ensures the stability of data transmission and integrates distributed memory management to guarantee data integrity in memory

Sirtuin 1 and Autophagy Attenuate Cisplatin-Induced Hair Cell Death in the Mouse Cochlea and Zebrafish Lateral Line

Cisplatin-induced ototoxicity is one of the major adverse effects in cisplatin chemotherapy, and hearing protective approaches are unavailable in clinical practice. Recent work unveiled a critical role of autophagy in cell survival in various types of hearing loss. Since the excessive activation of autophagy can contribute to apoptotic cell death, whether the activation of autophagy increases or decreases the rate of cell death in CDDP ototoxicity is still being debated. In this study, we showed that CDDP induced activation of autophagy in the auditory cell HEI-OC1 at the early stage. We then used rapamycin, an autophagy activator, to increase the autophagy activity, and found that the cell death significantly decreased after CDDP injury. In contrast, treatment with the autophagy inhibitor 3-methyladenine (3-MA) significantly increased cell death. In accordance with in vitro results, rapamycin alleviated CDDP-induced death of hair cells in zebrafish lateral line and cochlear hair cells in mice. Notably, we found that CDDP-induced increase of Sirtuin 1 (SIRT1) in the HEI-OC1 cells modulated the autophagy function. The specific SIRT1 activator SRT1720 could successfully protect against CDDP-induced cell loss in HEI-OC1 cells, zebrafish lateral line, and mice cochlea. These findings suggest that SIRT1 and autophagy activation can be suggested as potential therapeutic strategies for the treatment of CDDP-induced ototoxicity

EOS 2023 Workshop

Computational storage involves integrating compute resources with storage devices or systems to enable data processing within the storage device. This approach reduces data movement, enhances processing efficiency, and reduces costs. To facilitate in-situ data processing on storage servers, we developed a computational storage plugin that can be added to EOS FST. This plugin enables users to deploy compute resources directly within the storage servers, allowing them to perform data processing operations on the data stored in the FST nodes without having to move the data to a separate computing system. This can reduce latency and improve overall performance, especially when processing large volumes of data. The plugin can be extended to support a variety of data processing tasks, including data filtering, compression, encryption, and machine learning. The computational storage function is defined in a configuration that can be implemented in scripting languages or evolved independently of the storage system in the form of containers. When an FST node receives a request to open a file, the plugin is executed first. It then calls the target program on the storage server by parsing the parameters of the command to open the file. At this time, the input file must be on the FTS storage server, and the plugin also writes the output file to the node. At the end of the task execution, the output file is automatically registered into the MGM server. Client access is fully compatible with XRootD's API and EOS commands. Users can add tasks and parameters to be performed in the open option. The plugin has been tested and applied in the data processing of the Large High Altitude Air Shower Observatory (LHAASO), and the results show that the efficiency of data decoding is more than 5 times higher than the original method

Heat Prediction of High Energy Physical Data Based on LSTM Recurrent Neural Network

High-energy physics computing is a typical data-intensive calculation. Each year, petabytes of data needs to be analyzed, and data access performance is increasingly demanding. The tiered storage system scheme for building a unified namespace has been widely adopted. Generally, data is stored on storage devices with different performances and different prices according to different access frequency. When the heat of the data changes, the data is then migrated to the appropriate storage tier. At present, heuristic algorithms based on artificial experience are widely used in data heat prediction. Due to the differences in computing models of different users, the accuracy of prediction is low. A method for predicting future access popularity based on file access characteristics with the help of LSTM deep learning algorithm is proposed as the basis for data migration in hierarchical storage. This paper uses the real data of high-energy physics experiment LHAASO as an example for comparative testing. The results show that under the same test conditions, the model has higher prediction accuracy and stronger applicability than existing prediction models

Heat Prediction of High Energy Physical Data Based on LSTM Recurrent Neural Network

High-energy physics computing is a typical data-intensive calculation. Each year, petabytes of data needs to be analyzed, and data access performance is increasingly demanding. The tiered storage system scheme for building a unified namespace has been widely adopted. Generally, data is stored on storage devices with different performances and different prices according to different access frequency. When the heat of the data changes, the data is then migrated to the appropriate storage tier. At present, heuristic algorithms based on artificial experience are widely used in data heat prediction. Due to the differences in computing models of different users, the accuracy of prediction is low. A method for predicting future access popularity based on file access characteristics with the help of LSTM deep learning algorithm is proposed as the basis for data migration in hierarchical storage. This paper uses the real data of high-energy physics experiment LHAASO as an example for comparative testing. The results show that under the same test conditions, the model has higher prediction accuracy and stronger applicability than existing prediction models

EventDB: an event-based indexer and caching system for BESIII experiment

Beijing Spectrometer (BESIII) experiment has produced hundreds of billions of events. The traditional event-wise accessing of BESIII Offline Software System is not effective for the selective accessing with low rate during a physics analysis. In this paper, an event-based data management system (EventDB) is introduced, which can effectively alleviate the problems of low efficiency of data processing and low utilization of resources. Firstly, an indexing system based on NoSQL database is designed. By extracting specified attributes of events, the events interested to the physicists are selected and stored into the database, whilst the real data of event is still stored in ROOT files. For those hot events, the real event data can also be cached into EventDB to improve the access performance. The data analysis workflow of HEP experiments is needed to change if the EventDB system is applied. The analysis program queries the corresponding event index from database, then get event data from database if the event is cached, or get data from ROOT files if it is not cached. Finally, the test on more than one hundred billion physics events shows the query speed was greatly improved over traditional file-based data management systems

EventDB: an event-based indexer and caching system for BESIII experiment

Beijing Spectrometer (BESIII) experiment has produced hundreds of billions of events. The traditional event-wise accessing of BESIII Offline Software System is not effective for the selective accessing with low rate during a physics analysis. In this paper, an event-based data management system (EventDB) is introduced, which can effectively alleviate the problems of low efficiency of data processing and low utilization of resources. Firstly, an indexing system based on NoSQL database is designed. By extracting specified attributes of events, the events interested to the physicists are selected and stored into the database, whilst the real data of event is still stored in ROOT files. For those hot events, the real event data can also be cached into EventDB to improve the access performance. The data analysis workflow of HEP experiments is needed to change if the EventDB system is applied. The analysis program queries the corresponding event index from database, then get event data from database if the event is cached, or get data from ROOT files if it is not cached. Finally, the test on more than one hundred billion physics events shows the query speed was greatly improved over traditional file-based data management systems