Design for real-time data acquisition based on streaming technology

For the LHD project a long-pulse plasma experiment of 1-h duration is planned. In this quasi steady-state operation, the data acquisition system will be required to continuously transfer the diagnostic data from the digitizer front-end and display them in real-time. The CompactPCI standard is used to replace the conventional CAMAC digitizers in LHD, because it provides good functionality for real-time data streaming and also a connectivity with modern PC technology. The digitizer scheme, interface to the host computer, adoption of data compression, and downstream applications are discussed in detail to design and implement this new real-time data streaming system for LHD plasma diagnostics

Multi-Layer Distributed Storage of LHD Plasma Diagnostic Database

At the end of LHD experimental campaign in 2003, the amount of whole plasma diagnostics raw data had reached 3.16 GB in a long-pulse experiment. This is a new world record in fusion plasma experiments, far beyond the previous value of 1.5 GB/shot. The total size of the LHD diagnostic data is about 21.6 TB for the whole six years of experiments, and it continues to grow at an increasing rate. The LHD diagnostic database and storage system, i.e. the LABCOM system, has a completely distributed architecture to be sufficiently flexible and easily expandable to maintain integrity of the total amount of data. It has three categories of the storage layer: OODBMS volumes in data acquisition servers, RAID servers, and mass storage systems, such as MO jukeboxes and DVD-R changers. These are equally accessible through the network. By data migration between them, they can be considered a virtual OODB extension area. Their data contents have been listed in a “facilitator” PostgreSQL RDBMS, which now contains about 6.2 million entries, and informs the optimized priority to clients requesting data. Using the “glib” compression for all of the binary data and applying the three-tier application model for the OODB data transfer/retrieval, an optimized OODB read-out rate of 1.7 MB/s and effective client access speed of 3?25 MB/s have been achieved. As a result, the LABCOM data system has succeeded in combination of the use of RDBMS, OODBMS, RAID, and MSS to enable a virtual and always expandable storage volume, simultaneously with rapid data access

Steady-state data acquisition method for LHD diagnostics

The LHD experiment has gone through 5 campaign periods over the past 4 years, during which the diagnostics data continues to grow and the primary 28 measurements produce about 620 MB/shot in 150 shot/day 3-min cycles. In 2002, 30-min long-pulse experiments will be carried out in LHD, where real-time operations are indispensable for plasma measurements and data acquisition. The new scheme for utilizing conventional CAMAC digitizers in long-pulse experiments has been discussed and examined. As a result, in LHD, CAMACs will shift into 120?180 s cyclic operation, synchronized by the diagnostic timing system. The new CompactPCI-based digitizer frontend has performed about 84 MB/s continuous acquisition in benchmarks, and has been formulated with the conventional CAMAC system to make concurrent acquisitions

Nonstop Lose-Less Data Acquisition and Storing Method for Plasma Motion Images

Plasma diagnostic data analysis often requires the original raw data as they are, in other words, at the same frame rate and resolution of the CCD camera sensor. As a non-interlace VGA camera typically generates over 70 MB/s video stream, usual frame grabber cards apply the lossy compression encoder, such as mpeg-1/-2 or mpeg-4, to drastically lessen the bit rate. In this study, a new approach, which makes it possible to acquire and store such the wideband video stream without any quality reduction, has been successfully achieved. Simultaneously, the real-time video streaming is even possible at the original frame rate. For minimising the exclusive access time in every data storing, it has adopted the directory structure to hold every frame files separately, instead of one long consecutive file. The popular ‘zip’ archive method improves the portability of data files, however, the JPEG-LS image compression is applied inside by replacing its intrinsic deflate/inflate algorithm that has less performances for image data

Parallel Indexing Scheme for Data Intensive Applications

Abstract This paper proposes a parallel indexing scheme of a large amount of data in order to resolve the issues about time limitation. Three kinds of computing-nodes are introduced. These are reception-nodes, representative-nodes, and normal-nodes. A reception-node receives data for insertion. A representative-node receives queries. Normal-nodes retrieve data from indexes. Here, three kinds of indexes are introduced. These are a whole-index, a partial-index, and a reception-index. In a partial-index, data are stored. In a whole-index, partial-indexes are stored as its data. In a reception-index, additional data are stored. The reception-index is moved to a normal-node, and becomes a partial-index. The proposed scheme is also a data distribution scheme for shortening the insertion time. A reception-node accepts additional data even if the index is already built

Adaptive data migration scheme with facilitator database and multi-tier distributed storage in LHD

Recent “data explosion” induces the demand for high flexibility of storage extension and data migration. The data amount of LHD plasma diagnostics has grown 4.6 times bigger than that of three years before. Frequent migration or replication between plenty of distributed storage becomes mandatory, and thus increases the human operational costs. To reduce them computationally, a new adaptive migration scheme has been developed on LHD’s multi-tier distributed storage. So-called the HSM (Hierarchical Storage Management) software usually adopts a low-level cache mechanism or simple watermarks for triggering the data stage-in and out between two storage devices. However, the new scheme can deal with a number of distributed storage by the facilitator database that manages the whole data locations with their access histories and retrieval priorities. Not only the inter-tier migration but also the intra-tier replication and moving are even manageable so that it can be a big help in extending or replacing storage equipment. The access history of each data object is also utilized to optimize the volume size of fast and costly RAID, in addition to a normal cache effect for frequently retrieved data. The new scheme has been verified its effectiveness so that LHD multi-tier distributed storage and other next-generation experiments can obtain such the flexible expandability

Inter-application communication during LHD consecutive short pulse discharge experiment

LHD short pulse experiments are executed every three minutes. After the end of the discharge, the scientists must collect, analyze, visualize the last acquired data of the discharge, and prepare for the next discharge. From the beginning, the computer environment of the LHD (Large Helical Device) experiment has been built as a network distributed system, and various computers have been used for data acquisition or physical analysis. When one program is finished on one computer, that computer must send the results in order to the other computers to run programs. Smooth communication is required in order to finish all the tasks before the next discharge. To exchange the information among the applications running on the different computers, the authors have tried various methods, such as a commercial software to share the memory over the network, simple network file sharing method, IP multicast, web interfaces, and others. The purpose of this paper is to share our experiences of trial and error to build the network distributed systems for the consecutive plasma discharge experiments