We report on the status of the PAN-DA data acquisition system presented at the last Real Time Conference. Since that time, PAN-DA has been successfully used in the fixed target program at Fermilab. We also report on the plans and strategies for development of a new data acquisition system for the next generation of fixed target experiments at Fermilab. 10 refs., 3 figs

Anderson, J.

Berg, D.

Berman, E.

Brown, D.

Dorries, T.

Mackinnon, B.

Meadows, J.

Moore, C.

Nicinski, T.

Oleynik, G.

Petravick, D.

Pordes, R.

Rechenmacher, R.

Sergey, G.

Slimmer, D.

Streets, J.

Vittone, M.

Votava, M.

White, V.

Wilcer, N.

UNT Digital Library

* Fermi National Accelerator Laboratory -coIlf-9Y172 PAN-DA and BEYOND Data Acquisition for the Next Generation Experiments R. Pordes, J. Anderson, D. Berg, E. Berman, D. Brown, T. Dorries, B. MacKinnon, J. Meadows, C. Moore, T. Nicinski, G. Oleynik, D. Petravick, R. Rechenmacher, G. Sergey, D. Slimmer, J. Streets, M. Vittone, M. Votava, N. Wilcer, and V. White Online Support Department Fermi National Accelerator Laboratory P.O. Box 500, Batavia, Illinois 60510 June 1991 l Presented at the 7th Real Time Conference, June 2528,1991, Julich, Federal Republic of Germany. by Unlverrltlea Research Assoclstlon Inc. under contract with the United States Department of Energy PAN-DA AND BEYOND Data Acquisition For the Next Generation Experiments R.Pordes, J. Anderson, D. Berg, E. Berman, D. Brown, T. Doties, B. Mackinnon, J. Meadows, C. Moore, T. Nicinski, G. Oleynik, D. Petravick,R. Rechenmacher, G. Sergey, D. Slimmer, J. Streets, M. Vittone, M. Votava, N. Wilcer, V. White OnLine Support Depamnent Fermi National Accelerator Laboratory P.O. Box 500 Batavia, Il6O510, USA Sponsored by DOE Contract no DE-AC02-76CHO3COO Abstract We report on the status of the PAN-DA data acquisition system preaenwd at the last Real Tie Conference. Since that time, PAN-DA has been successfuly used in the fixed target program at Fermilab. We also report on the plans and strategies for development of a new data acquisition system for tbe next generation of fixed target experimenti atFennilab. I. INTRODU~ON The PAN-DA1 system provides for acquisition of data from FASTBUS, CAMAC and VME, software data filtering and monitoring in VME based processor nodes, logging of data from VME to 9 track tape or 8 mm tape cassettes, aad low rate ancillary distribution of data over Ethernet through VME to backend VMS and Unix systems. The system is centrally controlled from a VAX, which also provides message and statistics reporting for monitoring the divers data acquisition components .(See Figure 1). While PAN-DA can sustain front end rates of 7 Mbytes/set and logging rates of up to 1 Mbyte/w, these rates fall short of the requirements of the next generation of fixed target experiments at Fermilab. which will be. taking data in 1994. These experiments require front end ratea of 100 Mbyte&c, parallel filtering at a rate of IOKHz, and logging rates of 5-10 Mbytes/set. We report on OUT work towards design and implementation of this new system. II. STATUSOFPAN-DA The full PAN-DA data acquisition system has been successfully used by one experiment, E6872 during the 1990 fixed target run. It has since been adopted by a second experiment for the upcoming run. Components of PAN-DA are used at six other experiments. As implemented. PAN-DA supports a FASTBUS “event builder” host crate. A Struck General Purpose Master (GPM) reads oat sub-events from multiple L&my 1892 buffer memories and builds events within ils memory. Buffers of built events are written through branch bus to Fermilab ACP single board computers3 residing io VME. All data flows over the FASTBUS backplane twice. and the data rate from FASTBUS to tie VME buffers (including the buffer handling protocol overhead) gives an overall throughput of 7 Mbytes/set between FASTBUS and the VMB memories. The “Enhanced Event Pool” consists of a central “event flow control” processor - aa embedded 68020 VME single board computer. Multiple 2-6 Mbyte single board VME 68020 processors provide for event buffering and possible software filtering. Intelligent I/O controllers - which relieve tie burden of execution of the protocol of the devices from the main processor board - are used to log and distribute the data. The contml prccessor manages the pamIle logging of data and disaibution of events, and provision of available buffers to the front end data system.A logging rate of up to 960 Kbytes/sec to 4 exabytes in parallel is sustained. An event distribution rate of 150-180 Kbytes/sec to the analysis computers is achieved. (see figure 2) Since the last data taking run. E687 have added 3 Sun workstations to their Online Computing system. PAN-DA distributes events directly from tie VME buffers to the Suns in parallel with distributing events to the existing VMS monitoring computers. E681 ported their complete offline reconstruction code to SunOS to have sophisticated monitoring of the physics of the experiment in real time. 1 PANDA Front End Readout 10 Mby1.sIs.E (1821, FSCC, SCC) O”., *ma cbta pallI Distributed VMS Host DA control Event Builder Crate ’ (Host FASTBUS Crate. 1892’s, GPM, (CHI), FB 3C 8 Mbyt**wx. I Enhanced Event Pool 88 Etharnsl (event bufferlng, filtering, logging, and distribution) I Monitortna I Analwir Work,t~tlon. (“MS I “NIX) +19J 03 III. EXPERIENCES USING PAN-DA After initial commissioning, PAN-DA has turned out to be a very stable system. E687 reported minimal problems during the 6 months run - most of them caused by bad exabyte media or drives. The deliberate effort we expended on system diagnostic and monitoring tools _ both hardware and software - has proven its worth These tools include message logs from all sub-systems and canned “snapshot” command procedures which automatically mail the status of tbe system to the expxts. During the initial stages of data taking the PAN-DA support team was able to diagnose problems remotely and to store information to perform post-mortem analyses offline without impacting data taking. We used VME Bus monitors, high speed logic analysers. and SCSI bus monitors to find intermittent system problems quickly and systematically. IV. NEXT GENERATION DAQ The next generation of fixed target experiments at Fcrmilab, urgettcd io tic data in 1994, require a significant increase in the performance of the data acquisition system. We are collaborating with E781 as well as other potential customers, in specifying the architectural design, system design and implementation of a new data acquisition system. ~781’s4 requirements include a trigger rate of 10 Khz and an event size of 10 Kbyte, for an overall data rate. during the spill of 100 Mbytes/set. The events are buffered while a filtering algorithm requiring at least 10 msec/event of a 1 mip is used to provide a software filter. At present, it is expected that the software filter will require access to less than 10% of each event’s data. The falter is designed to reduce the event rate by a factor of 100, resulting in a logging requirement of about 1 Mbyte&c. E781 therefore. has imerestiog requirements for large buffer memories to hold the events during online filtering; large amounts of computing power in the data pipeline; and sniogent requirements for event building and acquisition rates. E799/P8325 and P831 have similar requirements for buffering. but much more stringent constraints on the front end dead time. v. MODELLING NEW-DAQ As the first stage in understanding the requirements and consmints of these new systems, we are building a model of the possible data acquisition architectures using MODSIM6 - an object oriented system modelling tool reported on elsewhere at this cooference.7 Parameterizing the system will allow us to investigate the interrelationships between trigger rate, event size. dead time, software filtering time and data throughput. We are also 2 collaborating with other groups in modelling of the data acquisition upgrade of Fertnilab collider experiment (CDF18. (see Figure 2). DAQ m-1 993 UJJ~L&Z -8 Scanner This will include a commercial Ultranet Switch as the event builder sub-system, being fed by eight input streams, reading out the 50 front end FASTBUS Crates. Intelligetu routing of the sub-events through the switch will provide completely built events to multiple-cpu Silicon Graphics cootputers which perform software filtering on the event data. Event rates of up to 100 Hz for event sizes of up to 250 Kbyte are atttainable. Again, software filtering will reduce the logging rate to 1 Hz. Several different architectures will be modelled and evaluated for the fixed target experiment requirements. VI. ARCHITECIZTRE OF NEW-DAQ The architecture being considered for this new Data Acquisition system consists of die following components (see Figure 3): . . parallel front end readout crates in FASTBUS, with very low deadrime readout controllers; very large sob-event buffers - up to several gigabytes; Event building across VSB or some other dedicated high speed backplane or synchronous bus into VME- tased memories or cpu bxw.ls; RISC-based VME and/or VSB accessible processors for software filtering of the data (single board or commercial UNIX workstation computers): parallel logging of data from each falter processor. or across VME onto available I/O media: distribution of events across ethernet or fddi: distributed control and monitoring of the system from a distributed system of Unix hosts. stored in parallel to the data, and read out in front of the main VII. HARDWARE FOR NEW-DAQ data stream. Interfaces for the VDAS memories to VME, We are evaluating the new generation of ADC and TDC CAMAC and FASTBUS, which support I/O at up to 40 modules - LeCroy modules, Phillips modules and in-house Mbyte&c, have been developed at Fermilab. We are also developed Fermilab modules - to meet the front end digitizing evaluating commercial VME dual and triple ported buffer mentorY boards. requirements. A new FASTBUS readout controller board may well be Thd computing requirements for a software filtering processor may easily be met by a number of off the shelf NEW- Front End Readout Ethernet, lFDDl I ( FSCC, CDF FRC,. 7, . _ Mbytewaec / Ethornot I I- L-10 \’ 14 1 OVB, many Suffe date path Distribute I d UNIX I Event Building I, !4-.-&?&> 15 Mb w Filter 1 1 Filter 11 Fi,ter -1 I EtherneffFDDl needed. In particular, the requirements of E781 will require a Unix worksrations. Tests are being done to evaluate which long distance (optic fibre?) connection from the front end can most effectively be used as online software filtering crates to the next level buffers. CDF have identified a need for processors. CDF is committed to using Silicon Graphics a new readout controller, which is being specified and designed Workstations in this role. Olher options include the Sun at this time. SpatcEngine, IBM R6OCG etc. Large, high speed memories were developed at Fermilab and have been used in several experiments to date. Though originally built as simple FIFO buffers, parallel control boards have been developed to allow record word counts to be We are continuing to evaluate new I/O media, although so far the promise of the double density, double speed Smm tape drives has not materialised in any uscable form. 4 Support for a Unix based host must include consideration of the control and monitoring requirements of a distributed data acquisdon system. We need to understand the throughput issues of logging devices from Rise based workstations. We are currently looking for SCSI/Data acquisition bus interfaces and as a start, evaluating the CES CBD parallel CAMAC branch highway driver. We are investigating the availability of SCSI or VME FASTBUS host intetfaccs. Our strategy is to not restrict support to any particular Unix platform. All applications we develop in house are released on four platforms. We intend to use this suatcgy to enable us to add support for new platforms as they are adopted by experiments. VIII. SOFIWARE FOR NEW-DAQ We are committed to designing and implementing the system requirements and design of NEW-DAQ using online CASE tools as described in a paper at this conferenceP We are using CADRE’s TEAMWORK for all new software development projects. We are planning to migrate from the pSOS real time kernel used in the embedded processors for PAN-DA to a kernel such as VxWorka or Lynx which have more fully developed host based software development support and networking applications. We found the lack of good program development tools to be a hindmnce in the development of PAN-DA. We are gaining experience in the use of Motif and X- windows as software support layers for graphical user interface input and information presentation paradigms. We expect to commit to these for use by the host based NEW-DAQ componenu. We are coding a Unix based message reporting system to gain experience with these to&. We are developing a prototype project in C++ in order to gain he experience needed to determine the appropriateness of Object Oriented Programming to our applications.lO Other prototype projects will be used to evaluate the Eiffel OOP Language and to investigate possible commercial distributed data bases for use in the Host Application layers of NEW-DAQ. IX. PLANS AND %XEDULE Our schedule is inevitably tied to the Fermilab experimental program. Component of NEW-DAQ that may be appropriate for the Collider experiments will be needed within eighteen months. Components for the fixed target experiments cau be developed over a longer time !?ame. We plan to converge on the detailed architecture of our new data acquisition system by the end of 1991. to be followed by the challenge of detailed design and implementation and the excitement of commissioning the system as part of a living physics experiment. x. REFERENCES ‘Bere. et al. “The PAN-DA Data Acauisition System’,’ IEEE’ Transactions on Nuclear icience, VoLNS-36, NOS. October 1989. 2E687, Memorandum of Understanding, Fermilab internal note 3T.Nash et al. The ACP Multiprocessor system at Fermilab, XXIII International Conference on High Energy Physics, Berkely, Ca, 1986. 4E781 Memorandum of Understanding, Fermilab internal note. sDA requirements for E799 Phase II, internal note, T. Yamanaka, Fermilab 6MODSIM - CACI Products Co ‘Data Acquisition Studies for the SSC. E. Milner, paper in Rl91 gCDF Data Acquisition Upgrade, October 1990. (pink book). 9Design of Digital Sky Survey DA and Online System, D. Peaavick et al, RT91 loObject Oriented Design and Programming for Experiment Online Applications, G. Oleynik, CHEP 91. 

PAN-DA and beyond: Data acquisition for the next generation experiments

https://digital.library.unt.edu/ark:/67531/metadc1085696/m2/1/high_res_d/5616816.pdf

PAN-DA and beyond: Data acquisition for the next generation experiments

Abstract

Similar works

Full text

Available Versions

UNT Digital Library