We propose to perform research in the area of a Persistant Object Manager for HEP. Persistant Objects are those which continue to exist upon process termination and may be accessed by other processes. It is expected that any system based upon this research will work primarily but not necessarily exclusively in an Object Oriented environment. Target applications include follow on or replacement products for existing packages such as GEANT, HEPDB, FATMEN, BHBOOK, experiment specific code event storage. In this respect, it is expected that more functionality will be required than simple persistance. It will be one of the goals of the of the project to define this extra layer of functionality. Strong emphasis will be placed on the use of standards and/or existing solutions wherever possible

Binko, P

Folger, G

Zybert, R

CERN Document Server

A persistent object manager for HEP

Buncic, P

.ll21;[·Lawrence Berkely Laboratory, Berkeley, California, USA OCR OutputBill Greiman, Dennis Hall, Doug Olson, Craig TullOPAL CollaborationOtto SchaileCERN/CN, Geneva, SwitzerlandGunter Folger, Jamie Shiers (spokesman)NA49Predrag Buncic, Ryzsard ZybertA Persistant Object Manager for HEP11 August 1994DRDC / P59CERN/DRDC/94-30RESEARCHEUROPEAN ORGANIZATION FOR NUCLEAR)S aPsc0000070s(lT“*§llllllllllllllllllllllllllllll IllllllllllllllCERN LIBRARIES, GENEVAsolutions where-ever possible. OCR OutputStrong emphasis will be placed on the use of standards and/or existingthis extra layer of functionality.simple object persistance. It will be one of the goals of the project to defineIn this respect, it is expected that more functionality will be required thanHEPDB, FATMEN, HBOOK, experiment speciiic code and event storage.follow on or replacement products for existing packages such as GEANT.exclusively in an Object Oriented environment. Target applications includeany system based upon this research will work primarily but not necessarilytermination and may be accessed by other processes. It is expected thatfor HEP. Persistant Objects are those which continue to exist upon processWe propose to perform research in the area of a Persistant Object Managera single derived data type. Common Fortran extensions (e.g. VMS structures), OCR Outputment facilities of ZEBRA in as simple a manner as possible - basically to supportprovided by the language. Others, such as FATMEN, use the memory managemanagement, have now been rewritten using the memory management facilitiespackage. Applications such as KUIP, which previously used ZEBRA for memoryment part is a relatively small fraction of the total - roughly 10% of the completenot be that of memory management. Already in ZEBRA the memory managemanagement systems, as stated above the principal role of the new system wouldYBOS [17], ZBOOK [18] and ZEBRA were principally considered as memoryAlthough systems such as BOS [13], CHEETAH [14], HYDRA [15], IAZELLE [16],1.1 Not a memory management systemafter several months’ brainstorming.that a much more complete document be produced in the first phase of the project,This document is not an attempt to specify the proposed system. It is proposedscalable. object oriented I/O system.package should not be considered as primarily a memory manager but as a flexible,desirable, if not mandatory. However, it is important to stress that the proposedfulfil a similar role to ZEBRA, and in which Object Oriented features are clearlytype I/O. For these reasons it is clear that some sort of package will be required tolanguages, even on a single machine, let alone machine independent derived datamemory at run time, I/O of derived data types is still an unsolved problem in theseprovide basic memory management features, e.g. allocation and deallocation ofmanager but more importantly as an I/O package. Although ’modern’ languagesProgram Library packages, which rely heavily on ZEBRA, both as a memoryThis high level functionality is used to advantage by many of the current CERNrequired in the future.persistancy and it is likely that many of these high level functions will still besuch as ZEBRA [4]. ZEBRA, however, offers much more than just data structureIn today’s environment, data suucture persistance is provided by packagesfrom ObjectDesign Inc, which is available for numerous platforms.named Ptool [2], and also worked with a commercial solution, ObjectStore [3]that of a persistant object manager. The PASS project wrote their own system,An essential component that was identilied early on in the PASS project [l] was1 Introductiona complete change of programming language. OCR Outputthe benefits of OO to be demonstrated in a real experiment without the trauma ofsystem. By offering OO features in both C and Fortran environments, it allowsZEBRA and can certainly be considered as an interesting prototype for any futureinvoked by sending a message to an object. DSPACK may be used with or withoutphism, methods and inheritance. In true OO style, the appropriate method isFortran programmer. DSPACK supports the following OO concepts; polymorcollaboration, introduces object oriented concepts that are available to the C orThe DSPACK [5] system, a data management system developed by the NA491.4 Object oriented featuressolutions will be sought and adopted where possible.catalogues and HEPDB databases. However, it is important to stress that standardGEANT geometry databases, HBOOK histogram and Ntuple files, FATMEN tilesolutions. In today’s terms, the proposed system should be capable of handlingproject to define this layer of functionality, matching it if possible to standardcall for more than simple object persistancy. It will be one of the goals of thein both on- and off-line environments. The requirements of the above packagescode and also client-server applications such as HEPDB [10]. It should be usablescale packages like GEANT [8], HBOOK [11], PAW [12] etc., experiment speciiicThe proposed system should clearly be capable of solving the problems of large1.3 Target applicationsimportant component of the proposed project.list of what is required. Prototyping is obviously required and will constitute anthis early stage. The features cited in this paper are by no means an exhaustiveexist in the LHC era, care should be taken to avoid overspecifying the system atAs there are many unknowns conceming the computing environment that will1.2 Avoid over specifying the systempermitted the construction of complex client-server systems.However, the powerful I/O capabilities of ZEBRA have proved essential, andFortran 90 derived data types or C structs could have been used instead.APIs might be applicable. cf: VMS LIBS routines and equivalent system services. OCR Output2The correct balance between complexity and power is hard to achieve. `Novice" and "expcn"more information.‘See URL 1mp://wwwn1khefn1/www/pub/oud/meetings/cou-940714/minuteslitm1#io formore complex than existing packages.expect that any new package would be both more powerful and thus inherentlyprovide significant enhancements over previous generations. Similarly, we canhaving unfriendly, complex interfaces. 2 The latest generation of such packagesThese have proved extremely successful, whilst at the same time being criticised forWe have many years of experience of both memory and I/O managers in HEP.package.the rest of this note, without implying any particular implementation of such amay be more appropriate (Object Oriented ZEbra). This term will be used inThe name ZOO has been proposed for the ZEBRA replacement, although OOZE2 Towards ZOOand requirements between the different groups.GEANT project [20]. One would hope for a free exchange of ideas, experienceprojects including MOOSE, the PASS project mentioned above and the proposedThis project, if approved, would clearly have to establish strong links with other1.6 Relationships with other projectsto be used as a building block for other applications, as was true for ZEBRA.strong support must be provided at CERN, particularly if any resulting package isHowever, given the importance of a persistant object manager, it is clear thatfollowed by the MOOSE [19] project.this area include the Object Database Standard, [7], which is also being closelywith CORBA [6] and CORBA compliant systems. Other important items inmembers of the PASS team. In particular, we wish to gain from their experiencemust clearly benefit from their experience and establish appropriate links withAlthough it is unclear whether the PASS project will continue to be funded, weMuch of the work performed in the PASS project is directly relevant to this area.1.5 The PASS projectthe case of the IBM PFS. this can be overridden by the user for a given file. OCR Output"Both the IBM and Meiko parallel file systems support a default mapping per jle system. Inof higher level utility routines.and MZDROP and is clearly insufficient. unless each application writer is to develop their own set3The functionality provided by malloc!) and free() can be compared to that of e.g. MZLIFT’Object’ based I/O packageInterfaces to packages such as ObjectStore and Ptool should be investigated.Exploitation of parallel file systems‘90 derived data types.Support for native language features, such as C structs, C++ classes, FortranSupport for industry standards formats, e. g. Sun xdr, cpio format etc.File based I/O package (on both sequential and random media)Dynamic creation of arbitrary objectsGeneralised data structure manipulation utilities (link lists, nees etc.)2.2 The components of ZOOsubsystems, eventually migating to a standard solution in some years’ time.ate. For example, one might interface to ZEBRA, Ptool and ObjectSt0re as I/Omodel such that components can be replaced with alternate solutions as appropripossible combination with mass storage systems, it makes sense to adopt a layerGiven the existance of commercial persistant object managers today and their2.1 A layer modelcomputing environment, it is unlikely to succeed.system is desigied in such a fashion that it can evolve to match changes in theknow that we can expect sigiihcant changes during its lifetime. Unless the newknow what the computing environment will be like at the start of LHC but we dowith message passing libraries etc, and future unknown developments. We do notLn the future any such package should be able to cope with shared memo1y,MPPsThe required features of a memory manager are well understood. 3 However.This is particularly important for machine-independent I/O. OCR Outputdata types be creatable at run time and that these data types be self describing.time identification of derived data types. It is essential that arbitrary user-definedA feature provided by the current systems but not by native languages is that of run2.5 Run time identification of derived data typesods and inheritance, for all dynamically defined objects.The system should support object oriented concepts, such as polymorphism, meth2.4 OO support for dynamically defined data typescompile and link in the code before it can be used.That is, the user must not be obliged to create a definition of a data type andThe system must support the creation of arbitrary data types defined at run time.2.3 Dynamically dehned data typeschange format only).Read access to existing ZEBRA FZ and RZ files (restricted to binary exMulti-language supportClient- server support(shared memory, sockets, pipes, message passing libraries etc.)Support for parallel processing using arbitrary communications mechanismsInteractive gaphical debug and monitoring packageSupport for DDLSimple debug package (a la DZEBRA)Which today implies C I/O on most systems. OCR Outputlowest level in the I/O buffers.replaced by copies to/from the I/O buffers. In the latter case. the byte ordering is inverted at theOSF and Ultrix are little endian. In the former case the conversion oode simply disappears and isThis is also the native format on most systems. although some. such as MS/DOS, Windows/N T.• Two’s complement• ASCII character set• IEEE Eloatingpoiut• 32 bit data. big endianSZEBRA defines exchange format as follows:of both RZ and FZ formats.tiles. For practical reasons this should be limited to binary exchange format tilesAs was the case with ZEBRA, the new system should be capable of reading existing2.9 Read access to existing ZEBRA filesbut must be copied in/out. This is both inefficient and inconvenient.such as CHARACTER and DOUBLE PRECISION, cannot be accessed directlyOne of the problems of existing systems, such as ZEBRA, is that certain data types,2.8 Direct access to data2.7 Support for native language derived data typesbe written as a series of fixed length records without control words.stream of bytes with no system control words. On sequential media, this wouldThe only tile format that should be supported is the ’bitfile’ - an unstructuredmachines, either through ftp (in binary mode) or via network tile systems.change format). Files written in this format can be freely exchanged betweenA great strength of the existing system is that of machine-independent I/O (ex2.6 Machine-independent I/Otools will clearly also be an invaluable tool during the development stage. OCR Outputline-oriented output and full graphical support will be required. The debuggingPowerful debugging tools are clearly required. As in the existing package, both2.13 Debugging aidsand/or standard products. This again emphasises the need for modularity.which will permit the replacement of various components by language featuresto be attained at the startup of LHC, we can expect developments though its lifetimefeatures eventually coupled with standard products. Whereas this goal is unlikelyIdeally, one would hope to obtain all of the required functionality from languagefrom the compiler’s inability to efficiently exploit the architecture.important on superscalar systems where significant performance hits can resultby the exploitation of features inherent to the language. This is particularlyThe current system involves a certain amount of inefficiency that could be avoided2.12 Efficiency through exploitation of language featuresa different style of computing to that in shared memory systems.such as the IBM SP2. It is important to point out that the latencies involved implymight be achieved using sockets, or message passing libraries in an environmentSupport for a less tightly coupled style of computing is also required. Typically, this2.11 Socket support (distributed processing)systems such as the Silicon Graphics Challenge, the Convex Exemplar range etc.for developing and debugging client/sewer applications, but also on multiprocessoras efficient a manner as possible. This will be used both on single processors, e. g.It is important that support for shared memory be provided by ZOO directly inwas performed ir1 a highly application specific manner.memory support, notably HBOOK first on VMS and later also Unix systems, thisAlthough some packages layered on top of ZEB RA provided some level of sharedMultiprocessor systems with shared memory are now becoming common place.2.10 Shared memory support10 OCR Outputthis phase.The experience of NA49 with DSPACK would be an important component ofbe written.After approximately one year of prototyping, a detailed design document wouldrequired. Some of these areas may be competing, others complementary.report would identify a number of areas where detailed prototyping would beAlthough phase 1 would also involve some prototyping, it is expected that the3.2 PHASE 2packages such as ObjectStore.This phase would include an analysis of the suitability of Ptool and commercialbe presented to various committees (DRDC, COLLECT etc.)between 3 and 6 months and would develop a detailed report of some 50 pages tobrainstorming meetings and discussions. This phase would be expected to lastexperts from both inside and outside CERN, CN, ECP and PPE in a serious ofAssuming that the project is approved, the first phase would be to involve various3.1 PHASE 13 TIMESCALESwill be required e.g. so that optimum blocksizes etc. are chosen.of parallel file systems will be important for performance and efficient integrationstriping of ZEBRA RZ files has recently been introduced into PIAE ExploitationA number of parallel file systems are now becoming available. In addition, file2.15 Parallel file systemscan be performed as part of the installation procedure.verification procedures be designed in to the system so that automatic verificationis very large, and clearly does not scale to the LHC. It is therefore essential thatit is actually working correctly. The effort required to test the existing softwareAs software becomes ever more complex. so does the requirement to ensure that2.14 Built·in verification and specification procedures1 1 OCR Outputhandled by a permanent member of staff.term support and maintenance issue should not be overlooked. This can only bewill require a team of dedicated, hard-working, programmers. However, the longto a very large fraction of experiment specific software. It is clear that this projectitself is central to much of the software supported by the CN/ASD group and alsoZEBRA and its precursors required many man-years of development. ZEBRA5 ManpowerPhase 3 24 man months per yearPhase 2 12 man months per yearPhase 1 6 man months per yearif the project is to succeed. For PHASE 1 we request 50 KSF.below. Note that these numbers should be considered as the bare minimum• Funds to pay for associates to work on the project at CERN as estimatedper year.• Funds for trial licenses of appropriate software (e. g. ObjectStore): 50 KSFthe following resources are requested:Our financial requests to the DRDC amount to 100 KSF per year. In particular,4 Resourcesbe expected to test early prototypes and provide feedback on required functionality.GEANT is expected to be one of the large customers of this work, they would alsoAt all stages, close interaction with other projects will clearly be required. As3.4 Interaction with other projectsZEBRA is required, this will probably last 1-2 years.phases would be realised. Assuming that a system of at least the complexity ofPhase 3 would be the implementation phase in which the results of the two previous3.3 PHASE 312 OCR Output99-01, DESY, Germany.[13] The BOS dynamic memory management system, DESY Internal report R1Q121.[ 12] The Physics Analysis Workstation, Long writeup. CERN Program LibraryCERN Program Library Y250.[11] l-IBOOK Statistical analysis and histogramming package, Long writeup.brary Ql80.[l0] HEPDB database management system, Long writeup, CERN Program LiCERN Program Library Q123.[9] FATMEN Distributed File and Tape Management System, Long writeup.W50l3.[8] GEANT detector simulation package, Long writeup, CERN Program Libraryl-55860-302-6, Morgan Kaufmann (publishers).[7] The Object Database Standard. ODMG-93, Edited by R.G.G.Cattell, ISBNture and Specification, Revision 1.1, OMG TC Document 91.12.1, 1991.[6] Object Management Group. The Common Object Request Broker: ArchitecLibrary Q125.[5] DSPACK Object Oriented Data Manager. DSPAC K tutorial CERN ProgramS ystern, CERN Program Library QIOO.[4] The ZEBRA Data Stmcture Management Package. Overview of the ZEBRA[3] Object Design Inc., Burlington, MA. ObjectStore Reference manual.1994.oratory for Advanced Computing, University of Illinois at Chicago, March[2] Ptool. The Design and Evaluation of a High Performance Object Store. LabProceedings of CHEP94 (to be published).[1] Petabyte Access Storage Solutions. The PASS Project Architectural Model.References13 OCR Outputtion in HER[20] DRDC proposal P58, GEANT 4 .· an Object Oriented toolkit for the simula[19] RD41 (MOOSE), An Object Oriented Software R&D project.Program Library Q210.[18] The ZBOOK dynamic storage management system, Long writeup, CERNCDF Computing Group, Fermilab.The YBOS memory management system, Programmers Reference Manual,[17]Proceedings of CHEP 90, Santa Fe, New Mexico.Jazelle - an Enhanced Data Management System for High Energy Physics,[16]The HYDRA data structure package, CERN TC Program Library.[15]E loisatron Project: Data Structures for Particle Physics, Erice, Italy, 1990.[14] The Cheetah Data Management System, Proceedings 0f the 14t/1 INFN

A persistent object manager for HEP

Abstract

Similar works

Full text

Available Versions

CERN Document Server

CERN Document Server