Adesanya, Yu

CERN Document Server

Data Analysis/Visualization for the Object-Oriented LHC eraY. Adesanya CERN, Geneva, SwitzerlandAbstractThe LHC++ [1] (Libraries for HEP Computing) working group was established inmid-1995, in response to a stated requirement from the LHC collaborations, to investigate a”CERNLIB-like” environment, but based on C++. The strategy of the working group, whichinvolves representatives from CERN experiments and outside, has been to adopt standards -either de-jure and de-facto - wherever appropriate and to concentrate HEP effort onHEP-specific problems. The presentation will focus on those areas of the proposed LHC++environment related to data analysis and visualization, and will include a discussion ofHEP-specific histogram classes, why they are needed, and their relationship with othercomponents of the strategy, including Object Databases (ODBMS) and industry-standardscientific visualization frameworks (AVS Express and IRIS Explorer).Keywords: Object-Oriented; C++; Standards; Histograms; Persistency; Visualization1 IntroductionThe LHC era introduces new requirements which cannot be met by the CERNLIB software foundation.Effective re-evaluation demands a change in our approach. In the past, there was just cause for CERN’sin-house development of mechanisms used to handle fundamental computing tasks. Today’s modularapproach, adopted through object-oriented and component-ware environments, enables commonelements of performance demanding applications to be identified and exploited. Such elements arefar more accessible today than yesteryear. The move towards object-oriented perspectives of problemsolving has left FORTRAN in its wake. The latest generation of high energy physicists regard C++as the current leader in high-level languages. The imminent arrival of the ANSI standard [2] shouldresolve large compiler differences, easing the pain of porting code. Many features are introducedas part of the language. Not all are ground breaking, but the point is that they form a standard, tobe adopted by all who use C++.The STL [3] (Standard Template Library) need not be used directly by users but we must ensure thatclass libraries are as compatible as possible while avoiding redundancy. We need to identify whichparts of a future framework should be developed specifically for LHC. Object-Oriented componentswill help us to create a consistent infrastructure, preserving objects as the units of operation throughoutall levels of the data analysis process. Nobody believes that todays implementation will address allthe needs of 2006. But we can fuse together a data analysis/visualization environment that is farmore future-proof than its predecessor, i.e. a software foundation that is coherent, maintainable andbased on recognized standards. The prototype data analysis/visualization framework will addressethe immediate needs of LHC physicists with an eye to the future.2 Areas associated with HEP unique requirements2.1 Histogram classesThe data analysis/visualization process model relies heavily upon suitable data structures. Theworking group considered existing solutions, both commercial and otherwise. General purposehistogramming packages seemed to fall short of our demands. This prompted the creation of theHistogram sub-group whose role it was to draw up a well defined set of requirements for a design.It appeared deceptively easy. What physicist couldn’t describe the features of a histogram? Therequirements are still being revised.2.2 Object PeristencyPersistency was not treated as an implicit feature of the histogram classes but it was advised thatprototype coding should begin as soon as possible, to help validate the analysis and see how much hadto be taken into consideration when making a class persistent capable. Object databases fitted intothe LHC++ scheme. Nobody wanted to see persistent data as rows and columns in a file. The RD45[4] group was established in 1995 to research object persistency and to produce recommendationsfor HEP projects such as GEANT 4. They understood the storage and performance demands offuture experiments and had chosen the Objectivity ODBMS [5] as a suitable choice for prototypehistogram class library.2.3 Modular interactive visualization for HEPInteractive data analysis and visualization is a major concern. Experience comes from several workinggroup members who have spent a considerable amount of time both developing and using the PAWvisualization package (amongst others). Over the course of time, lessons were learned relating tomaintainability of such systems. The AGOCG (Advisory Group On Computer Graphics) confirmedour beliefs in Modular Visualization Systems. They provide well structured functionality whichcan be easily extended. The modular approach allows supports encapsulation, vital for the reuseof legacy code. Although such systems tend to have larger learning curves, their benefits provideample justification.3 Designing histograms for HEPBefore the decision was taken to develop HEP histograms, some individualshad already contemplatedcreating their own ad-hoc classes. It was time for a common solution. An evaluation of Rogue WavesMath.h++ histogram class showed several limitations. Work began on formalizing histogram packagerequirements. The criteria include: support for weighted entries; multi-dimensional histograms; recording underflow and overflow; variety of binning systems which can be extended; persistence capability; general transformations; statistics.Once a draft requirements document had been produced, an initial class diagram was derived. It’smost striking feature is the distinct separation between histogram bins and the problem space whichdefines their boundaries. Histogram objects contain a collection of bins which store weights anderrors. Partition instances are responsible for generating a bin identifier from a data point.The classes are connected via a one-to-many relationship. This acknowledges the fact that usersoften wish to create create several histograms in a program which share the same characteristics.Bin contents can be set through the Histogram member functions: set bin value(X:Data point,value:Real) set bin error(X:Data point,error:Real)Both calls cause the recipient Histogram to retrieve the bin qualifier ID from its respective partition.The appropriate bin can then be referenced and the value or error is returned. Histograms can alsoundergo operations such as addition, multiplication and subtraction. The OMT diagram (Figure1.) lists operations which accept Histogram objects as arguments. This will be extended to scalartypes in the future. Two subclasses directly inherit from the Histogram class: DataHistogram isfor accumulating weights. The other subclass is ProfileHistogram.Histogramname:Stringget_bin_value(X:Data_Point):Realget_bin_error(X:Data_Point):Realset_bin_value(X:Data_Point,value:Real)set_bin_error(X:Data_Point,error:Real)operator+(h:Histogram):Histogramoperator-(h:Histogram):Histogramoperator*(h:Histogram):Histogramoperator/(h:Histogram):HistogramuseshasBinHisto_2D$create(name:String,nxBins:Integer,xLow:Real,xHigh:Real,nyBins:Integer,yLow:Real,yHigh:Real)fill(X:Real,Y:Real,w:weight)Histo_1D$create(name:String,nBins:Integer,xLow:Real,xHigh:Real)fill(X:Real,w:weight)Profile_Histogramcharacter_name:String$create(data_point_list:List<Data_Point>, character_value_list:List<Character_Value>, weight_list:List<Weight>)add_data_point(X:Data_Point, y:Character_Value, w:Weight)operator+(P:Profile_Histogram):Profile_Histogramoperator-(P:Profile_Histogram):Profile_Histogramtransform(T:Transformation)rebin(new_partition:Partition)Data_Histogramnb_of_data_points:Integer$create(data_point_list:List<Data_Point>, weight_list:List<Weight>)add_data_point(X:Data_Point,w:Weight)operator+(h:Data_Histogram):Data_Histogramoperator-(h:Data_Histogram):Data_Histogramtransform(T:Transformation)rebin(new_partition:Partition)Global_Statisticscharacter_name:Stringmean:Realrms:Realskew:Real$create(character_value_list:List<Character_Value>, weight_list:List<Weight>)add_value(w:Weight,y:Character_Value)operator+(stat:Global_Statistics):Global_Statisticsoperator-(stat:Global_Statistics):Global_Statisticsis associated to the same data points asis associated to the same data points andthe same partition asBin_Contentvalue:Realis_defined:Booleanerror:RealPartitionget_bin(X:Data_Point):BinFigure 1: Top-level OMT diagram of Histogram class structureThe Partition class holds a list of references to elementary partitions. An ElementaryPartitionobject is abstract, corresponding to a single dimension within a partition. Through sub-classing,different kinds of dimension are implemented such as fixed length, variable length, sets, etc. El-ementaryPartition objects can also be shared amongst different Partition objects. The design isfairly open in terms of extensibility. The initial class structure had been defined but concrete testcases would aid validation and refinement. Prototypes would act as a catalyst by promoting userfeedback. Other issues included the interface design. Persistency also had to be dealt with. Justhow much work is required in making C++ classes persistent?4 The impact of the Objectivity on class designA subset of the histogram class structure was chosen for immediate coding, enabling the creationof data histograms of n-dimension (fixed/variable width bins). The first stage was simple enough.Transient C++ classes were written using normal pointers to handle associations. The base types of thedata members belonging to persistent capable classes were switched to ODMG [6] (Object DatabaseManagement Group) compliant types. These types have a fixed implementation to overcome compilerand machine architecture differences. C++ memory pointers cannot be stored inside persistent classes.Objectivity provides a set of generic “smart” pointers used to reference persistent objects. The de-reference operators are overloaded so object data members and functions can be accessed just likeany normal object pointer. These smart pointers are also used to implement associations betweenpersistent objects. Objectivityalso provides persistent string classes and STL like containers includingvectors and ordered lists which can hold persistent objects or their references.4.1 The role of HepRefThe provisions are there but what if you wish to build a class that will exist in transient and persistentforms without having to make extensive code changes? The RD45 group were already tackling theproblem. Using smart pointers and macros a generic interface to references and containers wasdeveloped. In its persistent state, Objectivity references and containers were used. In transientform, containers were implemented using Rogue Wave’s Tools.h++ [7] class library.The genericreference is known as HepRef.4.2 Creating and accessing persistent histogramsIn Objectivity the key unit of database operations is the Transaction class. There can be onlyone active transaction open at any time (per-process). While a transaction is running, objects canbe created, deleted and amended but changes are only made to the database once the transaction isclosed. There is also the option of aborting a transaction. Everything must be done through persistentreferences. This is an important point to remember because when you de-reference in order to accessan objects member functions, objectivity hands the controls over to the C++ compiler.Some test code was written to perform the simple task of creating a Histogram, binning some valuesand storing it in a database. Objectivityallows for different object clustering schemes within databasesbased around “containers” (not to be confused with STL container classes). Hashing algorithms canbe used for optimized access. At this stage, we simply used the default database containers. Hereare the stages involved for creating a 2-D histogram: Start a transaction Obtain the desired database clustering handle Create an ElementaryPartition object for 1st Dimension Create an ElementaryPartition object for 2nd Dimension Create a Partition object Register ElementaryPartition objects with the Partition object Create a DataHistogram using the Partition object Fill the DataHistogram Close transactionThere are several steps involved, but one must be aware that the above example represents the mostdetailed case since all the associated objects are created from scratch. Partition and Elementary-Partition can be referenced by many client objects so if several histograms are created using similardimensions and base types, steps can be omitted. It’s straightforward but it was felt more could bedone to simplify matters. During many meetings, the subject of HBOOK was raised because itsrepresents the bare minimum which the histogram package has to provide. This covered the API too.Two new DataHistogram subclasses were created: Histo 1D and Histo 2D. They contain no newdata members, just more convenient functions. Their constructors create 1D and 2D fixed width binhistograms respectively. They take care of Partition creation and provide a HBOOK style interface.A Session class provides a high level interface encapsulating the behavior of Transaction, simpli-fying the task of database access. The C++ Objectivity interface did not produce any unwelcomesurprises. The API doesn’t pose any problems for competent C++ programmers. Clearly, thingswere taken into consideration when writing persistent capable classes but the effect on the designwas not overwhelming. Currently, the class library has been compiled for Silicon Graphics andHP-UX platforms.5 Histogram Interfaces for VisualizationIn reality, no vendor supplies a shrink-wrapped HEP visualization system ready to ship. An MVS(Modular Visualization System) is designed to allow a high degree of extensibility through codeintegration. A HEP tailored visualization package could include function fitting and minimalizationthrough a MINUIT interface as well as tools to manipulate the persistent histograms. The codeprototyping continued in conjunction with the evaluation of two leading MVS packages: AVS Express[8] and IRIS Explorer [9].5.1 Modular as opposed to monolithicMVS environments are a far cry from the unmaintainable “black box” designs of the past. Theprocess model is now seen as being composed of discrete, re-useable functional units connectedtogether to form a user application. Groups of modules can be encapsulated in order to derive newfeatures. Typically, an off-the-shelf MVS environment will consist of general purpose modules.Categories include: File readers/writers Filters for extracting data subsets Transformers which apply functions to datasets 2-D and 3-D graphics toolkitsNew modules can be built with existing code using an API provided by the MVS. Data is sharedbetween modules through a variety of types designed to handle different forms of visualizationinformation, from array-like datasets to graphics geometry. There is also the possibility to createuser-defined types: The protoyping objectives were: Read in histograms Provide an interface to operations Render the histogram data5.2 Share the histograms or database references?A decision had to be made as to how histograms were to be shared between the prototype modules.The choice was either transient C++ classes or persistent database references. MVS packages makeuse of shared memory mechanisms to reduce the need for data copying. Support of shared C++objects is of great importance but this would imply that the persisent object would have to be initiallyduplicated. The alternative was to share database references. Objectivity has a caching system forefficient object access but at the moment, the scope of each cache is limited per-process. The idealscenario would be either a global, machine scope database cache or an MVS which allows manymodules to be executed as a single process.GUI building facilties allow widgets to be associated to module parameters. HistogramReader wasthe first module to be written, allowing persistent histograms to be read into the MVS using a browserfront-end. Databases can be scanned for classes and their titles displayed in a scrolled list. Objectivityclass iterators are used to perform the scanning with the use of run-time type identification. Operatorfunctions had been partially implemented and they were complemented with an MVS interface whichallows binary operations to be performed interactively. In order to display the histogram data it wasnecessary to transform it into a suitable MVS datatype which could in turn be converted into geometrythrough graphics toolkit modules.The result was the ability to visualize histograms in two or three dimensions in a variety of ways.These included traditional graph plots, contours and 3D rendered scenes, annotated with axis andlegends. OpenGL [10] and Open Inventor [11] are widely regarded as the leading graphics libraries.They also provide standard means of accessing 3D scenes via the World-Wide-Web using the VRML[12] (Virtual Reality Modelling Language).6 Standards and vendor collaborationSo if industry can give solutions, they are worth our consideration. CERN’s program developmentshould center around HEP unique functionality. A great deal of manpower can be saved by sharingthe burden of software support but critics of the LHC++ approach raise two common questions: How long will vendors and their products last? Can you assure product interoperability?Many developments will take place during the next ten years. Our aim should be to try and makefuture transitions as smooth as possible. LHC++ strategy involves building layered foundations,based on leading standards. But that isn’t enough to guarantee compatibility between productsoriginating from different vendors. The intention is not only to form strong links between CERNand vendors but to get vendors collaborating between themselves.References1 The LHC++ group, “Libraries for HEP Computing - LHC++”,http://wwwcn1.cern.ch/asd/lhc++.html.2 P. J. Plauger, The Draft Standard C++ Library, Prentice Hall, (1994).3 M. Lee, A. Stepanov, The Standard Template Library, Hewlett-Packard, California,(1995).4 The RD45 group, RD45 - A Persistent Object Manager for HEP, CERN/LHCC,Geneva, (1996).5 Objectivity inc., Choosing an ODBMS, Objectivity, California, (1996).6 The ODMG, The Object Database Standard - ODMG-93, ed. R. G. G. Cattell, MorganKaufmann.7 Rogue Wave, “The Standard C++ Library and Tools.h++”,http://www.roguewave.com/products/stdlib/stdlibtools.html.8 Advanced Visual Systems, “AVS Express information and resources”,http://www.avs.com.9 The Numerical Algorithms Group, “IRIS Explorer Center”,http://www.nag.co.uk:70/Welcome IEC.html.10 T. Davis, Jackie, Neider, M. Woo, OpenGL Programming Guide: The Official Guide toLearning OpenGL, Addison-Wesley, Massachusetts, (1993).11 J. Wernecke, The Inventor Mentor, Addison-Wesley,Massachusetts, (1994).12 Silicon Graphics, “The Virtual Reality Modeling Language Specification”,http://vrml.sgi.com/moving-worlds/index.html.

Data analysis: visualization for the object-oriented LHC era

Abstract

Similar works

Full text

Available Versions

CERN Document Server