Bakker, R

Birke, T

Lange, R

Müller, R

CERN Document Server

CONTROL SYSTEM RELIABILITY REQUIRES CAREFUL SOFTWAREINSTALLATION PROCEDURES R. Mu¨llery , R. Bakker, T. Birke, R. Lange, BESSY, Berlin, GermanyAbstractThe demands on availability and reliability of the controlsystem reflects the stability needed by accelerators in pro-duction. On the other hand the system is continuously de-veloped and new features have to be included and adapted.With growing complexity of the system testing for unfore-seen side effects becomes increasingly difficult. Frequent-ly control access to pieces of equipment essential for theoperation is necessary. Maintenance periods suited for in-stallations are mostly extremely short. This paper describesthe release control system and the installation stages usedat BESSY to provide a high level of control system trans-parency and stability.1 ENVIRONMENTThe third generation light source BESSY II started user op-eration in 1999[1]. The control system in use at this ac-celerator is based on the EPICS toolkit[2]. This paper de-scribes the control system maintenance and development s-trategies providing a high level of both development speedand operational reliability.1.1 Network TopologyEssential prerequisite of manageable installation proce-dures is a proper separation of control system developmentarea and the system used for control and operation of theaccelerator. At BESSY this requirement is met by commonmeasures like segmentation of the computer network.Software is developed in a standard desktop environmen-t configured around a file server in an easily accessibleInternet domain (stage 0 in fig. 1). Here functionalitiesare tested for the first time at test stands. In contrast tothat relatively open environment the control system in pro-duction runs on a highly protected, non-routable Intranetwith autonomous file server, consoles and front end com-puters (IOCs). Independent network components as wellas fail over configurations guarantee minimized interrup-tion of service in case of hardware failures or other typicalcomputer problems.1.2 Test StagesWithin the control network there is a dedicated test ma-chine. Access restrictions for this computer are modifiedFunded by the Bundesministerium fu¨r Bildung, WissenschaftForschung und Technologie and by the Land Berliny Email: mueller@bii.bessy.deConsole ConsoleVMEVMEIDMONVMEConsole ConsoleVME VMEPSHFVACEPICS EPICSEPICSVMEEPICSEPICS(EPICS)TestData AquisitionMonochromatorCommon ServicesDAQPCPCPCVMEEPICSXBPMPinHoleVMEEPICSConsoleVMEEPICSLFBServerLabViews GP−IBBPMBLInterlPLC ConsoleConsole ServerINSTALLtesticcServerRouter0123       8/99  RMCOMVMEEPICSServerOnline Data  GatewayArchiverMessage LoggingProblemReport ...RDBMSCommunicationFacilityNetwork Accelerator  ControlsNetworkBeamlineControlsNetwork Machine DevelopmentFigure 1: Sketch of the Network Topology.to allow for downloads of new software releases from thedevelopment area. System configuration is an independen-t structural replication of the controls file server. Withinthe whole controls environment user accounts are mappedto functionalities like operator, maintainer etc. The trans-fer of developers ownership of modules to functionalitiesalready takes place on the test machine. Access securityallows full hardware control from this computer to admin-istrator accounts with installing and testing permission.If a new software module or version is scheduled for go-ing into operation it first has to be installed on the test ma-chine either in the boot area of the IOC computers or intothe tree of console applications (stage 1 in fig. 1). As soonas a time slot for tests becomes available the new versionsare tested by running console applications on the test ma-chine or booting the IOCs from the test area. Discoveredbugs are fixed, the improved version reinstalled and retest-ed. Whenever a test time slot ends the switch back to thesettled productional version requires in its worst case thereboot of the affected IOCs. If the new modules appear tobe sufficiently mature the version repository is updated, theInternational Conference on Accelerator and Large Experimental Physics Control Systems326International Conference  Accel rator and L rge Experim ntal Physics Control Systems, 1999, Trieste, Italymodules are installed on the file server and become the newproductional version (stage 2/3 in fig. 1).1.3 Multi-homed Communication FacilitySimilar to accelerator controls dedicated networks for thebeamline systems and major user groups are presently inthe phase of being set up. Independency of these areas aredesirable for operational security and availability reasonsbut require additional infrastructure.Data Switch-Yard: For numerous correlation andmonitoring tasks online data have to be passed between thedifferent networks. E.g. the beam intensity measured bythe accelerator control system has to be available also at thebeamlines. Equally the read-outs of the diagnostic XBPMsystems in the beamline area are needed by the orbit sta-bility control modules within the accelerator segment. Thedata transfers between the networks is provided by gate-way programs that control transfer of access security andimpact on network bandwidth. It is foreseen to run thesegateway processes on dedicated multi-homed communica-tion facilities (Fig. 1).Versioning of Production Systems: At BESSY thereare several layers (core system, toolkit libraries, genericand custom applications) and streams of development (ac-celerator, ID, beamline). If a subsystem has to go out ofsync and certain versions of the involved software have tobe frozen today this is mostly done by copying the requiredmodules to separate locations and using this extra tree forthe build. With increasing complexity of the systems it be-comes essential to replace this error-prone and inflexibleprocedure and maintain a uniform and transparent version-ing of all production quality software that allows an easyroll-back to preceding versions. The central communica-tion facility will provide the required file server functional-ity for this common repository (stage 2 in fig. 1).2 SOFTWARE DEVELOPMENTGuiding ideas behind the common and mandatory sourcecode development structure have been roll-back andbranch-merging capability, transparency and flexibility aswell as cooperative maintainability.2.1 Application Development EnvironmentUnified directory trees with preconfigured Makefiles, in-stall locations and handles to the proper version of the coremodules are provided to the application developer by a cen-tral script (makeBaseApp). The common structure easesmigration to newer releases of the base modules and thetransfer of modules to another developer.2.2 Source Code Control SystemCertain intermediate or stable stages of a software moduleare preserved by the check in into a source code reposi-tory. At BESSY the package Concurrent Versions System(CVS)[3] is used. CVS features concurrent access, con-flict resolution and branch merging strategies. Logging ofsource file changes and symbolically tagging of certain re-leases simplifies a secure roll-back to a previous version.Access and protection is controlled by standard Unix filepermissions. Developers cooperating on a certain packagehave a common group identification. Core developers haveto pay additional attention to licensing terms and utilize apseudo user identification. For inter-division exchange itis planned to set up a CVS server to allow for secure andnetworked code maintenance and retrieval.2.3 Module EncapsulationShared libraries are used to facilitate independent bug fix-ing and maintenance of modules used by several consoleapplications. Typically the interface definition of a moduleis not changed. Then it only requires to install the improvedversion of the shared library to fix all applications that usethe module. On the back side a careful bookkeeping of al-l affected software components is required if the interfacedefinition has to be changed and the installation of the newshared library is required.ModelEPICSCDEVtcl/TKDB AccessBPM MonitorPower Supply Magnet SupportCorrelationMeasurementData ConversionUtilityOrbitCorrectionApplicationsToolkitLibrariesCoreLibrariesFigure 2: Vertical Dependencies: Module Layers.2.4 Installation Environment and ProceduresA very convenient feature of the Application Developmen-t Environment (See 2.1) is the support of installation anddistribution procedures. For each module Makefiles allowto install binaries, libraries, include and resource files aswell as scripts controlling environment variables into theappropriate location. Distribution stages like the installa-tion into the developers working directory, into the globaldevelopment area, onto the test machine and finally into theproduction area file server are easily addressable. The uti-lized distribution mechanism rdist takes care of the actionsnecessary for a synchronization of the module.No structural solution is available yet for vertical inter-module dependencies. If a base or intermediate module ismodified it requires the definition of a coordinated rebuildand installation campaign to get all affected applicationsupdated. Even if the whole installation process has to runthrough the test stages described in 1.2 the vulnerability ofthe procedure requires versioning measures.327During commissioning many new installations and mod-ifications took place in parallel. Therefore full copies of theaccelerator control system in production have been savedroutinely to be able to return to a previous, operational ver-sion if the reason of an unexplainable malfunctioning couldnot be found in due time. This procedure quickly turnedout to become unmanageable. In the present stabilized sit-uation at least certain installation blocks are not overwrit-ten, but moved into the attic to provide a minimal versionhistory. As already sketched in 1.3 it is planned to set upa complete versioning of binaries, libraries and configura-tions that ever went into production.3 CONFIGURATION MANAGEMENTThe most difficult part of a ‘living’ control system is a sys-tematic, consistent and trouble free maintenance of the var-ious configuration data and their interdependencies.3.1 Reference Data RepositoryAt BESSY a relational database serves as a central refer-ence data repository. Configured around the notion of a‘device’ and its name this DB is supposed to hold all re-quired conversion factors, geometrical data, connectionsand relations[4].The whole DB is handled similar to the applications. In-dependent database instances are set up in the developmentarea, on the test machine and in the production area. Bystepwise replication the DB has to follow the test stagesthat are mandatory for the applications (see 1.2). As an ad-ditional back-up strategy the periodic export of the wholeDB to an external mass storage system takes care of the u-nique requirement on consistency, integrity and availabilityof the DB content.3.2 Propagation of Device ModificationsEven if a large fraction of the configuration data are gen-erated by scripts from the DB there is still a number ofhand-edited files and dependencies that require manual in-teraction.If e.g. a new device is added to the system or a device ismodified the ‘horizontal’ dependencies of the various toolshave to be checked to be able to identify affected entities. Ifthe new device belongs to an existing device class a re-runof the configuration scripts covers most of the adaptationwork. If a new device class is added the generating scriptshave to be adapted and the net of correlations and depen-dencies has to be adapted (Fig. 3).4 PROBLEM TRACKINGComplexity of the system, lack of time and limited avail-ability of the required hardware prevent the clear exclusionof problems newly induced with the modifications.In order to avoid unforeseen side effects a mailing listmandatory for the controls and application developers hasDataBaseGenericApplicationsStripToolmultiKnobAlarm ArchivingSave,RestoreCompareOnlineDataGatewayEntriesHierarchiesEngineeringScreensDrivingCorrectionMeasurementEvaluationHighLevelApplicationsIOCRT DBAppl.CDEVddlConfiguration ModificationsGeneratingScriptsWindows, NavigationPreconfiguredToolsFigure 3: Horizontal Dependencies: Propagation of Modi-fications.been set up. Pending new installations or possible problemsare announced on the list to prevent misunderstandings orneed for additional activities.The gnats problem reporting system has been configuredto cover the classical service areas (power supplies, RF,vacuum, controls) and notify the person(s) in charge of thesubsystem via email. Operators are encouraged to use thistool as a checkable, reliable and transparent tool to trans-mit complaints, observations or comments. A tcl/Tk and aWEB interface help to create and submit reports, check theprogress on a subject etc.5 SUMMARYThe combination of version control, staged installation pro-cedures and partly automatized configuration managementturned out to be a manageable compromise of effort andefficiency. The resulting availability, consistency and reli-ability justifies today’s unquestionable confidence into thecontrol system.6 REFERENCES[1] R. Bakker et al., ‘Status and Commissioning Results ofBESSY II’, Proceedings of the 1999 PAC, New York, 1999[2] R. Bakker et al., Proceedings of the 1998 EPAC, Stockholm,1998, p.1676[3] CVS has been developed by Per Cederqvist et. al. and is avail-able under the GNU General Public License[4] R. Bakker, T. Birke, B. Kuske, B. Martin, R. Mu¨ller, Proceed-ings of the 1997 ICALEPCS, Beijing, 1997, p.407328

Control System Reliability Requires Careful Software Installation Procedures

Abstract

Similar works

Full text

Available Versions

CERN Document Server