Schaffner, S K

CERN Document Server

SOFTWARE ENGINEERING PRACTICES FOR CONTROL SYSTEMRELIABILITY*S. K. Schaffner,  K. S. White#Thomas Jefferson National Accelerator Facility, Newport News, VA__________________________* Work supported by the U.S. Department of Energy, contract DE-AC05-84ER40150# Email: karen@jlab.orgAbstractThis paper will discuss software engineering practicesused to improve Control System reliability. We will beginwith a brief discussion of the Software EngineeringInstitute’s Capability Maturity Model (CMM) which is aframework for evaluating and improving key practicesused to enhance software development and maintenancecapabilities. The software engineering processesdeveloped and used by the Controls Group at the ThomasJefferson National Accelerator Facility (Jefferson Lab),using the Experimental Physics and Industrial ControlSystem (EPICS) for accelerator control, are described.Examples are given of how our procedures have been usedto minimize control system downtime and improvereliability. While our examples are primarily drawn fromour experience with EPICS, these practices are equallyapplicable to any control system. Specific issues addressedinclude resource allocation, developing reliable softwarelifecycle processes and risk management.1 INTRODUCTIONJefferson Lab, a multi-use facility consisting of aContinuous Electron Beam Accelerator simultaneouslyserving three experimental halls and a Free Electron Laseroutfitted with six user labs, is unique in its use of a singlecontrol system lab-wide. Both machines, along with theirrespective experimental areas and cryogenic facilities, useEPICS for their control and monitoring needs. EPICS isthe result of a collaboration, which began in the early1990’s, between Los Alamos National Laboratory and theAdvanced Photon Source at Argonne National Laboratory.Jefferson Lab is currently the second largest EPICS site ina collaboration that now encompasses over 100laboratories and universities worldwide.   While EPICS continues to develop to include morecapabilities, it is important to note that EPICS is a toolkit,providing development tools and runtime programs, toform the core of a complete control system based on thestandard model.1] Known as an attractive alternative tothe previously popular approach of custom coding allcontrol system functions at each facility, EPICS embodiessoftware sharing at its best. The functions provided by thecollaboration represent the core requirements for mostmodern control systems: control algorithm developmentand execution, data monitoring, storage, retrieval, andvisualization, and network communication between frontand back-end processors. Specific control algorithms,invariably different between sites, and some devicedrivers, must be implemented by each facility usingEPICS development tools. These tools allow programmersto generate a combination of code and configuration filesused to direct EPICS runtime programs. Using thisapproach, EPICS provides a well-tested core for acomplete control system that behaves according to site-specific instructions.   Using code generated by other laboratories savesprogramming effort by eliminating the duplication ofeffort present when recreating commonly used functionsat each site. Additionally, all sites benefit from theexperience and expertise gained by others through theiruse of EPICS. The base code provided to the EPICScommunity benefits from more development effort andtesting than could be achieved by a single site. However,using EPICS does not eliminate the need for softwareengineering and programming efforts at each site. TheEPICS installation itself, specific control programs, andconfiguration data must be supported. Most often, thissupport includes software development, maintenance, andupgrade activities. While managing this body ofcollaborative and site specific software in a way thatprovides consistent, reliable controls for machineoperations can be challenging, it is not substantiallydifferent than managing other software projects of similarsize and complexity. Such management requires reliableprocesses suited to the needs of the project and staff.  While Jefferson Lab is operational, there are still manyupgrades in progress and planned for the future thatrequire modifications to existing control programs or thedevelopment of new code. Additionally, these upgradesmust be accomplished during brief scheduled maintenanceperiods with limited time for testing. These projectrequirements mean we must be able to quickly install newsoftware, perform tests and restore the operationalmachine. These needs have driven the development of anappropriate Applications Development Environment andcorresponding software management processes.0-7803-5573-3/99/$10.00@1999 IEEE. 729Proceedings of the 1999 Particle Accelerator Conference, New York, 19992 STANDARD PRACTICESIt is common for software organizations to develop theirown internal standards and practices, but this effort can bestreamlined by studying current best practices andincorporating this information in a way that is locallyappropriate. Standard practices can provide properguidelines, help prioritize the implementation of newprocesses, and be used as a checklist to ensure allnecessary process areas are covered. As the software industry grows and matures, much hasbeen studied and written regarding software engineeringpractices. The most complete and definitive work on thistopic, CMM, comes from Carnegie Mellon University’sSoftware Engineering Institute (SEI).[2] This welldocumented model provides a framework for evaluatingand improving key practices used to provide reliablesoftware development and maintenance capabilities.Using a maturity based approach, CMM gives thesoftware engineering community an effective andstandardized means for modelling, measuring and definingsoftware development processes. The model defines fivelevels of maturity, summarized in Table 1, and details theunderlying principles and practices needed to improveprocess maturity and thus produce more robust, reliable,maintainable software systems. CMM recognizes thatsuch improvements require ongoing effort and provides aframework to help organize and prioritize this effort. Thismodel is very useful because the information is easilyaccessible and applicable regardless of the systems,projects, languages or techniques involved. The modelallows software engineers to assess their systems todetermine the current maturity level and the neededmaturity level based on risk factors. Once this assessmenthas been done, process improvement can be introducedover a period of time, in parallel with projectdevelopment. While it is ideal to have software processesdefined and in use from the beginning of a project, inreality it seldom works this way. CMM allows processesto be added as the project grows and transitions from thedevelopment stage to an operational system.Table 1: Level  Characteristics  Focus Areas 1 Initial Chaotic, few if anyprocessesNone 2Repeatable Processes formanagementRequirementsManagementProject Management 3 DefinedProcesses formanagement andengineeringTrainingProcess DefinitionRisk ManagementProject Performance 4 QuantifiedMechanisms tomeasure processes andqualityQuantitative ProcessMeasurementQuantitative QualityMeasurement 5 OptimiziedProcess optimization Innovation andImprovement    Most typical software efforts in non-commercial,research oriented environments, like laboratories, begin atMaturity Level 1. Employing even a few of the CMMtechniques can ensure that the project does not also endthere. Left to their own devices, software engineersusually develop some processes resulting in varying levelsof product quality. The challenge is getting all members ofa software development group to adopt the same reliablepractices. Adopting standard practices, like thosedescribed in CMM enables this to occur consistently andat the correct priority. In order to ensure success, it isimportant that the software development staff develop theactual process definitions. This ensures efficient processeswithout unnecessary overhead. The role of management isto specify the process goals and support the developmentand use of the resulting processes. Taking this approachusually results in maximum adherence to the processesand standards since the developers appreciate the benefitsof the methods. Additionally, a properly designed andimplemented development process will increase theefficiency of programmers by providing tools to executeredundant tasks, thus saving time and ensuringconsistency, a key to reliability.3 PROCESSES AND TOOLSAs with most projects, Jefferson Lab’s control systemdevelopment began as a chaotic effort. Over time, andwithout the benefit of CMM, the Controls Group naturallyimplemented processes designed to aid in the softwaredevelopment lifecycle. The group first began looking atthe process of requirements management. This arose outof a need to organize the haphazard approach to systemspecification that produced few if any useful requirementsdocuments. Requirements were received from manydifferent customers, sometimes verbally, somerequirements conflicted with others, and all were labelledhigh priority. Written requirements often did not exist ordid not contain the kind of information needed to proceedwith a project. To put order to this chaos, we developed atemplate for software requirements that was used tostructure the information received and ensure that allnecessary information was acquired. Additionally, forsoftware that was to be used by multiple groups at the lab,meetings were scheduled with representatives from eachgroup to aid in resolving conflicting requirements.Establishing this standard template quickly improved therequirements gathering process and ensured there was adocumented baseline for each major subsystem. In orderto continue to benefit, the documents must be updatedwhen new features are requested. Although updates can bedifficult to sustain, especially with frequent upgraderequests, they pay off during system maintenance andsubsequent upgrade work.   Faced with a seemingly endless list of tasks and a fixednumber of software engineers, we next focused on730Proceedings of the 1999 Particle Accelerator Conference, New York, 1999methods for project planning, tracking and resourcemanagement. These issues were addressed by thedevelopment of a software task database to documentrequests, assign priorities and resources, and trackprogress. For larger projects, preliminary meetings areheld to help determine the scope and complexity of thesoftware effort to make advance planning easier. Thegroup leader prioritizes requests based on lab-wide goals,customer input, and resource availability, and keeps thedatabase information up to date. Another tool is employedby the software engineers to track changes to software asupgrades occur. This information is integrated into theconfiguration management system accessible via theWorld Wide Web and is also used to facilitate softwarequality assurance.   A number of tools and techniques have been developedto improve software quality and minimize controls systemdowntime. Software testing can be difficult due to the lackof adequate off-line facilities and limited machine testtime. Before any software is installed in operations, atestplan is written documenting the application, softwaretest procedure, and roll back information in case problemsarise after beam operations commence. The testplan isreviewed and scheduled by a team leader. In addition tothe application specific tests, machine maintenanceperiods are followed by a comprehensive control systemquality assurance procedure. This ensures all controlscomputers, applications and communications arefunctioning correctly.  The Controls Group also provides24 hours on-call support for operations. The software on-call person is always available via pager and is trained toeither solve problems that arise or to contact the systemexpert if needed. Even if software quality assurance led toerror free operational code, control system on-call supportwould be needed to handle hardware failures andsituations where new operational procedures introducedisruptive resource loading. It is interesting to note thatthe frequency of off-hours calls has been significantlyreduced since the testplan process has been put in placeand control system downtime has dropped as well.[3]   Another aspect of our software process involvesconfiguration management. A source code structure hasbeen adopted making it possible to introduce automatedtools to create, version, and install operational software.These tools enable software developers to quickly loadnew code and roll back to previous versions, greatlyenhancing reliability by insuring that all process stepshave been consistently applied to each application. Inaddition to reliability improvements, such tools make thedevelopers’ job easier by automating lengthy installationsteps that were previously typed by hand.4 CONCLUSIONAt Jefferson Lab, the development and use of goodsoftware development processes has improved the qualityof our software and reduced machine downtime due tocontrol system problems. Because we inherited anunorganized development effort in progress, we haveintroduced new processes one at a time, phasing insupporting tools as time allowed. Our experiencedeveloping these processes and tools has led us to definethe following list of characteristics of good softwareprocesses:1. Processes are developed by those who will use them.2. Processes are supported by management.3. Processes take into account project and developerneeds.4. Processes make programmers’ work easier, moreefficient.5. Processes produce repeatable, consistent results andautomate repetitive steps.6. Processes are documented and publicly available.This list is useful to help design new processes andevaluate existing processes for possible improvements.We plan to continue developing and improving ourprocesses using SEI guidelines such as the CMMframework. By standardizing our development approachand prioritizing work, we have been able to improveprogrammer productivity and reduce the level of stressand pressure on the group. The number of operationalcontrol system failures has been reduced along with theassociated lost time. We have found the time invested indeveloping good processes has been well worth the effort,and believe that more improvements can be made.5 REFERENCES[1] M. E. Thuot, L. R. Dalesio, “Controls SystemArchitecture: The Standard and Non-StandardModels”, Proceeding of the 1993 Particle AcceleratorConference”, Washington D. C.[2]  Mark C. Paulk, Charles V. Weber, Bill Curtis, “TheCapability Maturity Model: Guidelines for improvingthe Software Process (SEI Series in SoftwareEngineering), Addison-Wesley Publishing Company,June 1995.[3]  K. S. White, H. Areti, O. Garza, “Control SystemReliability at Jefferson Lab”, ICALEPCS’97Proceeding, Beijing, China.731Proceedings of the 1999 Particle Accelerator Conference, New York, 1999

Software Engineering Practices For Control System Reliability

Abstract

Similar works

Full text

Available Versions

CERN Document Server