Dalesio, L

Moore, M

CERN Document Server

INTEGRATING ACQUIRED SUBSYSTEMSL. Dalesio, LANL, Los Alamos, New Mexico USAM. Moore, SRS, Savannah River, South Carolina, USAAbstractIntegrating subsystems from external sources into anintegrated control system is becoming more commonplaceas outsourcing is used to attempt to reduce costs.  Thefunctional requirements, integration plans, testing andmaintenance must all be considered to insure success.  Atthe Low Energy Demonstrator Accelerator (LEDA) atLos Alamos National Laboratory (LANL), systems wereacquired for high power RF, Radio Frequency Quadropole(RFQ) vacuum control and RFQ Resonance CoolingControl System (RCCS).  These applications arediscussed in order that we may discover good practices foracquiring external subsystems that fit into an integratedenvironment.1  INTRODUCTIONAcquiring subsystems from commercial sources is donefrequently to take advantage of external expertise, reducecosts, and augment resources.  These systems mostcommonly include some level of control system.  Theyare typically built and tested as stand-alone systems froma set of requirements.  Once they are delivered, they aremaintained by the receiving institute.  The deliveredsystem can occasionally be so difficult to maintain that ithas to be replaced or re-engineered, as in the case of theHigh Power RF at ESRF as reported at the lastICALEPCS[1].  In an attempt to better understand the secosts, a case study is done on three subsystems that wereacquired for LEDA at Los Alamos: RFQ vacuum control,RFQ cooling control and the high power RF control.2  RFQ VACUUM CONTROLThe RFQ vacuum control was developed by thevacuum control group at Lawrence Livermore NationalLaboratory (LLNL).  The requirement was to providecontrol for the RFQ and the RF windows of the RFQ.This system contains about 1,000 signals and manyequipment protection interlocks.  A meeting was held todiscuss the integration of this subsystem into the LEDAcontrols.  It was agreed that we would use Modicon PLCsand Granville-Phillips Ion Gauge Controllers.  It wasdecided that the equipment protection interlocks wouldreside in the PLC and the sequential logic for regenerationof the cryopumps and automatic pump down would bedone in Labview.  The system would be built, delivered,and tested in this fashion.  When testing was complete, theintegration into the LEDA control system, which is basedon the Experimental Physics and Industrial ControlSystem (EPICS), would be done by replacing the Labviewsystem with an EPICS Input/Output Controller (IOC).  Toaccomplish this only required moving the GPIB interfacesto the Ion Gauge Controllers and the serial interfaces tothe PLCs from a Mac running Labview, into the IOC.In an effort to quantify the expense of providing acontrol system for this, the system engineers were askedfor expended time.  None were able to give accuratenumbers, as time was not tracked at this level of detail.The following estimates were made by LLNL personnel:15 months to gather requirements, 6 months to implementthe vacuum system in the PLC and Labview and 1 monthto install.  The LANL personnel gave the followingestimates: .5 month to plan with LLNL, 2.5 months todevelop or adapt drivers, 2.5 months to produce screensand a database to integrate into EPICS and .5 month tointegrate.During the implementation of the vacuum controlsystem, the two control engineers met three times.  Theyworked together to coordinate long wire runs at LosAlamos and to integrate the vacuum system into EPICS.After the integration, several new requirements arrived.Strain gauges were added to the RFQ so that the stressadded by the RFQ window vacuum could be measured.These were required with very short delivery time andwere done directly in the EPICS IOC.  A requirement toprovide critical interlock from the vacuum readings to theHPRF was also added.  The interlock was to respondwithin 15 msecs of a rise in pressure.  The LLNL andLANL engineers determined that the PLC, with its presentload, would respond within 60 msec.  The EPICS IOCcould respond within 15 msec.  Failsafe hardware wasused in the IOC that requires an active output be writtenand additionally uses the state of all critical tasks in theIOC to maintain the interlock.In all, 22 months were used by LLNL and 6 months byLANL.  Of the 22 months used by LLNL, 15 were used togather requirements and design.  It would seem obviousthat the engineer working with the vacuum builders wouldbe able to gather those requirements more efficiently thanan engineer that was located at LANL would.  The 6months used to implement the ladder logic and Labviewscreens and sequences had some overlap in the 2.5 monthsused at LANL to read the PLC values into IOCs and makescreens for an EPICS workstation.  This is the overlap thathas to be compared to having the LLNL engineer learn toInternational Conference on Accelerator and Large Experimental Physics Control Systems412International Conference  Accel rator and L rge Experim ntal Physics Control Systems, 1999, Trieste, Italyuse EPICS and develop the application directly.  It wouldbe easier to assess this if we knew what part of the 6months used at LLNL was spent on Labview, as that is theportion that was reproduced in EPICS.  In addition, 5months was spent at LANL on developing drivers for theModicon PLC, adapting the Industry Pac (IP) GPIBdriver, adapting the IP carrier board driver and developingsome IP drivers.  This time needs to be compared to thetime that it would take to use the portable server and treatLabview as an IOC.  The point count and number of serialinterfaces and GPIB interfaces would have required morethan one Labview system.  More compelling is theinterlock requirement for the high power RF.  In additionto the interlock requirement, the modifications that wouldhave been needed to the Labview program to export everypiece of data that had been read from the PLC, wouldhave significantly increased the cost of the Labviewimplementation.  Labview only makes data availablethrough explicit commands to export the data.  EPICSvalues are all available to the rest of the network as part ofthe underlying functionality of the EPICS database andthe channel access protocol.All phases of the implementation were planned anddiscussed between the LLNL and LANL engineers.  In allcases, the requirements were considered along with aninterest in keeping the implementation costs low.3 HIGH POWER RFThe high power was acquired from two commercialcompanies. The RF system consists of approximately1,200 I/O signals. Continental Electronics provided thetransmitter using an Allen-Bradley PLC5/40.  MaxwellTechnologies supplied the power supply using an Allen-Bradley SLC5/03.  Both PLCs use the DF1 protocol andso the existing EPICS’ Allen-Bradley driver was modifiedto use it.  The high power RF system was controlled froman Allen-Bradley Panelmate display.  As the Panelmatewas left in place and there was no local/remote switch,driver modifications were made to have the EPICSdatabase reflect changes that were made from thePanelmate.  Integration was done by adding an EPICSIOC to the DataHighway Plus.  No communication wasdone between groups at LANL and the commercialcompanies until the systems had been delivered andtested.Again, we have no accurate time measure to use butonly the estimates provided by the engineers involved.The commercial vendors claim that the control effortrequired 12 months.  This was not broken down, so thereis no way to know what portion is used for requirements,implementation, or test.  LANL used 4 months to modifyan existing driver for this protocol and to support the ideathat control channels could be changed from anothermaster.  An additional 2 months were used to produce afault logger that behaved like the local dedicated logger,but made the data available over the network and kept allhistory instead of a limited number of events.  Finally, 12months was used to develop the EPICS database.  Thistime reflects the need to create 4,000 EPICS databaserecords to integrate the 1,000 PLC channels.The number of EPICS records was mostly inflated andcomplicated by the fact that the limits and gain factors foreach channel came in as separate channels.  In addition,the gain factors were not consistent in the PLC.  Thevendors had used the Panelmate display to change thegain factors.  Now these constants had to be firstdiscovered and then kept in EPICS so that the Panelmatedisplay and the EPICS display would match.  Duringintegration, several errors were discovered in the ladderlogic of the PLC.  These included registers beingoverwritten as scratch pad values in between updates ofthe Panelmate and control coils not being implemented toaffect the ladder logic.  Since the commercial companieshad underbid the contract and were loosing money,support was very difficult to get.  The problems in theladder logic were found by LANL personnel.  Some werefixed by the vendors, some by an RF engineer that wasrequired to take over maintenance.  The time spent by theLANL RF engineer is not included in these estimates.A total of 32 months was spent implementing thissubsystem.  It is easy to see that a lot of time could havebeen saved by coordinating some efforts.  The support formultiple masters could have been traded for aremote/local switch.  The time required to make extrachannels for handling gain mismatch could have beenhandled by cleaning up the ladder logic to be consistent.The time spent debugging the ladder logic could havebeen done more efficiently by having the vendorsinvolved in the integration.4 RFQ RESONANCE CONTROLCOOLING SYSTEM (RCCS)Τhe RCCS was developed by Allied Signal usingEPICS.  The RCCS controls water temperature to hold theRFQ on resonance.  In addition to the 1,000 I/O signals,feed-forward control loops were required to provide thesteady state control desired.  In this project, Allied Signaldid all of the work through integration and test and wouldonly turn over the system for maintenance.The time was not measured in any way to give usaccurate numbers.  Estimates given were that 12 monthswere used to gather requirements.  Allied used 9 monthsto develop the database and screens, six of which wereused to train a new engineer.  Industry Pac drivers wereneeded and 3 months was spent developing them.  Finally,four months was used to install and test the system andtune the loops with a RF into the RFQ.  Los Alamosoffered only peripheral support, probably less than a weekoverall.  System integration was completed by putting thepath for their displays into our operator account.413The engineers at Allied Signal thought that they wouldhave accomplished the job of defining the database inabout 3 months had they used PLCs instead of EPICS.This is a saving of 6 months.  It would have required adatabase be written for integration into EPICS bysomeone at LANL, which took about 2.5 months for thesimilarly sized vacuum controls potential savings of about.5 month.  A real loss in savings shows up in the 3 monthsto write drivers.  Similar drivers were written for the RFQvacuum and could have been provided directly for asaving of three person-months.  The lack of regularcommunication between the groups could be a reason thatthis opportunity was missed.5 LESSONS LEARNEDBefore we decide that there is any great insight gained,it is important to know that all time is estimated.  In spiteof this fact, the circumstances do allow us to make severalclaims.  Developing the requirements for subsystemcontrol at the place where the subsystem is beingdeveloped is most efficient.  The people producing thesubsystem control are going to be most efficient usingtools that are familiar to them. Then integration with thefacility needs to be considered.  Costs will increase whenthe facility must train people in new technology tomaintain a subsystem.  Integration requirements must beconsidered from the design phase through completion.  Itis clear that the subsystem requirements are only a subsetof the requirements for an integrated control system.  Inthe research physics community, it also seems clear thatthese requirements do no always show up in a timelyfashion.6 SUGGESTIONS TO AID INSUCCESSFFULLY ACQUIRINGSUBSYSTEM CONTROLSWhen selecting an approach to control a simplesubsystem, it is easiest to use adequate tools that arefamiliar to the engineer performing the work.  Controlengineers that are working on these subsystems should bein contact with the control engineers that are responsiblefor integration, as problems are best solved when acomplete understanding of local and global control issuesare represented.  Always be prepared for requirements tochange.  If a simple problem suddenly becomes complexbecause of a new requirement, it is important to be able torecognize and resolve the situation promptly.  Complexsystems are easiest to build and maintain by using asystem that is inherently capable of controlling complexsystems.  Stately simply, it is easier to have too muchcapability then not enough, as new requirements oftenconvert a simple problem into a difficult one.REFERENCES[1] J. Meyer, “Redesigning a Radio FrequencyControl System with TACO Why and How?”  ICALEPCS’99, Beijing, China, November 1997.http://www.aps.anl.gov/icalepcs97/paper97/p002.pdf414

Integrating Acquired Subsystems

Abstract

Similar works

Full text

Available Versions

CERN Document Server