Clarke, J A

Hodgkinson, C L

CERN Document Server

54THE REFILL WIZARD - IMPROVING EFFICIENCY AT DARESBURY LABORATORY’S SYNCHROTRON RADIATION SOURCEC.L. Hodgkinson and J.A. ClarkeCLRC, Daresbury Laboratory, UKAbstractThe SRS at Daresbury Laboratory is a 2 GeV 2nd generation synchrotronradiation source. The SRS is scheduled to operate almost 6000 hours per yearfor users. This places significant demands on the efficiency and reliability ofall accelerator components and systems. Although the SRS operates withonly one refill per day, each taking about an hour, with such a demanding userschedule the length of time taken to refill can be significant when calculatingthe annual performance statistics. The duty operations team leadertraditionally determined the refill sequence at the SRS without following astrict procedure. In December 1997, a very narrow aperture vacuum vesselwas installed. Following this, if the correct refill sequence was not followedthe potential for major component damage during injection was significant.The Refill Wizard, a series of tcl script programmes, was introduced to guidethe operations team through the correct procedure at every refill in a step-by-step manner. The introduction of this operations software has led to increasedefficiencies by making the refill process repeatable and predictable. Thispaper will discuss the significant advantages of such a system and will alsohighlight the unexpected disadvantages encountered during daily operationsusing the Refill Wizard.1. INTRODUCTIONThe SRS is a 2 GeV, 2nd generation synchrotron light source operating in the UK. It was designed touse the main dipole magnets as a primary source of radiation, however during its lifetime there havebeen several major upgrades to install insertion devices in the straight sections, these devices haverelatively large apertures.In 1997 a project was funded to install two 2 Tesla Multipole Wigglers (MPWs) in the SRS. TheMPWs were designed to provide high flux at 10 keV, therefore the vertical aperture was carefullyassessed to determine an optimum gap with respect to lifetime. It was concluded that the internal beamstay clear region in the straight sections could be reduced from 36 mm to 15 mm with a reduction inlifetime of approximately 15% [1]. In December 1997, a prototype MPW vessel was installed in theSRS to gain operational experience refilling the machine with such a small aperture. At the end of 1998,the two MPWs and vessels were installed in the SRS and have been in routine operation since the startof 1999.With the introduction of the two narrow aperture vessels the potential for major componentdamage during injection was significant. In order to prevent damage the correct refill sequence had tobe followed. This paper describes how the SRS was traditionally refilled, the solution employed toprevent damage to the narrow gap vessels and its subsequent effects on SRS operations.552. SRS OPERATIONSThe facility produces synchrotron radiation for users and is scheduled to operate almost 6000 hours peryear. This places significant demands on the efficiency and reliability of accelerator components andsystems. In addition to the number of user hours scheduled, the SRS is subject to Service LevelAgreements (SLAs) with our funding bodies. The overall reliability of the source must be in excess of90% for dipoles and MPWs and 85% for the superconducting wiggler sources. An allowance of 1 houris given for each multibunch refill.Due to the excellent lifetimes at the SRS, the facility can operate with only one fill per day.However, with such stringent requirements placed on the reliability of the source and the limited timeallowable for refills, it is critical that each refill is completed quickly and efficiently. Unless BeamStudies time is scheduled the duty operations team leader is responsible for conducting all SRS refills.The Main Control Room (MCR) is manned continuously by 3 shifts, each with 2 operators. Aduty operations team leader and a deputy operate the SRS on a day-to-day basis. Each team leaderworks with the same deputy, increasing team working and efficiency. There are 6 operations teams intotal. The additional cover enables annual leave and sick leave to be covered easily. In addition eachmember of the team can carry out project work.3. REFILL OF THE SRS PRIOR TO THE INTRODUCTION OF THE WIZARDPrior to the introduction of the narrow gap vessels each team leader had complete autonomy and carriedout refills to suit their own personal style. This method of refill produced very different procedures andrefill times varied from one operations team to another. However, provided that the time taken waswithin that allowable in the SLAs there was very little drive to standardise the system.Fig. 1: Multibunch Refills in 1997The graph shows the refill statistics for multibunch operations in 1997. Total time spent refilling permonth is shown on the left and the average time per fill for each month shown on the right.The average time taken for a multibunch refill in 1997 was 58 minutes.05101520253035404550JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DECMonthTotal Time (Hours)00.20.40.60.811.2Average Time Per Fill (Hours)Total Fill Time Average Time Per Fill564. THE INTRODUCTION OF THE WIZARDIn December 1997 the first narrow gap vessel was installed into the SRS storage ring. During the designphase of the project it became apparent that the possibility for major component damage during refillwas significant due to very high power levels in the photon beam and various vessel protection systemswere put in place. However, this did not eliminate the possibility of human error and it becamenecessary to ensure the correct refill sequence was followed. A system was required which allowed theoperations team to refill the SRS in a step-by-step manner. The solution employed was The SequenceManager, more commonly known as ‘The Wizard’.4.1 The WizardThe Wizard is a programme, which can read and execute a text file in a particular order. The file thewizard uses is the SRS Refill file, which is just a text file that contains a list of executable ToolCommand Language (tcl) programmes. Tcl is a general purpose, robust command language that can beeasily integrated into new applications. Tcl programmes are scripts consisting of tcl commands, whichare processed by an interpreter. The complete SRS Refill text file is given in Appendix 1. As the wizard reads each line of the textfile in order the associated tcl programme is executed. An example is shown below:The SRS Refill text file final line is:Run Goodbeam,2,GoodBeam.tclRun Goodbeam appears in the wizard window. Goodbeam.tcl is the name of the programme tobe executed. The number 2 indicates a level of priority. The wizard runs at level 5, therefore those linesthat have a number less than 5 must complete successfully to allow the next line in the file to beexecuted. If a line fails to complete there is the option to run the script again or to debug. The debugfunction allows the operator to move on to the next line even though the previous tcl script has not beencompleted correctly. This function is useful where the fault is likely to be communications problem,rather than a real fault. All lines in the SRS Refill file are at level 2, except one, the line reads ‘CycleOCEMs,6,CycleOcems.tcl’. As this number is higher than level 5, at which the wizard operates, thesequence will continue even if this tcl script does not execute properly. This particular script runs adegauss cycle on the magnet power supplies. Communications between the magnet power supplies andthe control system is carried out using RS232, which is relatively slow and an error message wasgenerated regularly. The debug command allowed the operator to continue with the refill in thesecircumstances, however the fault occurred often and caused unnecessary delays in refill. By changingits priority to 6 the wizard ignores the error message and continues with the sequence, eliminating theunnecessary delays. Before using this technique the implications of the script not running must becarefully considered. In this case the implications are minor if the degauss sequence does not run andinjection into the storage ring is not achieved, some time is lost fault finding and rerunning the sequencefrom the degauss cycle.Since it is a text file based it is easy to edit and add new tcl scripts. Several scripts have beenwritten by the Accelerator Physics group rather than relying on specialist controls programmes. Anexample of a tcl script Goodbeam.tcl is shown in Appendix 25. THE EFFECT OF THE WIZARD ON OPERATIONSThe wizard was introduced as a safety feature to prevent significant damage to the two narrow gapvessels and flexibility has been deliberately reduced, this caused considerable concern in the operationsgroup. It was perceived that the reduction in flexibility would reduce efficiency by preventing the dutyoperations team leader from intervening at particular points during the refill. However, flexibility canstill be maintained, by understanding what each of the tcl scripts are doing. Scripts can be modified andrun individually or arranged in a text file in a particular sequence for the wizard to execute.57In addition to its safety role the wizard has provided some significant benefits. Since theinstallation of the two new narrow gap vessels during December 1998, there have been two full years ofrefill statistics using the wizard. The two graphs below show multibunch refills in 1999 and 2000.Fig. 2: Multibunch Refills in 1999Fig. 3: Multibunch Refills in 2000The average time taken to refill the SRS in 1999 and 2000 was 48 minutes and 47 minutesrespectively. The refill times for 2000 include a significantly longer superconducting wiggler ramp(approximately 5 minutes longer), due to control system changes. The figure for 2000 also includes thehigh average refill time in October due to problems with the RF system.010203040506070JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DECMonthTotal Fill (Hours)00.20.40.60.811.21.4Average Time Per Fill (Hours)Total Refill Time Average Time Per Fill0510152025303540JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DECMonthTotal Fill (Hours)00.20.40.60.811.21.41.61.82Average Time Per Fill (Hours)Total Time Average Time Per Fill58The reduction in refill times has been achieved by allowing the precise timing and order of eventsto be modified in a systematic manner. The wizard locates and executes all of the small programmesthat the operations team had to search around for, prior to its introduction. By using a single wizard inthis way the interaction between the various front end controls systems and the operations team hasbeen simplified. A significant benefit has been the ease with which the operations team can changebetween various operating conditions. A selection box appears at the launch of the wizard, at whichpoint the operations team can select which insertion devices are available, the wizard then selects theappropriate scripts for that particular refill. This can save a significant amount of setting up time, whenan insertion device has to be withdrawn unexpectedly due to a fault.Another benefit is that there is no confusion over file names. For instance, the latest steering fileis already applied as are the best settings for obtaining injection. Occasionally in the past incorrect fileswere used because of poor communication, this is no longer an issue.In addition to the reduction in refill times, the overall reliability of the source for users has beenimproved by correct sequencing. For example soft starts on particular power supplies have beenroutinely introduced to prevent overload trips. In another case, a line has been added to the wizard,which checks the MPWs can wind out before allowing beam dump. This improves efficiency bykeeping the stored beam until the correct technical groups have been contacted and are ready to correctany fault immediately on beam dump. Beam stability has also been improved by keeping the magnetson for the maximum amount of time prior to refill, ensuring optimum thermal stability.The introduction of a step-by-step refill sequence has made the training of new operations staffmuch easier. In addition, as all operations team leaders now refill in the same way, when a member ofthe operations team is on leave, to replace that member with a stand-by causes much less disruption tothe two man team.Although the wizard has been proved to provide significant benefits to the operation of the SRS,there have been some unexpected disadvantages. All of these disadvantages are associated with the lackof flexibility and also the operators are now divorced from the procedures happening in the background.The operations staff who operated the SRS prior to the wizard gained significant understandingof the accelerators. This knowledge was necessary to refill the machine. The refill wizard has simplifiedthe refill to the extent that this knowledge is no longer necessary during routine operations. Althoughthe wizard has made it easier to train new operations staff, the length of time taken to acquire the sameknowledge of the accelerators and controls systems is much longer, as there is no longer a requirementfor it on a day-to-day basis. This reduction in expertise is noticeable when serious problems arise withthe operation of the accelerators.Due to the sequential nature of the wizard there are cases where the wizard has actually increasedthe refill time. The wizard has been developed to refill the SRS from the beginning; therefore thesesituations usually arise when a fault condition has occurred at some point during the refill. A significantpart of the sequence may need to be repeated increasing the time taken to re-establish injection. This isdue to conditions written into the scripts, which assume refill from a controlled beam dump.If the refill fails continually, it is more likely that a member of the Accelerator Physics Group isrequired to solve the fault. Fault finding and correction now relies upon the understanding of what eachof the tcl scripts do, where each of them are located and the interaction between tcls and other files.However, it is usually clear where the fault lies and so solving such problems is usually faster than inthe past.596. CONCLUSIONThe refill wizard was developed and introduced as a safety feature. It has fulfilled its role of protectingthe narrow gap vessels from damage for the last 3 years. Although received sceptically at first, due tothe deliberately reduced flexibility during refill, the wizard has performed well and provided someadditional benefits. The refill wizard is an excellent memory aid and minimises most of the repetitivetasks. It almost eliminates the possibility of operator error, particularly during busy periods in the MCR.As a result of using the wizard, refill times have reduced by approximately 10 minutes (17%) eventhough more actions are taken by the operations team due to the introduction of 2 additional insertiondevices. The user community have benefited from increased reliability of components and beamstability due to correct sequencing during refill. However, the reduction in flexibility does inevitablycause some frustration to the operations teams, particularly when the beam is lost part way through a filland the time taken to recover is much longer due to the sequential nature of the wizard. The wizardsimplifies the refill to the extent that new operators are able to refill the SRS very quickly. However, thedetailed knowledge that was required to refill the machine is no longer required on a day-to-day basis,which could prove to be a disadvantage in the long term.Since the introduction of the wizard the benefits to operations have far outweighed the minordisadvantages. The SRS Refill Wizard has proved to be a major improvement to the safety, efficiencyand reliability of the SRS.7. ACKNOWLEDGEMENTSThe authors of this paper would like to thank the members of the SRS Operations Team for theirinsights into daily operations using the wizard and for their contributions to this paper. Controls groupfor developing the software.References[1] James A Clarke, Neil Bliss, David Bradshaw, Cheryl Dawson, Barry Fell, Neville Harris, GaryHayes, Michael Poole and Ron Reid, ‘Design of a 2 T Multipole Wiggler Insertion Device for theSRS’, Journal of Synchrotron Radiation (1998), Volume 5, pp. 434-436.60APPENDIX 1 – SRS REFILL TEXT FILECheck MPW6 is OK,2,MPW6Check.tclCheck MPW14 is OK,2,MPW14Check.tclOpen MPW14 Gap,2,OpenMPW14Gap.tclUnlock MPW6,2,MPW6Unlock.tclOpen MPW6 Gap,2,OpenMPW6Gap.tclSwitch off servos,2,ServosOff.tclDump The Beam Using the RF,2,BeamDump.tclSet Wigglers to zero,2,WigDown.tclRun Injection sequence,2,OpsSequence.tclCheck DISPs,2,CheckDisps.tcl*** Open Undulator Gap ***,2,OpenUndulatorGap.tclShow refilling,2,ShowRefilling.tclReset Steering,2,ResetSteering.tclSet-up Transfer Path,2,TransferPathSetup.tclCycle OCEMs,6,CycleOcems.tclSet Injection Steering,2,InjectionSteering.tcl*** Set Timing mode ***,2,SetTiming.tclCheck cavity impedance calibration,2,ImpedanceCal.tclSwitch on RF,2,RFOn.tclSet OCEMs/RF for Injection,2,InjectionSetup.tcl*** Start Stacking ***,2,StartStacking.tcl*** Store Beam ***,2,StoreBeam.tclStart Ramp steering system,2,RampSteerOn.tclArm Vessel Protection system,2,VesselProtection.tclStart Orbit Trip,2,OrbitTripOn.tclStart Ramp,2,StartRampRF.tclSwitch Off DSHN,2,DSHNOff.tclInitialise MPW14,2,SendMPW14TrakMap.tclClose MPW14 Gap,2,SetMPW14Gap.tclInitialise MPW6,2,SendMPW6TrakMap.tclClose MPW6 Gap,2,SetMPW6Gap.tcl*** Ramp Wigglers ***,2,RampWigglers.tclStop Ramp Steering system,2,RampSteerOff.tclApply User Steering,2,UserSteering.tcl*** Check H Orbit ***,2,HOrbit.tcl*** Steer Line 5U ***,2,SteerLine5.tcl*** Steer All TVMs ***,2,Steer.tcl*** Run Global Servos ***,2,GlobalServos.tclShow User Beam,2,UserBeam.tcl*** Record Fill Structure ***,2,RecordFillStructure.tclRun Goodbeam,2,GoodBeam.tcl61APPENDIX 2 – GOODBEAM.TCL# Goodbeam.tcl# --------------## This file runs the injector down# # Revision history:# ------------------# 25th Nov 1998 BGM Script rewritten for MPW upgrade# 8th Feb 2000 BGM Added SI parameters# 20th Mar 2000 SFH Corrected error - SI.HTRBs now switch OFFpackage require Sequencesequence::init truesequence::completion 1 "Script interrupted"sequence::heading "GOODBEAM SEQUENCE \n\n"catch {sequence::setProp BM.ACPI.01 CCV 0.0}catch {sequence::setProp BM.DCPI.01 CCV 0.0}catch {sequence::setStatus BR.RFON.01 OFF} catch {sequence::setStatus {FR.PULS.01 FR.PULS.02 FR.HTV.01} OFF}catch {sequence::setStatus {BI.HTRB.01 BI.HTRB.02 BI.SEPT.01} OFF}catch {sequence::setStatus {BE.HTRB.01 BE.HTRB.02 BE.HTRB.03} OFF}catch {sequence::setStatus BE.BUMP.01 OFF}sequence::remark "\nBooster magnets running down. Please wait 60 seconds...\n"sequence::wait 60000catch {sequence::setStatus {BM.ACPI.01 BM.DCPI.01} OFF}catch {sequence::setStatus {LG.HTV.01 LG.RFHT.01} OFF}catch {sequence::setStatus LR.KHTV.01 OFF}catch {sequence::setStatus LR.HTRB.01 OFF}catch {sequence::setStatus LR.DHTR.01 OFF}catch {sequence::setStatus TM.DIP.02 OFF}sequence::remark "\nLinac Off and Booster magnets Off\n"sequence::wait 2000catch {sequence::setStatus SI.POSN.01 ON}catch {sequence::setProp SI.POSN.01 CCV 0.0}catch {sequence::setStatus {SI.HTRB.01 SI.HTRB.02 SI.HTRB.03 SI.HTRB.04} OFF}sequence::remark "\nSeptum retracted and Heaters off\n"sequence::wait 2000sequence::completion 0 "OK"exit

The Refill Wizard: Improving Efficiency At Daresbury Laboratory's Synchrotron Radiation Source

Abstract

Similar works

Full text

Available Versions

CERN Document Server