Bishop, D

Dale, D

Devereaux, S

Hui, H

Keitel, R

Waters, G

CERN Document Server

DISTRIBUTED MONITORING SYSTEM  FOR EPICS IOC HEALTHD. Dale, D. Bishop, S. Devereaux, H. Hui, R. Keitel, G. WatersTRIUMF, Vancouver, CanadaAbstractA microprocessor-based IOC VME crate monitoringsystem has been developed for the TRIUMF/ISACRadioactive Beam Facility control system. Distributedmonitoring modules reside in the IOC VME crates andbroadcast their address and status on a single CANBusloop. Each module builds a map of all the monitorstatuses that can be read via the VME backplane by thelocal IOC. The module is capable of monitoring crateparameters including voltages, temperature, fan speed,airflow etc. EPICS writeable watchdog registers allowdetection of software problems. When requested by aCANBus command the module will generate a VMEbussysreset and can cycle the power to the crate.1 INTRODUCTIONThe TRIUMF/ISAC Radioactive Beam Facilitycontrol system utilizes EPICS implemented in VME [1].Due to the distances from the control room and theseveral locations, where access is restricted, an IOChealth monitoring system has been implemented.2 DESIGN GOALSFor a VME crate monitoring system, severalparameters must be measured and compared toacceptable values. These parameters include: Temperature Voltage Fan speed VME SYSFAILIn addition, for IOC health, monitoring EPICS scantasks is essential. Although monitoring will detectproblems the system should provide a means forcorrecting the problem if possible. Processor hang-upsmay be corrected by resetting the processor. This can beaccomplished by driving the VME SYSRESET.However some processor or other module latch-ups canonly be remedied by cycling the power to the VMEcrate.In the TRIUMF/ISAC facility several of the IOCVME crates reside in areas that are biased at highvoltage. Communicating with these crates requiresgalvanic isolation.As previously described [2] CANBus is being used forcontrol of all beam steering elements. Due to its successand support by the EPICS collaboration slow controlcommunication should be implemented using CANBus.Each crate-monitoring module should broadcast itsstatus over the CANBus so that all modules can build astatus map of the other crate monitors. The local IOC,via the VME bus, should read this map. The moduleshould broadcast a single summary status word, thusminimizing CANBus activity.  The existing IOC andEthernet communications should be utilized to providedisplay and alarming to the operators.The monitoring system must be retrofitted into thepreviously chosen Elma VME crates [www.elma.com].3 SYSTEM DESIGNThe crate monitoring system consists of three parts:• a standard width 6U VME module,• an interface PCB,• and a hub3.1 VME ModuleA block diagram of the VME module is shown in Figure1. The module can measure the 5, +12, -12, and 5standby power supply voltages. It has an onboardFigure 1: Block Diagram of VME ModuleuCFLASHHardwareMonitorCANBusDriverFiberDriverI2CDeviceSPIDeviceFANCrateVoltagesAnalogTemperaturesEEPROMCANBusController VME InterfaceVME SysresetCrate PowerDownDigitalTemperaturesVME P2 Connector228International Conference on Accelerator and Large Experimental Physics Control Systems, 1999, Trieste, Italytemperature sensor, and supports the DallasSemiconductor 1-Wire TM temperature sensors[www.dalsemi.com]. In addition, two analogue inputsare available for use with temperature sensors such asthe National Semiconductor LM34/35[www.national.com]. Fan speed measurement requiresfans with a tachometer output. The module is capable ofmonitoring up to five fans. Other features include a serialRS-232 port as well as both SPI and I2C interfaces forsensor expansion. The module resides in VME A24memory space. All parameters can be read through VMEregisters and trip levels are written into VME registersand saved in EEPROM. The system supports up to 256modules. Visual and audible indications of problems areprovided. The module’s front panel displays include:• VME Access• CANBus Tx/Rx• RS-232 Tx/Rx• Fan Trip• Temperature Trip• Watchdog Timeout• +5V out of tolerance• +12V out of tolerance• -12V out of tolerance• +5V standby out of tolerance• Crate power inhibitIn addition to the CANBus interface, a fibre opticinterface can be used when galvanic isolation isrequired.3.2 HubThe fibres connect to a “HUB” much like an Ethernet10BaseT system as illustrated in Figure 2. The CANBustransceivers reside on the hub and all CANBus activity isrelayed to the module over the fibres. The fibres are keptshort enough so that the added delay does not effectproper CANBus communication.3.3 PCB InterfaceThe interface PCB allows connecting external devicessuch as fans, temperature sensors, and the power supplyinhibit via the VME P2 connector.4 OPERATIONA crate (A) reset or power cycle occurs when themodule in crate (A) receives a CANBus reset or powercommand from crate (B). Writing the destination crateID followed by the appropriate command to the crate-monitoring module in crate (B) via VME generates thesecommands.There are 16 possible watchdog registers formonitoring EPIC scan tasks. Currently for theTRIUMF/ISAC control system scan tasks of 0.01 to 10seconds are being used. Each of the 16 watchdogs can beset from 1 to 15 seconds in one-second increments. EachEPICS scan task must be modified to include a write to aVME memory location each time the task executes. Ifthe task fails to write to the location within the specifiedperiod a flag is set in the global status register for thatcrate monitor. The register contains a one-bit summaryfor each of fan, watchdog, temperature, and voltage.This global status register is transmitted at a 1Hz rate asa CANBus broadcast. Each crate monitor receives thisdata and stores it in the VME register, whichcorresponds with the address of the transmitting module.If the data contains fault information a flag is set in theVME fault register. The IOC needs only to check thisregister to determine if other crates are operatingproperly. If the flag has been set the IOC must thencheck all the mapped global status registers for the 256possible crates.5 STATUSA prototype module is currently undergoing initialtesting.REFERENCES[1] Keitel R. et al, “Design and Commissioning of theISAC Control System at TRIUMF”, theseproceedings[2] Bishop D. et al, “Distributed Power Supply Controlusing CANBus”, ICALEPCS 97, Beijing, November1997.Figure 2: Crate topology21 Slot VME CrateSlot 1=MVME162 IOCSlot 21=Crate Monitor21 Slot VME CrateSlot 1=MVME162 IOCSlot 21=Crate Monitor21 Slot VME CrateSlot 1=MVME162 IOCSlot 21=Crate MonitorTermTermCANBusOptical Fibres"HUB"Crate biased at60kV21 Slot VME CrateSlot 1=MVME162 IOCSlot 21=Crate Monitor229

Distributed Monitoring System for EPICS IOC Health

Abstract

Similar works

Full text

Available Versions

CERN Document Server