Adamczewski-Musch, J.

Bai, J.

Beck, D.

Bär, R.

Frühauf, J.

Hoffmann, J.

Kreider, M.

Kurz, N.

Prados, C.

Rauch, S.

Terpstra, W.

Voltz, S.

Zweig, M.

GSI Repository

Paving the Way for the FAIR General Machine Timing SystemD. Beck1, J. Adamczewski-Musch1, J. Bai2, R. Ba¨r1, J. Fru¨hauf1, J. Hoffmann1, M. Kreider1,N. Kurz1, C. Prados1, S. Rauch1, W. Terpstra1, S. Voltz1, and M. Zweig11GSI, Darmstadt, Germany; 2Goethe Universita¨t Frankfurt, GermanyAbstractThe task of the General Machine Timing system (GMT)is the hard real-time control of the GSI and FAIR accel-erator complex with sub-ns precision. This is achievedby a distributed event generation system based on thenotion of time in the facility. The prime features aretime-synchronization of 2000 nodes using White RabbitPrecision-Time-Protocol (WR-PTP), distribution of Inter-national Atomic Time (TAI) time stamps, generation oftrigger events for synchronization of equipment and pro-viding infrastructure for common services of the accelera-tor control system. During the past two years, the founda-tions of the GMT have been specified, designed and imple-mented.IntroductionThe GMT is based on a common notion of time of allconnected nodes, provided by WR-PTP [1]. A dedicatednetwork based on Ethernet and dedicated WR switches dis-tributes the commands broadcasted by a data master to allTiming Receivers (TR). Relevant commands are decodedand enqueued for timely execution. On time, none, one ormore specific actions depending on the local configurationof the TR are executed. A central clock master serves asgrand-master clock to which all nodes in the WR networksynchronize their clock, phase and time. As the GMT is atime based system, the precision for synchronizing actionsdepends only on the quality of clock and phase synchro-nization via WR-PTP and not on the propagation time ofcommands distributed by the data master.HardwareMore than 2000 TRs will be connected to the timingnetwork. About 1200 Scalable Control Units (SCU) withintegrated TR functionality will be the prime Front-EndComputers (FEC) for equipment control and have been de-veloped by the hardware group CSCOHW of the controlsystem department [2]. About 500-1000 dedicated TRsof different form factors are required by beam instrumen-tation and Data AcQuisition (DAQ). So far, TRs in thethree form factors PCIe, VME and stand-alone have beendeveloped in a cooperation between the departments ofControls and Experiment-Electronics. As all TRs are de-rived from the embedded TR in the SCU, they are basedon Field Programmable Gate Arrays (FPGA) from the Al-tera Arria R© families and share many identical electroniccomponents. The fundamental design principle is to maxi-mize re-usability and similarity only adapting the host-busbridge and I/O for each form factor.GatewareCode for FPGAs is called gateware. Although GSI’s andCERN’s TRs are based on FPGAs from different manufac-turers, the gateware is manufacturer independent to a largeextent. The concept is to separate functionality into differ-ent Intellectual Property (IP) cores which all have an inter-face to the System-on-Chip (SoC) Wishbone bus. All IPcores represent Wishbone devices that are interconnectedvia Wishbone crossbars. The crossbars do not only pro-vide clock domain crossings but also allow to connect formfactor specific Wishbone devices to a standard architecturecommon to all TRs. Each form factor needs to implementits own host-bus to Wishbone bus master bridge. The con-nection to the WR network providing a network interfaceand a WR link is always encapsulated in IP cores behind adedicated Wishbone crossbar. All Wishbone devices pro-vide Self-Describing Bus (SDB) records, that allow query-ing bus-addresses, vendor and product information. Ev-ery gateware includes at least one Lattice Micro 32 CPU(LM32) softcore [3] that allows to implement applicationspecific firmware. Gateware and firmware is developed incollaboration with CERN in the framework of the OpenHardware Repository [5].DriverThe main idea is to extend to SoC Wishbone bus outsidethe FPGA using the EtherBone protocol [4]. As EtherBonecan run over any serial protocol, it only requires a form fac-tor specific driver in the kernel implementing an interfaceto the Wishbone bus master interface on the TR’s FPGA.This allows EtherBone to connect to the Wishbone devicesof the TR over any host-bus to Wishbone bus master bridgeand even network protocols such as UDP and TCP/IP.StatusWhite Rabbit Network15 production quality WR switches [6] have been ob-tained in 2013. All of them are in operation and work reli-ably with respect to their hardware. However, gateware andsoftware are still missing some features. A clock masterswitch is already installed in the final position in the exist-ing building BG at GSI. From here, already five locationsFG-CS-08 GSI SCIENTIFIC REPORT 2013378 doi:10.15120/GR-2014-1-FG-CS-08at the existing facility are connected via optical fiber links.A part of the links will serve to connect equipment of theexisting facility to the new timing system. Another part isused for testing equipment or to gain experience with longterm operation of switches and equipment. Via appropriatepatches, some dedicated links of up to five kilometer lengthhave been set-up. Moreover, several kilometers of fiberhave been exposed to environmental conditions on a build-ing roof. Results of long term tests on WR clock and phasesynchronization including multiple layers of switches arevery promising and presented elsewhere [8].Data MasterThe data master schedules actions by broadcasting com-mands to the TRs via the WR network [9]. A first data mas-ter has been set-up based on a SCU and includes dedicatedfirmware in a LM32 softcore. The firmware generates thecommands that are broadcasted to the nodes. It can be con-figured and controlled via a dedicated API which is used bya Front End System Architecture (FESA) class. Since May2013 the data master is in operation successfully with verylittle downtime. This data master serves as a proof of prin-ciple and will be replaced by a parallel multicore version in2014.Timing ReceiversThe focus of the past two years has been the devel-opment of the TRs in a joint effort of different groups.With respect to hardware, work has been carried out byCSCOHW (SCU) [2] and CSCOEE (VME, PCIe, stan-dalone). CSCOTG has concentrated on gateware and soft-ware. The driver concept allows to access the TR fromLinux user space via the EtherBone protocol in a trans-parent way independent of the host bus bridge - the samefront-end software for equipment control can be used ondifferent FECs independently of the form factor of the TR.As a proof-of-principle, functionality dedicated to the tim-ing system has been integrated with FESA. This allows aFESA class subscribing to a so-called event-source withinFESA and receiving a notification upon an event.For the GMT, a main component of the gateware is theEvent-Condition-Action (ECA) unit. It is a complex filter,comparing the event of a command received from the datamaster against conditions pre-configured by the TR’s user.In case of a match, a pre-configured action will be sched-uled for timely execution in a so-called action channel. On-time, the action channel triggers a receiving component.Here, the action such as IRQ generation, digital output orequipment action such as ramping a magnet is executed.The precision for triggering is 8 ns due to the 125 MHzsystem clock. If supported by the receiving component, anadditional fine-delay feature allows more precise timing.All supported form factors allow fine-delay of 1 ns alreadyin the FPGA. As a perspective, further refinement by PLL-phase-shifting (125 ps) or delay chains in transceivers ofdifferential output pins (25 ps) might be possible inside theFPGA without the need of external fine-delay electronics.Next to timely execution of actions, the ability for pre-cise time stamping of triggers received by a TR is anotherkey requirement to the GMT. As all TRs share a commonnotion of time, a timestamp latch unit is common to thegateware of all TRs. This feature has already been inte-grated by the Multi Branch DAQ System [7] and success-fully tested with the form factors PCIe, VME and stan-dalone over many weeks [8].Conclusion and OutlookSignificant progress has been made on the hardware,gateware and software of the TRs. A timing network span-ning the existing facility has been set up with about 15WR switches. A first timing system has been set up inMay 2013. Key components like the ECA unit, the Wish-bone SoC architecture, the driver concept and the commu-nication library EtherBone have been developed and imple-mented.The next step is to set up the GMT for the re-commissioning of CRYRING at GSI. Here, about 50 TRsare required for synchronization of equipment of the con-trol system and beam instrumentation. At the beginning theGMT will only provide basic features. CRYRING serves asa test ground for the control system as a whole. This is im-portant for validating the concepts, gaining experience ofoperation under on-line conditions and assures the qualityof the components developed.References[1] J. Serrano, P. Alvarez, M. Cattin, E. G. Cota,P. M. J. H. Lewis, T. Włostowski et al., “The WhiteRabbit Project” Proceedings of ICALEPCS (2009) TUC004,Kobe, Japan.[2] S. Rauch et al., “Performance Tests of the StandardFAIR Equipment Controller Prototype”, Proceedings ofICALEPCS (2011) WEPMN018, Grenoble, France.[3] W. W. Terpstra, “The Case For Soft-CPUs in Accelera-tor Control Systems”, Proceedings of ICALEPCS (2011)THCHMUST05, Grenoble, France.[4] M. Kreider et al., “EtherBone - A Network Layer for theWishbone SoC Bus”, Proceedings of ICALEPCS (2011)THCHMUST05, Grenoble, France.[5] http://www.ohwr.org.[6] http://www.ohwr.org/projects/white-rabbit/wiki/Switch.[7] H. G. Essel et al., “The New Data Acquisition System atGSI”, IEEE Trans. of Nucl. Science, Vol.43, No.1 (1996) 132- 135.[8] N. Kurz et. al, “White Rabbit Applications for FAIR Experi-ments”, this report.[9] M. Kreider et al., “The FAIR Timing Master: A Discussionof Performance Requirements and Architectures for a High-precision Timing System”, Proceedings of ITA (2013), Wrex-ham, United Kingdom.GSI SCIENTIFIC REPORT 2013 FG-CS-08doi:10.15120/GR-2014-1-FG-CS-08 FAIR@GSI 379

Paving the Way for the FAIR General Machine Timing System

http://repository.gsi.de/record/68062/files/FG-CS-08.pdf

Paving the Way for the FAIR General Machine Timing System

Abstract

Similar works

Full text

Available Versions

GSI Repository