Brelet, J L

Busson, P

Crowley-Milling, Michael C

Dobrzynski, Ludwik

Défossez, M

Gannon, C

Karar, A

Macé, J R

Moreira, P

Romanteau, T

CERN Document Server

Embedding deserialisation of LHC experimental data insideField Programmable Gate Arrays.Ph. Busson, L.Dobrzynski, A. Karar, T. Romanteau for LLR, P.Moreira for CERN, J.L. Brelet, J.R. Macéfor XILINX France, M. Défossez for XILINX Benelux,  M. Crowley, C Gannon for XILINX DESIGNSERVICES (XDS)LLR, 91128 Palaiseau, France – CERN 1211 Geneva 23, Switzerland – XILINX France, 78353 Jouy-en-Josas, France –XILINX Benelux, 2400 Antwerpen, Belgium – XDS, Dublin, Irelanddobrzynski@cern.ch, romanteau@poly.in2p3.frAbstractLHC experiments will make use of thousands of seriallinks in order to transfer digital data from the electronicssitting in the detectors to the off detector electronics locatedmore than 100 meters away.  Due to the high levels ofradiation present in the detectors, CERN designed anddeveloped a radiation hard serialiser, the Gigabit OpticalLink (GOL) chip[1]. On the other hand, the off-detectorelectronics designed to process digital data received fromthe detectors will heavily rely on commercialprogrammable components like Field Programmable GateArrays.Commercial components will be used to de-serialize thedetector’s data prior to processing with FPGAs. The XilinxCompany is now offering a new type of FPGAs(Virtex2Pro) witch embed Multi Gigabit Transceivers(MGT[2]). The use of these components will allow a morepowerful and compact design of the off-detector electronicsprocessing boards. This paper will describe the results oftests performed to measure the performance of a link madewith a GOL chip and a Virtex2Pro circuit.I. INTRODUCTIONIn High Energy Physics (HEP) experiments, high-speedlinks operating in the Gbit/s range are chosen for datatransmission of both raw and specific trigger data. Lowlatency is required for the last one, to minimize the datastorage inside the detector until the trigger decision isavailable. For the transmission of raw data a low latency isnot required. It appears that a large reduction in number,size and cost of the data link boards can be achieved if ahigh integrated solution is possible for the deserialiser. Thenew Virtex2Po FPGA device from XILINX seems to be agood candidate for this application.At a 1st stage, we performed a performance study wherethe GOL was connected to the commercial referencedeserialiser TLKx501 device from Texas Instrument. Theseresults will be used as a reference in this paper, but thedetails will not be reportedThe present paper describes the work we performed incollaboration with XDS1 to check the functionality of theGOL when connected to the MGT of the Virtex2Pro.II. BASIC REQUIREMENTSThe evaluation focus on three measurements made onthe communication link:• Bit-error rate: While the MGT is synchronised tothe transmitted signal, the number of bit- errors arecounted and recorded periodically.• Loss of Synchronisation: Under certainconditions, the Rocket IO may lose synchronisationwith the signal transmitted by the GOL device. Thenumber of times that the MGT loosessynchronisation has been recorded. Statistics havebeen also collected regarding the resynchronisationtime.• Link latency: The Rocket IO receiver latency ismeasured.All these measurements have been carried out in slow(800Mbits/s) and fast (1.6Gbits/s) modes.The jitter of the clock provided to the MGT have beenmeasured and the eye diagrams of the serial transmitteddata observed.A test environment, described in the next section,  hasbeen implemented for these measurementsIII. TEST ENVIRONMENTA. OverviewFigure 1 shows the test environment. The GOL board isused to transmit a known data sequence to the V2PROboard. The data sequence is received by the FPGA deviceon the V2PRO board. The FPGA device contains the MGTtransceivers used to de-serialise the received stream. Inaddition, the FPGA contains dedicated logic and aPowerPC processor that are responsible for analysing thereceived data and making the measurements. The PowerPCis also responsible for offloading the measurements via an                                                           1 Xilinx Design Services based in IrelandRS232 interface whereby they can be displayed on a hyper-terminal residing on a PC.Figure 1: Test Environment setupB. V2Pro BoardXilinx has developed a dedicated high quality MGTcharacterisation board (ML320 Board). This board is a stateof the art and provide the best possible test platform for ourproject. XDS have been granted an early access to it.ML320 BoardThe verified designs have been ported to the ML320 inorder to run the final measurement tests. In addition to theV2PRO device, the board hosts also a UART driver that isused to support all test configuration and status functions.A single differential clock running at 40MHz (slow mode)or 80MHz (fast mode) clocks the board and can beconfigured using ACE  System.C. GOL BoardThe GOL board contains the GOL device and an AlteraFPGA. It has the pattern generator providing the datasequence to be transmitted. The Altera device isprogrammed from a PROM. Several switches are used tochange the operation modes of the board e.g. to switchbetween slow and fast mode.D. ClockingThe V2PRO board requires a single differential clock.This clock will be 40MHz for slow and 80MHz for fastmode.An Agilent data generator has been used to provide the tworequired clocks. The GOL board has its own on boardclock.These two clocks were not frequency or phase locked toeach other.E. OscilloscopeA high performance oscilloscope has been used forprobing the gigabit serial signals in order to help in anytroubleshooting. This oscilloscope was used for the latencymeasurement between the GOL and the MGT, see sectionIV.B.F. V2Pro FPGAFigure 2 illustrates the implemented frame work insidethe V2PRO FPGA. The design contains two MGTs. TheMGT12 is configured to receive serial Ethernet data at 800Mbits/s and MGT23 is configured to receive serial Ethernetdata at 1.6 Gbits/s. The SD+/SD- ports are connected to apair of SMA connectors on the board and the FD+/FD- areconnected to another pair of SMA connectors. The GOLserial data signals should be routed to one of these pairsdepending on whether the slow or fast mode is chosen.The MGT performs the serial to parallel conversion onthe serial inputs and passes the data and its own statusinformation to the measurement block. The measurementblock performs the necessary measurements based on ananalysis of the received data and the MGT status. It passesinterim measurement data to the processor periodically.The processor compiles and processes the measurementsand presents summary results on the RS232 UARTinterface whereby the measured data can be displayed. Theprocessor program and data memory will reside in FPGAblock RAM.Figure 2: V2Pro Design EnvironmentThe UART interface is bi-directional and it is conceived such thatthe user can enter commands on the hyper-terminal so thatmeasurement tests can be flexibly and quickly configured and                                                           2 MGT identification number on LM320 board3 MGT identification number on LM320 boardTx+JTAGRS232Tx-Stop SignalStart Signal Differential ClockV2ProBoardPCHyperTerminalAGILENTClockGeneratorGOLBoardOscilloscopeProcessor BusMGT1800 Mbits/sReceiverMeasurementBlockMGT21.6 Gbits/sReceiverPowerPCProcessorUART I/O RS232controlled. These commands would be interpreted and executedby the PowerPC by writing to the measurement block registers.G. PC / Hyper-terminalThe PC fulfils two functions. Firstly, it is used todownload the V2PRO bit stream to the board. Secondly, ithosts a hyper-terminal from where the measurement testswill be controlled, configured and on which themeasurement results will be displayed.IV. MEASUREMENT AND RESULTSA. Test Set-UpsThe tests described here after, rely on two specific set-ups:GOL Connection Set-upIn this connection set-up, the communication linkconnects the GOL and the ML320 boards. The transmitteddata rate is either 800 or 1600 Mbits/s depending on theBITS16 switch sitting on the Gol board. If the data rate is800 Mbits/s then the transmitted data should be connectedto the RX connectors for MGT1, otherwise MGT2 RXports should be used.This connection set-up allows bit error rate testing andlatency measurement.Loopback Connection Set-upIn this connection set-up, the GOL board is notrequired. In 800 Mbits/s mode, the TX ports of MGT94 areconnected to the RX ports of MGT9. In 1600 Mbits/smode, MGT45 ports are connected in the same way.Each Rocket IO transmitter is driven by a pattern generatorimplemented in the FPGA logic, the same as the one usedin the GOL board Altera device with the exception thatsome parts of the pattern contain  (intentionally) wrong datawords.In this loopback configuration, the intention is that thesame Rocket IO that transmits it, receives the transmitteddata. This connection set-up is used for two reasons:• It facilitates testing of the bit error measurementlogic and software.• It facilitates the measurement of resynchronisationtime as will be described below.B. Bit-Error Rate TestsBasic Measurement  MechanismA bit error counter (BEC) is implemented in the FPGAfabric. The receiver has the same pattern sequencer as the                                                           4 MGT identification number on LM320 board5 MGT identification number on LM320 boardone used at the transmitter. When the MGT is synchronisedand the receiver’s pattern sequencer is also locked to thereceived data sequence, then the BEC is incremented by X,where X is the number of bit errors within a word. X isobtained by comparing the output word of the receiver’spattern sequencer with the received data word from theMGT. The word width is 16 bit for slow mode and 32 bitfor fast mode.The GOL device remains in lock but consecutive sequences“S” are separated by one or more IDLE words. The biterror calculation is made over multiple sequences for a pre-specified time interval. The bit errors are only counted ifthey occur in a predefined range.To specify a time interval over which BER measurementsare made, there is a hardware timer implemented in theFPGA fabric. This timer is called the BEC Interval Timer.The timer creates a pulse signal periodically at a rate of onepulse per second, for example. This pulse causes the currentresult of the BEC to be stored in a memory-mapped registerand a dedicated timer interrupt to be sent to the processor.In response to the interrupt, the processor reads the BECvalue from the memory-mapped register and streams thevalue to the UART so that it is displayed on the hyper-terminal.It may be possible to make the BEC Interval Timeprogrammable from the hyper-terminal. In addition,software could be used to compute an (running) average biterror count over multiple BEC intervals.Pattern Loss of Lock ConditionsWhile the errors are being counted for a particulartransmitted sequence, it may happen that errors are detectedin each subsequent word or it may happen that theRXNOTINTABLE status bits from the MGT are active onmany consecutive bytes.If the MGT is synchronised (as indicated byRXLOSSOFSYNC), we have detected one of the followingconditions:• a framing error in the MGT where the bytealignment has been lost (indicated by consistenterrors on RXNOTINTABLE)• the pattern detector is not properly synchronisedwith the transmitted sequence, indicated byconsistent mismatch between the receiver’s patternsequence and the received pattern sequence.In either case, the receiver’s pattern sequence is halted andbit errors are no longer recorded. The receiver’s patterndetector must wait for the start of the next transmittedsequence before relocking and continuing the errorcounting process. The loss of lock condition is detectedwhen 8 consecutive bytes are received in error where a byteerror is defined by mismatch or RXNOTINTTABLE.If there are many occurrences of this loss of lock conditionduring a BEC measurement interval then the BECmeasurement may be considered invalid. For this reason,the number pattern detector loss of lock conditions is alsocounted by the hardware and the PowerPC may read thiscount at the end of the measurement interval.ResultsThe GOL connection set-up is used to connect the GOLdata lines to either MGT2 (1600Mbits/s) or MGT1 (800Mbits/s). The bit error rate test was conducted for bothdata-rates. The results are reported in table:Table 1: Bit Error Test ResultsData RateTestDuration(hours)Bit ErrorRateSynchronisation Losses800Mbits/s 17 : 006 0 01600Mbits/s 67 : 007 0 0Bit error rate is computed by the equation BER = B/(D*T)using the following quantities:• The time elapsed T.• The uncoded data rate D i.e. 640Mbits/s for slowmode and 1280Mbits/s for fast mode• The number of bit errors encountered during thetime elapsed i.e. B.AnalysisThe bit error rate testing has shown that a perfectcommunications channel can be established and maintainedover very long periods of time. The good results are likelyto be due to:• The high quality transmission signals coming fromthe GOL device.• The high quality of the ML320 board design interms of power supply network and the tracklayout.• The high quality of the REFCLK clock source onthe ML320 board.The only factor that will have to be significantly differentin the real application will be the transmission quality. Inthe real application, the transmission signals will exhibitconsiderable levels of jitter. At this time, it was not possibleto vary the jitter of the transmitted signals but this isdefinitely a future possibility.C. Link Latency TestsThe MGT receiver latency is of interest. However,there is not easy to measure just the receiver latency. It wasthen measured the end-to-end latency: L_TOTAL. The                                                           6 Equivalent to a BER limit of 6.6 10-14 at 95% of CL7 Equivalent to a BER limit of 8 10-15 at 95% of CLMGT receiver latency L_RX can then be obtained bysubtracting the transmitter latency L_TX and the cablelatency L_CABLE from L_TOTAL:L_RX = L_TOTAL - L_TX - L_CABLE.L_TX is obtained from reference[1], L_CABLE isgiven by the cable length and L_TOTAL is obtained by thefollowing way: When the Altera device provide the value FFFF to theGOL device for transmission it can produce a TX_STARTsignal, then when the receiver FPGA device detects thisFFFF, it can produce a RX_START signal. TheTX_START and RX_START signals are then probed by anoscilloscope and the time between them can be measured togive L_TOTAL. Repeated manual measurements have beenmade to determine the latency and study its variations.Alternatively, it may be possible to route theTX_START signal from the GOL board to the V2PROboard and the FPGA can then be used to make repeatedautomatic measurements.ResultsTo execute the measurement, the GOL connection set-up was used. An oscilloscope was used to probe theTX_START signal from the GOL board and theRX_START signal from the ML320 board. The timedifference between the rising edge of TX_START andRX_START has been  measuredThe results are quoted in Table 2. The time differencebetween TX_START and RX_START was found to bevariable but bounded below and above. The table quotesthe minimum and maximum time difference observed.Table 2: Latency ResultsData Rate Minimum (ns) Maximum (ns)800Mbits/s 732 7631600Mbits/s 414.4 434.2A detailed analysis of these results is given in reference[4].For comparison table 3 gives the GOL device latency [1].The measured latency for the GOL+TLK is about 5 clockcycles.Table 3: GOL Device Latency FiguresMode Minimum (ns) Maximum (ns)Slow 68 78Fast 54 64D. Resynchronisation Time TestsThe MGT indicates that synchronisation has been lost ifthere is a non-zero value on the RXLOSSOFSYNC[1:0]port. When this MGT port changes from zero to nonzero, ahardware timer is commenced. The timer stops when theMGT port returns to zero. The timer is a counterimplemented in the FPGA fabric. It runs of the 40MHzclock and it is 32 bit wide.When the timer stops, an interrupt goes to the processor.The processor reads the value of the timer and resets it. Thevalue read is the recovery time of the MGT and this isdisplayed on the hyper-terminal.In order to obtain this measurement, a mechanism isrequired so that the MGT is forced into a desynchronisedstate. Some ideas are as follows:• The processor will write to a register and thisregister will cause the MGT REFCLK to bedisabled. When the RXLOSSOFSYNC portchanges to non-zero, the REFCLK is re-enabled sothat resynchronisation becomes possible again.• The Agilent clock source frequency can beadjusted until the MGT is forced into adesynchronised state. The problem with this is thatthe user adjusting the frequency does not knowwhen exactly to restore the clock source to thenominal frequency in order to allowresynchronisation to occur.Because this is an unusual application, it was necessary toconsult internal Xilinx MGT experts for advice on this(Sean Koontz).1) DescriptionThe loopback connection set-up is used for this test.The transmitter data lines of the Rocket IO are routed backinto the receiver of the same Rocket IO. From the hyper-terminal user interface, the user can force the transmitter toinhibit the transmitter, causing the transmission line to beheld in a fixed state for a short period of time. This causesthe receiver to loss synchronisation. As soon as thetransmitter becomes uninhibited, a timer initialises andstarts counting. The timer increments until the receiverresynchronises. When this occurs, the timer stops countingand the value of the timer, i.e. the resynchronisation time isreported to the hyper-terminal. This process can be repeatedad infinitum and different results should be obtained eachtime. The maximum resynchronisation time is bounded bythe interval between transmitted comma characters.2) ResultsA number8 of measurements were made over time inboth fast and slow mode. For each measurement, thereceiver was made to loose synchronisation and theresynchronisation timer value was reported to the hyper-terminal. A different result was obtained for eachmeasurement. In addition, each result was in the range[321,30771] where the units are in 40MHz clock cycles.                                                           8 Approximately 50 measurements were made on eachmode.3) AnalysisThe results were as expected. The resynchronisationtime is dependent on where in the transmitted frame the de-synchronisation occurs. Thus, if the point of de-synchronisation is random, then the resynchronisation timeis random. In addition, since resynchronisation must waitfor a COMMA character (i.e. the start of the next frame) toarrive at the receiver, then the re synchronisation time mustbe bounded by the interval between COMMA characters.The COMMA character interval (frame size) is 32777clock cycles for both slow and fast mode. Thus, it would beexpected that the results be bounded in the range [0,32777].V. CONCLUSIONIntegration of deserialisers in FPGA devices is a keyfactor to reduce the global cost of the off-detectorelectronic for LHC experiments. The fast development ofindustrial standard chips (FPGA) for the telecom market isa good opportunity to try to use them for HEP applications.The first results of the projects reported in this paperdemonstrate that the GOL-MGT is a good combination formany LHC experiment requirements. Our future work inthis domain will be to find specific solutions, always basedon the use of FPGA fabrics and their embedded cores.Moreover, we think that it will be useful to perform aprecise characterisation of the clock jitter and the integrityof the Giga-bit data link.VI. REFERENCES[1] GOL Reference Manual, v1.2, May 2002, CERNMicroelectronics Group[2] Rocket I/O Transceiver User Guide, UG024, v1.3,June 2002, Xilinx[3] Requirements Specification (rs01-2250-0001),v1.3, Xilinx Design Services[4] Test Report (tr01-2250-0001), v1.3, Xilinx DesignServices

Embedding deserialisation of LHC experimental data inside field programmable gate arrays

Embedding deserialisation of LHC experimental data inside field programmable gate arrays

Abstract

Similar works

Full text

Available Versions

CERN Document Server