4 research outputs found
Development of FPGA-based High-Speed serial links for High Energy Physics Experiments
Get PDFHigh Energy Physics (HEP) experiments generate high volumes of data which need to be transferred over long distance. Then, for data read out, reliable and high-speed links are necessary. Over the years, due to their extreme high bandwidth, serial links (especially optical) have been preferred over the parallel ones. So that, now, high-speed serial links are commonly used in Trigger and Data Acquisition (TDAQ) systems of HEP experiments, not only for data transfer, but also for the distribution of trigger and control systems.
Examples of their wide use can be found at CERN, where each of the four big experiments mounted on the Large Hadron Collider (LHC) uses a huge amount of serial links in its read out system. Again at LHC, the Timing, Trigger and Control system (TTC), which broadcasts the timing signals, from the LHC machine to the experiments, uses optical serial link to distribute signals over kilometers of distance (diameter of LHC is 27 Km). Also for upgrades of LHC, physical layer components and protocol chips (ASIC) have been designed and are now under development: the Versatile Link and the GBT protocol (and ASICs) whose peculiarity relies in their radiation hardness.
This PhD project is intended to respond to the requests of HEP experiments, developing:
- a high-speed self-adapting serial link, which can be easily used in different application fields;
- the serial interface of a read out board in the end-cap region of ATLAS Experiment at LHC;
- the interface board for the barrel read out system of the ATLAS Experiments.
Both the two last projects have required the development of fixed latency, high-speed serial links.
In order to take advantage of flexibility, re-programmability and system integration of SRAM-based Field Programmable Gate Array devices (FPGAs), their serializer-deserializer (SERDES) embedded modules have been chosen for the development of the links. However, as a drawback, FPGA embedded SERDESes are typically designed for applications that do not require a deterministic latenc. Then, an accurate study of their architecture has been necessary, in order to find a configuration and a clocking scheme to guarantee a deterministic transmission delay in data transfers.
The frequency agile, auto-adaptive serial link is capable to analyze the incoming data stream, by scanning the Unit Interval, and to find the highest transmission line rate, according to a given tolerated Bit Error Ratio (BER).
It uses a new feature (RX eye margin analysis) of the RX side of the Xilinx 7 series FPGAs high-speed transceivers (GTX/GTH), in order to measure and display the receiver eye margin after the equalizer.
When the new eye scan functionality is running, an additional sampler is activated in the GTX. It acquires a new sample (Offset Sample), with programmable (horizontal and vertical) offsets from the data sample point (Data Sample) used in standard operation.
An eye scan measurement run is performed by acquiring a large number of Data Samples (which can range from tens of thousands to 1014 or more) and by counting the number of times the Offset Sample has a different value with respect to the Data Sample; the latter number is often called Error Count. The BER at a specific vertical and horizontal offset is given by the ratio between the Error Count and the Sample Count. By repeating the eye scan measurement for each horizontal and vertical offset in the Unit Interval (or in a part of the U.I.) a 2-D BER map can be produced which is usually called Statistical Eye.
The auto-adaptive derail ink is designed around an FPGA-embedded microprocessor, which drives the programmable ports of the GTX, in order to perform a 2-D eye-scan, and takes care of the reconfiguration of the GTX parameters, in order to fully benefit from the available link bandwidth.
Xilinx provides a standalone tool that allows performing the Eye Scan Analysis on the receiver side of the GTX/GTH transceiver, using the MicroBlaze Micro Controller System macro; the toolkit also includes the Eye Scan algorithm (providing the C code). Moreover, Xilinx supplies the hardware sources files for the implementation of a link based on the XAUI protocol, in which the GTXs are arranged in a loopback configuration.
The original contribution of this work consists in the build-up, design and optimization of a full architecture, on top of the basic Xilinx tool, which:
- drives the programmable ports of the GTX in order to modify the line rate of the link;
- runs consecutive eye scans for various line rate;
- analyses the results of the different scans, in order to find the maximum line rate sustainable by the link;
- manages the synchronization between the transmitter and the receiver of the link, that will be needed at each line rate change.
The application can be deployed as a monitoring tool in HEP experiments, in order to remotely monitor a transmission system or detect issues in the serial link physical layer. An application example could be some of the many experiments at Large Hadron Collider (LHC) at CERN, which have been intensively using different serial links, both for transmission of TTC signals and for trigger and data readout.
Besides, this solution could be easily adapted in wide, different frameworks, as it can be used on top of any user’s existing link, as it has no specific requirement about link specification or protocol.
The other two serial interface developed in this project are in the framework of the ATLAS experiment.
ATLAS is one of the four detectors installed on the LHC proton-proton collider built at CERN. It was designed to collide two opposing particle beams at an energy of 14 TeV and to reach a luminosity of 1034 cm-2/s. In order to reach the design parameters, the LHC system will be upgraded in several phases. In order to take advantage of the improved LHC operation, the ATLAS detector must be upgraded following the same schedule as the LHC upgrade.
The main focus of the Phase-I ATLAS upgrade (to be completed by 2018) is on the Level-1 trigger where upgrades are planned for both the muon and the calorimeter trigger systems.
In particular, for the end-cap region of the muon spectrometer, the installation of a new set of precision tracking and trigger detectors was approved, called the ‘New Small Wheels’ (NSW). It will be instrumented with micro-mesh gaseous structure detectors (MM) and small-strip Thin Gap Chambers (sTGC). These detectors will solve two points of particular importance at high luminosity: high rate of fake high-pt level-1 muon triggers, and high L1 muon rate with the current momentum threshold. With the introduction of new detectors, new electronics need to be developed, in particular new trigger electronics for both the MM and sTGC. I was involved in the development of serial interface of the FPGA-based sTGC trigger board that uses information from the coarse sTGC readout pads.
The sTGC pad trigger board receives serial information coming from 24 front-end chips at 4.8 Gb/s. On the board, data are deserialised, aligned and analyzed by the trigger algorithm. The trigger logic processes the data and choses two candidates at each Bunch Crossing. The result is then serialised and used for selective fine-grained strip readout. I developed the pad trigger board interface logic. The data format from the front-end chips has been agreed upon, and defines the requirements on the receiver and decoding logic. The number of output lines is 24 and the data are 8B/10B formatted. While the receiver uses the Xilinx Kintex-7 GTX transceivers, the output lines are driven by double data rate (DDR) shift registers at 640 Mb/s. A fixed latency in the sTGC trigger chain was guaranteed through the implementation and configuration of all serialisers and deserialisers. In order to test the project, I also developed a simple microprocessor-based protocol for accessing the board via terminal (rs232). A demonstrator board is now being developed.
Another Phase-I Level-1 trigger upgrade consists of a new Muon to Central Trigger Processor Interface (MUCTPI). The MUCTPI receives muon candidate information from each of the muon detectors, selects muon candidates and sends them to the Central Trigger Processor (CTP). In the first runs of ATLAS, the L1 Barrel trigger candidate data were transferred to the MuCTPI via copper cables. In order to cope with the trigger upgrade, serial optical links are necessary. The optical links will provide a much higher bandwidth (up to 6.4 Gb/s) which will be used to transfer additional information from the sector logic modules, for example data for more than two muon candidates. They will also provide a lower transmission latency. I developed the interface board between the new MUCTPI and the Resistive Plate Chambers (RPC) muon trigger, using the Xilinx Artix-7 FPGA GTP transceivers. I took care of the study of feasibility of the new serial optical transmitter and the logic for the new data format. Also in this case, the fixed latency has been a requirement to be fulfilled
