Optimization of multi-gigabit transceivers for high speed data communication links in HEP Experiments

The scheme of the data acquisition (DAQ) architecture in High Energy Physics (HEP) experiments consist of data transport from the front-end electronics (FEE) of the online detectors to the readout units (RU), which perform online processing of the data, and then to the data storage for offline analysis. With major upgrades of the Large Hadron Collider (LHC) experiments at CERN, the data transmission rates in the DAQ systems are expected to reach a few TB/sec within the next few years. These high rates are normally associated with the increase in the high-frequency losses, which lead to distortion in the detected signal and degradation of signal integrity. To address this, we have developed an optimization technique of the multi-gigabit transceiver (MGT) and implemented it on the state-of-the-art 20nm Arria-10 FPGA manufactured by Intel Inc. The setup has been validated for three available high-speed data transmission protocols, namely, GBT, TTC-PON and 10 Gbps Ethernet. The improvement in the signal integrity is gauged by two metrics, the Bit Error Rate (BER) and the Eye Diagram. It is observed that the technique improves the signal integrity and reduces BER. The test results and the improvements in the metrics of signal integrity for different link speeds are presented and discussed

Trigger and Timing Distributions using the TTC-PON and GBT Bridge Connection in ALICE for the LHC Run 3 Upgrade

The ALICE experiment at CERN is preparing for a major upgrade for the third phase of data taking run (Run 3), when the high luminosity phase of the Large Hadron Collider (LHC) starts. The increase in the beam luminosity will result in high interaction rate causing the data acquisition rate to exceed 3 TB/sec. In order to acquire data for all the events and to handle the increased data rate, a transition in the readout electronics architecture from the triggered to the trigger-less acquisition mode is required. In this new architecture, a dedicated electronics block called the Common Readout Unit (CRU) is defined to act as a nodal communication point for detector data aggregation and as a distribution point for timing, trigger and control (TTC) information. TTC information in the upgraded triggerless readout architecture uses two asynchronous high-speed serial links connections: the TTC-PON and the GBT. We have carried out a study to evaluate the quality of the embedded timing signals forwarded by the CRU to the connected electronics using the TTC-PON and GBT bridge connection. We have used four performance metrics to characterize the communication bridge: (a)the latency added by the firmware logic, (b)the jitter cleaning effect of the PLL on the timing signal, (c)BER analysis for quantitative measurement of signal quality, and (d)the effect of optical transceivers parameter settings on the signal strength. Reliability study of the bridge connection in maintaining the phase consistency of timing signals is conducted by performing multiple iterations of power on/off cycle, firmware upgrade and reset assertion/de-assertion cycle (PFR cycle). The test results are presented and discussed concerning the performance of the TTC-PON and GBT bridge communication chain using the CRU prototype and its compliance with the ALICE timing requirements

Optimization of multi-gigabit transceivers for high speed data communication links in HEP Experiments

The scheme of the data acquisition (DAQ) architecture in High Energy Physics (HEP) experiments consist of data transport from the front-end electronics (FEE) of the online detectors to the readout units (RU), which perform online processing of the data, and then to the data storage for offline analysis. With major upgrades of the Large Hadron Collider (LHC) experiments at CERN, the data transmission rates in the DAQ systems are expected to reach a few TB/sec within the next few years. These high rates are normally associated with the increase in the high-frequency losses, which lead to distortion in the detected signal and degradation of signal integrity. To address this, we have developed an optimization technique of the multi-gigabit transceiver (MGT) and implemented it on the state-of-the-art 20nm Arria-10 FPGA manufactured by Intel Inc. The setup has been validated for three available high-speed data transmission protocols, namely, GBT, TTC-PON and 10 Gbps Ethernet. The improvement in the signal integrity is gauged by two metrics, the Bit Error Rate (BER) and the Eye Diagram. It is observed that the technique improves the signal integrity and reduces BER. The test results and the improvements in the metrics of signal integrity for different link speeds are presented and discussed

A flexible readout board for HEP experiments

This thesis will present my contributions to the development of the PiLUP board along with a general overview of its features and capabilities. The PiLUP board is a general-purpose FPGA-based readout board for data acquisition systems under development by the University of Bologna and the Instituto Nazionale Fisica Nucleare (INFN) and intended for high energy physics experiments, where the sheer amount of data generated by detectors often requires custom hardware solutions. This board was initially proposed for the next upgrade of the ATLAS Pixel detector. In this context its purpose would be to interface the Front-End readout chip RD53A with the FELIX card and provide advanced testing features such as an emulator for the RD53A that will help the development of the other parts of the data acquisition chain. Nonetheless, since the early stages of development, the hardware has been designed to offer great flexibility so that the same hardware platform could be directly used in other applications. To this purpose an important feature of the board is the great extendibility offered by the presence of different interfaces, such as and 3 FMC connectors (two low density and one high density), a PCI Express x8 interface, gigabit ethernet and an integrated SFP connector. The computing power of the PiLUP is provided by of two FPGAs, a Zynq-7 SoC and a Kintex-7 produced by Xilinx, intended to be used in master-slave configuration. In this case the Zynq, with its dual-core ARM processor and the possibility to run an embedded linux distribution, would be used as main interface with the other functionalities in the board. The main objective of this thesis is the development of such software and firmware control infrastructure, starting from the firmware solutions for the inter-FPGA communication to the low-level software to control the system

Characterizing Single Event Upsets within the lpGBT-based End-of-Substructure Card

The CERN ATLAS particle physics experiment is currently undergoing a significant system upgrade (ATLAS Phase II upgrade). As a result of the upgrade the experiment's Inner Tracker (ITk) and the front-end electronics of the ITk are being redesigned to handle increased data rates and a higher radiation environment. Within the ITk, the End Of Substructure (EoS) card is a new custom designed digital board that will provide the data, command, and power interface between on and off-detector electronics. Each EoS card makes use of one or two custom CERN designed low power Gigabit Transceivers (lpGBTs) ASICS that have been created for the purposes of supporting high bandwidth optical links in high radiation environments throughout CERN experiments. An estimated 1552 EoS cards will be installed in the ITk, each representing a potential point of failure. Given the complexity and quantity of new hardware designs involved, and that the EoS cards will be not be accessible or serviceable after the upgrade has been completed, there is a need for rigorous quality assurance (QA) and quality control (QC) testing. This thesis therefore describes an independent test setup commissioned, by the author, at the University of Cape Town (UCT) Physics Department for characterising aspects of EoS card's operation under representative radiation conditions. Specifically, the radiation environment of the ITk poses a challenge to electronics as energetic particles can deposit their energy within the circuit material resulting in an erroneous change in logic known as a Single Event Upset (SEU). The lpGBT is a radiation tolerant ASIC and employs digital signal processing (DSP) and triple modular redundancy (TMR) techniques to mitigate against the effects of SEUs on transmitted data. This thesis presents an experiment setup which tests this hypothesis that the DSP stages are susceptible to data corruption caused by SEUs. In addition the setup also attempts to characterize the susceptibility of the scrambler, encoder, and interleaver stages within the lpGBT to SEUs. This experiment is carried out by actively irradiating an EoS card with a neutron source (energy spectrum of up to 11 MeV), while emulating each stage on a non-irradiated off-board FPGA. Additionally and in support of this experiment, the existing firmware and LabView automation software developed at DESY are extended. Results from this thesis indicate that the DSP stages within the lpGBT are susceptible to data corruption caused by SEUs. It was also shown that the susceptibility of the experiment itself did not effect the measured SEU rates. Finally, preliminary results suggest that susceptibility of the DSP stages within the lpGBT can be characterized as the Bit Error Rate (BER) increases depending on the number of active stages

Investigating the Optical Link Performance of the End-of Substructure Card and Susceptibility to SEUs

Particle physics experiments carried out by CERN attempt to investigate the fundamental forces of matter. One of these experiments is the ATLAS experiment, which studies the proton-proton collisions in the LHC. A series of upgrades are planned to increase the luminosity by a factor of five, leading to the high-luminosity LHC (HL-LHC). This upgrade will increase the potential for new discoveries but brings with it design challenges in relation to the harsh radiation environment and significant data throughput required. The ATLAS experiment is building a new detector to cope with these challenges, titled the Inner Tracker (ITk). A crucial part of this new detector is the End-of-Substructure (EoS) card, which constitutes the interface between the ondetector electronics and the off-detector systems. In addition to the operational challenges, the HL-LHC does not allow for repairs or replacing of EoS cards once operation commences, emphasizing the need for thorough testing and qualification of this component. This thesis focuses on characterizing the performance of the EoS card in the presence of radiation, under non-ideal operating conditions and the impact of optical link parameters. The first set of tests is centered on qualifying the radiation tolerance of the EoS card. The radiation environment within the ITk poses a threat to the stable operation of electronics as energetic particles have the potential to cause erroneous changes in device logic, known as Single Event Upsets (SEU). The SEU susceptibility of the EoS card, with a focus on the Versatile Link Plus Transceiver (VTRx+) component, is studied by irradiating the EoS card with a neutron source with a distributed energy spectrum and a peak energy of 11MeV while performing a bit error rate (BER) test to monitor for radiation induced errors. The second set of tests deals with characterizing the impact of an irregular power supply on the EoS card's performance through simulating noise on the supply lines and monitoring the response in BER. The final set of tests investigates the impact the VTRx+ configuration parameters have on the quality of the optical signal. These tests were carried out at the University of Cape Town (UCT) with the support of DESY, a national research institute in Germany, responsible for the production of the EoS cards. A number of new firmware, software and hardware modules were developed as part of this work in order to carry out the tests required. The most significant of which comprised a novel firmware addition allowing for the evaluation of the optical signal quality with an FPGA. This contribution is now being integrated into the quality control proceedings at DESY, to be used in assessing optical signal quality of the entire set of approximately 1552 EoS cards being produced

Contributions to Phase Two of AGATA electronics

En el campo de la física nuclear, la espectroscopia de rayos gamma de alta resolución es un método preciso para estudiar la estructura del núcleo, extrayendo la energía y la distribución angular de los fotones gamma emitidos en las transiciones entre estados nucleares. Para obtener núcleos en un estado excitado y por tanto emitan rayos gamma, hemos de hacer chocar la materia, produciendo reacciones nucleares (espectroscopia de haz) o recurrir a desintegraciones radiactivas (espectroscopia de desintegración). Los detectores de semiconductor de germanio de alta pureza (HPGe) han demostrado tener una buena respuesta interaccionando con rayos gamma. Al igual que otros detectores de basados en semiconductores, cuando se los somete a alto voltaje, los detectores HPGe producen una alta corriente de medida proporcional a la energía de los rayos gamma incidentes. El multi-detector HPGe AGATA (Advanced GAmma Tracking Array) es uno de los espectrómetros gamma de alta resolución más avanzados que existen dedicado al estudio de la física nuclear. Para maximizar la sensibilidad, los detectores HPGe de AGATA tienen los contactos exteriores divididos en 36 segmentos, de este modo se puede determinar la posición del fotón y la energía depositada en cada una de estas partes. Con la información sobre la posición y la energía de los fotones es posible reconstruir las interacciones de los rayos gamma a través de los algoritmos de tracking. Gracias a esta técnica, es posible maximizar la sensibilidad del detector (resolución energética y factor P/T) sin necesidad de utilizar parte del ángulo sólido de detección para otros detectores dedicados a la supresión del efecto Compton. Además de los detectores mismos, los detectores de HPGe sensibles al posicionamiento requieren una electrónica de muestreo con ratios señal a ruido de calidad espectroscópica, que capturen y digitalicen las trazas para ser procesadas por los algoritmos de análisis de forma de pulso (Pulse Shape Analysis). Para conseguir la máxima sensibilidad y eficiencia, el proyecto AGATA busca construir el multi-detector cubriendo una superficie total con 4π de ángulo sólido, optimizando la información obtenida, algo especialmente crítico en experimentos que usan costosos haces de iones radiactivos. Otro objetivo en la construcción de AGATA es su movilidad. El multi-detector AGATA se instala en diferentes laboratorios para aprovechar la variedad de haces e instrumentación complementaria que existen en los diferentes centros europeos. El proyecto AGATA se encuentra actualmente en su Fase 1, que busca cubrir hasta 1π de ángulo sólido y se encuentra funcionando con la segunda generación de electrónica. Los 45 detectores instalados actualmente utilizan en parte la anterior generación o Fase 0 de electrónica, que fue diseñada y producida entre 2005 y 2007. El principal objetivo a nivel de electrónica en la colaboración AGATA es el desarrollo de la nueva generación para la Fase 2, que busca instrumentar 180 detectores y la cual se ha desarrollado parcialmente en esta tesis. Los principales objetivos de la electrónica para la Fase 2 son la integración de en un solo dispositivo, desde la digitalización hasta la salida de datos y el protocolo Ethernet como comunicación para dicha salida. La tecnología Ethernet permitirá una conexión multipunto y la posibilidad de leer los datos desde cualquier sitio de la granja de procesado de AGATA. También se han tenido en cuenta, en el diseño, facilitar el mantenimiento y evitar la obsolescencia de los componentes utilizados. Uno de los grandes problemas que se encuentran en la integración del sistema electrónico de AGATA es la optimización de los recursos en la FPGA por parte del Pre-procesado. Con el avance de la tecnología, a pesar del aumento de la tasa de datos por transceptores de alta velocidad en estos dispositivos (entre 16 y 32 Gbps), el número de transceptores en las FPGAs no se ha incrementado sustancialmente. Además, el coste de los dispositivos FPGA aumenta considerablemente con el número de transceptores. Esto es un problema crítico en AGATA, ya que requiere un gran número de canales digitalizados por dispositivo, pero no a una velocidad especialmente alta (sobre 2 Gbps). Para reducir la complejidad del sistema, el coste y la potencia total, el número de líneas de alta velocidad se ha optimizado mediante agregación de datos por multiplexado en tiempo, incrementando la velocidad de tasa de datos, pero con una reducción en el número total de éstas de 4 a 1. Esta solución se ha llevado a cabo a través de la tarjeta Input Data Mezzanine, concebida y desarrollada enteramente en esta tesis. El objetivo principal desde el punto de vista científico es demostrar la posibilidad de leer 40 canales bajo el protocolo JESD204 o uno equivalente, vía fibra óptica o por cable físico, únicamente con 10 transceptores de alta velocidad de una FPGA, gracias a la técnica de multiplexado por división en el tiempo. La base de la que se parte es la electrónica actual de AGATA y se apoya en tecnología del estado del arte sobre diseño hardware y software para FPGA, diseño digital de alta velocidad y comunicaciones digitales. A pesar de que este diseño se ha realizado principalmente para el proyecto AGATA, consideramos que esta tecnología será de interés para otros instrumentos y aplicaciones.In the field of Nuclear Physics, high-resolution gamma ray spectroscopy is an accurate method to perform nuclear structure studies, retrieving the energy and angular distributions from gamma photons emitted in the transition between nuclear states. In order to obtain the nucleus in an excited state, such that will emit gamma-rays, we are forced to collide matter, doing nuclear reactions (in the in-beam spectroscopy) or resort to the radioactive decay (decay spectroscopy). The High Purity Germanium (HPGe) semiconductor detectors have shown to provide good response as gamma-ray detector. As other semiconductor detectors, HPGe produce, with high sensitiveness, a current proportional to gamma ray energies while there are subject to high voltage inverse polarization, in cryogenic conditions. The AGATA (Advanced GAmma Tracking Array) HPGe detector array is a state-of-the-art detector array for the gamma ray spectroscopy technique in nuclear physics. In order to improve the sensitivity, AGATA HPGe detectors have the outer contact divided in 36 segments in order to determine photon position and energy deposited in each segment. With the interaction energy and position information is possible to reconstruct (Track) the gamma-ray interaction sequence using tracking algorithms. With such technique is possible to maximize the sensitivity of the detector array (energy resolution and P/T) without using part of the detection solid angle for the anti-Compton active shields. In addition to the segmented detectors, the positions sensitive HPGe arrays require sampling electronics with spectroscopic signal-to-noise ratios, which provides the traces to be processed by the Pulse Shape Analysis algorithms. To provide maximum efficiency and sensitivity, the AGATA project aims to construct a 4π solid angle detector array. This geometry optimizes as well the information obtained, something that is especially important in experiments using expensive radioactive ion beams. Another goal in the construction of AGATA is the mobility of the array. AGATA is installed in different laboratories to take advantage of the variety of beams and complementary instrumentation existing in different European centres. The AGATA project is currently in its Phase 1, using a second generation electronics, which aims at building a 1 π solid angle coverage. This requires 45 detectors, that today are partly instrumented with the previous Phase 0 electronics, mostly design and produced in the period from 2005 to 2007. Presently, the main goal for the AGATA collaboration, regarding electronics, is the development of the Phase 2 version, with the objective of instrumenting 180 detectors, which is partly done by the work described in this thesis. The main improvements for this Phase 2 electronics are: the integration of all the electronics from digitizers to readout, including Pre-processing, in one standalone system and the use of Ethernet as the readout protocol. The Ethernet technology will enable a multipoint connection and the possibility to distribute the data anywhere within the AGATA processing farm. One of the main problems found in the integration of all the system is the optimization of the FPGA resources used in the Pre-processing. Despite of the increase in the high-speed transceiver data rates of the last FPGA developed in the industry, the number of transceivers on the devices is limited. Furthermore, the FPGA cost increases largely with the amount of transceivers, which is an issue for the AGATA detectors, with a need for a large number of transceivers but not at an especially high data rate. To reduce system complexity, cost and power, the number of high speed digital lines is optimized through data aggregation, increasing the speed data rate of each line but with a reduction of 4 to 1 in the total number of transceiver lines. The solution is carried out through the Input Data Mezzanine board, conceived and developed completely under this thesis work. From a technological point of view, the main objective of the thesis is to prove the possibility of reading up to 40 optical or copper low rate inputs, using JESD204 or equivalent protocol, in the FPGA using only 10 transceivers through a time division multiplexing technique. The work is done with state-of-the-art in hardware-software FPGA design, high-speed digital design and digital communications, as well as with the knowhow of the AGATA current electronics. Although this device is designed for AGATA, we consider that this technology will be of interest for other instruments and applications