Free Electron Laser in Hamburg (FLASH) and XRay Free Electron Laser (XFEL) are linear accelerators that require a complex and accurate Low Level Radio Frequency (LLRF) control system. Currently working systems are based on aged Versa Module Eurocard (VME) architecture. One of the alternatives for the VME bus is the Advanced Telecommunications and Computing Architecture (ATCA) standard. The ATCA based LLRF controller mainly consists of a few ATCA carrier boards and several Advanced Mezzanine Cards (AMC). AMC modules are available in variety of functions such as: ADC, DAC, data storage, data links and even CPU cards. This paper focuses on the software that allows user to collect and plot the data from commercially available TAMC900 board

Butkowski, L

Jablonski, G

Makowski, D

Szachowalow, S

CERN Document Server

EuCARD-CON-2010-012European Coordination for Accelerator Research and DevelopmentPUBLICATIONSoftware for Data Acquisition AMCModule with PCI Express InterfaceSzachowalow, S (Technical University of Lodz) et al04 February 2010The research leading to these results has received funding from the European Commissionunder the FP7 Research Infrastructures project EuCARD, grant agreement no. 227579.This work is part of EuCARDWork Package 10: SC RF technology for higher intensityproton accelerators and higher energy electron linacs.The electronic version of this EuCARD Publication is available via the EuCARD web site<http://cern.ch/eucard> or on the CERN Document Server at the following URL :<http://cdsweb.cern.ch/record/1237834EuCARD-CON-2010-012Software for Data Acquisition AMC Module with PCI Express Interface Sergiusz Szachowalow, Grzegorz Jablonski, Dariusz Makowski Department of Microelectronics and Computer Science Technical University of Lodz Lodz, Poland Lukasz Butkowski Institute of Electronic Systems Warsaw University of Technology Warsaw, Poland Waldemar Koprek, Stefan Simrock Deutsches Elektronen-Synchrotron Hamburg, Germany   Abstract— Free Electron Laser in Hamburg (FLASH) and X-Ray Free Electron Laser (XFEL) are linear accelerators that require a complex and accurate Low Level Radio Frequency (LLRF) control system. Currently working systems are based on aged Versa Module Eurocard (VME) architecture. One of the alternatives for the VME bus is the Advanced Telecommunications and Computing Architecture (ATCA) standard. The ATCA based LLRF controller mainly consists of a few ATCA carrier boards and several Advanced Mezzanine Cards (AMC). AMC modules are available in variety of functions such as: ADC, DAC, data storage, data links and even CPU cards. This paper focuses on the software that allows user to collect and plot the data from commercially available TAMC900 board. Index Terms— Linear accelerator, Advanced Mezzanine Card, Analog to Digital Converter, PCI Express. I. INTRODUCTION Linear accelerators such as FLASH (Free Electron Laser in Hamburg) or XFEL (X-Ray Free Electron Laser) need powerful and reliable digital Low Level Radio Frequency (LLRF) system. Stabilization of electromagnetic field in accelerating modules is done by a real-time soft controller with a digital fast feedback and adaptive feed-forward.   Figure 1.  Block diagram of LLRF controller The digital control system is implemented in SimCon 3.1L, which is based on the Versa Module Eurocard (VME) architecture. The VMEbus specification was introduced in 1981 and has been updated a few times since that time. Beside the digital interface used for acquisition purposes the LLRF controller also needs several custom-defined analogue signals. The VME architecture does not offer any backplane connections that can be easily adopted as user-defined signals therefore all analogue connections are routed in front panel via coaxial cables. The Advanced Telecommunication Computing Architecture (ATCA) has been chosen as the next standard for LLRF controller. The fully functional ATCA system is composed of a crate with Shelf Manager (ShM), carrier board with bays for Advanced Mezzanine Cards (AMCs) and a Rear Transition Module (RTM) which is connected in the back of the carrier board. The block diagram of the distributed LLRF controller based on ATCA architecture is presented in Fig. 2. The ATCA carrier board (blade) provides three types of connectors in the backplane [2]. Z1 (Zone-1) connector provides necessary voltages and signals from shelf manager, Z2 (Zone-2) provides connection to the base and fabric interface and finally Z3 (Zone-3) connector provides signals that are unspecified in the standard. The Z3 connector is used in a custom developed carrier-board at DESY as a source of analogue signals for AMC modules.   Figure 2.  Block diagram of ATCA based controller   The acquisition of signals that come from cavities is done using a TEWS TAMC900 module [3]. All necessary signals for its ADCs are routed from the backplane via Z3 connector through an AMC-A PCB and RTM module. In near future AMC-A PCB and RTM will provide all necessary analogue signals for ATCA based LLRF controller (e.g. external clock signals, external triggers and signals for ADCs). Development of the hardware fitted for analogue signals distribution requires that simple software for data acquisition is provided. II. TEWS TAMC900 BOARD The TAMC900 module is a high speed analog-to-digital converter with the AMC form factor. It consists of 8 fast LT2254 ADCs, powerful Virtex-5 FPGA and 36 Mb of QDR-II SRAM. Data from ADCs can be transmitted to the CPU by up to PCI Express x8 link. The configuration of the on-board peripherals is maintained by XC95144XL CPLD. The block diagram of the TAMC900 board is shown in Fig. 3. The TAMC900 provides three clock inputs and three trigger inputs. The three external clock inputs and the PCI Express reference clock are routed to a flexible clocking scheme that allows independent clocking of the ADCs in two groups. The trigger inputs are routed directly to the FPGA. Detailed board features are presented below. A. On-board Clocks The TAMC900 board has several clock sources: three external differential clocks, one 100 MHz clock from AMC connector, one local 250 MHz clock and 2 clock outputs from FPGA.  Figure 3.  Block diagram of TAMC900  Figure 4.  TAMC900 clocks distribution The clocks are distributed to the ADCs and the FPGA. Clock configuration is driven by on-board CPLD. Clocks routing is shown on the Fig. 4. B. ADCs The TAMC900 board includes eight LTC2254 ADCs. The LTC2254 are 14-bit 105 Msps, designed for digitizing high frequency, wide dynamic range signals. Operating frequency of the LCT2254 ADC is ranged from 1 Msps to 105 Msps. The input voltage range of the ADCs is set to ±1V (2V peak-to-peak) by default. Configuration of the operation mode is accessible via the CPLD’s registers. C. CPLD The on-board Xilinx X95144XL CPLD chip is used to configure all board peripherals. All functions of the CPLD are accessible to the user FPGA via an easy to use interface. The clock is driven by the CPLD. The timing of FPGA-CPLD interface is shown on Fig. 5. The most important CPLD registers are listed in Tab. I. D. Samtec Connector All analog signals, external clocks and triggers are routed to the board through 120 pin Samtec connector [3]. The TAMC900 board ships with TAMC900-A1 which is a simple Signal Conditioning Adapter (SiCA) which allows the user to provide signals to the board from the front panel. This particular board has been changed to a custom designed PCB that provides analogue signals to the TAMC900 through AMC-A connector. Conditioning of signals is performed on the RTM module connected to Z3 connector on carrier board.  Figure 5.  FPGA to CPLD interface  TABLE I.  CPLD CONTROL REGISTERS CPLD address (hex) Description Reg. access Value after reset 0x00 – 0x07 ADC0 - ADC7 R/W 0x02 0x10 Crosspoint Switch R/W 0x00 0x11 Clock Mux R/W 0x00 0x12 Jitter Attenuator R/W 0x00 0x18 General Board R/W 0x00 0x1F Revision R 0x42   III. DESCRIPTION OF APPLICATION A simplified firmware block diagram is shown in Fig. 6. The firmware allows user to store 32 kilosamples from each ADC. Data acquisition is done in a multiplexed mode with an acknowledgement flag. After setting active ADC with ADC_SELECT register and clearing the BIT_STATUS flag, application waits for an external trigger to start filling up memory (implemented as Block RAM in FPGA) with samples from chosen ADC. After the memory is filled up the BIT_STATUS is set to ‘1’. This informs user that newly collected data is ready to be read. User can easily change board configuration by accessing the CPLD_AREA. Board is initially configured to route external clock one (EXT_CLK1) to ADCs and FPGA, turn on every ADC and set its data format to binary offset representation. All memories and registers are available through Integral Interface (II) and have been implemented with iidoocsgen application [4]. User can access the data through PCI Express interface via II to PCIE bridge [5]. Registers and areas are described in Table II. User can change default acquisition clock with CLK_SEL register. Selected acquisition clock is connected to CLKOUT0 and CLKOUT1 outputs of FPGA and can be used as clock for ADCs.  Figure 6.  Block diagram of TAMC900 firmware TABLE II.  TAMC900 FIRMWARE AREAS AND REGISTERS Register/area name Access Description AREA_DST0 R Samples acquired from selected ADC BIT_STATUS R/W Acquisition start command/ready flag ADC_SELECT W Number of active ADC TRIG_EDGE W Trigger’s active edge selection - ‘0’ – rising, ‘1’ – falling CLK_SEL W Acquisition clock selection – 83.3 MHz, CLKOUT0, CLKOUT1 or LVDS_8 AREA_CPLD R/W Area of CPLD registers  The VHDL project is complemented with all necessary applications needed to visualize acquired data from ADCs (PCIE driver, scripts for storing the acquired data to files, Matlab script for plotting, etc.). IV. MEASUREMENTS AND RESULTS The tests have been done in laboratory, where input signals were sine waves generated with laboratory generator and in Accelerator Module 6 (ACC6), where signals from eight cavities have been provided (probe signals) A. Laboratory Measurements The block diagram of the test setup is presented in Fig. 7. Main frequency (1.3 GHz) comes from Master Oscillator (MO). This frequency is synchronized with 1.354 GHz frequency needed by down-converter placed on RTM in the Local Oscillator (LO) Generation Box. The box also produces 81 MHz reference clock for the TAMC900 board. The clock can be connected to the board in two different ways. Firstly, it can be routed directly to the LVDS_8 link on TAMC900 board via single ended differential signal transformer. Secondly, it can be connected to external clock source one (EXT_CLK1) via single ended Low Voltage Differential Signaling (LVDS) adapter. Because there is no real cavity signal in the lab, ADCs measure 54 MHz sinusoidal signal that is produced by down converter sourced by MO (via splitter). The TAMC900 board communicates with a PC (which act as root complex) through a PCI Express switch PEX8532 and external PCI Express connection. Photo of the laboratory test setup is shown in Fig. 8. Test results are presented in Tab. III. Indexes with “S” refer to sine wave measurements and indexes with “N” refer to ADCs noise measurements. The noise signal has been measured with no signal provided and terminated ADCs inputs. The ratio of measured signal frequency to acquisition frequency is 1:1.5, that gives three samples for every two sine wave periods. When signals are perfectly synchronized, such condition appears on figures as three parallel lines (graph ought to be cropped as a dots). Example measurements for ADC0 are shown in Fig. 9. Inset shows part where acquisition clock jitter has the biggest influence on accuracy of measured signal (biggest sine wave inclination).  Figure 7.  Block diagram of laboratory test setup  Figure 8.  Photo of laboratory test setup     Figure 9.  Example ADC0 laboratory readout  Figure 10.  Block diagram of ACC6 test setup TABLE III.  LABORATORY TEST RESULTS ADC RMSN [mV] RMSS [V] SNR [dB] VNp-p [mV] VSp-p [V] 0 4.5 0.574 42 19 1.41 1 4.9 0.572 41 12 1.4 2 2 0.575 49 4.6 1.38 3 4.7 0.579 42 4.8 1.42 4 1.2 0.551 53 5.9 1.23 5 0.6 0.001 0.5 5 0.01 6 0.4 0.011 28 2.6 0.03 7 13 0.311 28 39 0.75 TABLE IV.  ACC6 TEST RESULTS ADC RMSN [mV] RMSS [V] SNR [dB] VNp-p [mV] VSp-p [V] 0 5.2 0.44 39 15 1.02 1 2.9 0.43 44 505 1.05 2 1.8 0.54 49 2.9 1.28 3 4.6 0.51 41 5.2 1.59 4 0.8 0.38 54 4.3 0.93 5 0.3 0.41 62 3.5 0.96 6 0.3 0.44 63 2.8 1.1 7 26 0.26 20 41 0.60 B. ACC6 Measurements The block diagram of the test setup used in the ACC6 is shown in Fig. 10. The main difference is that signals measured by ADCs come from cavity as opposed to coming from a generator. The trigger signal is connected to the board. It comes from accelerator trigger infrastructure and is routed to TAMC900 board through a simple resistor divider that converts TTL to 2.5V level. The 81MHz clock is connected directly to the board from the VME timing module (where it is converted to LVDS standard). Tests have been performed with a different firmware delivered by DESY. Results are presented in Tab. IV. V. CONCLUSION The ATCA is a very flexible standard and it can easily fulfil demands of such complex systems as linear accelerators. The major drawback of this architecture is the complexity of used communication interfaces. Real-time events require usage of low latency links. In such case connection transfer-rates are not as important as fast response to requests. Completion of non-real-time events can be performed with higher latency links. In such cases the transfer rate of connection is more important than fast response to request. The main advantage of ATCA standard is its scalability - system can be easily expanded. The Z3 connector on carrier blade simplifies the distribution of analogue signals needed in the control system. Proposed software offers the ability to acquire data from ADCs on TAMC900 board through PCI Express interface.  Tests done with two different version of firmware and under two different conditions gave quite similar and comparable results. The signal-to-noise ratio of acquired signals shows the influence of acquisition clock jitter on accuracy of measured signals. High clocks frequency demand that impedance of source, line and load is perfectly matched. This fact should be considered during the design process of a new version of AMC-A and RTM PCBs. ACKNOWLEDGEMENT The research leading to these results has received funding from the European Commission under the EuCARD FP7 Research Infrastructures grant agreement no. 227579. The author is a scholarship holder of project entitled "Innovative education ..." supported by European Social Fund. REFERENCES [1] D. Makowski, W. Koprek, T. Jezynski, A.Piotrowski, G.Jablonski, S.Simrock “Interfaces and Communication Protocols in ATCA-based LLRF Control Systems”, NSS/MIC 2008. [2] D.Makowski, A.Piotrowski, A.Napieralski, “Universal Communication Module Based on AMC Standard”, 15th International Conference Mixed Design of Integrated Circuits and Systems, MIXDES 2008. [3] “TAMC900 AMC with 8 high Speed ADCs 105MSps, 14Bit”, User Manual. [4] A. Piotrowski, S. Tarnowski, G. Jablonski, A. Napieralski, “Integral Interface - Universal Communication Interface for FPGA-based Projects”, 14th International Conference Mixed Design of Integrated Circuits and Systems, MIXDES 2007. [5] G. Jablonski, D.Makowski, A. Piotrowski, “Low level drivers for PCI express communication (FPGA)”, XFEL Project Communication Links WP 3.43.     

Software for Data Acquisition AMC Module with PCI Express Interface

Abstract

Similar works

Full text

Available Versions

CERN Document Server