The Detector Data Link (DDL) is the high-speed optical link for the ALICE experiment. This link shall transfer the data coming from the detectors at 100 MB/s rate. The main components of the link have been developed: the destination Interface Unit (DIU), the Source Interface Unit (SIU) and the Read-out Receiver Card (RORC). The first RORC version is based on the VME bus. The performance tests show that the maximum VME bandwidth could be reached. Meanwhile the PCI bus became very popular and is used in many platforms. The development of a PCI-based version has been started. The document describes the prototype version in three sections. An overview explains the main purpose of the card: to provide an interface between the DDL and the PCI bus. Acting as a 32bit/33MHz PCI master the card is able to write or read directly to or from the system memory from or to the DDL, respectively. Beside these functions the card can also be used as an autonomous data generator. The card has been designed to be well adapted to applications, which require small software overhead such the high-speed data acquisition systems. The implementation of the firmware will be presented. For the logic design we are using VHDL and schematic draw. Software library routines were written in C and are available on Linux OS. The results of performance measurements will be available to allow the comparison between the VME-RORC and PCI-RORC. In the conclusion the future plans and the idea of the improved (64bit/66MHz) PCI-RORC will be shown

Carena, W

Csató, P

Divià, R

Dénes, E

Schossmaier, K

Soós, C

Sulyán, J

Van de Vyvre, P

Vascotto, Alessandro

CERN Document Server

EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCHEuropean Laboratory for Particle PhysicsPCI Based Read-out Receiver Cardin the ALICE DAQ SystemPublicationALICE reference numberALICE-PUB-2001-47 version 1.0Institute reference numberDate of last change2001-10-18Authors:W. Carena, P. Csató, E. Dénes, R. Divià, K. Schossmaier, C. Soós, J. Sulyán,A. Vascotto, P. Vande Vyvrefor the ALICE CollaborationPCI Based Read-out Receiver Cardin the ALICE DAQ SystemW. Carena (1), P. Csato´ (2), E. De´nes (2),R. Divia` (1), K. Schossmaier (1), C. Soo´s (1),J. Sulya´n (2), A. Vascotto (1), P. Vande Vyvre (1)for the ALICE CollaborationCERN EP-AID (1), CH-1211 Geneva 23, SwitzerlandKFKI-RMKI (2), H-1121 Budapest, HungaryAbstractThe Detector Data Link (DDL) is the high-speed optical link for the ALICEexperiment. This link shall transfer the data coming from the detectors at100 MB/s rate. The main components of the link have been developed: thedestination Interface Unit (DIU), the Source Interface Unit (SIU) and theRead-out Receiver Card (RORC). The first RORC version is based on theVME bus. The performance tests show that the maximum VME bandwidthcould be reached. Meanwhile the PCI bus became very popular and is used inmany platforms. The development of a PCI-based version has been started.The document describes the prototype version in three sections. An overviewexplains the main purpose of the card: to provide an interface between theDDL and the PCI bus. Acting as a 32bit/33MHz PCI master the card isable to write or read directly to or from the system memory from or to theDDL, respectively. Beside these functions the card can also be used as anautonomous data generator. The card has been designed to be well adapted toapplications, which require small software overhead such the high-speed dataacquisition systems. The implementation of the firmware will be presented.For the logic design we are using VHDL and schematic draw. Software libraryroutines were written in C and are available on Linux OS. The results ofperformance measurements will be available to allow the comparison betweenthe VME-RORC and PCI-RORC. In the conclusion the future plans and theidea of the improved (64bit/66MHz) PCI-RORC will be shown.1 OverviewThe Detector Data Link (DDL) is an integral part of the ALICE [1] data acquisitionsystem (DAQ). The DDL shall provide duplex data transfer between the detectorsand Local Data Concentrators (LDC) at up to 100 MB/s speed. The DDL consistsof three main parts, the Source Interface Unit (SIU), the optical medium and theDestination Interface Unit (DIU). This naming convention is based on the fact thatthe main data ﬂow is coming from the detectors. Accordingly the SIU card is sittingon the detector side connected to the Front-end Electronics Cards (FEE) and theDIU is plugged in the LDC computers. A Read-out Receiver Card (RORC) shallprovide the interface between the Detector Data Link (DDL) and the LDCs. TheDIU interfaces are speciﬁed by [2].(FEIC)ElectronicsFront-endUnit (SIU)InterfaceSource DestinationInterfaceUnit (DIU)Read-outReceiverCard(RORC)Backward channelForward channelDDL hardware = Source Interface Unit + Physical Medium + Destination Interface UnitFigure 1: The Detector Data LinkThe connection between the LDC and the RORC depends on the local I/O busemployed in the computer. Few years ago the PCI local bus was not as widespreadas today and an evolution of the VME bus toward higher performance was expected.Therefore the ﬁrst version of the RORC card was developed for VME systems [3].This card was used for testing the prototype version of the DDL components aswell as to measure the DDL performance connected to the VME bus. The results ofthose measurements show that the DDL transfer speed was only limited by the VMEmaximum speed. It was also clear that the VME based RORC card (VME-RORC)was expensive and complex. At present, the PCI local bus exists in any computerand provides enough bandwidth for the DDL. The theoretical speed is 132 MB/swith the 32bit/33MHz version and 528 MB/s with the 64bit/66MHz version and theexpected evolution of the VME bus toward higher performance has not materialized.2 DevelopmentThe development was split into two phases. The ﬁrst version of the PCI based RORCcard was deﬁned as 32bit/33MHz PCI master card. To reduce the complexity of theboard, its cost and development time it was decided not to use on board memory.The data shall be transferred directly into the memory of the LDC. Since the cardcan operate as PCI master the software will only have to allocate enough space inthe memory of the computer to let the RORC move the data to the destination.This will reduce the latency and the software intervention, so the DAQ softwarewill be able to make better use of the PCI bus. The required bus functions areimplemented in a dedicated PCI interface circuit from AMCC [4]. The custom logicfunctions for the DIU interface are realized in a Programmable Logic Device (PLD)from ALTERA. The ﬁrmware is divided into three parts according to functions theyprovide (Figure 2):• The add-on logic section handles the signals between the PLD and the PCIbridge. It also manages the add-on logic bus, performs the arbiter functionfor the diﬀerent ﬁrmware parts and reads the mailboxes.• The internal control and command manager receives the commands and con-trols the ﬁrmware to carry out the required actions. This part contains thewrite and read DMA engines and the memory manager as well.• The DIU interface, is dealing with the DIU. It contains FIFOs to store thedata and a pattern generator that can be used for test purposes.OMBPCI toAOL toPCI FIFOIMBAOL interfacePCIAOL FIFODIU interfaceFIFOReceivergeneratorPatternFIFOTransmittercontrollerWrite DMAinterpretercommandandmanagerMemorycontrollerRead DMADIUInternal ControlFigure 2: Architecture of the pRORCThe control of the card is done by commands. A command may have parametersas well. The commands are passed through the incoming mailboxes of the PCIinterface circuit. There are four incoming mailboxes in the device. Each mailbox is32-bit wide. The add-on logic is notiﬁed if there is a new command waiting in themailboxes. This is done by a hardware interrupt that is raised if one predeﬁned byteof the incoming mailboxes becomes full. The response of the RORC is sent backthrough the outgoing mailboxes. Since the PCI interrupt should be avoided there isno particular notiﬁcation to the software about the state of the outgoing mailboxes.However, the RORC shall not generate any response without a request from thesoftware. The DDL is using the same mechanism for the status and error reporting.The add-on logic keeps track of the outgoing mailboxes to avoid overwriting anydata, which is waiting for the software. The PCI bridge has a 32-bit bi-directionaladd-on local bus with separate address lines. The add-on logic can use this bus toaccess the registers, mailboxes and FIFO channels of the bridge. There are diﬀerentparts of the ﬁrmware competing for the bus access. The add-on logic contains abus manager, which plays the role of the arbiter. The arbitration is based on apredeﬁned priority scheme and performed by internal request and grant signals.According to this scheme the command transfer has the highest priority, whichensure the controllability of the RORC even during DMA transfer.The DDL interface makes the connection between the DIU and other parts ofthe ﬁrmware like the DMA channels. It contains FIFOs on both the transmit andthe receiving side in order to separate the diﬀerent clock domains and to balance thedata transfer speed on the link. There are two FIFOs in the receiver and one FIFOand a command register in the transmitter. The data and status in the receivermay come from the DDL or from the transmitter via an internal loop-back. Thisloop-back can be used for self-test. The input of the transmit FIFO can also beconnected to two diﬀerent sources. It is either the output of the read DMA channelor the output of the internal test pattern generator. The test pattern generator isable to produce data blocks with diﬀerent lengths and patterns. The cycle numberis selectable or it can be set to inﬁnite.The internal control and command manager receives the commands from theadd-on logic. Depending on the destination the command will be interpreted or sentover the link via the DDL interface. The partial memory management is carried outby this part of the ﬁrmware. It contains the DMA engines, which control the datatransfer to and from the PC memory. The write DMA (WDMA) is writing to thePC memory while the read DMA (RDMA) is reading from the PC memory. TheDMA engines are using the FIFOs implemented in the PCI bridge to do the transfer.The PCI bridge contains primary registers to control the data transfer. When thebridge operates as PCI master these registers are accessible from the add-on logic.One register deﬁnes the base address of the DMA transfer and the transfer countis stored in other register. There are two sets of such registers for read and writeDMA.The applied memory management supports the so-called scattered memory model.This will be used in the next version of the Data Acquisition and Test Environment(DATE) software [5]. In this model the data blocks are transferred to diﬀerentmemory locations in the PC memory. The software running on the machine ismaintaining a list of free memory pages. In this context page means a buﬀer of acertain size in the PC memory. To initiate the WDMA the RORC has to know thebase address and size of such a memory page. The software shall provide the RORCwith this information, so that it can accomplish the transfer. The RORC storesthis information in the so-called Free FIFO. Each entry of the Free FIFO consists ofthree ﬁelds containing the base address, the page size and the Ready FIFO index.The latter requires explanation. The hardware has to report the software about thereadiness of a memory page. This will happen in two cases, when the page boundaryis reached during the DMA or when the block ﬁnishes, which is indicated by a statusword generated by the DDL. In both case the ﬁrmware will close the given page bysending two words to a location in the PC memory. The address of this location isdeﬁned by the Ready FIFO index and the Ready FIFO base address. The ﬁrst wordcontains the number of words transferred to that memory page. The second wordcontains zero if the block stretches over the page boundary or the DTSTW statusword in case of the block ﬁnished. Using this mechanism the software can detectthe end of a DMA cycle by simply checking the Ready FIFO in the host memory.(Figure 3)BA - Base AddressBL - Block LengthIDX - Ready FIFO Entry IndexRFBA - Ready FIFO Base AddressBA BL IDXFree FIFO(128 entry)bufferDataRFBA IDXFIFOReadyFigure 3: Memory management3 Performance testSince the DDL is ready to provide 100 MB/s bandwidth on the forward channel itis important to know the performance of the RORC card. With the VME basedcard the measurements showed that the transfer speed is limited by the read-outsystem. The measured speed was less than 40 MB/s. In order to evaluate thetransfer speed using the PCI local bus test ﬁrmware and software were developed.The test ﬁrmware contains a simple DMA engine and a pattern generator. Thepattern generator generates data blocks with diﬀerent lengths. The DMA alwaysgoes to the same location in the memory. At the end of every block a counter isincremented and then transferred to the host memory (Figure 4). From time totime the software checks this memory location and calculates the transfer speed.The most important signals, which control the DMA, are placed on logic analyzerconnectors, so that they can be monitored during the operation. These signalsrepresent well the traﬃc on the PCI.Block counterBase addressBase address + 4Base address + 4(n+1)Data bufferFigure 4: pRORC performance test 1In the ﬁrst test the performance measured in stand-alone operation. There wasnot any other user process running on the computer. The computer had singleprocessor and the operating system was Linux with 2.2.x kernel. The DMA speedwas measured using diﬀerent block sizes to ﬁnd out the variation of the speed versusthe block size. The result of the test is shown in Figure 5.40 400 4K 40K 400K 2M 4M 6MData length [Byte]20406080100120140DMA speed [MB/s]Figure 5: DMA speed vs. block size (stand-alone operation)In the second test a more realistic environment was set up. The host computerhad two Intel Pentium III processors running at 400 MHz. The installed Linuxkernel version was 2.2.x. In order to increase the traﬃc on the PCI bus a GigabitEthernet network interface card was also installed. Beside the test software twoother processes was using the PC resources. A benchmarking tool, called stream lwas doing intensive memory operations in order to emulate a process, which wouldperform data processing in the LDC. Network traﬃc was generated by iperf. Inthe last scenario the DATE software consisted of the typical LDC components, likeread-out and recorder. The performance of the diﬀerent processes were measuredas well as the combined performance. The DMA size was constant (400 KByte)during the measurement. During the test the DMA speed, the available memoryand network bandwidth were measured. The usage of the CPUs was also monitoredduring the test in order to see the remaining resources for other applications runningon the LDCs, like software components of the High Level Trigger (HLT) system. Theresults are summarized in Figure 6.--------1 1252 360[MB/s] [MB/s]Memory Network[MB/s]3113 31080 2180 180-210 stream_l(100%)pRORC DMA(0%) + DATE(66%) +pRORC DMA(0%) + iperf(66%)pRORC DMA(0%) +stream_l(100%)iperf (100%)stream_l (100%)pRORC DMA (0%)4019-20654DMAScenario Components’ CPU usageFigure 6: Performance of the LDC components4 Conclusions and Future WorkThe ﬁrst version of the PCI based Read-out Receiver Card for 32-bit 33MHz PCIwas developed. This card is a functional replacement of the VME based RORCand it supersedes the old design in terms of performance. The hardware is rathersimple and cheap. The ﬁrmware connects the DDL to the LDC computer via DMAchannels. The performance tests show that DMA bandwidth allows the DDL tooperate at the maximum speed. The software library was developed to provideinterface for the read-out software.The next version of the RORC will be based on the 64bit/66MHz PCI. ThePCI interface functions will be realized in the PLD by using IP core. Beside thedata ﬂow operation the card shall provide data processing functions for the HLTsystem. These functions require on-board memory too. The option of an integratedRORC-DIU card will be investigated as well.References[1] ALICE Collaboration, “ALICE - Technical Proposal for A Large Ion ColliderExperiment at the CERN LHC”, CERN/LHCC 1995-71, December 1995.[2] G. Rubin, P.V. Vyvre, “ALICE Detector Data Link (DDL) - Interface ControlDocument”, ALICE Internal Note, INT-96-43.[3] G. Rubin, J. Sulya´n, “The Read-out Receiver Card (RORC) Hardware User’sGuide”, ALICE Internal Note, INT-97-14.[4] Applied Micro Circuit Corporation, “S5935 PCI Product Data Book”,http://www.amcc.com.[5] ALICE Data Acquisition group (ALICE/DAQ), private communication

PCI Based Read-out Receiver Card in the ALICE DAQ System

PCI Based Read-out Receiver Card in the ALICE DAQ System

Abstract

Similar works

Full text

Available Versions

CERN Document Server