Interface modules between PCI local bus and HIPPI are described. The modules are intended to aid the implementation of the high performance computer network. the aimed maximum throughput of the interface module is 100MBytes/sec while sustained data transfer rate depends of the particular system performance

Anguelov, T

Antchev, G H

Georgiev, G

Gillot, D

Guglielmi, A M

McLaren, R A

Orel, O

Piperov, S

Van Praag, A

Vankov, I

CERN Document Server

ORGANISATION EUROPEAN POUR LA RECHERCHE NUCLEAIRE CERN EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCHPCI - HIPPI Interface Modules T. Anguelov, G. Antchev, G. Georgiev, S. Piperov, I. Vankov, INRNE-BAS, Sofia, BulgariaR. A. McLaren, A. van Praag, CERN, Geneva, Switzerland.O. Orel, IHEP, ProdvinoD. Gillot, A. Guglielmi, Digital Equipment Corporation, Joint project office, CERNCERN/ECP 96-14, 18 September 1996Summary Interface modules between PCI local bus and HIPPI are described. The modules are intended to aidthe implementation of the high performance computer network. The aimed maximum throughput ofthe interface module is 100 MBytes/sec while sustained data transfer rate depends of the particularsystem performance. Presented at the SOZOPOL-96 Workshop of Relastivistic Nuclear Physics, Sozopol, Bulgaria, 30 September - 6 October Introduction PCI - the next generation local bus An introduction to the HIPPI Hardware Achieved results Acknowledgements References IntroductionThe High Energy Physics (HEP) experiments require complex Data Acquisition Systems (DAQ).Very often the DAQ is a multilevel systems, allowing filtering and data processing for a hugeamount of input data to an acceptable for off-line analysis output data. The size of the events,delivered from the accelerators in the HEP experiments, ranges from Megabytes to Terabytes andthe event rate is up to hundred of kilohertz. This big amount of data implies fast data movingbetween DAQ’s levels. The major importance is the throughput of the DAQ in the upstream direction. Using standardinterfaces and powerful workstations is acceptable approach to build up cost effective DAQ. Thedata transport between different levels in the complex DAQ uses the high bandwidth interfaces asFDDI, FASTBUS, FCS, SCI or ATM. One of the fastest interfaces, intended to meet the requirement of the high speed data transfer is theHigh Performance Parallel Interface (HIPPI) [1]. Originally developed for use in the supercomputerarea, now the HIPPI is widely used in other applications. An interface module from HIPPI to therecently introduced high speed local bus as Peripheral Component Interconnect (PCI) [2] will beuseful to create powerful DAQ based on HIPPI. PCI - the next generation local busMost of the computer companies use different I/O buses in their computers thus making the add-inmodules for one machine incompatible to others. In 1992 Intel introduced the bus named PeripheralComponent Interconnect. More than 168 computer companies as Apple, DIGITAL, IBM, Intel,NEC and others have adopted this bus as internal bus of their computers. Now the PCI dominates inthe computer industry. Many OEMs offer PCI compatible products as video-controllers andinterface modules for Ethernet, SCSI, FDDI. The PCI is a high performance local bus architecture intended to meet the requirements of today’scomputer systems. In technical aspect PCI is powerful bus architecture with many advancesfeatures: Synchronous bus with clock frequency 33 or 66 MHz. 32/64 bits address/data path. Peek transfer rates as high as 528 MBytes/sec. Multiplexed address-data lines. Dynamic data width sizing by means of byte enables strobes. Double handshake mechanism. Sixteen type bus operations. Hidden (access based) central arbitration mechanism. Full Multi-master capability. Exclusive access using a locking mechanism to an accessed device. Cache coherency support. Hierarchical (tree likes) architecture based on PCI-to-PCI bridges. Auto configuration through a configuration space register set. Predictable target latency. Defined bus utilisation by means of a master latency timer. Fundamental new approach of bus driving by using of the reflected wave. Even parity for address/data and command/byte enables lines. Expansion ROM for device specific initialisation and system boot. JTAG/Boundary scan capabilities. In the application aspect the PCI is moving in multiple dimensions - Figure 1.  Figure 1. PCI Applications Universal bus for all platforms from low-end PCs to high-end workstations Processor independent 32 or 64 bit local bus. Forward and backward compatibility of 32 and 64 bit add-in boards. Auto configuration enabling "plug and play" add-in board insertion. Operation at 5V, 3.3 or both. The PCI data transfer mechanism is burst oriented, that is a train of data words during one bustransaction is moved between the initiator module and the target module. The PCI specificationdoes not limit the burst length, however an infinite data transfer is not possible because of: System memory limitations. Presence of other bus initiators that might arbitrate the bus. Software programmable and hardware implemented bus utilisation timer. Type of the bus operation. There are two generic types of bus master devices. A device that very often needs a fast access tothe bus and transfer small data block is called "latency sensitive master". Device that needs the busaccess occasionally but transfer large block of data is called "high throughput master". PCI toleratesboth types of devices - latency sensitive devices and high throughput by means of hardwareimplemented latency timer and software controllable type of bus operation. This makes a PCI idealsolution when different type of bus mastering interface modules are connected to the system. Figure2  Figure 2. PCI based system configuration A PCI based system may dynamically tune itself according to some conflicting requirements as thefast access to the system devices and the large data transfer to the system memory. An introduction to the HIPPIThe HIPPI standardisation was started in 1987 at American National Standards Institute (ANSI).The HIPPI standard includes HIPPI-PH (Physical layer), HIPPI-SC (Switch Control), HIPPI-FP(Framing Protocol) and HIPPI-LE (Link Encapsulation). All of them are ANSI Standards. The HIPPI-PH specifies mechanical, electrical and signalling parameters of the HIPPI over 25meters cable of the twisted pares caring differential ECL level signals. The HIPPI is unidirectional,synchronous channel with fixed clock frequency of 25 MHz. The theoretical maximum throughputof the HIPPI is 100 MBytes/sec for 32 bit and 200 MBytes/sec for 64 bit version. Separate links canbe interconnected through HIPPI switches in HIPPI networks having a very high throughput. TheHIPPI switches are commercially available from NCS, Essential Communications and GigabitLabs.. The HIPPI is a point to point link from a data transmitting device "HIPPI SOURCE" to a datareceiving device "HIPPI DESTINATION" - Figure 3.  Figure 3. HIPPI Interface signal summary The protocol is based on the relationship between the "REQUEST" and "CONNECT" and alook-ahead data flow control by the "READY" pulses. The physical framing hierarchy is: HIPPIconnection, HIPPI packet and HIPPI burst - Figure 4. The HIPPI connection consists of one or several packets. Upon requesting a connection the sourceasserts the I-Field, a 32 bit word, used as routing information when the connection is establishedthrough one or more HIPPI switches.  Figure 4. Physical framing hierarchy The packet consists of one or several bursts. The HIPPI burst is a train of up to 256  Figure 5. The second level of the DAQ for NA48 experiment words of 32 or 64 data bits each and 4 or 8 byte parity bits. Every burst is followed by alength-longitudinal redundancy checkword (LLRC) which provides a proper data integrity foreventual error detection and correction. Most of the DAQs for the HEP experiments need the movement of data only in one direction - fromprevious to the next DAQ level. In such case the use of a simplex HIPPI segment is relevant. Thefirst application of the HIPPI based computer network is the second level of the DAQ for NA48experiment at CERN [3]. The second level of the NA48 DAQ is an array of powerful workstationsreceiving data from an event builder - Figure 5. The HIPPI switch is used as a distributor, forwarding large data blocks (approximately 180MBytes) to the currently available workstation while other workstations are processing data,already transferred to them. HIPPI can be used to build a local area network (LAN) with various configurations. A simplenetwork, configured as "star" has a HIPPI switch and number of network stations connected to theswitch - Figure 6.  Figure 6. HIPPI based LAN LAN applications requiring a full duplex use two HIPPI channels in both directions. The totalthroughput of the network is a sum of the particular throughput of all network segments. HardwareTwo interface modules were developed - a HIPPI "Destination" and a HIPPI "Source". The"Destination" module block diagram is depicted on Figure 7. The incoming data is transferred to thesystem memory by means of a Direct Memory Access (DMA) transactions. The advantage of DMAis that the system is free to perform other tasks on the same time the DMA occupies the PCI bus.For large data transfer the DMA requires on-board Scatter-Gather Table Memory (SGTM). TheSGTM contains the correspondence between the physical and logical addresses of the systemmemory buffer, reserved for the current HIPPI connection. The DMA engine automaticallyretrieves the physical address of the next memory page in the case of page segmented systemmemory. This mechanism allows large data blocks to be transferred to the system memory withoutthe time consuming interrupt procedure.  Figure 7. HIPPI "Destination" block diagram A FIFO memory is used as an elastic buffer of data words. It is clocked by the PCI clock on theinput port and by the HIPPI clock on the output port. The HIPPI protocol information and eventualHIPPI data errors are stored into the History Memory. If HIPPI data packets contain one or moreparity errors, they can be located in the system memory after the end of the current HIPPIconnection. This mechanism avoids the delay due to interrupt handling if the protocol informationand data errors would be reported to the system by an interrupt request. The interrupt requestbreaking the data transfer is generated only after the HIPPI connection end or after a seriousinterface error. The HIPPI "Source" module block diagram is shown on Figure 8.  Figure 8. HIPPI "Source" block diagram In addition to the physical addresses, the SGTM contains the HIPPI packet delimiters andconnection end as well as the memory buffer size. The DMA engine moves data from the systemmemory to the HIPPI until the end of memory buffer is reached, or if the HIPPI connection has tobe terminated. In the "Source" and in the "Destination" the AMCC device S5933 is used as PCIcontroller. It is a general purpose PCI controller with flexible internal structure intended to satisfyvarious user requirements to the PCI interface. Achieved resultsThe speed measurements were carried out on Digital workstations Alpha 400, Alpha 600 and VMEmodule AXBVME 160. The listedthroughput in Table 1 was measured on the HIPPI port of themodules without taking in account the software overhead.               DEC Alpha 600   DEC Alpha 400   AXBVME 160        "Destination"  91 MBytes/sec   72 MBytes/sec   65 MBytes/sec     "Source"       20 MBytes/sec   20 MBytes/sec   20 MBytes/sec     AcknowledgementsThe Autor thanks the National Found for Scientific Research of Bulgarian Ministry of Education,Science and Technology for sponsoring this project. References1. PCI Local Bus Specification, Rev 2.0, April 1993, PCI Special Interest Group, Oregon, USA 2. HIPPI - Mechanical, Electrical and Signalling Protocol Specification REV 8.1, June 24, 1991 3. J-P. Matheys, et al. "Data Transfer and Distribution at 70 Mbytes/s", CERN/ECP 93-7.Presented at IEEE RT 1993  This is one of the CERN High Speed Interconnect pages - 10 December 1996 - Arie van Praag 

PCI - HIPPI Interface Modules

PMC - HIPPI interface modules

PCI - HIPPI Interface Modules

Abstract

Similar works

Full text

Available Versions

CERN Document Server

CERN Document Server