The Gigabyte System Network (GSN) standard specifies two media for connection: a Copper Cable connection, used up to now, and a Fibre Optics connection using a 12 channel flat cable. As no work had been done on the optical connection up to now, it was decided to start a project in this direction at CERN in collaboration with industry. It is explained why a different approach was taken to that foreseen as a GSN optical standard and how an optical connection that works at the full GSN bandwidth was realised. It is now decided that this technology will be the new GSN Optical standard

Van Praag, A

CERN Document Server

ORGANISATION   EUROPEAN   POUR   LA   RECHERCHE NUCLEAIRECERN EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCHCERN-IT-2000-0083 April 2000GSN Optical: How to solve the problemArie van Praag, CERN, 3 April 2000AbstractThe Gigabyte System Network (GSN) standard specifies two media forconnection: a Copper Cable connection, used up to now, and a FibreOptics connection using a 12 channel flat cable. As no work had been doneon the optical connection up to now, it was decided to start a project in thisdirection at CERN in collaboration with industry. It is explained why adifferent approach was taken to that foreseen as a GSN optical standardand how an optical connection that works at the full GSN bandwidth wasrealised. It is now decided that this technology will be the new GSN Opticalstandard.2GSN Optical: How to solve the problemArie van Praag, CERN, 3 April 2000AbstractThe Gigabyte System Network (GSN) standard specifies two media forconnection: a Copper Cable connection, used up to now, and a FibreOptics connection using a 12 channel flat cable. As no work had beendone on the optical connection up to now, it was decided to start aproject in this direction at CERN in collaboration with industry. It isexplained why a different approach was taken to that foreseen as a GSNoptical standard and how an optical connection that works at the fullGSN bandwidth was realised. It is now decided that this technology willbe the new GSN Optical standard.A Short Introduction to GSN:The Gigabyte System Network is a 10 Gigabit/s network standardised by ANSI NCITSand ISO/IEC under the project name HIPPI-6400 [1]. This high speed network wasfrom the beginning developed for a very high transfer speed with a very low networklatency. A maximum of functionality is executed in hardware. The transfer is done insmall micropackets and the transfer medium is divided into four multiplexed VirtualChannels to avoid congestion by large data streams. Each of the Virtual Channels has itsown end to end CRC error checking extended with an overall link CRC. A refined look-ahead mechanism avoids end point congestion [2]. All this interface functionality of thephysical link is built into a single silicon chip called SUMAC. The use of this chip byall manufacturers of GSN equipment leads to a very high factor of compatibility. Inorder to keep transfer latency low GSN is accompanied with an operating system bypassprotocol "ST" (Scheduled Transfer) that works also with IP and SCSI transfers [3][4].The ST protocol is also successfully demonstrated for other network technologies suchas Fibre Channel and Gigabit Ethernet.For the actual transfer medium the standard describes two different connectiontechnologies, one for Copper Cable and one for Fibre Optics [5]. The copper cableconnection uses a 24 channel cable and the fibre connection is made for a 12 fibre flat-cable connection. The line port of the SUMAC chip has two selectable operationalmodes: the optical connection with half the number of lines uses twice the clockfrequency. To extend the range of connection the SUMAC line input has a verycomplex skew suppression circuit that is autoadjusting by a training cycle with a specialdata pattern and slower clock cycle on one of its two clock signals. The followingdescribes in detail the tests done to come to a working version of the fibre connectionfor GSN. Some knowledge of the SUMAC chip may be necessary to understand theproblems that are handled here in detail.The SuMAC Chip Line Interface:First of all there is the note from the developers of the SUMAC chip at SGI that it isvery questionable that the optical mode will work at the full clock frequency of 1 GHzneeded for the optical ( 12 bit ) mode.The next question is the clock oscillator itself. The best external oscillator that iscommercially available has a frequency of 825 MHz. This is independent from thequestion how to handle this class of frequencies, via a printed circuit board with all itsstray capacities and inductances, into the SUMAC chip, without signal degradation. For3the designers of the SUMAC chip it would only have taken three more Asic gates tomake a fixed frequency doubler inside the chip.The last question is if the optics would ever have worked, as the reference clock givenby the SUMAC chip in optical mode is "CLOCK", a signal disturbed by the trainingcycle bringing the optics out of DC balance (or at least it will disturb the stability of anyPLL that may be in the loop).It was decided at CERN to look at the GSN-Optical problem. The only commerciallyavailable components that can handle a 12 bit parallel fibre connection are the InfineonParoli’s. They exist in two versions [8]:1. A 12 channel wide direct coupled optical converter in versions for 1 Gbaud,1.6 Gbaud and 2.5 Gbaud.  (V23814/15-K1306-M130)2. A 22 channel multiplexed version available for 200 - 500 Mbits/s perchannel. (V23814/15-K1306-M230)From the problems with the SUMAC chip it is clear that using the 12 bit optical modewith the 12 channel component is a risk. The 24 bit mode foreseen for copper cableconnections is a proven concept that is used already in GSN equipment. As aconclusion, the work done for a GSN Optical connection concentrated on the 22 channelmultiplexed component coupled to the SUMAC chip working in 22 bit mode.The Paroli modules use LVDS levels for the data path and LVCMOS for controlsignals.Adapting the Paroli to the SUMAC:In some way the coupling between PECL logic of the SUMAC and the LVDS logic hasto be made. These levels look compared to each other as follows:SUMAC Chip Paroli InMin Typ Max Min Max unitInput Common mode Voltage 1.05 1.30 1.55 1.2 VDifferential Input Voltage 0.150 - 2.70 0.1 1 VOutput Common mode Voltage - 1.25 - 1.2 VDifferential Output Voltage 2.20 2.50 2.70 0.25 0.4 VIn the direction SUMAC to PAROLI we see that:The Paroli demands a min. -max. differential input signal of 100 - 1000 mVWhere the Sumac chip delivers 2200 mVOutput common mode for the Sumac is 1250 mVFor the Paroli I calculated the common mode at 1200 mVIn order to stay in a safe operating area some adaptation has to be done. It is solved byusing the AC coupling of the copper connection and with an attenuating termination asin Fig 1.SUMAC 2703636220PAROLIFig 14The result measured on the clock signal is 800 mV on the PAROLI input pins. Fig 2gives the waveform for Clock_2 at the input of the Paroli transmitter and fig 3 the samesignal at the output of the Paroli receiver.Fig 2 Fig 3In the direction from Paroli to SUMAC the following data is validThe Sumac chip demands as diff. input signal minimum 150 mVThe Paroli has a minimum diff. output signal of 250 mVInput common mode for the Sumac is + 250 mV 1300 mVOutput common mode for the Sumac is 1250 mVThis makes it possible to use a DC coupling, keeping the 75 ohm Tevenin equivalentterminator resistors as for the cable connection in place.Output Timing:Both Transmitter and Receiver synchronise with PLL oscillators, doubling orquadrupling the clock frequency on the fibre, corresponding to the "Clock" or "SCI"mode. The second mode corresponds to the SUMAC functionality.Due to the PLL’s the receiver chip brings clock and the data on phase to the output pins.Such a 1 nsec external clock delay must be made as is given in Fig 4. Test Set-Up:For the test a small piggyback board with the optical components was made that plugsinto the auxiliary connector of the GSN port on a Genroco bridge. The board uses a 5Vpower supply with a 3.3 V stabiliser. Every separate power group is individuallydecoupled with a ferrite component and a decoupling capacitor per power pin.All data lines are paired and calculated for a 75 Ohm impedance. Placing transmitterand receiver on opposite sides of the board, avoids crossings and via’s in the data lines.DATACLOCK+ 1 nsecCLOCKFig. 45A PECL clock driver was added in case a second clock had to be generated, or if adivide by 2 effect would occur on one of the clocks. Its final function was to generate aclock delay. The board has only 4 layers. See also Fig 5 for the board and Fig 6 for thetest set-up.As test generator the Genroco test board, a modified FC converter for the bridge is used.Fig 5 Fig 6Test Results:One of the optical transmitters was not working properly so I had only one workingboard available. This limited the testing to loop-back mode on a single bridge. (It wasoriginally planned to connect two bridges both with a GigE board and run data betweenthem).It has to be remarked that in this set-up and during these tests Clock_2 has only 270psec delay. After some initial set-up tests the following results were obtained:As a first attempt all tests available on with test board were run for about 1 hour with a5 m fibre and using all VC’s.1. Loop & Check Pings.2. Loop & Check Single Packet Messages.3. Loop & Check Multi Packet Messages.During these tests no error was found. The fibre was opened repeatedly to check thaterror messages would be given.This was followed by a long term test using the third test with 25 micropacket messagesand using 4 fibres of 5 m each patched together to 20 m. (this is all I have available).After 8 hours an error occurred in the form of a missing high byte. This errorreproduced itself at regular distances. Concentrating the ventilator air-stream to thetransmitter circuit ( fig 7 ) solved the problem and the circuit is now working more than48 hours without any error detected.6Conclusion.By doing this optical test, CERN is the first place to have a GSN optical connectionworking. It shows that the Paroli type V23814/15-K1306-M230 is a well workingcomponent that can be used for GSN optical connections up to 75 m. It can also be afast optical connection in HEP experiments with up to 22 channels in a singleconnection.According to Infineon, distances of more than 75 m lead to fibre skew problems. Forthis distance skew tested cables are available.A critical point with the Paroli components is good cooling, especially for thetransmitter.It should also be noted that the connector guidance pins are inside the componentconnector. The cable has no such pins. In the case of cable patches these guidance pinshave to be placed manually. Infineon proposes a solution that uses extension cables witha male and a female connector.In order to obtain interoperability between all GSN equipment from differentmanufacturers, it is necessary that common signal-pin relations get standardised. Aproposal is given as an Annex.Technical documentation of this project was presented at the HNF HIPPI-6400standardisation meeting of 4 April 2000 in San Diego, USA [7], with the result that theactual version of the HIPPI-6400 OPT standard is withdrawn from public review inNCITS, to be modified with the proposed solution for GSN optical, in the T11.1 group.AcknowledgementI want to thank all those who have made it possible to work on the GSN-Opticalproblem, especially: at CERN, Ben Segal who allowed me the time to do so; at Infineonin Berlin, Germany, Lothar Droege for his technical assitance around the Parolimodules; at Genroco in Slinger, USA, Brian Breuer for technical assistance withproblems around the SuMAC chip, and Don Woeltz for presenting my work in the HNFtechnical meeting in San Diego; and Genroco in general for making GSN bridges andGSN test boards available.Fig 7.7References [1] High-Performance Parallel Interface -6400, Mbit/s Physical Layer (HIPPI-6400 PH),  Don TolmieLANl et al, ANSI NCITS 323-1998. 2.4, June 19, 1999, ISO/IEC 11518-10,http://www.hippi.org/c6400PH.html.[2] Potential HEP Applications of a New High Performance Networking Technology, Arie Van Praag,Ben Segal,( CERN, IT/PDP/TE, 21 August 1999 ), Presented at CHEP 2000, Padoua, Italy.http://hsi.web.cern.ch/HSI/gsn/reports/GSN-CHEP.PDF[3] Scheduled Transfer Protocol (ST), T11.1/Project 1245-D/Rev 3.2, ANSI NCITS xxx-199x., ISO/IEC11518-xx. http://www.hippi.org/cST.html[4] SCSI on Scheduled Transfer (SST), ANSI NCITS xxx-199x., ISO/IEC 11518-xx,http://www.hippi.org/cSCSI.html[6] HIPPI-6400-OPT Rev 1.2, ANSI NCITS xxx-199x., ISO/IEC 11518-xx.,http://www.hippi.org/c6400OPT.html[7] HIPPI ad-hoc Optical Working Group Minutes, San Diego, 4 April 2000.http://www.hippi.org/min0004opt.pdf.[8] Data sheets for Infineon Paroli components can be found on the web at the following URL:http://www.infineon.com/products/fiber/index_paroli.htm8ANNEXProposal for Paroli DC standard pin Pin ConnectionsSUMAC_OutFUNTIONSClock2_Out_LClock2_Out_HD08_Out_LD08_Out_HD09_Out_LD09_Out_HD00_Out_LD00_Out_HD10_Out_LD10_Out_HD01_Out_LD01_Out_HD11_Out_LD11_Out_HD02_Out_LD02_Out_HD12_Out_LD12_Out_HD03_Out_LD03_Out_HD13_Out_LD13_Out_HD04_Out_LD04_Out_HD14_Out_LD14_Out_HD05_Out_LD05_Out_HD15_Out_LD15_Out_HD06_Out_LParoliPIN No12345678910111213141516171819202122232425262728293031323334353637383940414243DC TXFunctionVcc 1t.b.l.o.t.b.l.o.t.b.l.o.t.b.l.o.LAVeeVeeVcc 3MACI NCI PDI 01 NDI 01 PDI 12 NDI 12 PDI 02 NDI 02 PDI 13 NDI 13 PDI 03 NDI 03 PDI 14 NDI 14 PVcc 3DI 04 NDI 04 PVeeDI 15 NDI 15 PDI 05 NDI 05 PDI 16 NDI 16 PDI 06 NDI 06 PDI 17 NDI 17 PDI 07 NDI 07 PDI 18 NDI 18 PDI 08 NDC RXPIN NoVeeVcc 1Vcc 2t.b.l.o.-RESETFrame-DetVcc 3VeeVcc 4Lock-DetCO PCO NDO 01 PDO 01 NDO 12 PDO 12 NDO 02 PDO 02 NDO 13 PDO 13 NDO 03 PDO 03 NDO 14 PDO 14 NVcc 4DO 04 PDO 04 NVeeDO 15 PDO 15 NDO 05 PDO 05 NDO 16 PDO 16 NDO 06 PDO 06 NDO 17 PDO 17 NDO 07 PDO 07 NDO 18 PDO 18 NDO 08 PSUMAC_InFUNTIONSClock2_In_H (1Clock2_In_L (1D08_In_HD08_In_L D09_In_HD09_In_LD00_In_HD00_In_LD10_In_HD10_In_LD01_In_HD01_In_LD11_In_HD11_In_LD02_In_HD02_In_LD12_In_HD12_In _LD03_In_HD03_In_LD13_In _HD13_In _LD04_In_HD04_In_LD14_In _HD14_In _LD05_In_LD05_In_HD15_In_LD15_In_HD06_In_L9D06_Out_HC2_Out_LC2_Out_HD07_Out_LD07_Out_HC3_Out_LC3_Out_HC0_Out_LC0_Out_HFrame_Out_LFrame_Out_HC1_Out_LC1_Out_HCLOCK_Out_LCLOCK_Out_H4445464748495051525354555657585960616263646566676869707172DI 08 PVeeDI 19 NDI 19 PVcc 3DI 09 NDI 09 PDI 20 NDI 20 PDI 10 NDI 10 PDI 21 NDI 21 PDI 11 NDI 11 PDI 22 NDI 22 PCLK-SELt.b.l.o.Vcc 3-RESETVeeVeeLE-LEt.b.l.o.t.b.l.o.t.b.l.o.Vcc 1DO 08 NVeeDO 19 PDO 19 NVcc 4DO 09 PDO 09 NDO 20 PDO 20 NDO 10 PDO 10 NDO 21 PDO 21 NDO 11 PDO 11 NDO 22 PDO 22 NCLK-SELo.e.t.b.l.o.Vcc 4VeeVcc 3-SD 11ENSDt.b.l.o.Vcc 2Vcc 1VeeD06_In_HC2_In_HC2_In_LD07_In_LD07_In_HC3_In_HC3_In_LC0_In_LC0_In_HFrame_In_LFrame_In_HC1_In_LC1_In_HCLOCK_In_HCLOCK_In_L

GSN Optical: How to solve the problem

Abstract

Similar works

Full text

Available Versions

CERN Document Server