An overview of a read-out driver (ROD) for silicon detectors in the ATLAS experiment at the Large Hadron Collider (LHC) is presented. Two silicon-based ATLAS tracking systems, referred to as the Pixel Detector and the Semiconductor Tracker (SCT), are controlled and read-out using a common 9U VME board. A hybrid design of Field Programmable Gate Arrays (FPGAs) and Digital Signal Processors (DSPs) has allowed the Silicon ROD to meet the challenges of format error-counting and event trapping without interfering in the construction and transmission of event fragments to the next level in the read-out system. Performance of the ROD during detector assembly, calibrations and cosmic-ray data-taking are also discussed

Vickey, T

CERN Document Server

ATL-INDET-SLIDE-2009-34502November2009A Read-Out Driver for Silicon Detectors in ATLASTrevor Vickey•University of Wisconsin, Madison•ATLAS CollaborationSeptember 26, 200612th Workshop on Electronics for the LHC and Future Experiments2A Toroidal LHC ApparatuS (ATLAS)General purpose experiment at the LHC22 mpp3The ATLAS Inner DetectorCritical for charged-particle tracking, B hadron and τ taggingConsists of three detector sub-systems (and more than 86 million channels)ppη = 0η = 2.5Pixel Detector(3 Barrels and 3 x 2 End-Cap Disks)Semiconductor Tracker (SCT)(4 Barrels and 9 x 2 End-Cap Disks)Transition Radiation Tracker (TRT)(Axial Barrel Straws and Radial End-Cap)4Pixel Tracker ModuleEach Pixel Module has16 Front-End ICs• 2880 channels / IC • 18 columns x 160 rows• Bump bonds to sensorOne Module Controller Chip• Collects data from                                              the 16 FE chips• Translates commands into    chip signalsTotal Number of Modules1456 (Barrels)144 x 2 (End-caps)Read-out Rate:B-Layer 160 Mbit/sLayer-1 and Endcaps80 Mbit/sLayer-2 40 Mbit/s6.0 cm2.2 cmThere are over 80 million channels in the ATLAS Pixel TrackerATLAS Pixel Module (46080 channels)  Pixel Size: 50 x 400 μmModule Controller Chip (MCC)Front-End ICs (x16)Flexible PCB Sensor (between PCB and IC)5Semiconductor Tracker ModuleEach side of an SCT Module hasTwo silicon sensors• Manufactured by Hamamatsu• 768 instrumented strips at an 80 μm pitchAn array of six binary readout chips• ABCD3TA ASICs• Discriminator• Pipeline• Data Compression Logic• Read-out BufferTotal number of modules:2112 (Barrels)988 x 2 (End-caps)Read-out Rate:Barrels and End-caps 40 Mbit/s12.8 cm6.4 cm40 mrad40 milliradian stereo anglebetween sides provides resolution in the axial directionThere are over 6.2 million channels in the ATLAS Semiconductor TrackerATLAS SCT Module (1536 channels)  Pitch: 80 μm6Data Acquisition HardwareThe ROD (Read Out Driver) is central to the DAQ chainHardware common between Pixel and SCT Trackers (firmware differences exist)ROD BOC MODULE48ROD BOC MODULE48ROD BOC MODULE48ROD BOC MODULE48SBCTIMVME CrateTTCROSHOST(Read Out Subsystem)(Timing, Trigger & Control) (TIming Module)Up to 16 RODs(Read Out Driver)per VME Crate(Back Of Crate)(Single Board Computer). . .. . .To Event-Builderand Level-2Timing and TriggerOptical Fibers for:Clock, Config, TriggerEvent DataConfigurationHistogramsS-Link7The Silicon Read-Out Driver (ROD)MODULE BOCSBCTIMROS RODController& Master DSPFormatters Event FragmentBuilderRouter& Slave DSPs(Read Out Subsystem)(Back Of Crate)(Single Board Computer)(TIming Module)To Event-Builderand Level-2CalibrationHistogramsSecondary purpose: Calibrations / MonitoringFPGAs for time-critical functions (Event Data Path)DSPs for configuration, ROD control, calibrations and monitoringPrimary purpose: Module configuration, Trigger propagation, Data formattingA hybrid of FPGAs and DSPsEvent DataConfiguration & TriggersBackpressure / Halt Output S-Link8The Silicon Read-Out Driver (ROD)MasterDSPControllerFPGAFormatterFPGAsRouter FPGAEventFragmentBuilderFPGASlaveDSPs9U VME9ROD Controller (FPGA) and Master DSPThe Master DSP (MDSP) has ROD and BOC registers connected to one of its EMIFs (External Memory InterFace)Host Port Interface to Slave DSPsRead/Write to ROD FPGAs, BOCMaster DSPRead/Write to Master DSPDSP Serial Port 1 (SP1)DSP Serial Port 0 (SP0)48To BOC / ModulesControl LinksBit MasksTIM Data and CommandsGenerate DynamicMask & Mode BitsTo FormatterFromFormatterTo Event FragmentBuilderTrigger Operation& ControlIncrement /DecrementTrigger CountSerial ExpansionMake Event DataDetect CommandsEvent ID DataTrigger TypeDynamic MaskROD L1ID, BCID& Trigger TypeThe Master DSP is a Texas Instruments 6201 integer DSPs running at 160 MHz it is used for configuration & control Purpose: Setup ROD data-path, module configuration, trigger propagation10Link Data FormatThe Pixel MCC and the SCT ABCD3TA each return a bit streamThe BOC splits these into individual 40 MHz streamsThe ROD Input Links receive the data streams from the BOC• 40 MHz output: The Pixel Barrel Layer-2 and the SCT• 80 MHz output: The Pixel Barrel Layer-1 and the Pixel Endcaps• 160 MHz output: The Pixel Barrel B-LayerThanks to Andreas Korn (LBNL) for the Input Memory FIFO capturesEvent Header: 11101TimeInput Link Number [0,47]40 MHz Mode 80 MHz Mode160 MHz Mode11ROD Formatter (FPGA)8 Formatters on the ROD; capable of header error detectionInput MemoryFIFO(48 bits/32k Words)Input MUXFMT ModuleEnableMode Bits(play/skip/hold-off)Serial-to-Parallel ConversionFIFO(add event headers)Read-Out Control& CounterFrom BOC / ModulesTo Event Fragment BuilderTo/From ControllerMode Bits FIFO12The Input Memory FIFO is capable ofrecording event data from the Links and playing these through the ROD(a useful debugging feature)Input LinksTokenTokenGood Header: 11101Acceptable Headers: 1110011111110010110110101Header & Trailer CounterLink IDTime-outHalt OutputFormatter Num.Data ValidPurpose: Serial-to-parallel conversion (word data), detect module packet errors 12ROD Event Fragment Builder (FPGA)FIFOFIFOEngine 0Engine 1EventMemAEventMemCEventMemB To/From RouterTo/From FormatterTo/From FormatterTo/From ControllerOutput FIFO Data (43 bits)Output FIFO Data (43 bits)Header and Trailer Data (32 bits)Halt OutputHalt OutputHeader/TrailerGeneratorError Summary WordError Summary WordFormat & CountFormat & CountL1/BC ID CheckL1/BC ID CheckError CheckError CheckIncrement /Decrement L1IDIncrement /Decrement L1IDL1 & BC IDsEvent Data& Trigger TypeDynamic Mask EventMems A, B and C are useful for debugging work Link Num. (6 bits)Link Num. (6 bits)Time-out ErrorTime-out ErrorL1ID ErrorL1ID ErrorBCID ErrorBCID ErrorData (32 bits)Data (32 bits)Formatter Num. (2 bits)Formatter Num. (2 bits)Link Num. (4 bits)Link Num. (4 bits)Data (32 bits)Data (32 bits)Data ValidData ValidTime OutTime OutPurpose: Collect Formatter output, check L1 and BC IDs, count errors, generate Event Header & Trailer 13Error Counting and HandlingOne of the primary purposes of the Formatters and the EFB:To identify and count errors found inside of the event dataThe types of errors are stored inside of the event header and trailer words:Name Bits [15:0] or [31:16]Event Header 001PTLBKxMMMMMMMEvent Trailer 010ZHVxxxxxxxxxxSingle Hit 1FFFFCCCCCCCxfx0Double Hit 1FFFFCCCCCCCsfx1Example for the SCT event formatis shown B = BC ID ErrorC = Cluster base addressF = Front-End numberf = Error in the first hitH = Header/Trailer Limit ErrorK = Condensed modeL = L1 ID ErrorM = Link NumberP = Preamble Errors =  Error in the second hitT = Time-out ErrorV = Data Overflow Errorx = (filled with 0 by the ROD)Z = Trailer Bit Error14ROD Router (FPGA) & Slave DSPsSDSP 3SDSP 2SDSP 1SDSP 0Read, formatand direct theevent data2 TrapsEvent Type: ATLAS, ROD or TIM2 TrapsEvent Type: ATLAS, ROD or TIM2 TrapsEvent Type: ATLAS, ROD or TIM2 TrapsEvent Type: ATLAS, ROD or TIMDMA Transfer Engine256 32-bit word blocksDMA Transfer Engine256 32-bit word blocksDMA Transfer Engine256 32-bit word blocksDMA Transfer Engine256 32-bit word blocksS-LINK To/From ROSTo/From Event Fragment BuilderXOn / XOff S-LinkS-Link DataDsp Halt OutputHalt OutputError Format DataData ValidOutput FIFO (43 bits)Event Header andTrailer Data (32 bits)Texas Instruments 6713 floating point DSPs running at 220 MHz for monitoring and calibration histogrammingPurpose: Route formatted data to   Level-2 and/or Slave DSPs (histogram)  15ROD Communication: RegistersThese communication registers are blocks of 32-bit words at the start of the DSP’s internal memoryThe Master DSP regularly checks Communication RegistersThis is done while inside of the main loop (infinite) of the DSP codeMDSP polls these registers watching for requests from the Host (user)Quite a few different types of Communication RegistersGeneral Status (number of current tasks running, is event trapping engaged, etc.)Dedicated Histogramming Registers (scan statistics, coordinate actions)Event Trapping Command & Status Registers (set trapping parameters, event statistics)General Purpose Command Register (receive host requests)Memory-map Register (set at initialization; points to the memory map in external memory)Task Status Register (indicates current state of tasks running on the ROD)Registers specific to Inter-DSP communicationCounter Register (ticks over with every iteration of the main loop)Reserved Misc. registers for debugging purposes 16ROD Communication: Lists and PrimitivesROD FPGA registers are mapped in the Master DSP memory spaceA system of primitives allows the MDSP to remain in control of its memory while receiving commands from the host Primitives exist for:• Reading and writing FPGA registers• Reading and writing regions of Master DSP memory• Load or modify Silicon Module configurations• Starting the Slave DSPsThe MDSP can send primitive lists to the Slave DSPs• Start calibration histogrammingIn general, primitives are executed once by the DSPDSP software is versatile enough to handle new primitives• These are simple and easy to writePrimitive replies (if any)Stored inside of a special “reply” bufferReply buffer is received by the Host after primitive execution. . .List LengthList IndexList Primitive CountList VersionList LengthList ChecksumPrimitive LengthPrimitive IndexPrimitive IDPrimitive PayloadDataPrimitive VersionPrimitive LengthPrimitive IndexPrimitive IDPrimitive PayloadDataPrimitive VersionList HeaderList TrailerPrimitive 1Primitive NPrimitive structure17Primitive List State MachinePrimitive List execution: Host-to-MDSP and MDSP-to-SDSPPrimitive List ExecutionIDLEPAUSED EXECUTING(one primitive per Iteration)PREPAREREPLYACKNOWLEDGE List Inserted?YESPause?Finished or Abort?NOYESAbort?Resume?YESinList Bit ClearedBy Host?NONOYESNONOYESNOYES18Text Buffer State MachineText Buffer reply: MDSP-to-Host and SDSP-to-MDSPError, Info, Diagnostic and Transfer (MDSP is a way-station for SDSP text buffers)Text Buffer TransmissionBuffer Occupied?Buffer Read?Read Request?IDLEWAITINGFROZENYESYESNONONOYES19ROD Communication: TasksPrimitives are typically executed once, Tasks execute ROD functions over an extended period of timeThese operate independent of the Primitive Lists• The ROD can still process Primitive Lists while various Tasks are runningTasks start and stop by the Host (or MDSP) sending Primitives• Run until they complete (or are halted manually)Examples of tasks:• Event Trapping Task (e.g., in preparation for calibration histogramming)• Histogramming Task (issuance of triggers, processing and binning events)20Pixel Calibration ScansA couple of common Pixel Scans:Threshold (on chip charge-injection for each individual pixel; scan the number of hits for each injected charge to obtain the discriminator threshold)Noise (a threshold scan without charge injection to measure noise)Threshold Scan resultshown for a single Pixel ModuleThreshold for each pixel(note the 16 FE ICs) 21SCT Calibration ScansSome of the commonly run SCT Calibration Scans:Rx Threshold Test (optimize the Rx threshold value in the BOC data-receiver chip)NMask Test (demonstrate that the chip mask register functions properly)Pipeline Test (scan the chip pipeline for defects; stuck on or off)Full Bypass Test (does the chip bypass feature function properly)Noise Occupancy Test (a threshold scan without charge injection to measure noise)Synchronous Trigger Noise Test (triggers distributed synchronously across the system)N-Mask Scan resultshown for a single SCT Module22ROD use during detector production / assemblyCalibration scans have been routinely used during module productionVerify module functionality before and after mounting or transportClassification of modules• Rank by the number of defects; only mount and install the very bestThis phase also gave feedback to ROD developers (and still continues to!)A few DSP software and FPGA Firmware problems have been uncovered this way• Some unique bugs appeared when stressing the ROD with a full complement of modules 23Cosmics Data-TakingTRT/SCT combined run in May 2006ROD performed well; successfully collected more than 400k eventsUseful DAQ/Detector shake-down, data for efficiency and alignment studiesOn-ROD Histogramming used for timingin the SCT during cosmics running(i.e., delay the signal by the correct amount)Transmission Delay(bunch-crossings)Number of coincident hitson top and bottom stripsNoiseSCTTransition Radiation Tracker (TRT)A total of 504 SCT Modules into 12 RODs were used during the combined cosmics data-taking run20 cm of concreteScintillatorScintillatorScintillator24ConclusionsThe Silicon Read-Out Driver (ROD)9U VME board hybrid of DSPs and FPGAs• Primary functions: module configuration, trigger propagation and data formatting• Secondary functions: module calibration and monitoring~260 production boards have been assembled (~160 for Pixels and ~100 for the SCT)• Many of these are now installed in their final crates inside of the ATLAS counting room The ROD has already been used extensivelyDuring module production and detector assembly at various sites around the worldTest-beam runs at CERN (most recently Summer 2004)Cosmics data taking during the combined TRT/SCT run in May 2006• Soon with the Pixel Endcap!Looking forward to larger-scale multi-crate testsThe SCT Barrels have been lowered into the ATLAS Collision Hall (testing in Winter 2006)Focus of future workSpeeding up the DSP code (shorten the time taken for configuration and calibration scans)Implementation of data-quality/detector monitoring during physics data-takingAutomated response for module recovery/reconfiguration (in conjunction with DAQ/DCS)25Additional Slides26SCT and TRT Detectors in the Collision HallBarrels inserted into the cryostat in late August 2006 27SCT RODs in USA-15Production RODs in their final location for ATLAS running28The SCT EndcapModule and Disk29The Pixel EndcapSector and Disks

A Read-out Driver for Silicon Detectors in ATLAS

Abstract

Similar works

Full text

Available Versions

CERN Document Server