We present a kernel implementation of the LHCb MEP protocol. MEP is implemented in the IP stack as a loadable module. This allows for better monitoring at the network level and can potentially reduce the overhead associated with the reception of the data

Neufeld, N

Olejnik, L

CERN Document Server

LHCb-PUB-2010-00427/08/2012Linux implementation of the MEP protocol for theLHCb experiment Lukasz Olejnik∗Niko Neufeld†December 8, 2010AbstractWe describe a kernel implementation of the LHCb Multi Event Pro-tocol. MEP is implemented in the IP stack as a loadable module. Thisallows for better monitoring at the network level and can potentiallyreduce the overhead associated with the reception of the data.Document created September 15, 2009Last revised December 8, 2010Report number LHCb-PUB-2010-00 CERN-LHCb-PUB-2010-004∗PSNC, Poznan†CERN, PH-Department1 Introduction1.1 The LHCb trigger and DAQ systemsThe LHC experiments will generate enormous amounts of data per second.This results from the fact that every 25 ns (40 MHz) the collisions willspark with events that will be detected. Even though the LHCb experimentwill operate with a reduced (with respect to the general-purpose detectors)luminosity of 2× 1032 cm−1s−1 (therefore, the crossing rate is also reducedto rate of 10 MHz). In order to efficiently cope with such an unprecedentedamount of data, every experiment takes advantage of a trigger mechanism.A trigger is a system that indicates the useful events which should be furtheranalyzed. In the LHCb experiment, there are two triggers[2, 3]:1. The first level trigger (L0) - the first trigger, located very close tothe detectors. It consists of custom electronics and performs read-outat the rate of 1 MHz, using partial information of the detector. Thedata goes to Front-End Electronics (FEE) where A/D conversion isperformed1. Then, the data (if accepted) is sent via wires to Tell1boards[4]. Those boards give a shape to the network traffic by encap-sulating the events in an MEP packet and sending them via Ethernet(with the use of IP).2. The High Level Trigger (HLT) - the software trigger is dividedinto two levels• HLT1 the first level of the software trigger with a rate of 30 kHz• HLT2 - the second level of the software trigger has a rate of2 kHz and this data are forwarded to storage - for oﬄine analysis.The High Level Trigger (HLT) is a computing farm located in the Point8, 100 m underground, in a shielded cavern close to the detector. Itconsists of about 1700 CPU nodes.1.2 Introduction to the MEP protocolTypically, LHCb event2 is of the 100 B size, per trigger. In order to reducethe overhead of network headers, multiple events are packed into singlenetwork packet. For this purpose, Multi Event Protocol (MEP)[6] is used fortransmission of large numbers of events. It consists of a 12 B header, whichis followed by events (every event in a packet has one additional pseudo-header which describes it and allows reconstruction) and it works in theTransport Layer (layer 4) of the OSI Model - Like UDP and TCP, on top ofthe IP protocol. The protocol is, like UDP, connection-less, unsynchronized1With the exception of VELO detector2 An event is a current state of the detectors. For more information, refer to [5].1and unreliable - there are no retransmissions. Therefore, in order to reducethe number of lost packets there is a need for a communication channel withgood characteristics (quality) and enough memory in the nodes (to minimizethe queues on their sides). On the HLT side, the MEP protocol is being usedfor receiving data, exclusively.2 MEP implementationCurrently, the high level trigger software is utilizing the Linux RAW socketmechanism to obtain data from DAQ. In the Linux kernel’s IP protocolhandler function, if a packet under consideration is neither UDP nor TCP,it goes into the RAW section. However, the more elegant solution would beto have the MEP protocol code available inside the kernel - in the form of akernel module. It would aid the testing phase, and also could theoreticallybe slightly faster.2.1 AnalysisIt was decided to implement the MEP protocol with a blueprint of the UDPimplementation, as it was the most simple solution (the RAW implementa-tion was also considered at one point).Linux kernel source had to be consulted and analyzed, in particular -UDP, RAW and elements of IP implementations, various data structuresthat are essential in the process of the loading of a protocol and other doc-uments. It was not the only problem at the very beginning of the project.At the time of designing, the HLT farm consisted exclusively of ScientificLinux 4 with Linux kernel version 2.6.9. However, a short migration to thenewer Scientific Linux 5, with Linux 2.6.18 was already scheduled. Tak-ing into account that the years between the two of those kernel releasesresulted in a number of API changes, two possible approaches could havebeen used: either to develop a module that would be consistent with bothkernel releases, or to develop only one version and therefore focus solely onthe task itself (and not the kernel evolution). The focus was then on a 2.6.18implementation.Another problem was the unavoidable nature of kernel development,which often means frequent oopses and kernel panics. This is why it wasdecided to utilize a virtual environment for the development. VirtualBoxwas used from the start to the end of the project. This proved to be a gooddecision, since not only it was a good development aid, but also provideddebugging during the kernel panics - the full information could had beensent via a virtual serial console (to the host OS) for later debugging.22.2 MEP - implementationThis part describes how does the module work at the hight level. Since MEPis implemented as a module, it has got the usual module-related functions -one for the (correct) loading and unloading (which performs necessary clean-ings). These functions are called mep_init and mep_exit. As was statedbefore, MEP implementation is based on the source code of the kernel’s UDPimplementation, with some modifications. There are several important dif-ferences. One example may be the concept of “ports“. Since DAQ is justone-way data receiving task in one ”tunnel“, there is no need of the UDP’s65535 ports being in use. So the simplest way was to explicitly choose one“port” - a socket, and free the remaining memory. Other significant distinc-tion is not removing the headers, because information contained in them isrequired for the HLT event reconstruction.2.2.1 Protocol insertionThe purpose of mep_init - a function that loads the module - is to reg-ister the MEP protocol. It performs this task by executing the proto_-register function. This function takes the mep_prot - an instance ofproto structure as an argument. The next step is to add a protocol bycalling the standard kernel function inet_add_protocol (it takes mep_-protocol as an argument, which is an instance of the net protocol structurewith a pointer to a crucial function for the receiving process; it is the placewhere the ipproto’s value3[7] of the MEP protocol is stated as 242). Thefinal call of inet_register_protosw finishes the protocol insertion. Aftera successful protocol registration, the appropriate entry should be visiblein /proc/net/protocols. The mep_proc_init and mep_proc_init2 functionsprepare the files with statistics in the virtual proc file system.2.2.2 Protocol removalThe protocol removal performs a clean removal of the MEP protocol imple-mentation (module). It does that by executing functions that undo every-thing that was done during the loading phase. This is of course done in theopposite order.2.3 MEPThis section describes the core functions of the MEP module. The architec-ture is based on the standard Linux kernel coding style. Additionally, somespecific functionality is implemented to enable debugging.3More up to date resources are http://www.iana.org/assignments/protocol-numbers/protocol-numbers.xml or just /etc/protocols on a Linux system32.3.1 The core functionsThe two core functions are:• mep_rcv - this function is always called when a packet with ipproto =242 field arrives. It is called by the lower (IP) layer. First of all,the function checks if the packet contains a header and if there is alistening socket. If this is true, the udp_queue_rcv_skb function iscalled. Then, two indexes are computed (according to the source IPaddress in the packet). If this index is a legitimate one, the mep_-stats[index]4 is incremented. If not, the mep_err variable is. Theprocess of querying whether there is a listening socket is performed bythe udp_v4_lookup_longway function.• mep_recvmsg is the last function working in layer 4. It calls the stan-dard skb_copy_datagram_iovec function which in turn copies thedata to the user-space and fills the sockaddr in structure. Unlike UDP,it leaves the MEP header.2.3.2 The CountersThe two statistics-gathering files are called mep and mep2. They reside inthe /proc/net directory. The first one is similar to /proc/net/udp (and/ortcp). The second one is a per-node statistics counter. It has two columns -the first one being the node-number (which can be translated into an IP, butfor performance reasons this is done in the userspace by a simple Pythonscript), and the other - a counter, tracking the number of packets whichhave arrived so far from that node). Like the UDP counterparts, they canbe accessed when a socket is open.2.4 The use of MEP socketThe MEP socket can be used just like every other socket, by a call to thesocket(2) function, such as:socket(AF_INET, SOCK_DGRAM, 242)And then, the subsequent calls to, for example, recvfrom (after a bind) willresult in the new data being received.4It is an array of 5376 elements, sufficient (in reality: much more than sufficient) tolog the packets from the Tell1 boards which have the addresses of the form X.X.[0-20].X.In reality, the last octet is always much smaller than 255, but it is hard to tell how manychannels (and boards) there will be in the future, especially considering the scenario wherethe full read-out of L0 will be deployed42.5 Maintenance considerationsThe first maintenance decisions had to be made even before anything wasdone: which kernel version(s) to address. It is obvious that the Linux kernelsource changes with time. Even though this particular region of kernel isusually not the one that is frequently (e.g. in two subsequent releases)changed, these eventual (future) changes has to be taken into considerationbefore a large deployment is made. The kernel in the deployed systemschanges only with the upgrade of distribution (Scientific Linux), so it canbe assumed that the new version of SCL differs by a number of kernel releasesto the previous one. Therefore, before the general system upgrade, it has tobe established: whether anything (in the kernel source) was changed and willit still work after the system upgrade, updating the socket’s implementationwhen needed. So the operating principles behaviour in environments withcustom kernel modules apply.3 ConclusionThe project proved to be very interesting. First of all, a substantial amountof Linux Linux kernel source code has to be read and understood and sec-ondly, coping with various problems that arised during the kernel develop-ment was a significant part of the work on the kernel development. In orderto succeed with this task, an advanced knowledge of fundamental conceptsof network operation and protocols (TCP/IP) had to be used, as well astools - such as standard web reconnaissance, but also the ones capable ofsending custom network packets (like scapy, hping3), the ones for debuggingprocess, virtual environment from VirtualBox. In addition, the implemen-tation was kept secure against any (unlikely) security threats. Performancecomparison with respect to the RAW implementation was also done on dataprovided by the FEST-Injector. This showed that the implementation is atleast as fast as the RAW implementation is. Varying speedups were in therange of 0.5− 2%.The working (stable) version has only receiving capabilities. This, how-ever is enough, since DAQ is meant to do a one way data transfer 5 - readthe dedectors data. The implementation is slightly faster with comparisonto the RAW (and UDP) socket.The MEP module described in this note is under considerations forintegration with the (production) HLT farm and will become an integralpart of the LHCb Online Data Acquisition System. In closing, it is worthnoting that the implementation discussed in this note is suitable for anyenvironment[8] where Tell1 boards are utilized, and not only LHCb.5A development version exists, capable of sending MEPs. But, since additional symbolsneed to be exported in the Linux kernel, a kernel recompilation would be absolutelyrequired. Therefore it is unclear whether there is any need for such a solution.5References[1] LHCb Collaboration, LHCb Online System Technical Design Report,CERN-LHCC-2001-040, 2001[2] LHCb Collaboration, LHCb Trigger System Technical Design Report,CERN-LHCC-2003-031, 2003[3] E. Rodriguesa, The LHCb Trigger System, Nuclear Physics B (Proc.Suppl.), vol. 170, pp. 298–302, 2007. 23, 24[4] G. Haefeli, A. Bay, A. Gong, H. Gong, M. Muecke, N. Neufeld, O. Schnei-der, The LHCb DAQ interface board TELL1, Nucl. Instrum. and Meth.A 560 (2006) 494–502[5] G. Liu, F. Frontera, M. Savrie`, N. Neufeld, Management, Optimizationand Evolution of the LHCb Online Network, CERN-THESIS-2010-040,2010[6] B. Jost and N. Neufeld, Raw-data transport format, 2004. technical doc-ument.[7] J. Postel, ASSIGNED NUMBERS, RFC: 790, http://tools.ietf.org/html/rfc790[8] B. Angeluccia, C. Avanzinia, G. Collazuol, S. Galeottic, E. Imbergamod,G. Lamannab, G. Magazzu, G. Ruggierob, M. Sozzia, S. Vendittia, In-tegrated Trigger and Data Acquisition system for the NA62 experimentat CERN, TWEPP-09: Topical Workshop on Electronics for ParticlePhysics, Paris, France, 21 - 25 Sep 2009, pp. 179-182A Other tasksAppendices are stripped from this note’s version.6

Linux implementation of the MEP protocol for the LHCb experiment

Abstract

Similar works

Full text

Available Versions

CERN Document Server