Makowski, D

Napieralski, A

Swiercz, B

CERN Document Server

EU contract number RII3-CT-2003-506395 CARE-Conf-06-071-SRF   SRF       Research of Fault-Tolerant Computing Using COTS Elements   B. ´Swiercz, D. Makowski, A. Napieralski  Department of Microelectronics and Computer Science Technical University of Lodz Al. Politechniki 11, 93-590 Lodz, Poland     Abstract  The problem of designing radiation tolerant systems is crucial for modern physics. Reliable control systems able to operate in radiation environment are necessary to carry out physics experiments. This paper highlights the operating system, which is used to develop software fault tolerant environment for the reliable computing. The main idea is to implement fault tolerant algorithms into a kernel instead of user-space programs. In this paper a new real-time kernel, called sCore, is proposed. The sCore is a modern, multitasking kernel, designed for real-time applications.     Contribution to the NSTI Nanotech 2006, Boston (USA)     Work supported by the European Community-Research Infrastructure Activity under the FP6 “Structuring the European Research Area” programme (CARE, contract number RII3-CT-2003-506395) Research of Fault-Tolerant Computing Using COTS ElementsB. S´wiercz, D. Makowski, A. NapieralskiDepartment of Microelectronics and Computer ScienceTechnical University of LodzAl. Politechniki 11, 93-590 Lodz, Polandswierczu@dmcs.pl, dmakow@dmcs.pl, napier@dmcs.plABSTRACTThe problem of designing radiation tolerant systemsis crucial for modern physics. Reliable control systemable to operate in radiation environment are necessaryto carry out physics experiments. This paper highlightsthe operating system, which is used to develop a soft-ware fault tolerant environment for the reliable com-puting. The main idea is to implement fault tolerantalgorithms into a kernel instead of user-space programs.In this paper a new real-time kernel, called sCore, isproposed. The sCore is a modern, multitasking kernel,designed for real-time applications.Keywords: Single Event Upset, Radiation Environ-ment, Fault Tolerant System, Hardened, Real-Time Sys-tem1 BACKGROUND OF THERESEARCHX-Ray Free Electron Laser (X-FEL) [1, 2], designedat DESY Research Center in Hamburg, is the 4th gen-eration of synchrotron accelerator [3, 4]. Total lengthof the X-FEL will be 3.4 km with 2.1 km of acceleratortunnel length. Distributed control system is necessaryto control all subsystems of X-FEL’s modules. Somepart of electronic equipment (DSP and FPGA boards,embedded computers and microprocessors systems) de-signed for control and data acquisition will be placedinside accelerators tunnel close to cavities – the mainpart of accelerator. During normal operation cavitiesproduce gamma and neutron radiation. Neutrons (espe-cially slow called thermal neutrons) can affect memoryand registers in digital circuits and cause on illegal op-eration and thus loss of the system functionality [5, 6].Single memory malfunction is known as a Single EventUpset (SEU) [7–9]. Microprocessor system placed in-side the accelerator tunnel are subjected to neutron ra-diation. To ensure reliable operation it is necessary toprotect digital circuits against radiation [10]. The sim-ple way to protect control systems against SEU is touse hardend computer systems designed for cosmic andmilitary application but keeping in mind total length ofthe accelerator tunnel it will be too expensive for X-FELproject. Other solution, much cheaper but more com-plex, is to use Commercial of The Shelf Components(COTS) connected with system redundancy and safetyalgorithms. Protective algorithms can by used on hard-ware layer (for example embedded inside FPGA chips)or on software one. The authors decided to implementSEU-tolerant algorithms inside the sCore operating sys-tem kernel. The sCore is designed to work with stan-dard computer architecture such as PC computers orembedded versions used by industry. There is knownmany frameworks and libraries supporting fault tolerantenvironments but this approach requires good knowl-edge about SEUs nature from application programmers.Other approach is to use a special kernel version withembedded protection algorithms which allows to developand run applications not able to tolerate SEUs.2 THE SCORE KERNELThe sCore is multitasking preemptive kernel basedon the microkernel architecture [11]. It is design to workon IA-32 architecture (standard PC platform and alsoembedded industrial computers) but thanks to C++ ab-straction layer sCore code could be translated to dif-ferent platforms (there are plans to port on ARM andPowerPC architectures). The sCore can by used todevelop real-time applications with a constant time oftasks switching. The sCore has built in Round Robinscheduler with 256 priority levels queue. All internalstructures are static and initialised during system boot-strap. This technique ensures stability and predictabletime dependences. Dynamic memory allocation insidekernel are not served by kernel.2.1 The EDAC TaskRadiation tolerant sCore should provide mechanismto find and correct bit-flip errors inside memory. StaticRandom Access Memory (SRAM) technology, used todevelop fast buffers memory and registers, is suscepti-ble to the SEU effects. Dynamic Random Access Mem-ory (DRAM), payload for the large main system mem-ory, is also sensitivity to the SEU effect. Soft errorsinside memory can break the system down. To protectsystem memory special task, called EDAC Task, wasimplemented. The EDAC Task (fig. 1) divides systemmemory into three parts.Memory avilable for system and applicationsEDAC TaskProgram memoryTextDataSystem memoryDataTextCopy 1Copy 2Figure 1: EDAC Task and memory organisationscheme.The first memories region is used to work by sys-tem (sCore kernel) and running applications. The sec-ond and third part of memory are used exclusively byEDAC Task for storage copy of read-only data from thefirst memory region. Read-only data (machine codesand constant variables) are organized by linker to spe-cial program section called text section. When systemis started EDAC Task copies text section of kernel andall application into the second and third memory region.During system operation EDAC Task periodically com-pare the first and second memory region. If EDAC Taskfind a difference between the first and second memoryparts it will compare all three memory regions to choosethe correct one and restore first memory region usingvoting technique. The EDAC Task is run periodicallyand simultaneously to other applications.3 EXPERIMENTS WITH THESCORE KERNELThere were carried out two experiments with thesCore kernel run on the standard PC computer. PCcomputer was placed near neutron source (fig. 2). Thefirst experiment was done with the 241AmBe neutronsource and the second was done inside Liniac2 accelera-tors tunnel near the electron-positron converter. IArad-Sim [12], special version of Bochs IA-32 architecture em-ulator was used to simulate kernel sensitivity to SEUbefore experiments. The standard PC computer withPentium III 500MHz processor and 128 MB of RAMmemory was used for experiments. Apart from EDACTask, two other application was run on the sCore. Thefirst application was simple shell and the second wasMD5 calculation application. MD5 algorithm was cho-sen to calculate MD5 sum for a initialised memory re-gion. MD5 application was run in infinite loop and cal-culated value was compared with a constant value cal-culated during kernel compilation. If MD5 sum equals aconstant it means that there was not any error in mem-ory or errors were corrected by EDAC Task.3.1 Radiation environment around the241AmBeThe 241AmBe izotop was used as a neutrons source(fig. 2). The 241AmBe was placed inside the water mod-erator to slow down neutrons (thermal neutrons gener-ate more SEUs then fast neutrons).AmBeModeratorsCoreFigure 2: Experiment with 241AmBe.The experiment was conducted during 22 hours. Sys-tem activity during experiment is shown in fig. 3. Therewas nine errors (SEUs) in memory (level 6 and 4 in fig3). All errors were corrected by EDAC Task. Only oneSEU was detected by MD5 application and was not cor-rected by EDAC Task. 0 1 2 3 4 5 6 7 816/0518:0016/0520:0016/0522:0017/0500:0017/0502:0017/0504:0017/0506:0017/0508:0017/0510:0017/0512:0017/0514:0017/0516:00(2) System initialization(4) MD5 app error(5) MD5 app finished(6) EDAC Task error(7) EDAC Task finishedFigure 3: System activity diagram during experimentwith 241AmBe.3.2 Radiation environment inside theLiniac2 tunnelLiniac2 accelerator is used for experiments insidetunnel because of VUV-FEL and X-FEL acceleratorsare still under development. The computer with sCoresystem was installed close to the electron-positron con-verter (fig. 4). Electron-positron converter generatespositrons for physic experiments but also gamma andneutrons are generated as a parasitic effect. There wasdone two experiments inside Liniac2. The first experi-ment was done during 26 hours and the second takes 24hours. When the electron-positron converter was turnedon, neutron radiation was generated and sCore receivedunknown interrupt (see in fig. 5, level 3) number 13.Interrupt number 13 is called General Fault by Intelnomenclature. The sCore kernel was not able to workinside Liniac2 tunnel when the electron-positron con-verter was running.4 CURRENT AND FUTURE WORKThe current work is focused on the advantage of vir-tual memory management provided by modern proces-sors. Instead of scanning all system memory (the firstmemory region scanned by EDAC Task) all memorypages has set dirty bit which means that every accessto the memory generates interrupt. The ISR procedurewhich serve interrupt compares demanded pages withits copy and clears dirty bit. This technique is fasterand more effective then EDAC Task but it still requiresmemory redundancy for keeping working copies. Moresophisticated technique is to use hardware debuggingmode provided by processors. Currently, the service ofsCoreconverterelectron−positronRestricted areaFigure 4: Experiment inside the Liniac2 tunnel. 0 1 2 3 4 5 6 7 820/0510:0020/0512:0020/0514:0020/0516:0020/0518:0020/0520:0020/0522:0021/0500:0021/0502:0021/0504:0021/0506:0021/0508:0021/0510:0021/0512:00(0) System restarted by watchdog(2) System initialization(3) Interrupt(4) MD5 app error(5) MD5 app finished(6) EDAC Task error(7) EDAC Task finishedFigure 5: System activity diagram during experimentinside the Liniac2 tunnel.debug mode is implementing inside sCore kernel. Usingdebug mode is possible to run application step by stepand it is easy to watch every access to the memory cell.Moreover, sCore with built in debug mode service willbe able to repeat every process instruction three timesand compare results (this technique is based on transac-tion mechanism). Therefore sCore should be ported tothe other architecture (such as ARM and PowerPC) tocooperate with many types of microprocessors systemsand embedded computers.5 CONCLUSIONDuring experiments it was possible to observe be-haviour of PC architecture under neutron radiation im-pact. Experiment with the 241AmBe shown that sim-ple redundant technique (EDAC Task) can protect userapplication and prevent SEUs influence on the systemmemory. On the other hand, because of high radia-tion level, experiment inside Liniac2 tunnel close to theelectron-positron converter shown that EDAC Task isalmost useless. When the electron-positron converterwas working average six SEUs events per second insidememory was observed. Neutrons intensity inside Liniac2caused errors not only in main memory but also in pro-cessors registers. The EDAC Task can be useful onlyfor weak radiation environment. The future work willfocus on fault tolerant technique to protect micropro-cessor system from high radiation level comparable tothe radiation generated by electron-positron converter.6 ACKNOWLEDGEMENTSWe acknowledge the support of the EuropeanCommunity-Research Infrastructure Activity under theFP6 ”Structuring the European Research Area” pro-gram (CARE, contract number RII3-CT-2003-506395),and Polish National Science Council Grant ”138/E-370/SPB/6.PR UE-DIE 354/2004-2007”.REFERENCES[1] A. Schwarz, ”The European X-Ray free electronlaser project at DESY”, 26th International Free-Electron Laser Conference, pp. 85”89, Sierpien´2004.[2] W. Shi, ”SASE X-Ray Free Electron Laser InDESY”, Journal of the Society of Chinese Physists,vol. 6, pp. 5 ”16, Grudzien´ 2000.[3] R. Brinkmann, K. Flottmann, J. Rosbach, P.Schmuser, N. Walker, H. Weise, ”TESLA TechnicalDesign Report - The Accelerator, part II”, DESY,2001.[4] G. Materlik, T. Tschentscher, ”TESLA TechnicalDesign Report. The X-Ray Free Electron Laser,PART V”, DESY, 2001.[5] D.Makowski, M. Grecki, B. Mukherjee, B. S´wiercz,S. Simrock, ”Radiation Tolerant System for Neu-trons Measurement”, 12th Mixed Design of Inte-grated Circuits and Systems, MIXDES, lipiec 2005.[6] G. Messenger, M. Ash, ”The Effects of Radiationon Electronic Systems”, ISBN 0-442-25417-2. VanNostrand Reinhold Company Inc., 1986.[7] R. Peterson, ”Radiation-induced errors in memorychips”, Brazilian Journal of Physics, vol. 33, nr 2,pp. 246”249, czerwiec 2003.[8] F. Giustino, ”Radiation Effects on SemiconductorDevices”, PhD thesis, Politecnico di Torino, Marzec2001.[9] D.M. Fleetwood, H. A. Eisen, ”Total-dose radia-tion hardness assurance”, Nuclear Science, IEEETransactions, vol. 50, pp. 552” 564, czerwiec 2003.[10] D. Makowski, B. S´wiercz, M. Grecki, A.Napieralski, (in Polish) ”Projektowanie sys-temo´w niewraz˙liwych na wpÃlyw promieniowania napotrzeby akceleratora X-FEL”, Elektronika - Kon-strukcje, Technologie, Zastosowania, nr 7/2005.[11] B. S´wiercz, D. Makowski, A. Napieralski, ”ThesCore - Operating System for Research of Fault-Tolerant Computing”, 12th Mixed Design of Inte-grated Circuits and Systems, MIXDES, lipiec 2005.[12] B. S´wiercz, D. Makowski, A. Napieralski, ”IArad-Sim - IA32 architecture under high radiation en-vironment simulator”, 2005 NSTI NanotechnologyConference and Trade Show, Nanotech 2005, SmartSensors and Systems

Research of Fault-Tolerant Computing Using COTS Elements

Abstract

Similar works

Full text

Available Versions

CERN Document Server