14 research outputs found
Parallel, distributed and GPU computing technologies in single-particle electron microscopy
An introduction to the current paradigm shift towards concurrency in software
Neue Algorithmen zur Strukturbestimmung von MakromolekĂŒlen
Genaue Informationen ĂŒber die dreidimensionale Struktur biologischer MakromolekĂŒle sind von essentieller Bedeutung fĂŒr die Untersuchung biologischer Systeme. Mit Hilfe dieser Informationen ist es möglich, Ursachen von Erkrankungen die durch verĂ€nderte Proteinformen verursacht werden nachzuvollziehen und spezifische Medikamente zu entwickeln welche diese MolekĂŒle inhibieren. Leider ist es bis heute nicht möglich auf direktem, experimentellem Wege die dreidimensionale Struktur eines MolekĂŒls zu bestimmen. Alle derzeit verfĂŒgbaren Methoden (wie z.B. Röntgenkristallographie, EinzelmolekĂŒl Kryo-Elektronenmikroskopie oder NMR Spektroskopie) liefern experimentelle Daten die zunĂ€chst computergestĂŒtzt aufbereitet werden mĂŒssen um eine aussagekrĂ€ftige dreidimensionale Struktur zu erhalten. Somit haben die QualitĂ€t und die Effizienz dieser Berechnungen einen groĂen Einfluss auf die gesamte Strukturbiologie. Die derzeitigen Verbesserungen in Instrumentierung und Computerhardware erlauben die Entwicklung und Anwendung von Algorithmen welche frĂŒher unvertretbar lange Rechenzeiten in Anspruch genommen hĂ€tten. Dies gilt insbesondere fĂŒr die Methode der Kryo-Elektronenmikroskopie, welche im Vergleich die gröĂten Anforderungen an die computergestĂŒtzten Berechnungen stellt. In der hier besprochenen Arbeit werden neue Methoden zur Verwaltung und Bearbeitung der Ă€uĂerst groĂen Datenmengen, wie sie in der Kryo-Elektronenmikroskopie anfallen, vorgestellt und diskutiert. Unter Ausnutzung aktuellster Technologien (wie z.B. paralleles Programmieren auf Graphikkartenprozessoren) wurden neue Algorithmen zur Verbesserung der Bildverarbeitung entwickelt. Diese wurden in eine zu diesem Zweck entwickelte flexible, objekt-orientierte und intuitiv zu bedienende Bildverarbeitungssoftware integriert. Insbesondere wurde ein Algorithmus entwickelt der automatisch diejenigen EinzelmolekĂŒlbilder identifiziert, welche die dreidimensionale Rekonstruktion beeintrĂ€chtigen
Karabo-GUI: The Multi-Purpose Graphical Front-End for the Karabo Framework
The Karabo GUI is a generic graphical user interface (GUI) which is currently developed at the European XFEL GmbH. It allows the complete management of the Karabo distributed control and data acquisition system. Remote applications (devices) can be instantiated, operated and terminated. Devices are listed in a live navigation view and from the self-description inherent to every device a default configuration panel is generated. The user may combine interrelated components into one project. Such a project includes persisted device configurations, custom control panels and macros. Expert panels can be built by intermixing static graphical elements with dynamic widgets connected to parameters of the distributed system. The same panel can also be used to graphically configure and execute dataanalysis workflows. Other features include an embedded IPython scripting console, logging, notification and alarm handling. The GUI is user-centric and will restrict display or editing capability according to the userâs role and the current device state. The GUI is based on PyQt technology and acts as a thin network client to a central Karabo GUI-Server
The Karabo distributed control system
The Karabo distributed control system has been developed to address the challenging requirements of the European X-ray Free Electron Laser facility, including complex and custom-made hardware, high data rates and volumes, and close integration of data analysis for distributed processing and rapid feedback. Karabo is a pluggable, distributed application management system forming a supervisory control and data acquisition environment as part of a distributed control system. Karabo provides integrated control of hardware, monitoring, data acquisition and data analysis on distributed hardware, allowing rapid control feedback based on complex algorithms. Services exist for access control, data logging, configuration management and situational awareness through alarm indicators. The flexible framework enables quick response to the changing requirements in control and analysis, and provides an efficient environment for development, and a single interface to make all changes immediately available to operators and experimentalists
The Calibration Pipeline for the Large 2D Pixel Detectors with Mhz Readout at the European XFEL
Integrated Detector Control and Calibration Processing at the European XFEL
The European X-ray Free Electron Laser is a high-intensity X-ray light source currently being constructed in the area of Hamburg, that will provide spatially coherent X-rays in the energy range between and . The machine will deliver , consisting of up to , with a repetition rate. The LPD, DSSC and AGIPD detectors are being developed to provide high dynamic-range Mpixel imaging capabilities at the mentioned repetition rates. A consequence of these detector characteristics is that they generate raw data volumes of up to . In addition the detectors on-sensor memory-cell and multi-/non-linear gain architectures pose unique challenges in data correction and calibration, requiring online access to operating conditions and control settings. We present how these challenges are addressed within XFELs control and analysis framework Karabo, which integrates access to hardware conditions, acquisition settings (also using macros) and distributed computing. Implementation of control and calibration software is mainly in Python, using self-optimizing (py) CUDA code, numpy and iPython parallels to achieve near-real time performance for calibration application
Karabo: An Integrated Software Framework Combining Control, Data Management, and Scientific Computing Tasks
The expected very high data rates and volumes at the European XFEL demand an efficient concurrent approach of performing experiments. Data analysis must already start whilst data is still being acquired and initial analysis results must immediately be usable to re-adjust the current experiment setup. We have developed a software framework, called Karabo, which allows such a tight integration of these tasks. Karabo is in essence a pluggable, distributed application management system. All Karabo applications (called âDevicesâ) have a standardized API for self-description/configuration, program-flow organization (state machine), logging and communication. Central services exist for user management, access control, data logging, configuration management etc. The design provides a very scalable but still maintainable system that at the same time can act as a fully-fledged control or a highly parallel distributed scientific workflow system. It allows simple integration and adaption to changing control requirements and the addition of new scientific analysis algorithms, making them automatically and immediately available to experimentalists
Calibration and Calibration Data Processing Concepts at the European XFEL
The European X-ray Free Electron Laser (Altarelli, 2006) is a high-intensity X-ray light source currently being constructed in Hamburg, Germany, that will provide spatially coherent X-rays in the energy range between 0.25 keV and 25 keV. The machine will deliver a unique time structure, consisting of up to 2700 pulses, with a 4.5 MHz repetition rate, 10 times per second at very high photon fluxes up to 1017 photons/s (Tschentscher, 2012). The LPD (Hart, 2012; Koch, 2013), DSSC (Porro, 2010, 2012; Lutz, 2010) and AGIPD (Graafsma, 2009) detectors are being developed to provide Mpixel imaging capabilities at the aforementioned repetition rates for a dynamic range spanning from single photon sensitivity to 104 â105 photons per pixel. The detectors are optimized for specific energy ranges. A direct consequence of the aforementioned detectorsâ characteristics is that they generate raw data volumes unprecedented in photon science, ranging up to 1Mpixel x 640 memory cells x 10 pulse/s x 16 bit, i.e. 12.8 Gbyte/s. On-detector vetoing may not necessarily lower these rates much - a memory cell freed by a vetoed pulse may be used by data from one of the remaining 2700 pulses a train consists of. The PC-layer may reduce this data amount by additional software triggering, but this is not guaranteed. Figure 1 gives an overview of the different data products at European XFEL, as well as their flows and involved user roles, under the assumption that processing takes place within XFELâs Karabo framework (Heisen, 2013). In addition to the high data rates, the Mpixel detectorsâ on-sensor memory-cell and multi-gain-stage architectures necessary for the high dynamic range, pose unique challenges in detector-specific data corrections and calibration (Weidenspointner, 2012; Sztuk-Dambietz, 2013a). These challenges are addressed by providing a dedicated and thoroughly characterized set of test stands, which utilize continuous sources (Fe-55, X-ray tubes) as well as a pulsed setup: PulXar (Sztuk-Dambietz, 2013b), which is designed to produce X-ray pulses of 50-150 ns duration, within a 0.6 ms burst followed by a 99.4 ms gap. The radiation it produces thus closely matches the XFEL pulse structure. Additionally, simulation tools are being developed to assist in detector characterization (Bohlen and Joy, 2013)
The Karabo distributed control system
The Karabo distributed control system has been developed to address the challenging requirements of the European X-ray Free Electron Laser facility, including complex and custom-made hardware, high data rates and volumes, and close integration of data analysis for distributed processing and rapid feedback. Karabo is a pluggable, distributed application management system forming a supervisory control and data acquisition environment as part of a distributed control system. Karabo provides integrated control of hardware, monitoring, data acquisition and data analysis on distributed hardware, allowing rapid control feedback based on complex algorithms. Services exist for access control, data logging, configuration management and situational awareness through alarm indicators. The flexible framework enables quick response to the changing requirements in control and analysis, and provides an efficient environment for development, and a single interface to make all changes immediately available to operators and experimentalists.</p