Parallel, distributed and GPU computing technologies in single-particle electron microscopy

An introduction to the current paradigm shift towards concurrency in software

Neue Algorithmen zur Strukturbestimmung von Makromolekülen

Genaue Informationen über die dreidimensionale Struktur biologischer Makromoleküle sind von essentieller Bedeutung für die Untersuchung biologischer Systeme. Mit Hilfe dieser Informationen ist es möglich, Ursachen von Erkrankungen die durch veränderte Proteinformen verursacht werden nachzuvollziehen und spezifische Medikamente zu entwickeln welche diese Moleküle inhibieren. Leider ist es bis heute nicht möglich auf direktem, experimentellem Wege die dreidimensionale Struktur eines Moleküls zu bestimmen. Alle derzeit verfügbaren Methoden (wie z.B. Röntgenkristallographie, Einzelmolekül Kryo-Elektronenmikroskopie oder NMR Spektroskopie) liefern experimentelle Daten die zunächst computergestützt aufbereitet werden müssen um eine aussagekräftige dreidimensionale Struktur zu erhalten. Somit haben die Qualität und die Effizienz dieser Berechnungen einen großen Einfluss auf die gesamte Strukturbiologie. Die derzeitigen Verbesserungen in Instrumentierung und Computerhardware erlauben die Entwicklung und Anwendung von Algorithmen welche früher unvertretbar lange Rechenzeiten in Anspruch genommen hätten. Dies gilt insbesondere für die Methode der Kryo-Elektronenmikroskopie, welche im Vergleich die größten Anforderungen an die computergestützten Berechnungen stellt. In der hier besprochenen Arbeit werden neue Methoden zur Verwaltung und Bearbeitung der äußerst großen Datenmengen, wie sie in der Kryo-Elektronenmikroskopie anfallen, vorgestellt und diskutiert. Unter Ausnutzung aktuellster Technologien (wie z.B. paralleles Programmieren auf Graphikkartenprozessoren) wurden neue Algorithmen zur Verbesserung der Bildverarbeitung entwickelt. Diese wurden in eine zu diesem Zweck entwickelte flexible, objekt-orientierte und intuitiv zu bedienende Bildverarbeitungssoftware integriert. Insbesondere wurde ein Algorithmus entwickelt der automatisch diejenigen Einzelmolekülbilder identifiziert, welche die dreidimensionale Rekonstruktion beeinträchtigen

Karabo-GUI: The Multi-Purpose Graphical Front-End for the Karabo Framework

The Karabo GUI is a generic graphical user interface (GUI) which is currently developed at the European XFEL GmbH. It allows the complete management of the Karabo distributed control and data acquisition system. Remote applications (devices) can be instantiated, operated and terminated. Devices are listed in a live navigation view and from the self-description inherent to every device a default configuration panel is generated. The user may combine interrelated components into one project. Such a project includes persisted device configurations, custom control panels and macros. Expert panels can be built by intermixing static graphical elements with dynamic widgets connected to parameters of the distributed system. The same panel can also be used to graphically configure and execute dataanalysis workflows. Other features include an embedded IPython scripting console, logging, notification and alarm handling. The GUI is user-centric and will restrict display or editing capability according to the user’s role and the current device state. The GUI is based on PyQt technology and acts as a thin network client to a central Karabo GUI-Server

The Karabo distributed control system

The Karabo distributed control system has been developed to address the challenging requirements of the European X-ray Free Electron Laser facility, including complex and custom-made hardware, high data rates and volumes, and close integration of data analysis for distributed processing and rapid feedback. Karabo is a pluggable, distributed application management system forming a supervisory control and data acquisition environment as part of a distributed control system. Karabo provides integrated control of hardware, monitoring, data acquisition and data analysis on distributed hardware, allowing rapid control feedback based on complex algorithms. Services exist for access control, data logging, configuration management and situational awareness through alarm indicators. The flexible framework enables quick response to the changing requirements in control and analysis, and provides an efficient environment for development, and a single interface to make all changes immediately available to operators and experimentalists

Integrated Detector Control and Calibration Processing at the European XFEL

The European X-ray Free Electron Laser is a high-intensity X-ray light source currently being constructed in the area of Hamburg, that will provide spatially coherent X-rays in the energy range between $0.25\,\mathrm{keV}$ and $25\,\mathrm{keV}$. The machine will deliver $10\,\mathrm{trains/s}$, consisting of up to $2700\,\mathrm{pulses}$, with a $4.5\,\mathrm{MHz}$ repetition rate. The LPD, DSSC and AGIPD detectors are being developed to provide high dynamic-range Mpixel imaging capabilities at the mentioned repetition rates. A consequence of these detector characteristics is that they generate raw data volumes of up to $15\,\mathrm{Gbyte/s}$. In addition the detectors on-sensor memory-cell and multi-/non-linear gain architectures pose unique challenges in data correction and calibration, requiring online access to operating conditions and control settings. We present how these challenges are addressed within XFELs control and analysis framework Karabo, which integrates access to hardware conditions, acquisition settings (also using macros) and distributed computing. Implementation of control and calibration software is mainly in Python, using self-optimizing (py) CUDA code, numpy and iPython parallels to achieve near-real time performance for calibration application

Karabo: An Integrated Software Framework Combining Control, Data Management, and Scientific Computing Tasks

The expected very high data rates and volumes at the European XFEL demand an efficient concurrent approach of performing experiments. Data analysis must already start whilst data is still being acquired and initial analysis results must immediately be usable to re-adjust the current experiment setup. We have developed a software framework, called Karabo, which allows such a tight integration of these tasks. Karabo is in essence a pluggable, distributed application management system. All Karabo applications (called “Devices”) have a standardized API for self-description/configuration, program-flow organization (state machine), logging and communication. Central services exist for user management, access control, data logging, configuration management etc. The design provides a very scalable but still maintainable system that at the same time can act as a fully-fledged control or a highly parallel distributed scientific workflow system. It allows simple integration and adaption to changing control requirements and the addition of new scientific analysis algorithms, making them automatically and immediately available to experimentalists

Calibration and Calibration Data Processing Concepts at the European XFEL

The European X-ray Free Electron Laser (Altarelli, 2006) is a high-intensity X-ray light source currently being constructed in Hamburg, Germany, that will provide spatially coherent X-rays in the energy range between 0.25 keV and 25 keV. The machine will deliver a unique time structure, consisting of up to 2700 pulses, with a 4.5 MHz repetition rate, 10 times per second at very high photon fluxes up to 1017 photons/s (Tschentscher, 2012). The LPD (Hart, 2012; Koch, 2013), DSSC (Porro, 2010, 2012; Lutz, 2010) and AGIPD (Graafsma, 2009) detectors are being developed to provide Mpixel imaging capabilities at the aforementioned repetition rates for a dynamic range spanning from single photon sensitivity to 104 –105 photons per pixel. The detectors are optimized for specific energy ranges. A direct consequence of the aforementioned detectors’ characteristics is that they generate raw data volumes unprecedented in photon science, ranging up to 1Mpixel x 640 memory cells x 10 pulse/s x 16 bit, i.e. 12.8 Gbyte/s. On-detector vetoing may not necessarily lower these rates much - a memory cell freed by a vetoed pulse may be used by data from one of the remaining 2700 pulses a train consists of. The PC-layer may reduce this data amount by additional software triggering, but this is not guaranteed. Figure 1 gives an overview of the different data products at European XFEL, as well as their flows and involved user roles, under the assumption that processing takes place within XFEL’s Karabo framework (Heisen, 2013). In addition to the high data rates, the Mpixel detectors’ on-sensor memory-cell and multi-gain-stage architectures necessary for the high dynamic range, pose unique challenges in detector-specific data corrections and calibration (Weidenspointner, 2012; Sztuk-Dambietz, 2013a). These challenges are addressed by providing a dedicated and thoroughly characterized set of test stands, which utilize continuous sources (Fe-55, X-ray tubes) as well as a pulsed setup: PulXar (Sztuk-Dambietz, 2013b), which is designed to produce X-ray pulses of 50-150 ns duration, within a 0.6 ms burst followed by a 99.4 ms gap. The radiation it produces thus closely matches the XFEL pulse structure. Additionally, simulation tools are being developed to assist in detector characterization (Bohlen and Joy, 2013)

The Karabo distributed control system

The Karabo distributed control system has been developed to address the challenging requirements of the European X-ray Free Electron Laser facility, including complex and custom-made hardware, high data rates and volumes, and close integration of data analysis for distributed processing and rapid feedback. Karabo is a pluggable, distributed application management system forming a supervisory control and data acquisition environment as part of a distributed control system. Karabo provides integrated control of hardware, monitoring, data acquisition and data analysis on distributed hardware, allowing rapid control feedback based on complex algorithms. Services exist for access control, data logging, configuration management and situational awareness through alarm indicators. The flexible framework enables quick response to the changing requirements in control and analysis, and provides an efficient environment for development, and a single interface to make all changes immediately available to operators and experimentalists.</p