kiwiPy: Robust, high-volume, messaging for big-data and computational science workflows

In this work we present kiwiPy, a Python library designed to support robust message based communication for high-throughput, big-data, applications while being general enough to be useful wherever high-volumes of messages need to be communicated in a predictable manner. KiwiPy relies on the RabbitMQ protocol, an industry standard message broker, while providing a simple and intuitive interface that can be used in both multithreaded and coroutine based applications. To demonstrate some of kiwiPy's functionality we give examples from AiiDA, a high-throughput simulation platform, where kiwiPy is used as a key component of the workflow engine

Determining crystal phase purity in c-BP through X-ray absorption spectroscopy

Citation: Determining crystal phase purity in c-BP through X-ray absorption spectroscopy. S. P. Huber, V. V. Medvedev, E. Gullikson, B. Padavala, J. H. Edgar, R. W. E. van de Kruijs, F. Bijkerk, and D. Prendergast. Phys. Chem. Chem. Phys. 19 8174--8187 (2017) 10.1039/c6cp06967cWe employ X-ray absorption near-edge spectroscopy at the boron K-edge and the phosphorus L2,3-edge to study the structural properties of cubic boron phosphide (c-BP) samples. The X-ray absorption spectra are modeled from first-principles within the density functional theory framework using the excited electron core-hole (XCH) approach. A simple structural model of a perfect c-BP crystal accurately reproduces the P L2,3-edge, however it fails to describe the broad and gradual onset of the B K-edge. Simulations of the spectroscopic signatures in boron 1s excitations of intrinsic point defects and the hexagonal BP crystal phase show that these additions to the structural model cannot reproduce the broad pre-edge of the experimental spectrum. Calculated formation enthalpies show that, during the growth of c-BP, it is possible that amorphous boron phases can be grown in conjunction with the desired boron phosphide crystalline phase. In combination with experimental and theoretically obtained X-ray absorption spectra of an amorphous boron structure, which have a similar broad absorption onset in the B K-edge spectrum as the cubic boron phosphide samples, we provide evidence for the presence of amorphous boron clusters in the synthesized c-BP samples

Materials Cloud, a platform for open computational science

Materials Cloud is a platform designed to enable open and seamless sharing of resources for computational science, driven by applications in materials modelling. It hosts 1) archival and dissemination services for raw and curated data, together with their provenance graph, 2) modelling services and virtual machines, 3) tools for data analytics, and pre-/post-processing, and 4) educational materials. Data is citable and archived persistently, providing a comprehensive embodiment of the FAIR principles that extends to computational workflows. Materials Cloud leverages the AiiDA framework to record the provenance of entire simulation pipelines (calculations performed, codes used, data generated) in the form of graphs that allow to retrace and reproduce any computed result. When an AiiDA database is shared on Materials Cloud, peers can browse the interconnected record of simulations, download individual files or the full database, and start their research from the results of the original authors. The infrastructure is agnostic to the specific simulation codes used and can support diverse applications in computational science that transcend its initial materials domain.Comment: 19 pages, 8 figure

How to verify the precision of density-functional-theory implementations via reproducible and universal workflows

In the past decades many density-functional theory methods and codes adopting periodic boundary conditions have been developed and are now extensively used in condensed matter physics and materials science research. Only in 2016, however, their precision (i.e., to which extent properties computed with different codes agree among each other) was systematically assessed on elemental crystals: a first crucial step to evaluate the reliability of such computations. We discuss here general recommendations for verification studies aiming at further testing precision and transferability of density-functional-theory computational approaches and codes. We illustrate such recommendations using a greatly expanded protocol covering the whole periodic table from Z=1 to 96 and characterizing 10 prototypical cubic compounds for each element: 4 unaries and 6 oxides, spanning a wide range of coordination numbers and oxidation states. The primary outcome is a reference dataset of 960 equations of state cross-checked between two all-electron codes, then used to verify and improve nine pseudopotential-based approaches. Such effort is facilitated by deploying AiiDA common workflows that perform automatic input parameter selection, provide identical input/output interfaces across codes, and ensure full reproducibility. Finally, we discuss the extent to which the current results for total energies can be reused for different goals (e.g., obtaining formation energies).Comment: Main text: 23 pages, 4 figures. Supplementary: 68 page

Common workflows for computing material properties using different quantum engines

The prediction of material properties based on density-functional theory has become routinely common, thanks, in part, to the steady increase in the number and robustness of available simulation packages. This plurality of codes and methods is both a boon and a burden. While providing great opportunities for cross-verification, these packages adopt different methods, algorithms, and paradigms, making it challenging to choose, master, and efficiently use them. We demonstrate how developing common interfaces for workflows that automatically compute material properties greatly simplifies interoperability and cross-verification. We introduce design rules for reusable, code-agnostic, workflow interfaces to compute well-defined material properties, which we implement for eleven quantum engines and use to compute various material properties. Each implementation encodes carefully selected simulation parameters and workflow logic, making the implementer’s expertise of the quantum engine directly available to non-experts. All workflows are made available as open-source and full reproducibility of the workflows is guaranteed through the use of the AiiDA infrastructure.This work is supported by the MARVEL National Centre of Competence in Research (NCCR) funded by the Swiss National Science Foundation (grant agreement ID 51NF40-182892) and by the European Union’s Horizon 2020 research and innovation program under Grant Agreement No. 824143 (European MaX Centre of Excellence “Materials design at the Exascale”) and Grant Agreement No. 814487 (INTERSECT project). We thank M. Giantomassi and J.-M. Beuken for their contributions in adding support for PseudoDojo tables to the aiida-pseudo (https://github.com/aiidateam/aiida-pseudo) plugin. We also thank X. Gonze, M. Giantomassi, M. Probert, C. Pickard, P. Hasnip, J. Hutter, M. Iannuzzi, D. Wortmann, S. Blügel, J. Hess, F. Neese, and P. Delugas for providing useful feedback on the various quantum engine implementations. S.P. acknowledges support from the European Unions Horizon 2020 Research and Innovation Programme, under the Marie Skłodowska-Curie Grant Agreement SELPH2D No. 839217 and computer time provided by the PRACE-21 resources MareNostrum at BSC-CNS. E.F.-L. acknowledges the support of the Norwegian Research Council (project number 262339) and computational resources provided by Sigma2. P.Z.-P. thanks to the Faraday Institution CATMAT project (EP/S003053/1, FIRG016) for financial support. KE acknowledges the Swiss National Science Foundation (grant number 200020-182015). G.Pi. and K.E. acknowledge the swissuniversities “Materials Cloud” (project number 201-003). Work at ICMAB is supported by the Severo Ochoa Centers of Excellence Program (MICINN CEX2019-000917-S), by PGC2018-096955-B-C44 (MCIU/AEI/FEDER, UE), and by GenCat 2017SGR1506. B.Z. thanks to the Faraday Institution FutureCat project (EP/S003053/1, FIRG017) for financial support. J.B. and V.T. acknowledge support by the Joint Lab Virtual Materials Design (JLVMD) of the Forschungszentrum Jülich.Peer reviewe

Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.Peer reviewe

Workflows in AiiDA:Engineering a high-throughput, event-based engine for robust and modular computational workflows

Over the last two decades, the field of computational science has seen a dramatic shift towards incorporating high-throughput computation and big-data analysis as fundamental pillars of the scientific discovery process. This has necessitated the development of tools and techniques to deal with the generation, storage and processing of large amounts of data. In this work we present an in-depth look at the workflow engine powering AiiDA, a widely adopted, highly flexible and database-backed informatics infrastructure with an emphasis on data reproducibility. We detail many of the design choices that were made which were informed by several important goals: the ability to scale from running on individual laptops up to high-performance supercomputers, managing jobs with runtimes spanning from fractions of a second to weeks and scaling up to thousands of jobs concurrently, and all this while maximising robustness. In short, AiiDA aims to be a Swiss army knife for high-throughput computational science. As well as the architecture, we outline important API design choices made to give workflow writers a great deal of liberty whilst guiding them towards writing robust and modular workflows, ultimately enabling them to encode their scientific knowledge to the benefit of the wider scientific community