Accelerating Molecular Graph Neural Networks via Knowledge Distillation

Recent advances in graph neural networks (GNNs) have enabled more comprehensive modeling of molecules and molecular systems, thereby enhancing the precision of molecular property prediction and molecular simulations. Nonetheless, as the field has been progressing to bigger and more complex architectures, state-of-the-art GNNs have become largely prohibitive for many large-scale applications. In this paper, we explore the utility of knowledge distillation (KD) for accelerating molecular GNNs. To this end, we devise KD strategies that facilitate the distillation of hidden representations in directional and equivariant GNNs, and evaluate their performance on the regression task of energy and force prediction. We validate our protocols across different teacher-student configurations and datasets, and demonstrate that they can consistently boost the predictive accuracy of student models without any modifications to their architecture. Moreover, we conduct comprehensive optimization of various components of our framework, and investigate the potential of data augmentation to further enhance performance. All in all, we manage to close the gap in predictive accuracy between teacher and student models by as much as 96.7% and 62.5% for energy and force prediction respectively, while fully preserving the inference throughput of the more lightweight models.Comment: Accepted as a conference paper at NeurIPS 202

Challenges with unsupervised LLM knowledge discovery

We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent. The idea behind unsupervised knowledge elicitation is that knowledge satisfies a consistency structure, which can be used to discover knowledge. We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a particular leading unsupervised knowledge-elicitation method, contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a series of experiments showing settings in which unsupervised methods result in classifiers that do not predict knowledge, but instead predict a different prominent feature. We conclude that existing unsupervised methods for discovering latent knowledge are insufficient, and we contribute sanity checks to apply to evaluating future knowledge elicitation methods. Conceptually, we hypothesise that the identification issues explored here, e.g. distinguishing a model's knowledge from that of a simulated character's, will persist for future unsupervised methods.Comment: 12 pages (38 including references and appendices). First three authors equal contribution, randomised orde

GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets

Recent years have seen the advent of molecular simulation datasets that are orders of magnitude larger and more diverse. These new datasets differ substantially in four aspects of complexity: 1. Chemical diversity (number of different elements), 2. system size (number of atoms per sample), 3. dataset size (number of data samples), and 4. domain shift (similarity of the training and test set). Despite these large differences, benchmarks on small and narrow datasets remain the predominant method of demonstrating progress in graph neural networks (GNNs) for molecular simulation, likely due to cheaper training compute requirements. This raises the question -- does GNN progress on small and narrow datasets translate to these more complex datasets? This work investigates this question by first developing the GemNet-OC model based on the large Open Catalyst 2020 (OC20) dataset. GemNet-OC outperforms the previous state-of-the-art on OC20 by 16% while reducing training time by a factor of 10. We then compare the impact of 18 model components and hyperparameter choices on performance in multiple datasets. We find that the resulting model would be drastically different depending on the dataset used for making model choices. To isolate the source of this discrepancy we study six subsets of the OC20 dataset that individually test each of the above-mentioned four dataset aspects. We find that results on the OC-2M subset correlate well with the full OC20 dataset while being substantially cheaper to train on. Our findings challenge the common practice of developing GNNs solely on small datasets, but highlight ways of achieving fast development cycles and generalizable results via moderately-sized, representative datasets such as OC-2M and efficient models such as GemNet-OC. Our code and pretrained model weights are open-sourced

First crenarchaeal chitinase found in Sulfolobus tokodaii

This is the first description of a functional chitinase gene within the crenarchaeotes. Here we report of the heterologues expression of the ORF BAB65950 from Sulfolobus tokodaii in E. coli. The resulting protein degraded chitin and was hence classified as chitinase (EC 3.2.4.14). The protein characterization revealed a specific activity of 75 mU/mg using colloidal chitin as substrate. The optimal activity of the enzyme was measured at pH 2.5 and 70 °C, respectively. A dimeric enzyme configuration is proposed. According to amino acid sequence similarities chitinases are attributed to the two glycoside hydrolase families 18 and 19. The derived amino acid sequence of the S. tokodaii gene differed from sequences of these two glycoside hydrolase families. However, within a phylogenetic tree of protein sequences, the crenarchaeal sequence of S. tokodaii clustered in close proximity to members of the glycoside hydrolase family 18

Tortuosity of Battery Electrodes: Validation of Impedance-Derived Values and Critical Comparison with 3D Tomography

Tortuosity values of porous battery electrodes determined using electrochemical impedance spectroscopy in symmetric cells with a non-intercalating electrolyte are typically higher than those values based on numerical analysis of 3D tomographic reconstructions. The electrochemical approach assumes that the electronic resistance in the porous coating is negligible and that the tortuosity of the porous electrode can be calculated from the ionic resistance determined by fitting a transmission line equivalent circuit model to the experimental data. In this work, we validate the assumptions behind the electrochemical approach. First, we experimentally and theoretically investigate the influence of the electronic resistance of the porous electrode on the extracted ionic resistances using a general transmission line model, and provide a convenient method to determine whether the electronic resistance is sufficiently low for the model to be correctly applied. Second, using a macroscopic setup with known tortuosity, we prove that the ionic resistance quantified by the transmission line model indeed yields the true tortuosity of a porous medium. Based on our findings, we analyze the tortuosities of porous electrodes using both X-ray tomography and electrochemical impedance spectroscopy on electrodes from the same coating and conclude that the distribution of the polymeric binder phase, which is not imaged in most tomographic experiments, is a key reason for the underestimated tortuosity values calculated from 3D reconstructions of electrode microstructures.ISSN:0013-4651ISSN:1945-711

Quantitative and time-resolved detection of lithium plating on graphite anodes in lithium ion batteries

The ability of fast and safe charging is critical for the further success of lithium ion batteries in automotive applications. In state-of-the-art lithium ion batteries, the charging rate is limited by the onset of lithium plating on the graphite anode. Despite its high importance, so far no analytical technique has been available for directly measuring lithium plating during battery charge. Herein, we introduce operando electron paramagnetic resonance (EPR) spectroscopy as the first technique capable of time-resolved and quantitative detection of lithium metal plating in lithium ion batteries. In an exemplary study, the C-rate dependence of lithium metal plating during low-temperature charging at −20 °C is investigated. It is possible to quantify the amount of ‘dead lithium’ and observe the chemical reintercalation of plated lithium metal. In this way, it is possible to deconvolute the coulombic inefficiency of the lithium plating/stripping process and quantify the contributions of both dead lithium and active lithium loss due to solid electrolyte interphase (SEI) formation. The time-resolved and quantitative information accessible with operando EPR spectroscopy will be very useful for the optimization of fast charging procedures, testing of electrolyte additives, and model validation

Surface Oxygen Depletion of Layered Transition Metal Oxides in Li-Ion Batteries Studied by Operando Ambient Pressure X‑ray Photoelectron Spectroscopy

A new operando spectro-electrochemical setup was developed to study oxygen depletion from the surface of layered transition metal oxide particles at high degrees of delithiation. An NCM111 working electrode was paired with a chemically delithiated LiFePO4 counter electrode in a fuel cell-inspired membrane electrode assembly (MEA). A propylene carbonate-soaked Li-ion conducting ionomer served as an electrolyte, providing both good electrochemical performance and direct probing of the NCM111 particles during cycling by ambient pressure X-ray photoelectron spectroscopy. The irreversible emergence of an oxygen-depleted phase in the O 1s spectra of the layered oxide particles was observed upon the first delithiation to high state-of-charge, which is in excellent agreement with oxygen release analysis via mass spectrometry analysis of such MEAs. By comparing the metal oxide-based O 1s spectral features to the Ni 2p3/2 intensity, we can calculate the transition metal-to-oxygen ratio of the metal oxide close to the particle surface, which shows good agreement with the formation of a spinel-like stoichiometry as an oxygen-depleted phase. This new setup enables a deeper understanding of interfacial changes of layered oxide-based cathode active materials for Li-ion batteries upon cycling

Operando electron paramagnetic resonance spectroscopy – formation of mossy lithium on lithium anodes during charge–discharge cycling

The formation of mossy lithium and lithium dendrites so far prevents the use of lithium metal anodes in lithium ion batteries. To develop solutions for this problem (e.g., electrolyte additives), operando measurement techniques are required to monitor mossy lithium and dendrite formation during electrochemical cycling. Here we present a novel battery cell design that enables operando electron paramagnetic resonance (EPR) spectroscopy. It is shown that time-resolved operando EPR spectroscopy during electrochemical cycling of a lithium-metal/LiFePO4 (LFP) cell provides unique insights into the lithium plating/dissolution mechanisms, which are consistent with ex situ scanning electron microscopy (SEM) analysis. To demonstrate the viability of the operando EPR method, two cells using different electrolytes were studied. When using an electrolyte containing fluoroethylene carbonate (FEC) additive, a higher reversibility of the lithium anode and reduced formation of micro-structured (mossy/dendritic) lithium were observed