Fast hyperparameter optimisation of graph neural network for molecular property prediction

In the evolving domain of graph neural networks, there is a growing effort focused on predicting molecular properties. However, a noticeable gap persists, as much of the research overlooks the comprehensive exploration of hyperparameters—a crucial aspect for achieving positive outcomes in graph neural network applications. This underscores the vital role of hyperparameter optimisation, despite the challenge posed by resource-intensive procedures. To address this gap and overcome the challenge, our study achieves significant advancements. Firstly, we summarise graph neural networks for molecular property prediction into a structured framework, systematically identifying key hyperparameters for optimisation. Secondly, we introduce an innovative hierarchical evaluation strategy embedded in a genetic algorithm named HESGA, expediting optimisation by early elimination of unpromising solutions. This approach demonstrates improved efficiency and cost-effectiveness compared to traditional Bayesian optimisation. Thirdly, we propose the implementation of a binary tree to model the hyperparameter space, further enhancing HESGA’s effectiveness. Lastly, guided by empirical insights, we present a hybrid evaluation strategy that surpasses advanced optimisation methods, demonstrating reduced computational costs and accelerated optimisation. Overall, our research not only addresses the challenge of elevated computational expenses in hyperparameter optimisation but also enhances graph neural network performance, effectively bridging the research gap in hyperparameter optimisation for graph neural networks in the context of predicting molecular properties

Towards machine learning approaches for predicting the self-healing efficiency of materials

Acknowledgement This research is supported by the Engineering and Physical Sciences Research Council (EPSRC) funded Project on New Industrial Systems: Manufacturing Immortality (EP/R020957/1). The authors are also grateful to the Manufacturing Immortality consortium.Peer reviewedPublisher PD

FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy

Measuring the distance between machine-produced and human language is a critical open problem. Inspired by empirical findings from psycholinguistics on the periodicity of entropy in language, we propose FACE, a set of metrics based on Fourier Analysis of the estimated Cross-Entropy of language, for measuring the similarity between model-generated and human-written languages. Based on an open-ended generation task and the experimental data from previous studies, we find that FACE can effectively identify the human-model gap, scales with model size, reflects the outcomes of different sampling methods for decoding, correlates well with other evaluation metrics and with human judgment scores. FACE is computationally efficient and provides intuitive interpretations

AI3SD Video: Hyperparameter Optimisation for Graph Neural Networks

Traditional deep learning has made significant progress on various problems, from computer vision to natural language processing. For graph problems, there are still many challenges. Graph neural networks (GNNs) have been proposed for a wide range of learning tasks in the graph domain. In particular, in recent years, an increasing number of GNN models were applied to model molecular graphs and predict the properties of the corresponding molecules. However, a direct impediment to achieve good performance with the lower computational cost is to select appropriate hyperparameters. Meanwhile, many molecular datasets are far smaller than many other datasets in typical deep learning applications. Most hyperparameter optimization (HPO) methods for deep learning have not been explored in terms of their efficiencies on such small datasets in the molecular domain. We conducted theoretical analyses for popular HPO methods (random search, TPE, and CMA-ES) and proposed a genetic algorithm with hierarchical evaluation strategy and tree-structured mutation for HPO. Finally, we believe that our work will motivate further research to GNNs as applied to molecular machine learning problems and facilitate scientific discovery

Humans of AI3SD: Yingfang Yuan

This interview forms part of our Humans of AI3SD Series.In this Humans of AI4SD interview he discusses the challenges involved in graph networks and hyperparameter optimisation, making the most of interdisciplinary research, how AI can speed things up, and his advice for early career researchers

A Systematic Comparison Study on Hyperparameter Optimisation of Graph Neural Networks for Molecular Property Prediction

Graph neural networks (GNNs) have been proposed for a wide range of graph-related learning tasks. In particular, in recent years there has been an increasing number of GNN systems that were applied to predict molecular properties. However, in theory, there are infinite choices of hyperparameter settings for GNNs, and a direct impediment is to select appropriate hyperparameters to achieve satisfactory performance with lower computational cost. Meanwhile, the sizes of many molecular datasets are far smaller than many other datasets in typical deep learning applications, and most hyperparameter optimization (HPO) methods have not been explored in terms of their efficiencies on such small datasets in molecular domain. In this paper, we conducted a theoretical analysis of common and specific features for two state-of-the-art and popular algorithms for HPO: TPE and CMA-ES, and we compared them with random search (RS), which is used as a baseline. Experimental studies are carried out on several benchmarks in MoleculeNet, from different perspectives to investigate the impact of RS, TPE, and CMA-ES on HPO of GNNs for molecular property prediction. In our experiments, we concluded that RS, TPE, and CMA-ES have their individual advantages in tackling different specific molecular problems. Finally, we believe our work will motivate further research on GNN as applied to molecular machine learning problems in chemistry and materials sciences