Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges

Generative Artificial Intelligence (AI) is one of the most exciting developments in Computer Science of the last decade. At the same time, Reinforcement Learning (RL) has emerged as a very successful paradigm for a variety of machine learning tasks. In this survey, we discuss the state of the art, opportunities and open research questions in applying RL to generative AI. In particular, we will discuss three types of applications, namely, RL as an alternative way for generation without specified objectives; as a way for generating outputs while concurrently maximizing an objective function; and, finally, as a way of embedding desired characteristics, which cannot be easily captured by means of an objective function, into the generative process. We conclude the survey with an in-depth discussion of the opportunities and challenges in this fascinating emerging area.Comment: Published in JAIR at https://www.jair.org/index.php/jair/article/view/1527

Combining evolutionary algorithms with reaction rules towards focused molecular design

Designing novel small molecules with desirable properties and feasible synthesis continues to pose a significant challenge in drug discovery, particularly in the realm of natural products. Reaction-based gradient-free methods are promising approaches for designing new molecules as they ensure synthetic feasibility and provide potential synthesis paths. However, it is important to note that the novelty and diversity of the generated molecules highly depend on the availability of comprehensive reaction templates. To address this challenge, we introduce ReactEA, a new open-source evolutionary framework for computer-aided drug discovery that solely utilizes biochemical reaction rules. ReactEA optimizes molecular properties using a comprehensive set of 22,949 reaction rules, ensuring chemical validity and synthetic feasibility. ReactEA is versatile, as it can virtually optimize any objective function and track potential synthetic routes during the optimization process. To demonstrate its effectiveness, we apply ReactEA to various case studies, including the design of novel drug-like molecules and the optimization of pre-existing ligands. The results show that ReactEA consistently generates novel molecules with improved properties and reasonable synthetic routes, even for complex tasks such as improving binding affinity against the PARP1 enzyme when compared to existing inhibitors.Centre of Biological Engineering (CEB, University of Minho) for financial and equipment support. Portuguese Foundation for Science and Technology (FCT) under the scope of the strategic funding of UIDB/04469/2020 unit and through a Ph.D. scholarship awarded to João Correia (SFRH/BD/144314/2019). European Commission through the project SHIKIFACTORY100 - Modular cell factories for the production of 100 compounds from the shikimate pathway (Reference 814408).info:eu-repo/semantics/publishedVersio

Graph Neural Networks for Molecules

Graph neural networks (GNNs), which are capable of learning representations from graphical data, are naturally suitable for modeling molecular systems. This review introduces GNNs and their various applications for small organic molecules. GNNs rely on message-passing operations, a generic yet powerful framework, to update node features iteratively. Many researches design GNN architectures to effectively learn topological information of 2D molecule graphs as well as geometric information of 3D molecular systems. GNNs have been implemented in a wide variety of molecular applications, including molecular property prediction, molecular scoring and docking, molecular optimization and de novo generation, molecular dynamics simulation, etc. Besides, the review also summarizes the recent development of self-supervised learning for molecules with GNNs.Comment: A chapter for the book "Machine Learning in Molecular Sciences". 31 pages, 4 figure

Leveraging Transformer Models for Accelerated Drug Discovery

In the realm of AI-accelerated drug discovery, particularly in de novo drug design, significant challenges include unpredictable drug responses in clinical trials, biases in predictive models, and the opaque nature of AI methodologies that complicate the understanding of a drug's mechanism of action. These issues have limited the progression of AI-discovered drugs into clinical trials and regulatory approval. Concurrently, the development of me-too drugs, which involve modifications of existing drugs within the same therapeutic class, presents a less risky and potentially more effective avenue. However, the potential of AI to enhance their development remains largely underexplored. This dissertation aims to transform the development of me-too drugs through the application of AI, with a focus on transformer and large language models (LLMs). It introduces innovative frameworks that utilize the representation learning and generative capabilities of transformer models to refine and expedite the me-too drug development process. These methodologies, referred to as "drug optimization", seek to further accelerate the production of effective me-too drugs. This work makes four significant contributions to the field: (1) It proposes two fusion methods that integrate transformer models with graph neural networks, enhancing the precision of binding affinity predictions. (2) It assembles a comprehensive dataset of 10 million binding affinity values across a diverse array of proteins and drugs, providing an invaluable resource for model training and validation. (3) It proposes two generative models for drug optimization, fine-tuned through reinforcement learning, with the goal of automating and expediting the creation of effective me-too drugs. (4) It introduces an innovative bidirectional GPT model for molecular textual sequences (SMILES), enabling precise generative mask infilling for targeted drug optimization. And by conducting comprehensive evaluations on real world viral and cancer target proteins, we demonstrate that the proposed drug optimization frameworks can consistently enhance existing molecules/drugs

Reinforcement learning with supervision beyond environmental rewards

Reinforcement Learning (RL) is an elegant approach to tackle sequential decision-making problems. In the standard setting, the task designer curates a reward function and the RL agent's objective is to take actions in the environment such that the long-term cumulative reward is maximized. Deep RL algorithms---that combine RL principles with deep neural networks---have been successfully used to learn behaviors in complex environments but are generally quite sensitive to the nature of the reward function. For a given RL problem, the environmental rewards could be sparse, delayed, misspecified, or unavailable (i.e., impossible to define mathematically for the required behavior). These scenarios exacerbate the challenge of training a stable deep-RL agent in a sample-efficient manner. In this thesis, we study methods that go beyond a direct reliance on the environmental rewards by generating additional information signals that the RL agent could incorporate for learning the desired skills. We start by investigating the performance bottlenecks in delayed reward environments and propose to address these by learning surrogate rewards. We include two methods to compute the surrogate rewards using the agent-environment interaction data. Then, we consider the imitation-learning (IL) setting where we don't have access to any rewards, but instead, are provided with a dataset of expert demonstrations that the RL agent must learn to reliably reproduce. We propose IL algorithms for partially observable environments and situations with discrepancies between the transition dynamics of the expert and the imitator. Next, we consider the benefits of learning an ensemble of RL agents with explicit diversity pressure. We show that diversity encourages exploration and facilitates the discovery of sparse environmental rewards. Finally, we analyze the concept of sharing knowledge between RL agents operating in different but related environments and show that the information transfer can accelerate learning

Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity

This survey addresses the crucial issue of factuality in Large Language Models (LLMs). As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital. We define the Factuality Issue as the probability of LLMs to produce content inconsistent with established facts. We first delve into the implications of these inaccuracies, highlighting the potential consequences and challenges posed by factual errors in LLM outputs. Subsequently, we analyze the mechanisms through which LLMs store and process facts, seeking the primary causes of factual errors. Our discussion then transitions to methodologies for evaluating LLM factuality, emphasizing key metrics, benchmarks, and studies. We further explore strategies for enhancing LLM factuality, including approaches tailored for specific domains. We focus two primary LLM configurations standalone LLMs and Retrieval-Augmented LLMs that utilizes external data, we detail their unique challenges and potential enhancements. Our survey offers a structured guide for researchers aiming to fortify the factual reliability of LLMs.Comment: 62 pages; 300+ reference