Designing algorithms to aid discovery by chemical robots

Recently, automated robotic systems have become very efficient, thanks to improved coupling between sensor systems and algorithms, of which the latter have been gaining significance thanks to the increase in computing power over the past few decades. However, intelligent automated chemistry platforms for discovery orientated tasks need to be able to cope with the unknown, which is a profoundly hard problem. In this Outlook, we describe how recent advances in the design and application of algorithms, coupled with the increased amount of chemical data available, and automation and control systems may allow more productive chemical research and the development of chemical robots able to target discovery. This is shown through examples of workflow and data processing with automation and control, and through the use of both well-used and cutting-edge algorithms illustrated using recent studies in chemistry. Finally, several algorithms are presented in relation to chemical robots and chemical intelligence for knowledge discovery

Synthetic Biology: A Bridge between Artificial and Natural Cells.

Artificial cells are simple cell-like entities that possess certain properties of natural cells. In general, artificial cells are constructed using three parts: (1) biological membranes that serve as protective barriers, while allowing communication between the cells and the environment; (2) transcription and translation machinery that synthesize proteins based on genetic sequences; and (3) genetic modules that control the dynamics of the whole cell. Artificial cells are minimal and well-defined systems that can be more easily engineered and controlled when compared to natural cells. Artificial cells can be used as biomimetic systems to study and understand natural dynamics of cells with minimal interference from cellular complexity. However, there remain significant gaps between artificial and natural cells. How much information can we encode into artificial cells? What is the minimal number of factors that are necessary to achieve robust functioning of artificial cells? Can artificial cells communicate with their environments efficiently? Can artificial cells replicate, divide or even evolve? Here, we review synthetic biological methods that could shrink the gaps between artificial and natural cells. The closure of these gaps will lead to advancement in synthetic biology, cellular biology and biomedical applications

Printable microscale interfaces for long-term peripheral nerve mapping and precision control

The nascent field of bioelectronic medicine seeks to decode and modulate peripheral nervous system signals to obtain therapeutic control of targeted end organs and effectors. Current approaches rely heavily on electrode-based devices, but size scalability, material and microfabrication challenges, limited surgical accessibility, and the biomechanically dynamic implantation environment are significant impediments to developing and deploying advanced peripheral interfacing technologies. Here, we present a microscale implantable device – the nanoclip – for chronic interfacing with fine peripheral nerves in small animal models that begins to meet these constraints. We demonstrate the capability to make stable, high-resolution recordings of behaviorally-linked nerve activity over multi-week timescales. In addition, we show that multi-channel, current-steering-based stimulation can achieve a high degree of functionally-relevant modulatory specificity within the small scale of the device. These results highlight the potential of new microscale design and fabrication techniques for the realization of viable implantable devices for long-term peripheral interfacing.https://www.biorxiv.org/node/801468.fullFirst author draf

On adaptive control and particle filtering in the automatic administration of medicinal drugs

Automatic feedback methodologies for the administration of medicinal drugs offer undisputed potential benefits in terms of cost reduction and improved clinical outcomes. However, despite several decades of research, the ultimate safety of many--it would be fair to say most--closed-loop drug delivery approaches remains under question and manual methods based on clinicians' expertise are still dominant in clinical practice. Key challenges to the design of control systems for these applications include uncertainty in pharmacological models, as well as intra- and interpatient variability in the response to drug administration. Pharmacological systems may feature nonlinearities, time delays, time-varying parameters and non-Gaussian stochastic processes. This dissertation investigates a novel multi-controller adaptive control strategy capable of delivering safe control for closed-loop drug delivery applications without impairing clinicians' ability to make an expert assessment of a clinical situation. Our new feedback control approach, which we have named Robust Adaptive Control with Particle Filtering (RAC-PF), estimates a patient's individual response characteristic in real-time through particle filtering and uses the Bayesian inference result to select the most suitable controller for closed-loop operation from a bank of candidate controllers designed using the robust methodology of mu-synthesis. The work is presented as four distinct pieces of research. We first apply the existing approach of Robust Multiple-Model Adaptive Control (RMMAC), which features robust controllers and Kalman filter estimators, to the case-study of administration of the vasodepressor drug sodium nitroprusside and examine benefits and drawbacks. We then consider particle filtering as an alternative to Kalman filter-based methods for the real-time estimation of pharmacological dose-response, and apply this to the nonlinear pharmacokinetic-pharmacodynamic model of the anaesthetic drug propofol. We ultimately combine particle filters and robust controllers to create RAC-PF, and test our novel approach first in a proof-of-concept design and finally in the case of sodium nitroprusside. The results presented in the dissertation are based on computational studies, including extensive Monte-Carlo simulation campaigns. Our findings of improved parameter estimates from noisy observations support the use of particle filtering as a viable tool for real-time Bayesian inference in pharmacological system identification. The potential of the RAC-PF approach as an extension of RMMAC for closed-loop control of a broader class of systems is also clearly highlighted, with the proposed new approach delivering safe control of acute hypertension through sodium nitroprusside infusion when applied to a very general population response model. All approaches presented are generalisable and may be readily adapted to other drug delivery instances

Evaluation of blood glucose level control in Type 1 diabetic patients using online and offline reinforcement learning

[SPA] Los pacientes con diabetes tipo 1 deben monitorear de cerca sus niveles de glucemia y administrar insulina para controlarlos. Se han propuesto métodos de control automatizado de la glucemia que eliminan la necesidad de intervención humana, y recientemente, el aprendizaje por refuerzo, un tipo de algoritmo de aprendizaje automático, se ha utilizado como un método efectivo de control en entornos simulados. Actualmente, los métodos utilizados para los pacientes con diabetes, como el régimen basal- bolus y los monitores continuos de glucemia, tienen limitaciones y todavía requieren intervención manual. Los controladores PID se utilizan ampliamente por su simplicidad y robustez, pero son sensibles a factores externos que afectan su efectividad. Las obras existentes en la literatura de investigación se han enfocado principalmente en mejorar la precisión de estos algoritmos de control. Sin embargo, todavía hay margen para mejorar la adaptabilidad a los pacientes individuales. La siguiente fase de investigación tiene como objetivo optimizar aún más los métodos actuales y adaptar los algoritmos para controlar mejor los niveles de glucemia. Una solución potencial es usar el aprendizaje por refuerzo (RL) para entrenar los algoritmos en base a datos individuales del paciente. En esta tesis, proponemos un control en lazo cerrado para los niveles de glucemia basado en el aprendizaje profundo por refuerzo. Describimos la evaluación inicial de varias alternativas llevadas a cabo en un simulador realista del sistema glucorregulador y proponemos una estrategia de implementación particular basada en reducir la frecuencia de las observaciones y recompensas pasadas al agente, y usar una función de recompensa simple. Entrenamos agentes con esa estrategia para tres grupos de clases de pacientes, los evaluamos y los comparamos con otras alternativas. Nuestros resultados muestran que nuestro método con Proximal Policy Optimization es capaz de superar a los métodos tradicionales, así como a propuestas similares recientes, al lograr períodos más prolongados de estado glicémico seguro y de bajo riesgo. Como extensión del aporte anterior, constatamos que la aplicación práctica de los algoritmos de control de glucemia requeriría interacciones de prueba y error con los pacientes, lo que es una limitación para entrenar el sistema de manera efectiva. Como alternativa, el aprendizaje reforzado sin conexión no requiere interacción con humanos y la investigación previa sugiere que se pueden lograr resultados prometedores con conjuntos de datos obtenidos sin interacción, similar a los algoritmos de aprendizaje automático clásicos. Sin embargo, aún no se ha evaluado la aplicación del aprendizaje reforzado sin conexión al control de la glucemia. Por lo tanto, en esta tesis, evaluamos exhaustivamente dos algoritmos de aprendizaje reforzado sin conexión para el control de glucemia y examinamos su potencial y limitaciones. Evaluamos el impacto del método utilizado para generar los conjuntos de datos de entrenamiento, el tipo de trayectorias (secuencias de estados, acciones y recompensas experimentadas por un agente en un entorno,) empleadas (método único o mixto), la calidad de las trayectorias y el tamaño de los conjuntos de datos en el entrenamiento y el rendimiento, y los comparamos con las alternativas como PID y Proximal Policy Optimization. Nuestros resultados demuestran que uno de los algoritmos de aprendizaje reforzado sin conexión evaluados, Trajectory Transformer, es capaz de rendir al mismo nivel que alternativas, pero sin necesidad de interacción con pacientes reales durante el entrenamiento.[ENG] Patients with Type 1 diabetes are required to closely monitor their blood glucose levels and administer insulin to manage them. Automated glucose control methods that eliminate the need for human intervention have been proposed, and recently, reinforcement learning, a type of machine learning algorithm, has been used as an effective control method in simulated environments. Currently, the methods used for diabetes patients, such as the basal-bolus regime and continuous glucose monitors, have limitations and still require manual intervention. The PID controllers are widely used for their simplicity and robustness, but they are sensitive to external factors affecting their effectiveness. The existing works in the research literature have mainly focused on improving the accuracy of these control algorithms. However, there is still room for improvement regarding adaptability to individual patients. The next phase of research aims to further optimize the current methods and adapt the algorithms to better control blood glucose levels. Machine learning proposals have paved the way partially, but they can generate generic models with limited adaptability. One potential solution is to use reinforcement learning (RL) to train the algorithms based on individual patient data. In this thesis, we propose a closed-loop control for blood glucose levels based on Deep reinforcement learning. We describe the initial evaluation of several alternatives conducted on a realistic simulator of the glucoregulatory system and propose a particular implementation strategy based on reducing the frequency of the observations and rewards passed to the agent, and using a simple reward function. We train agents with that strategy for three groups of patient classes, evaluate and compare it with alternative control baselines. Our results show that our method with Proximal Policy Optimization is able to outperform baselines as well as similar recent proposals, by achieving longer periods of safe glycemic state and low risk. As an extension of the previous contribution, we have noticed that, practical application of blood glucose control algorithms would necessitate trial-and-error interaction with patients, which could be a limitation for effectively training the system. As an alternative, offline reinforcement learning does not require interaction with subjects and preliminary research suggests that promising results can be achieved with datasets obtained offline, similar to classical machine learning algorithms. However, application of offline reinforcement learning to glucose control has to be evaluated yet. Thus, in this thesis, we comprehensively evaluate two offline reinforcement learning algorithms for blood glucose control and examine their potential and limitations. We assess the impact of the method used to generate training datasets, the type of trajectories employed (sequences of states, actions, and rewards experienced by an agent in an environment over time), the quality of the trajectories, and the size of the datasets on training and performance, and compare them to commonly used baselines such as PID and Proximal Policy Optimization. Our results demonstrate that one of the offline reinforcement learning algorithms evaluated, Trajectory Transformer, is able to perform at the same level as the baselines, but without the need for interaction with real patients during training.Escuela Internacional de Doctorado de la Universidad Politécnica de CartagenaUniversidad Politécnica de CartagenaPrograma Doctorado en Tecnologías de la Información y las Comunicacione

MicroBioRobots for Single Cell Manipulation

One of the great challenges in nano and micro scale science and engineering is the independent manipulation of biological cells and small man-made objects with active sensing. For such biomedical applications as single cell manipulation, telemetry, and localized targeted delivery of chemicals, it is important to fabricate microstructures that can be powered and controlled without a tether in fluidic environments. These microstructures can be used to develop microrobots that have the potential to make existing therapeutic and diagnostic procedures less invasive. Actuation can be realized using various different organic and inorganic methods. Previous studies explored different forms of actuation and control with microorganisms. Bacteria, in particular, offer several advantages as controllable micro actuators: they draw chemical energy directly from their environment, they are genetically modifiable, and they are scalable and configurable in the sense that any number of bacteria can be selectively patterned. Additionally, the study of bacteria inspires inorganic schemes of actuation and control. For these reasons, we chose to employ bacteria while controlling their motility using optical and electrical stimuli. In the first part of the thesis, we demonstrate a bio-integrated approach by introducing MicroBioRobots (MBRs). MBRs are negative photosensitive epoxy (SU8) microfabricated structures with typical feature sizes ranging from 1-100 μm coated with a monolayer of the swarming Serratia marcescens. The adherent bacterial cells naturally coordinate to propel the microstructures in fluidic environments, which we call Self-Actuation. First, we demonstrate the control of MBRs using self-actuation, DC electric fields and ultra-violet radiation and develop an experimentally-validated mathematical model for the MBRs. This model allows us to to steer the MBR to any position and orientation in a planar micro channel using visual feedback and an inverted microscope. Examples of sub-micron scale transport and assembly as well as computer-based closed-loop control of MBRs are presented. We demonstrate experimentally that vision-based feedback control allows a four-electrode experimental device to steer MBRs along arbitrary paths with micrometer precision. At each time instant, the system identifies the current location of the robot, a control algorithm determines the power supply voltages that will move the charged robot from its current location toward its next desired position, and the necessary electric field is then created. Second, we develop biosensors for the MBRs. Microscopic devices with sensing capabilities could significantly improve single cell analysis, especially in high-resolution detection of patterns of chemicals released from cells in vitro. Two different types of sensing mechanisms are employed. The first method is based on harnessing bacterial power, and in the second method we use genetically engineered bacteria. The small size of the devices gives them access to individual cells, and their large numbers permit simultaneous monitoring of many cells. In the second part, we describe the construction and operation of truly micron-sized, biocompatible ferromagnetic micro transporters driven by external magnetic fields capable of exerting forces at the pico Newton scale. We develop micro transporters using a simple, single step micro fabrication technique that allows us to produce large numbers in the same step. We also fabricate microgels to deliver drugs. We demonstrate that the micro transporters can be navigated to separate single cells with micron-size precision and localize microgels without disturbing the local environment