Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior

We propose a framework for the design of feedback controllers that combines the optimization-driven and model-free advantages of deep reinforcement learning with the stability guarantees provided by using the Youla-Kucera parameterization to define the search domain. Recent advances in behavioral systems allow us to construct a data-driven internal model; this enables an alternative realization of the Youla-Kucera parameterization based entirely on input-output exploration data. Perhaps of independent interest, we formulate and analyze the stability of such data-driven models in the presence of noise. The Youla-Kucera approach requires a stable "parameter" for controller design. For the training of reinforcement learning agents, the set of all stable linear operators is given explicitly through a matrix factorization approach. Moreover, a nonlinear extension is given using a neural network to express a parameterized set of stable operators, which enables seamless integration with standard deep learning libraries. Finally, we show how these ideas can also be applied to tune fixed-structure controllers.Comment: Preprint; 18 pages. arXiv admin note: text overlap with arXiv:2304.0342

Reinforcement Learning with Partial Parametric Model Knowledge

We adapt reinforcement learning (RL) methods for continuous control to bridge the gap between complete ignorance and perfect knowledge of the environment. Our method, Partial Knowledge Least Squares Policy Iteration (PLSPI), takes inspiration from both model-free RL and model-based control. It uses incomplete information from a partial model and retains RL's data-driven adaption towards optimal performance. The linear quadratic regulator provides a case study; numerical experiments demonstrate the effectiveness and resulting benefits of the proposed method.Comment: IFAC World Congress 202

Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach

Meta-learning is a branch of machine learning which trains neural network models to synthesize a wide variety of data in order to rapidly solve new problems. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that can be used to tune proportional--integral controllers. Our meta-RL agent has a recurrent structure that accumulates "context" to learn a system's dynamics through a hidden state variable in closed-loop. This architecture enables the agent to automatically adapt to changes in the process dynamics. In tests reported here, the meta-RL agent was trained entirely offline on first order plus time delay systems, and produced excellent results on novel systems drawn from the same distribution of process dynamics used for training. A key design element is the ability to leverage model-based information offline during training in simulated environments while maintaining a model-free policy structure for interacting with novel processes where there is uncertainty regarding the true process dynamics. Meta-learning is a promising approach for constructing sample-efficient intelligent controllers.Comment: 23 pages; postprin

Meta-Reinforcement Learning for Adaptive Control of Second Order Systems

Meta-learning is a branch of machine learning which aims to synthesize data from a distribution of related tasks to efficiently solve new ones. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training, such as a model structure. The meta-RL agent is trained over a distribution of model parameters, rather than a single model, enabling the agent to automatically adapt to changes in the process dynamics while maintaining performance. A key design element is the ability to leverage model-based information offline during training, while maintaining a model-free policy structure for interacting with new environments. Our previous work has demonstrated how this approach can be applied to the industrially-relevant problem of tuning proportional-integral controllers to control first order processes. In this work, we briefly reintroduce our methodology and demonstrate how it can be extended to proportional-integral-derivative controllers and second order systems.Comment: AdCONIP 2022. arXiv admin note: substantial text overlap with arXiv:2203.0966

Transformation of human bronchial epithelial cells alters responsiveness to inflammatory cytokines

BACKGROUND: Inflammation is commonly associated with lung tumors. Since inflammatory mediators, including members of the interleukin-6 (IL-6) cytokine family, suppress proliferation of normal epithelial cells, we hypothesized that epithelial cells must develop mechanisms to evade this inhibition during the tumorigenesis. This study compared the cytokine responses of normal epithelial cells to that of premalignant cells. METHODS: Short-term primary cultures of epithelial cells were established from bronchial brushings. Paired sets of brushings were obtained from areas of normal bronchial epithelium and from areas of metaplastic or dysplastic epithelium, or areas of frank endobronchial carcinoma. In 43 paired cultures, the signalling through the signal transducer and activator of transcription (STAT) and extracellular regulated kinase (ERK) pathways and growth regulation by IL-6, leukemia inhibitory factor (LIF), oncostatin M (OSM), interferon-γ (IFNγ) or epidermal growth factor (EGF) were determined. Inducible expression and function of the leukemia inhibitory factor receptor was assessed by treatment with the histone deacetylase inhibitor depsipeptide. RESULTS: Normal epithelial cells respond strongly to OSM, IFNγ and EGF, and respond moderately to IL-6, and do not exhibit a detectable response to LIF. In preneoplastic cells, the aberrant signaling that was detected most frequently was an elevated activation of ERK, a reduced or increased IL-6 and EGF response, and an increased LIF response. Some of these changes in preneoplastic cell signaling approach those observed in established lung cancer cell lines. Epigenetic control of LIF receptor expression by histone acetylation can account for the gain of LIF responsiveness. OSM and macrophage-derived cytokines suppressed proliferation of normal epithelial cells, but reduced inhibition or even stimulated proliferation was noted for preneoplastic cells. These alterations likely contribute to the supporting effects that inflammation has on lung tumor progression. CONCLUSION: This study indicates that during the earliest stage of premalignant transformation, a modified response to cytokines and EGF is evident. Some of the altered cytokine responses in primary premalignant cells are comparable to those seen in established lung cancer cell lines

Diverse perspectives on interdisciplinarity from Members of the College of the Royal Society of Canada

Various multiple-disciplinary terms and concepts (although most commonly interdisciplinarity, which is used herein) are used to frame education, scholarship, research, and interactions within and outside academia. In principle, the premise of interdisciplinarity may appear to have many strengths; yet, the extent to which interdisciplinarity is embraced by the current generation of academics, the benefits and risks for doing so, and the barriers and facilitators to achieving interdisciplinarity, represent inherent challenges. Much has been written on the topic of interdisciplinarity, but to our knowledge there have been few attempts to consider and present diverse perspectives from scholars, artists, and scientists in a cohesive manner. As a team of 57 members from the Canadian College of New Scholars, Artists, and Scientists of the Royal Society of Canada (the College) who self-identify as being engaged or interested in interdisciplinarity, we provide diverse intellectual, cultural, and social perspectives. The goal of this paper is to share our collective wisdom on this topic with the broader community and to stimulate discourse and debate on the merits and challenges associated with interdisciplinarity. Perhaps the clearest message emerging from this exercise is that working across established boundaries of scholarly communities is rewarding, necessary, and is more likely to result in impact. However, there are barriers that limit the ease with which this can occur (e.g., lack of institutional structures and funding to facilitate cross-disciplinary exploration). Occasionally, there can be significant risk associated with doing interdisciplinary work (e.g., lack of adequate measurement or recognition of work by disciplinary peers). Solving many of the world\u27s complex and pressing problems (e.g., climate change, sustainable agriculture, the burden of chronic disease, and aging populations) demands thinking and working across long-standing, but in some ways restrictive, academic boundaries. Academic institutions and key support structures, especially funding bodies, will play an important role in helping to realize what is readily apparent to all who contributed to this paper-that interdisciplinarity is essential for solving complex problems; it is the new norm. Failure to empower and encourage those doing this research will serve as a great impediment to training, knowledge, and addressing societal issues

Diverse perspectives on interdisciplinarity from the Members of the College of the Royal Society of Canada

Various multiple-disciplinary terms and concepts (although most commonly “interdisciplinarity”, which is used herein) are used to frame education, scholarship, research, and interactions within and outside academia. In principle, the premise of interdisciplinarity may appear to have many strengths; yet, the extent to which interdisciplinarity is embraced by the current generation of academics, the benefits and risks for doing so, and the barriers and facilitators to achieving interdisciplinarity represent inherent challenges. Much has been written on the topic of interdisciplinarity, but to our knowledge there have been few attempts to consider and present diverse perspectives from scholars, artists, and scientists in a cohesive manner. As a team of 57 members from the Canadian College of New Scholars, Artists, and Scientists of the Royal Society of Canada (the College) who self-identify as being engaged or interested in interdisciplinarity, we provide diverse intellectual, cultural, and social perspectives. The goal of this paper is to share our collective wisdom on this topic with the broader community and to stimulate discourse and debate on the merits and challenges associated with interdisciplinarity. Perhaps the clearest message emerging from this exercise is that working across established boundaries of scholarly communities is rewarding, necessary, and is more likely to result in impact. However, there are barriers that limit the ease with which this can occur (e.g., lack of institutional structures and funding to facilitate cross-disciplinary exploration). Occasionally, there can be significant risk associated with doing interdisciplinary work (e.g., lack of adequate measurement or recognition of work by disciplinary peers). Solving many of the world’s complex and pressing problems (e.g., climate change, sustainable agriculture, the burden of chronic disease, and aging populations) demand thinking and working across long-standing, but in some ways restrictive, academic boundaries. Academic institutions and key support structures, especially funding bodies, will play an important role in helping to realize what is readily apparent to all who contributed to this paper—that interdisciplinarity is essential for solving complex problems; it is the new norm. Failure to empower and encourage those doing this research will serve as a great impediment to training, knowledge, and addressing societal issues