Deep Residual Policy Reinforcement Learning as a Corrective Term in Process Control for Alarm Reduction: A Preliminary Report

Conventional process controllers (such as proportional integral derivative controllers and model predictive controllers) are simple and effective once they have been calibrated for a given system. However, it is difficult and costly to re-tune these controllers if the system deviates from its normal conditions and starts to deteriorate. Recently, reinforcement learning has shown a significant improvement in learning process control policies through direct interaction with a system, without the need of a process model or the system characteristics, as it learns the optimal control by interacting with the environment directly. However, developing such a black-box system is a challenge when the system is complex and it may not be possible to capture the complete dynamics of the system with just a single reinforcement learning agent. Therefore, in this paper, we propose a simple architecture that does not replace the conventional proportional integral derivative controllers but instead augments the control input to the system with a reinforcement learning agent. The agent adds a correction factor to the output provided by the conventional controller to maintain optimal process control even when the system is not operating under its normal condition

Interpretable Input-Output Hidden Markov Model-Based Deep Reinforcement Learning for the Predictive Maintenance of Turbofan Engines

An open research question in deep reinforcement learning is how to focus the policy learning of key decisions within a sparse domain. This paper emphasizes on combining the advantages of input-output hidden Markov models and reinforcement learning. We propose a novel hierarchical modeling methodology that, at a high level, detects and interprets the root cause of a failure as well as the health degradation of the turbofan engine, while at a low level, provides the optimal replacement policy. This approach outperforms baseline deep reinforcement learning (DRL) models and has performance comparable to that of a state-of-the-art reinforcement learning system while being more interpretable

Specialized Deep Residual Policy Safe Reinforcement Learning-Based Controller for Complex and Continuous State-Action Spaces

Traditional controllers have limitations as they rely on prior knowledge about the physics of the problem, require modeling of dynamics, and struggle to adapt to abnormal situations. Deep reinforcement learning has the potential to address these problems by learning optimal control policies through exploration in an environment. For safety-critical environments, it is impractical to explore randomly, and replacing conventional controllers with black-box models is also undesirable. Also, it is expensive in continuous state and action spaces, unless the search space is constrained. To address these challenges we propose a specialized deep residual policy safe reinforcement learning with a cycle of learning approach adapted for complex and continuous state-action spaces. Residual policy learning allows learning a hybrid control architecture where the reinforcement learning agent acts in synchronous collaboration with the conventional controller. The cycle of learning initiates the policy through the expert trajectory and guides the exploration around it. Further, the specialization through the input-output hidden Markov model helps to optimize policy that lies within the region of interest (such as abnormality), where the reinforcement learning agent is required and is activated. The proposed solution is validated on the Tennessee Eastman process control

Hierarchical Framework for Interpretable and Probabilistic Model-Based Safe Reinforcement Learning

The difficulty of identifying the physical model of complex systems has led to exploring methods that do not rely on such complex modeling of the systems. Deep reinforcement learning has been the pioneer for solving this problem without the need for relying on the physical model of complex systems by just interacting with it. However, it uses a black-box learning approach that makes it difficult to be applied within real-world and safety-critical systems without providing explanations of the actions derived by the model. Furthermore, an open research question in deep reinforcement learning is how to focus the policy learning of critical decisions within a sparse domain. This paper proposes a novel approach for the use of deep reinforcement learning in safety-critical systems. It combines the advantages of probabilistic modeling and reinforcement learning with the added benefits of interpretability and works in collaboration and synchronization with conventional decision-making strategies. The BC-SRLA is activated in specific situations which are identified autonomously through the fused information of probabilistic model and reinforcement learning, such as abnormal conditions or when the system is near-to-failure. Further, it is initialized with a baseline policy using policy cloning to allow minimum interactions with the environment to address the challenges associated with using RL in safety-critical industries. The effectiveness of the BC-SRLA is demonstrated through a case study in maintenance applied to turbofan engines, where it shows superior performance to the prior art and other baselines.Comment: arXiv admin note: text overlap with arXiv:2206.1343

Human MAIT cells respond to and suppress HIV-1

Human MAIT cells sit at the interface between innate and adaptive immunity, are polyfunctional and are capable of killing pathogen infected cells via recognition of the Class IB molecule MR1. MAIT cells have recently been shown to possess an antiviral protective role in vivo and we therefore sought to explore this in relation to HIV-1 infection. There was marked activation of MAIT cells in vivo in HIV-1-infected individuals, which decreased following ART. Stimulation of THP1 monocytes with R5 tropic HIVBAL potently activated MAIT cells in vitro. This activation was dependent on IL-12 and IL-18 but was independent of the TCR. Upon activation, MAIT cells were able to upregulate granzyme B, IFNγ and HIV-1 restriction factors CCL3, 4, and 5. Restriction factors produced by MAIT cells inhibited HIV-1 infection of primary PBMCs and immortalized target cells in vitro. These data reveal MAIT cells to be an additional T cell population responding to HIV-1, with a potentially important role in controlling viral replication at mucosal sites

Practising lawyers in Nova Scotia: cognitive style and preferences for practice

The Cognitive Style Index and a demographic survey were administered to 524 practising lawyers in Nova Scotia. Results indicate that lawyers, as a group, have a more analytical than intuitive cognitive style. Differences between men and women and between partners and associates were nonsignificant statistically. This finding suggests lawyers are a more homogeneous group in terms of cognitive style than other groups such as law students and various groups of business managers. However, lawyers differed significantly in cognitive style across various preferred areas of practice. For example, those preferring criminal law scored statistically significantly lower on the Cognitive Style Index than those who preferred Real Estate and Construction law. Organizational behavior implications are discussed

Mucosal-associated invariant T (MAIT) cells are activated in the gastrointestinal tissue of patients with combination ipilimumab and nivolumab therapy-related colitis in a pathology distinct from ulcerative colitis

The aim of this study was to investigate the pathogenesis of combination ipilimumab and nivolumab-associated colitis (IN-COL) by measuring gut-derived and peripheral blood mononuclear cell (GMNC; PBMC) profiles. We studied GMNC and PBMC from patients with IN-COL, IN-treated with no adverse-events (IN-NAE), ulcerative colitis (UC) and healthy volunteers using flow cytometry. In the gastrointestinal-derived cells we found high levels of activated CD8+ T cells and mucosal-associated invariant T (MAIT) cells in IN-COL, changes that were not evident in IN-NAE or UC. UC, but not IN-C, was associated with a high proportion of regulatory T cells (Treg). We sought to determine if local tissue responses could be measured in peripheral blood. Peripherally, checkpoint inhibition instigated a rise in activated memory CD4+ and CD8+ T cells, regardless of colitis. Low circulating MAIT cells at baseline was associated with IN-COL patients compared with IN-NAE in one of two cohorts. UC, but not INCOL, was associated with high levels of circulating plasmablasts. In summary, the alterations in T cell subsets measured in IN-COL-affected tissue, characterized by high levels of activated CD8+ T cells and MAIT cells and a low proportion of Treg, reflected a pathology distinct from UC. These tissue changes differed from the periphery, where T cell activation was a widespread on-treatment effect, and circulating MAIT cell count was low but not reliably predictive of colitis

Circulating gluten-specific FOXP3 + CD39 + regulatory T cells have impaired suppressive function in patients with celiac disease

Background Celiac disease is a chronic immune-mediated inflammatory disorder of the gut triggered by dietary gluten. Although the effector T-cell response in patients with celiac disease has been well characterized, the role of regulatory T (Treg) cells in the loss of tolerance to gluten remains poorly understood. Objective We sought to define whether patients with celiac disease have a dysfunction or lack of gluten-specific forkhead box protein 3 (FOXP3)+ Treg cells. Methods Treated patients with celiac disease underwent oral wheat challenge to stimulate recirculation of gluten-specific T cells. Peripheral blood was collected before and after challenge. To comprehensively measure the gluten-specific CD4+ T-cell response, we paired traditional IFN-γ ELISpot with an assay to detect antigen-specific CD4+ T cells that does not rely on tetramers, antigen-stimulated cytokine production, or proliferation but rather on antigen-induced coexpression of CD25 and OX40 (CD134). Results Numbers of circulating gluten-specific Treg cells and effector T cells both increased significantly after oral wheat challenge, peaking at day 6. Surprisingly, we found that approximately 80% of the ex vivo circulating gluten-specific CD4+ T cells were FOXP3+CD39+ Treg cells, which reside within the pool of memory CD4+CD25+CD127lowCD45RO+ Treg cells. Although we observed normal suppressive function in peripheral polyclonal Treg cells from patients with celiac disease, after a short in vitro expansion, the gluten-specific FOXP3+CD39+ Treg cells exhibited significantly reduced suppressive function compared with polyclonal Treg cells. Conclusion This study provides the first estimation of FOXP3+CD39+ Treg cell frequency within circulating gluten-specific CD4+ T cells after oral gluten challenge of patients with celiac disease. FOXP3+CD39+ Treg cells comprised a major proportion of all circulating gluten-specific CD4+ T cells but had impaired suppressive function, indicating that Treg cell dysfunction might be a key contributor to disease pathogenesis

Development of novel methods for non-canonical myeloma protein analysis with an innovative adaptation of immunofixation electrophoresis, native top-down mass spectrometry, and middle-down de novo sequencing

OBJECTIVES: Multiple myeloma (MM) is a malignant plasma cell neoplasm, requiring the integration of clinical examination, laboratory and radiological investigations for diagnosis. Detection and isotypic identification of the monoclonal protein(s) and measurement of other relevant biomarkers in serum and urine are pivotal analyses. However, occasionally this approach fails to characterize complex protein signatures. Here we describe the development and application of next generation mass spectrometry (MS) techniques, and a novel adaptation of immunofixation, to interrogate non-canonical monoclonal immunoproteins. METHODS: Immunoprecipitation immunofixation (IP-IFE) was performed on a Sebia Hydrasys Scan2. Middle-down de novo sequencing and native MS were performed with multiple instruments (21T FT-ICR, Q Exactive HF, Orbitrap Fusion Lumos, and Orbitrap Eclipse). Post-acquisition data analysis was performed using Xcalibur Qual Browser, ProSight Lite, and TDValidator. RESULTS: We adapted a novel variation of immunofixation electrophoresis (IFE) with an antibody-specific immunosubtraction step, providing insight into the clonal signature of gamma-zone monoclonal immunoglobulin (M-protein) species. We developed and applied advanced mass spectrometric techniques such as middle-down de novo sequencing to attain in-depth characterization of the primary sequence of an M-protein. Quaternary structures of M-proteins were elucidated by native MS, revealing a previously unprecedented non-covalently associated hetero-tetrameric immunoglobulin. CONCLUSIONS: Next generation proteomic solutions offer great potential for characterizing complex protein structures and may eventually replace current electrophoretic approaches for the identification and quantification of M-proteins. They can also contribute to greater understanding of MM pathogenesis, enabling classification of patients into new subtypes, improved risk stratification and the potential to inform decisions on future personalized treatment modalities