Reinforcement Learning With Reward Machines in Stochastic Games

We investigate multi-agent reinforcement learning for stochastic games with complex tasks, where the reward functions are non-Markovian. We utilize reward machines to incorporate high-level knowledge of complex tasks. We develop an algorithm called Q-learning with reward machines for stochastic games (QRM-SG), to learn the best-response strategy at Nash equilibrium for each agent. In QRM-SG, we define the Q-function at a Nash equilibrium in augmented state space. The augmented state space integrates the state of the stochastic game and the state of reward machines. Each agent learns the Q-functions of all agents in the system. We prove that Q-functions learned in QRM-SG converge to the Q-functions at a Nash equilibrium if the stage game at each time step during learning has a global optimum point or a saddle point, and the agents update Q-functions based on the best-response strategy at this point. We use the Lemke-Howson method to derive the best-response strategy given current Q-functions. The three case studies show that QRM-SG can learn the best-response strategies effectively. QRM-SG learns the best-response strategies after around 7500 episodes in Case Study I, 1000 episodes in Case Study II, and 1500 episodes in Case Study III, while baseline methods such as Nash Q-learning and MADDPG fail to converge to the Nash equilibrium in all three case studies

DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations

The diffusion-based text-to-image model harbors immense potential in transferring reference style. However, current encoder-based approaches significantly impair the text controllability of text-to-image models while transferring styles. In this paper, we introduce DEADiff to address this issue using the following two strategies: 1) a mechanism to decouple the style and semantics of reference images. The decoupled feature representations are first extracted by Q-Formers which are instructed by different text descriptions. Then they are injected into mutually exclusive subsets of cross-attention layers for better disentanglement. 2) A non-reconstructive learning method. The Q-Formers are trained using paired images rather than the identical target, in which the reference image and the ground-truth image are with the same style or semantics. We show that DEADiff attains the best visual stylization results and optimal balance between the text controllability inherent in the text-to-image model and style similarity to the reference image, as demonstrated both quantitatively and qualitatively. Our project page is https://tianhao-qi.github.io/DEADiff/.Comment: Accepted by CVPR 202

Identification of disulfidptosis related subtypes, characterization of tumor microenvironment infiltration, and development of DRG prognostic prediction model in RCC, in which MSH3 is a key gene during disulfidptosis

Disulfidptosis is a newly discovered mode of cell death induced by disulfide stress. However, the prognostic value of disulfidptosis-related genes (DRGs) in renal cell carcinoma (RCC) remains to be further elucidated. In this study, consistent cluster analysis was used to classify 571 RCC samples into three DRG-related subtypes based on changes in DRGs expression. Through univariate regression analysis and LASSO-Cox regression analysis of differentially expressed genes (DEGs) among three subtypes, we constructed and validated a DRG risk score to predict the prognosis of patients with RCC, while also identifying three gene subtypes. Analysis of DRG risk score, clinical characteristics, tumor microenvironment (TME), somatic cell mutations, and immunotherapy sensitivity revealed significant correlations between them. A series of studies have shown that MSH3 can be a potential biomarker of RCC, and its low expression is associated with poor prognosis in patients with RCC. Last but not least, overexpression of MSH3 promotes cell death in two RCC cell lines under glucose starvation conditions, indicating that MSH3 is a key gene in the process of cell disulfidptosis. In summary, we identify potential mechanism of RCC progression through DRGs -related tumor microenvironment remodeling. In addition, this study has successfully established a new disulfidptosis-related genes prediction model and discovered a key gene MSH3. They may be new prognostic biomarkers for RCC patients, provide new insights for the treatment of RCC patients, and may inspire new methods for the diagnosis and treatment of RCC patients

Loss of circSRY reduces γH2AX level in germ cells and impairs mouse spermatogenesis.

Sry on the Y chromosome is the master switch of sex determination in mammals. It has been well established that Sry encodes a transcription factor that is transiently expressed in somatic cells of the male gonad, leading to the formation of testes. In the testis of adult mice, Sry is expressed as a circular RNA (circRNA) transcript. However, the physiological function of Sry circRNA (circSRY) remains unknown since its discovery in 1993. Here we show that circSRY is mainly expressed in the spermatocytes, but not in mature sperm or somatic cells of the testis. Loss of circSRY led to germ cell apoptosis and the reduction of sperm count in the epididymis. The level of γH2AX was decreased, and failure of XY body formation was noted in circSRY KO germ cells. Further study demonstrated that circSRY directly bound to miR-138-5p in spermatocytes, and in vitro assay suggested that circSRY regulates H2AX mRNA through sponging miR-138-5p. Our study demonstrates that, besides determining sex, Sry also plays an important role in spermatogenesis as a circRNA

Inhibition of highly productive HIV-1 infection in T cells, primary human macrophages, microglia, and astrocytes by Sargassum fusiforme

BACKGROUND: The high rate of HIV-1 mutation and increasing resistance to currently available antiretroviral (ART) therapies highlight the need for new antiviral agents. Products derived from natural sources have been shown to inhibit HIV-1 replication during various stages of the virus life cycle, and therefore represent a potential source of novel therapeutic agents. To expand our arsenal of therapeutics against HIV-1 infection, we investigated aqueous extract from Sargassum fusiforme (S. fusiforme) for ability to inhibit HIV-1 infection in the periphery, in T cells and human macrophages, and for ability to inhibit in the central nervous system (CNS), in microglia and astrocytes. RESULTS: S. fusiforme extract blocked HIV-1 infection and replication by over 90% in T cells, human macrophages and microglia, and it also inhibited pseudotyped HIV-1 (VSV/NL4-3) infection in human astrocytes by over 70%. Inhibition was mediated against both CXCR4 (X4) and CCR5 (R5)-tropic HIV-1, was dose dependant and long lasting, did not inhibit cell growth or viability, was not toxic to cells, and was comparable to inhibition by the nucleoside analogue 2', 3'-didoxycytidine (ddC). S. fusiforme treatment blocked direct cell-to-cell infection spread. To investigate at which point of the virus life cycle this inhibition occurs, we infected T cells and CD4-negative primary human astrocytes with HIV-1 pseudotyped with envelope glycoprotein of vesicular stomatitis virus (VSV), which bypasses the HIV receptor requirements. Infection by pseudotyped HIV-1 (VSV/NL4-3) was also inhibited in a dose dependant manner, although up to 57% less, as compared to inhibition of native NL4-3, indicating post-entry interferences. CONCLUSION: This is the first report demonstrating S. fusiforme to be a potent inhibitor of highly productive HIV-1 infection and replication in T cells, in primary human macrophages, microglia, and astrocytes. Results with VSV/NL4-3 infection, suggest inhibition of both entry and post-entry events of the virus life cycle. Absence of cytotoxicity and high viability of treated cells also suggest that S. fusiforme is a potential source of novel naturally occurring antiretroviral compounds that inhibit HIV-1 infection and replication at more than one site of the virus life cycle

Clinical application of superselective transarterial embolization of renal tumors in zero ischaemia robotic-assisted laparoscopic partial nephrectomy

ObjectiveTo assess the feasibility and safety of zero ischaemia robotic-assisted laparoscopic partial nephrectomy (RALPN) after preoperative superselective transarterial embolization (STE) of T1 renal cancer.MethodsWe retrospectively analyzed the data of 32 patients who underwent zero ischaemia RALPN after STE and 140 patients who received standard robot-assisted laparoscopic partial nephrectomy (S-RALPN). In addition, we selected 35 patients treated with off-clamp RALPN (O-RALPN) from September 2017 to March 2022 for comparison. STE was performed by the same interventional practitioner, and zero ischaemia laparoscopic partial nephrectomy (LPN) was carried out by experienced surgeon 1-12 hours after STE. The intraoperative data and postoperative complications were recorded. The postoperative renal function, routine urine test, urinary Computed Tomography (CT), and preoperative and postoperative glomerular filtration rate (GFR) data were analyzed.ResultsAll operations were completed successfully. There were no cases of conversion to opening and no deaths. The renal arterial trunk was not blocked. No blood transfusions were needed. The mean operation time was 91.5 ± 34.28 minutes. The mean blood loss was 58.59 ± 54.11 ml. No recurrence or metastasis occurred.ConclusionFor patients with renal tumors, STE of renal tumors in zero ischaemia RALPN can preserve more renal function, and it provides a safe and feasible surgical method

PwHAP5, a CCAAT-binding transcription factor, interacts with PwFKBP12 and plays a role in pollen tube growth orientation in Picea wilsonii

The HAP complex occurs in many eukaryotic organisms and is involved in multiple physiological processes. Here it was found that in Picea wilsonii, HAP5 (PwHAP5), a putative CCAAT-binding transcription factor gene, is involved in pollen tube development and control of tube orientation. Quantitative real-time reverse transcription-PCR showed that PwHAP5 transcripts were expressed strongly in germinating pollen and could be induced by Ca2+. Overexpression of PwHAP5 in pollen altered pollen tube orientation, whereas the tube with PwHAP5RNAi showed normal growth without diminishing pollen tube growth. Furthermore, PwFKBP12, which encodes an FK506-binding protein (FKBP) was screened and a bimolecular fluorescence complementation assay performed to confirm the interaction of PwHAP5 and PwFKBP12 in vivo. Transient expression of PwFKBP12 in pollen showed normal pollen tube growth, whereas the tube with PwFKBP12RNAi bent. The phenotype of overexpression of HAP5 on pollen tube was restored by FKBP12. Altogether, our study supported the role of HAP5 in pollen tube development and orientation regulation and identified FKBP12 as a novel partner to interact with HAP5 involved in the process

CONTRACT-BASED MULTI-RATE CONTROL DESIGN

This thesis investigates a contract-based control design framework that aims to solve the constraint partition problem raised in multi-rate control. There are three components in such a framework: the contract coordinator, the high-level motion planner, and the low-level tracking controller. The primary idea of the contract coordinator is to resolve the online constraint partition problem, thereby enhancing coordination between levels and enabling a more adaptable multi-rate control framework. This flexibility in the control framework translates to more choices in selecting the initial state. The contract coordinator is designed based on solving a linear matrix inequality (LMI) problem in real-time, which takes the online information from both levels and outputs the new constraint partition online. Given the input and state constraints, a robust model predictive control with varying constraints is proposed for high-level motion planning. The low level implements the robust control barrier function-based quadratic programming to ensure safety given the constraints from the LMI coordinator. The theoretical guarantees of each level, including feasibility, safety, and stability, are provided. One numerical simulation example is given to show the effectiveness of the proposed method

Cognitive modelling of visual configuration reproduction tasks

The process of reproducing visual patterns from memory raises questions about processing and encoding of perceived visual information in memory and strategies to overcome visual memory limitations. The thesis investigates the cognitive processes of drawing visual configurations from memory. It investigates the role of chunking in visual pattern reproduction and how it is affected by stimulus viewing time, features of the stimulus, or existing knowledge of participants. Two empirical datasets of visual configuration reproduction tasks were reanalysed. And cognitive models were developed to evaluate the results of analyses.The first disc location memorisation task concerns reproducing a number of randomly placed discs with various presenting durations. Analysis focusing on temporal and spatial measures to examine the presence of chunking in the disc production procedure of participants. The analyses indicate that in short trials (≤ 200 ms), participants are likely to retain the stimulus as a mental image. However, in long exposure trials (≥ 1 s), participants were also likely to build chunks based on salient geometric patterns which contain a larger number of discs. Two models about the disc location memorisation task were developed within the ACT-R cognitive architecture to better understand the cognition by comparing the models’ performance to that of human data. Moreover, the testing of models which only use the standard ACT-R showed that the task cannot be modelled without adding additional capabilities. Thus, a snapshot model with a fovea-peripheral based activation mechanism was developed, which simulates how more attention and processing resources are given to the centre of the visual field for short stimulus exposure trials. For long exposure trials, a chunking model was developed based on the snapshot model by adding chunking processes which can encode geometric patterns.The second complex diagram drawing task is reproducing a single complex abstract diagram in four types of tasks including tracing, copying, immediate recall and delayed recall across 10 sessions. Cluster size and response strategy analyses found that participants used chunking to aid the memorising processes, and most of them changed their strategy from an initial generic nearest neighbour first strategy to a chunk dependent strategy with increasing familiarity of the stimulus. An ACT-R model was developed following the procedure of drawing the complex diagram. The model includes a pattern identifying mechanism based on the visibilities of lines, a cognitive processes competition between visual information acquisition and memory retrieval processes, and a symmetry based visual feature encoding process. The model simulated the effects of strategy changes and chunking hierarchy formation on human participants’ performance in this drawing task.The overall studies found that chunking plays a substantial role in visual configuration re-production tasks. The implications of the cognitive models are discussed with regard to existing accounts of diagrammatic reasoning and problem solving.</p