Conservative State Value Estimation for Offline Reinforcement Learning

Offline reinforcement learning faces a significant challenge of value over-estimation due to the distributional drift between the dataset and the current learned policy, leading to learning failure in practice. The common approach is to incorporate a penalty term to reward or value estimation in the Bellman iterations. Meanwhile, to avoid extrapolation on out-of-distribution (OOD) states and actions, existing methods focus on conservative Q-function estimation. In this paper, we propose Conservative State Value Estimation (CSVE), a new approach that learns conservative V-function via directly imposing penalty on OOD states. Compared to prior work, CSVE allows more effective in-data policy optimization with conservative value guarantees. Further, we apply CSVE and develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states \emph{around} the dataset, and the actor applies advantage weighted updates extended with state exploration to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods and is strongly competitive among recent SOTA methods

Experimental investigation of two-phase relative permeability of gas and water for tight gas carbonate under different test conditions

Currently, tight carbonate gas reservoir has received little attention due to few discoveries of them. In this study, gas–water two-phase relative permeability was measured under two different conditions: High Temperature High Pore Pressure (HTHPP – 80 °C, 38 MPa), as well as Ambient Condition (AC), using whole core samples of tight gas carbonate. Relative permeability curves obtained at HTHPP showed two contrary curve profiles of gas relative permeability, corresponding to the distinctive micro-pore structure acquired from CT-Scanning. Then, based on Klinkenberg theory and a newly developed slip factor model for tight sandstone, slippage effect under AC is calibrated and the overestimation of gas relative permeability prove up to 41.72%–52.34% in an assumed heterogeneity. In addition, relative permeability curves obtained at HTHPP switch to higher gas saturation compared to that under AC with the rock wettability change from water-wet to less water-wet. And the wettability alteration is believed to be caused by charge change on mineral surface

Solving the Batch Stochastic Bin Packing Problem in Cloud: A Chance-constrained Optimization Approach

This paper investigates a critical resource allocation problem in the first party cloud: scheduling containers to machines. There are tens of services and each service runs a set of homogeneous containers with dynamic resource usage; containers of a service are scheduled daily in a batch fashion. This problem can be naturally formulated as Stochastic Bin Packing Problem (SBPP). However, traditional SBPP research often focuses on cases of empty machines, whose objective, i.e., to minimize the number of used machines, is not well-defined for the more common reality with nonempty machines. This paper aims to close this gap. First, we define a new objective metric, Used Capacity at Confidence (UCaC), which measures the maximum used resources at a probability and is proved to be consistent for both empty and nonempty machines, and reformulate the SBPP under chance constraints. Second, by modeling the container resource usage distribution in a generative approach, we reveal that UCaC can be approximated with Gaussian, which is verified by trace data of real-world applications. Third, we propose an exact solver by solving the equivalent cutting stock variant as well as two heuristics-based solvers -- UCaC best fit, bi-level heuristics. We experimentally evaluate these solvers on both synthetic datasets and real application traces, demonstrating our methodology's advantage over traditional SBPP optimal solver minimizing the number of used machines, with a low rate of resource violations.Comment: To appear in SIGKDD 2022 as Research Track pape

Association between angiotensin-converting enzyme inhibitor-induced cough and the risk of lung cancer: a Mendelian randomization study

Background: Observational studies and meta-analyses have demonstrated a positive correlation between the use of angiotensin-converting enzyme inhibitors (ACEIs) and lung cancer. However, the findings remain controversial; furthermore, the relationship between ACEI-induced cough and lung cancer development remains unknown. We used Mendelian randomization (MR) to verify the association between ACEI use, ACEI-induced cough, and the risk of lung cancer.Methods: We performed a two-sample MR analysis to determine the unconfounded relationships between ACE inhibition, which mimics the effects of ACEIs, and genetic proxies for ACEI-induced cough and lung cancer. Single nucleotide polymorphisms that imitate ACE receptors and ACEI-induced cough were collected and integrated into a meta-analysis of existing genome-wide association studies for various lung cancers. The relationship was quantified using inverse variance weighting, weighted median, and MR-Egger methods.Results: A statistically significant association was observed between ACE inhibition and the risk of small cell lung cancer for Europeans (excluding rs118121655/rs80311894). Associations were identified between ACEI-induced cough and the risk of lung cancer for Europeans, although not for Asians, and between ACEI-induced cough and lung adenocarcinoma (excluding rs360206).Conclusion: Our findings reveal a relationship between ACE inhibition and lung cancer development, as well as a significant association between ACEI-induced cough and a higher risk of lung cancer for Europeans. Patients with hypertension who experience dry cough as a side effect of ACEI use should consider switching to an alternative antihypertensive treatment

Polyclonal antibody against the DPV UL46M protein can be a diagnostic candidate

<p>Abstract</p> <p>Background</p> <p>The duck plague virus (DPV) UL46 protein (VP11/12) is a 739-amino acid tegument protein encoded by the <it>UL46 </it>gene. We analyzed the amino acid sequence of UL46 using bioinformatics tools and defined the main antigenic domains to be between nucleotides 700-2,220 in the <it>UL46 </it>sequence. This region was designated UL46M. The DPV <it>UL46 </it>and <it>UL46M </it>genes were both expressed in <it>Escherichia coli </it>Rosetta (DE3) induced by isopropy1-β-<smcaps>D</smcaps>-thiogalactopyranoside (IPTG) following polymerase chain reaction (PCR) amplification and subcloning into the prokaryotic expression vector pET32a(+). The recombinant proteins were purified using a Ni-NTA spin column and used to generate the polyclonal antibody against UL46 and UL46M in New Zealand white rabbits. The titer was then tested using enzyme-linked immunosorbent assay (ELISA) and agar diffusion reaction, and the specificity was tested by western blot analysis. Subsequently, we established Dot-ELISA using the polyclonal antibody and applied it to DPV detection.</p> <p>Results</p> <p>In our study, the DPV UL46M fusion protein, with a relative molecular mass of 79 kDa, was expressed in <it>E. coli </it>Rosetta (DE3). Expression of the full <it>UL46 </it>gene failed, which was consistent with the results from the bioinformatic analysis. The expressed product was directly purified using Ni-NTA spin column to prepare the polyclonal antibody against UL46M. The titer of the anti-UL46M antisera was over 1:819,200 as determined by ELISA and 1:8 by agar diffusion reaction. Dot-ELISA was used to detect DPV using a 1:60 dilution of anti-UL46M IgG and a 1:5,000 dilution of horseradish peroxidase (HRP)-labeled goat anti-rabbit IgG.</p> <p>Conclusions</p> <p>The anti-UL46M polyclonal antibody reported here specifically identifies DPV, and therefore, it is a promising diagnostic tool for DPV detection in animals. UL46M and the anti-UL46M antibody can be used for further clinical examination and research of DPV.</p

Development and validation of nomogram for predicting cognitive frailty with multimorbidity: a cross-sectional study

ObjectivesThis study aims to develop and validate a nomogram for cognitive frailty in older patients with multimorbidity.MethodsFrom April 2022 to December 1, 2024, a total of 711 older patients participated in the study. The study was conducted at a tertiary hospitals in Changsha, Hunan Province, China. We employed LASSO regression to identify initial variables associated with risk factors for older adults with multimorbidity and subsequently utilized regression analysis to develop predictive models. We collected encompassing demographic information, FRAIL scale scores, Mini-Mental State Examination (MMSE) results, Mini Nutritional Assessment Short Form (MNA-SF) evaluations, Patient Health Questionnaire-9 (PHQ-9) responses, and Athens Insomnia Scale (AIS) ratings. Statistical analyses were performed using R version 4.3.2. The model’s predictive performance was evaluated using receiver operating characteristic (ROC) and area under the curve (AUC). Calibration was assessed via calibration curves, and clinical utility through decision curve analysis (DCA). Internal consistency was validated using bootstrap, and external validity with an independent test dataset.ResultsIn this study, the training and validation sets were 498 and 213 patients, respectively. In the training set, there were 183 patients with cognitive frailty with a prevalence of 36.9%. Six initial variables were selected for the LASSO regression, including drinking, constipation, polypharmacy, chronic pain, nutrition, and depression. These six variables were included in the final predictive model. The model demonstrated a concordance index (C-index) of 0.818. Furthermore, AUC for the training and validation sets were determined to be 0.827 and 0.784, underscoring the model’s robust predictive capability.ConclusionThe high prevalence of cognitive frailty in older patients with multimorbidity should be noted. Efforts to diagnose cognitive frailty and develop targeted interventions in the context of an ageing population and young onset of dementia are of significance in delaying and reversing cognitive frailty

The Broad Host Range Phage vB_CpeS_BG3P Is Able to Inhibit Clostridium perfringens Growth

Clostridium perfringens is an important pathogen for both humans and animals, causing human foodborne disease and necrotic enteritis in poultry. In the present study, a C. perfringens-specific phage, vB_CpeS_BG3P (designated as BG3P hereafter), was isolated from chicken farm sewage. Both electron microscopy and phylogenetic analysis suggested that phage BG3P is a novel phage belonging to Siphoviridae family. Phage BG3P exhibited a broad host range against different C. perfringens isolates (90.63% of strains were infected). Sequencing of the complete genome revealed a linear double-stranded DNA (43,528 bp) with 28.65% GC content. After sequence analysis, 73 open reading frames (orf s) were predicted, of which only 13 were annotated with known functions. No tRNA and virulence encoding genes were detected. It should be noted that the protein of orf 15 has 97.92% homology to C. perfringens-specific chloramphenicol resistance protein, which has not been reported for any C. perfringens phage. Phylogenetic analysis of the ssDNA binding protein demonstrated that this phage is closely related to C. perfringens phages phiSM101 and phi3626. In considering future use as an antimicrobial agent, some biological characteristics were observed, such as a good pH (3–11) stability and moderate temperature tolerance (<60 C). Moreover, bacteriophage BG3P showed a good antimicrobial effect against C. perfringens liquid cultures. Thus, phage treatment with MOI ≥ 100 completely inhibited bacterial growth compared to untreated cultures. Although phage BG3P shows good lytic efficiency and broad host range in vitro, future development and application may need to consider removal of the chloramphenicol-like resistance gene or exploring its lysin for future antibacterial applications.This work was supported by the National Key Research and Development Program of China (No. 2018YFE0101900) and the Jiangsu Agricultural Science and Technology Foundation (No. cx(21)1004)