Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach

Compared to on-policy counterparts, off-policy model-free deep reinforcement learning can improve data efficiency by repeatedly using the previously gathered data. However, off-policy learning becomes challenging when the discrepancy between the underlying distributions of the agent's policy and collected data increases. Although the well-studied importance sampling and off-policy policy gradient techniques were proposed to compensate for this discrepancy, they usually require a collection of long trajectories and induce additional problems such as vanishing/exploding gradients or discarding many useful experiences, which eventually increases the computational complexity. Moreover, their generalization to either continuous action domains or policies approximated by deterministic deep neural networks is strictly limited. To overcome these limitations, we introduce a novel policy similarity measure to mitigate the effects of such discrepancy in continuous control. Our method offers an adequate single-step off-policy correction that is applicable to deterministic policy networks. Theoretical and empirical studies demonstrate that it can achieve a "safe" off-policy learning and substantially improve the state-of-the-art by attaining higher returns in fewer steps than the competing methods through an effective schedule of the learning rate in Q-learning and policy optimization

Are treatment guides and rational drug use policies adequately exploited in combating respiratory system diseases?

SummaryThe aim of the present study was to increase awareness regarding the rational use of medicines. The data were obtained via the Material Resources Management System Module of the Ministry of Health. For the appropriateness of treatments, the Global Initiative for Asthma, the Global Initiative for Chronic Obstructive Lung Disease, and the guidelines for the rational use of medicines were used. We also investigated whether any de-escalation method or physical exercise was performed. Statistical analyses were performed using descriptive statistics to determine the mean, standard deviation, and frequency. The results showed that healthcare providers ignored potential drug reactions or adverse interactions, and reflecting the lack of adherence to the current treatment guides, 35.8% irrational use of medicines was recorded. Thus, de-escalation methods should be used to decrease costs or narrow the antibiotic spectrum, antibiotic selection should consider the resistance patterns, culturing methods should be analyzed, and monotherapy should be preferred over combination treatments

Improving Phase Change Memory Performance with Data Content Aware Access

A prominent characteristic of write operation in Phase-Change Memory (PCM) is that its latency and energy are sensitive to the data to be written as well as the content that is overwritten. We observe that overwriting unknown memory content can incur significantly higher latency and energy compared to overwriting known all-zeros or all-ones content. This is because all-zeros or all-ones content is overwritten by programming the PCM cells only in one direction, i.e., using either SET or RESET operations, not both. In this paper, we propose data content aware PCM writes (DATACON), a new mechanism that reduces the latency and energy of PCM writes by redirecting these requests to overwrite memory locations containing all-zeros or all-ones. DATACON operates in three steps. First, it estimates how much a PCM write access would benefit from overwriting known content (e.g., all-zeros, or all-ones) by comprehensively considering the number of set bits in the data to be written, and the energy-latency trade-offs for SET and RESET operations in PCM. Second, it translates the write address to a physical address within memory that contains the best type of content to overwrite, and records this translation in a table for future accesses. We exploit data access locality in workloads to minimize the address translation overhead. Third, it re-initializes unused memory locations with known all-zeros or all-ones content in a manner that does not interfere with regular read and write accesses. DATACON overwrites unknown content only when it is absolutely necessary to do so. We evaluate DATACON with workloads from state-of-the-art machine learning applications, SPEC CPU2017, and NAS Parallel Benchmarks. Results demonstrate that DATACON significantly improves system performance and memory system energy consumption compared to the best of performance-oriented state-of-the-art techniques.Comment: 18 pages, 21 figures, accepted at ACM SIGPLAN International Symposium on Memory Management (ISMM

The miR-644a/CTBP1/p53 axis suppresses drug resistance by simultaneous inhibition of cell survival and epithelialmesenchymal transition in breast cancer

Tumor cells develop drug resistance which leads to recurrence and distant metastasis. MicroRNAs are key regulators of tumor pathogenesis; however, little is known whether they can sensitize cells and block metastasis simultaneously. Here, we report miR-644a as a novel inhibitor of both cell survival and EMT whereby acting as pleiotropic therapy-sensitizer in breast cancer. We showed that both miR-644a expression and its gene signature are associated with tumor progression and distant metastasis-free survival. Mechanistically, miR-644a directly targets the transcriptional co-repressor C-Terminal Binding Protein 1 (CTBP1) whose knock-outs by the CRISPRCas9 system inhibit tumor growth, metastasis, and drug resistance, mimicking the phenotypes induced by miR-644a. Furthermore, downregulation of CTBP1 by miR-644a upregulates wild type- or mutant-p53 which acts as a 'molecular switch' between G1-arrest and apoptosis by inducing cyclin-dependent kinase inhibitor 1 (p21, CDKN1A, CIP1) or pro-apoptotic phorbol-12-myristate-13-acetate-induced protein 1 (Noxa, PMAIP1), respectively. Interestingly, an increase in mutant-p53 by either overexpression of miR-644a or downregulation of CTBP1 was enough to shift this balance in favor of apoptosis through upregulation of Noxa. Notably, p53- mutant patients, but not p53-wild type ones, with high CTBP1 have a shorter survival suggesting that CTBP1 could be a potential prognostic factor for breast cancer patients with p53 mutations. Overall, re-activation of the miR-644a/CTBP1/p53 axis may represent a new strategy for overcoming both therapy resistance and metastasis

Growth, tolerance and safety outcomes with use of an extensively hydrolyzed casein-based formula in infants with cow’s milk protein allergy

ObjectiveTo evaluate growth, tolerance and safety outcomes with use of an extensively hydrolyzed casein-based formula (eHCF) in infants with cow’s milk protein allergy (CMPA).MethodsA total of 226 infants (mean ± SD age: 106.5 ± 39.5 days, 52.7% were girls) with CMPA who received eHCF comprising at least half of the daily dietary intake were included. Data on anthropometrics [weight for age (WFA), length for age (LFA) and weight for length (WFL) z-scores] were recorded at baseline (visit 1), while data on infant feeding and stool records, anthropometrics and Infant Feeding and Stool Patterns and Formula Satisfaction Questionnaires were recorded at visit 2 (on Days 15 ± 5) and visit 3 (on Days 30 ± 5).ResultsFrom baseline to visit 2 and visit 3, WFA z-scores (from −0.60 ± 1.13 to −0.54 ± 1.09 at visit 2, and to −0.44 ± 1.05 at visit 3, p < 0.001) and WFL z-scores (from −0.80 ± 1.30 to −0.71 ± 1.22 at visit 2, and to −0.64 ± 1.13 at visit 3, p = 0.002) were significantly increased. At least half of infants never experienced irritability or feeding refusal (55.7%) and spit-up after feeding (50.2%). The majority of mothers were satisfied with the study formula (93.2%), and wished to continue using it (92.2%).ConclusionsIn conclusion, eHCF was well-accepted and tolerated by an intended use population of infants  ≤ 6 months of age with CMPA and enabled adequate volume consumption and improved growth indices within 30 days of utilization alongside a favorable gastrointestinal tolerance and a high level of parental satisfaction

Reducing the environmental impact of surgery on a global scale: systematic review and co-prioritization with healthcare workers in 132 countries

Abstract Background Healthcare cannot achieve net-zero carbon without addressing operating theatres. The aim of this study was to prioritize feasible interventions to reduce the environmental impact of operating theatres. Methods This study adopted a four-phase Delphi consensus co-prioritization methodology. In phase 1, a systematic review of published interventions and global consultation of perioperative healthcare professionals were used to longlist interventions. In phase 2, iterative thematic analysis consolidated comparable interventions into a shortlist. In phase 3, the shortlist was co-prioritized based on patient and clinician views on acceptability, feasibility, and safety. In phase 4, ranked lists of interventions were presented by their relevance to high-income countries and low–middle-income countries. Results In phase 1, 43 interventions were identified, which had low uptake in practice according to 3042 professionals globally. In phase 2, a shortlist of 15 intervention domains was generated. In phase 3, interventions were deemed acceptable for more than 90 per cent of patients except for reducing general anaesthesia (84 per cent) and re-sterilization of ‘single-use’ consumables (86 per cent). In phase 4, the top three shortlisted interventions for high-income countries were: introducing recycling; reducing use of anaesthetic gases; and appropriate clinical waste processing. In phase 4, the top three shortlisted interventions for low–middle-income countries were: introducing reusable surgical devices; reducing use of consumables; and reducing the use of general anaesthesia. Conclusion This is a step toward environmentally sustainable operating environments with actionable interventions applicable to both high– and low–middle–income countries

Effect of various enzymatic treatments on the mechanical properties of coir fiber/poly(lactic acid) biocomposites

###EgeUn###The effects of enzymatic treatments on the properties of coir fiber-reinforced poly(lactic acid) (PLA) were not found in the literature. Accordingly, the effects of various enzymatic treatments on the mechanical performance of the coir fiber-reinforced PLA composites were investigated in the current study. Four different enzymes, namely lipase, lactase, pectinase, and cellulase, were used. The mechanical properties of the composites were determined by the tensile, flexural, impact tests, and dynamic mechanical analysis. According to the test results, the use of enzyme treated coir fibers affected the mechanical properties except for the flexural properties with different extents depending upon their type. The tensile strength increased with the treatments of lipase and lactase, while the treatments with pectinase and cellulase had no remarkable effect. The impact strength was improved with enzymatic treatments except for pectinase. All enzymatic treatments improved the elastic modulus below the glass transition temperature. In brief, enzymatic treatments improved the interfacial adhesion between coir fiber and PLA via the waxes and fatty acids removal and/or the increment in surface roughness.Erciyes University Scientific Research UnitErciyes University [BAP-FYL-2017-7635]The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by Erciyes University Scientific Research Unit under grant no. BAP-FYL-2017-7635