Accelerated Policy Gradient: On the Nesterov Momentum for Reinforcement Learning

Policy gradient methods have recently been shown to enjoy global convergence at a $\Theta(1/t)$ rate in the non-regularized tabular softmax setting. Accordingly, one important research question is whether this convergence rate can be further improved, with only first-order updates. In this paper, we answer the above question from the perspective of momentum by adapting the celebrated Nesterov's accelerated gradient (NAG) method to reinforcement learning (RL), termed \textit{Accelerated Policy Gradient} (APG). To demonstrate the potential of APG in achieving faster global convergence, we formally show that with the true gradient, APG with softmax policy parametrization converges to an optimal policy at a $\tilde{O}(1/t^2)$ rate. To the best of our knowledge, this is the first characterization of the global convergence rate of NAG in the context of RL. Notably, our analysis relies on one interesting finding: Regardless of the initialization, APG could end up reaching a locally nearly-concave regime, where APG could benefit significantly from the momentum, within finite iterations. By means of numerical validation, we confirm that APG exhibits $\tilde{O}(1/t^2)$ rate as well as show that APG could significantly improve the convergence behavior over the standard policy gradient.Comment: 51 pages, 8 figure

Coordinate Ascent for Off-Policy RL with Global Convergence Guarantees

We revisit the domain of off-policy policy optimization in RL from the perspective of coordinate ascent. One commonly-used approach is to leverage the off-policy policy gradient to optimize a surrogate objective -- the total discounted in expectation return of the target policy with respect to the state distribution of the behavior policy. However, this approach has been shown to suffer from the distribution mismatch issue, and therefore significant efforts are needed for correcting this mismatch either via state distribution correction or a counterfactual method. In this paper, we rethink off-policy learning via Coordinate Ascent Policy Optimization (CAPO), an off-policy actor-critic algorithm that decouples policy improvement from the state distribution of the behavior policy without using the policy gradient. This design obviates the need for distribution correction or importance sampling in the policy improvement step of off-policy policy gradient. We establish the global convergence of CAPO with general coordinate selection and then further quantify the convergence rates of several instances of CAPO with popular coordinate selection rules, including the cyclic and the randomized variants of CAPO. We then extend CAPO to neural policies for a more practical implementation. Through experiments, we demonstrate that CAPO provides a competitive approach to RL in practice.Comment: 47 pages, 4 figure

Subject-relevant Document Recommendation: A Reference Topic-Based Approach

Knowledge-intensive workers, such as academic researchers, medical professionals or patent engineers, have a demanding need of searching information relevant to their work. Content-based recommender system (CBRS) makes recommendation by analyzing similarity of textual contents between documents and users’ preferences. Although content-based filtering has been one of the promising approaches to document recommendations, it encounters the over-specialization problem. CBRS tends to recommend documents that are similar to what have been in user’s preference profile. Rationally, citations in an article represent the intellectual/affective balance of the individual interpretation in time and domain understanding. A cited article shall be associated with and may reflect the subject domain of its citing articles. Our study addresses the over-specialization problem to support the information needs of researchers. We propose a Reference Topic-based Document Recommendation (RTDR) technique, which exploits the citation information of a focal user’s preferred documents and thereby recommends documents that are relevant to the subject domain of his or her preference. Our primary evaluation results suggest the outperformance of the proposed RTDR to the benchmarks

A rare complication in a child undergoing chemotherapy for acute lymphoblastic leukemia: Superior sagittal sinus thrombosis

AbstractWe report the case of a 4-year-old boy with acute lymphoblastic leukemia in high-risk group who suffered from generalized tonic-colonic seizure evolving into status epilepticus, and subsequent left hemiparesis during his first reinduction chemotherapy, consisting of dexamethasone, vincristine, l-asparaginase, and epirubicin. Superior sagittal sinus and cerebral venous thrombosis, predominantly in right side, were proved by brain magnetic resonance imaging. After aggressive treatment with low-molecular weight heparin (LMWH), left hemiparesis improved in 1 week. And he was fully ambulatory 3 weeks later. The second cycle of reinduction chemotherapy was conducted smoothly with the concomitant use of LMWH. This case illustrates the strong correlation of the rare thrombotic complication, superior sagittal sinus thrombosis, and hypercoagulable status secondary to combination use of l-asparaginase and corticosteroid. Early and vigilant recognition of superior sagittal sinus thrombosis and prompt anticoagulation with LMWH may prevent further neurological damage

Autophagy Inhibition Enhances Apoptosis Induced by Dioscin in Huh7 Cells

Extensive research results support the application of herbal medicine or natural food as an augment during therapy for various cancers. However, the effect of dioscin on tumor cells autophagy has not been clearly clarified. In this study, the unique effects of dioscin on autophagy of hepatoma cells were investigated. Results found that dioscin induced caspase-3- and -9-dependent cell apoptosis in a dose-dependent manner. Moreover, inhibition of ERK1/2 phosphorylation significantly abolished the dioscin-induced apoptosis. In addition, dioscin triggered cell autophagy in early stages. With autophagy inhibitors to hinder the autophagy process, dioscin-induced cell apoptosis was significantly enhanced. An inhibition of caspase activation did not affect the dioscin-induced LC3-II protein expression. Based on the results, we believed that while apoptosis was blocked, dioscin-induced autophagy process also diminished in Huh7 cells. In conclusion, this study indicates that dioscin causes autophagy in Huh7 cells and suggests that dioscin has a cytoprotective effect

General Versus Spinal Anesthesia: Which is a Risk Factor for Octogenarian Hip Fracture Repair Patients?

SummaryBackgroundMost studies have shown no difference between the two types of anesthesia administered to hip fracture patients. This study compared postoperative morbidity and mortality in octogenarian patients who received either general or spinal anesthesia for hip fracture repair.MethodsWe retrospectively analyzed the hospital records of 335 octogenarian patients who received hip fracture repair in our teaching hospital between 2002 and 2006. A total of 167 and 168 patients received general and spinal anesthesia, respectively. Morbidity, mortality, and intraoperative and preoperative variables were compared between groups.ResultsThere were no mortality differences between spinal and general anesthesia groups. However, the overall morbidity was greater in the general anesthesia group than in the spinal anesthesia group (21/167 [12.6%] vs. 9/168 [5.4%]; p = 0.02). Respiratory system-related morbidity was also higher in the general anesthesia group than in the spinal anesthesia group (11/167 [6.6%] vs. 3/168 [1.8%]; p = 0.03). Logistic regression analysis revealed two significant predictors of postoperative morbidity: anesthesia type (general; odds ratio, 2.39) and preexisting respiratory diseases (odds ratio, 3.38).ConclusionGeneral anesthesia increased the risk of postoperative morbidity in octogenarian patients after hip fracture repair, and patients with preexisting respiratory diseases were especially vulnerable. Spinal anesthesia is strongly recommended in such individuals

Role of acid-sensing ion channel 3 in sub-acute-phase inflammation

<p>Abstract</p> <p>Background</p> <p>Inflammation-mediated hyperalgesia involves tissue acidosis and sensitization of nociceptors. Many studies have reported increased expression of acid-sensing ion channel 3 (ASIC3) in inflammation and enhanced ASIC3 channel activity with pro-inflammatory mediators. However, the role of ASIC3 in inflammation remains inconclusive because of conflicting results generated from studies of <it>ASIC3 </it>knockout (<it>ASIC3</it>^-/-) or dominant-negative mutant mice, which have shown normal, decreased or increased hyperalgesia during inflammation.</p> <p>Results</p> <p>Here, we tested whether ASIC3 plays an important role in inflammation of subcutaneous tissue of paw and muscle in <it>ASIC3</it>^-/-mice induced by complete Freund's adjuvant (CFA) or carrageenan by investigating behavioral and pathological responses, as well as the expression profile of ion channels. Compared with the <it>ASIC3</it>^+/+controls, <it>ASIC3</it>^-/-mice showed normal thermal and mechanical hyperalgesia with acute (4-h) intraplantar CFA- or carrageenan-induced inflammation, but the hyperalgesic effects in the sub-acute phase (1–2 days) were milder in all paradigms except for thermal hyperalgesia with CFA-induced inflammation. Interestingly, carrageenan-induced primary hyperalgesia was accompanied by an <it>ASIC3</it>-dependent <it>Nav1.9 </it>up-regulation and increase of tetrodotoxin (TTX)-resistant sodium currents. CFA-inflamed muscle did not evoke hyperalgesia in <it>ASIC3</it>^-/-or <it>ASIC3</it>^+/+mice, whereas carrageenan-induced inflammation in muscle abolished mechanical hyperalgesia in <it>ASIC3</it>^-/-mice, as previously described. However, <it>ASIC3</it>^-/-mice showed attenuated pathological features such as less CFA-induced granulomas and milder carrageenan-evoked vasculitis as compared with <it>ASIC3</it>^+/+mice.</p> <p>Conclusion</p> <p>We provide a novel finding that ASIC3 participates in the maintenance of sub-acute-phase primary hyperalgesia in subcutaneous inflammation and mediates the process of granuloma formation and vasculitis in intramuscular inflammation.</p

Effects of and satisfaction with short message service reminders for patient medication adherence: a randomized controlled study

BACKGROUND: Medication adherence is critical for patient treatment. This study involved evaluating how implementing Short Message Service (SMS) reminders affected patient medication adherence and related factors. METHODS: We used a structured questionnaire to survey outpatients at three medical centers. Patients aged 20 years and older who were prescribed more than 7 days of a prescription medication were randomized into SMS intervention or control groups. The intervention group received daily messages reminding them of aspects regarding taking their medication; the control group received no messages. A phone follow-up was performed to assess outcomes after 8 days. Data were collected from 763 participants in the intervention group and 435 participants in the control group. RESULTS: After participants in the intervention group received SMS reminders to take medication or those in the control group received no messages, incidences of delayed doses were decreased by 46.4 and 78.8% for those in the control and intervention groups, respectively. The rate of missed doses was decreased by 90.1% for participants in the intervention group and 61.1% for those in the control group. We applied logistic regression analysis and determined that participants in the intervention group had a 3.2-fold higher probability of having a decrease in delayed doses compared with participants in the control group. Participants in the intervention group also showed a 2.2-fold higher probability of having a decrease in missed doses compared with participants in the control group. CONCLUSIONS: Use of SMS significantly affected the rates of taking medicine on schedule. Therefore, daily SMS could be useful for reminding patients to take their medicine on schedule

Vision-Based Finger Detection, Tracking, and Event Identification Techniques for Multi-Touch Sensing and Display Systems

This study presents efficient vision-based finger detection, tracking, and event identification techniques and a low-cost hardware framework for multi-touch sensing and display applications. The proposed approach uses a fast bright-blob segmentation process based on automatic multilevel histogram thresholding to extract the pixels of touch blobs obtained from scattered infrared lights captured by a video camera. The advantage of this automatic multilevel thresholding approach is its robustness and adaptability when dealing with various ambient lighting conditions and spurious infrared noises. To extract the connected components of these touch blobs, a connected-component analysis procedure is applied to the bright pixels acquired by the previous stage. After extracting the touch blobs from each of the captured image frames, a blob tracking and event recognition process analyzes the spatial and temporal information of these touch blobs from consecutive frames to determine the possible touch events and actions performed by users. This process also refines the detection results and corrects for errors and occlusions caused by noise and errors during the blob extraction process. The proposed blob tracking and touch event recognition process includes two phases. First, the phase of blob tracking associates the motion correspondence of blobs in succeeding frames by analyzing their spatial and temporal features. The touch event recognition process can identify meaningful touch events based on the motion information of touch blobs, such as finger moving, rotating, pressing, hovering, and clicking actions. Experimental results demonstrate that the proposed vision-based finger detection, tracking, and event identification system is feasible and effective for multi-touch sensing applications in various operational environments and conditions