Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence

Many policy optimization approaches in reinforcement learning incorporate a Kullback-Leilbler (KL) divergence to the previous policy, to prevent the policy from changing too quickly. This idea was initially proposed in a seminal paper on Conservative Policy Iteration, with approximations given by algorithms like TRPO and Munchausen Value Iteration (MVI). We continue this line of work by investigating a generalized KL divergence -- called the Tsallis KL divergence -- which use the $q$-logarithm in the definition. The approach is a strict generalization, as $q = 1$ corresponds to the standard KL divergence; $q > 1$ provides a range of new options. We characterize the types of policies learned under the Tsallis KL, and motivate when $q >1$ could be beneficial. To obtain a practical algorithm that incorporates Tsallis KL regularization, we extend MVI, which is one of the simplest approaches to incorporate KL regularization. We show that this generalized MVI($q$) obtains significant improvements over the standard MVI($q = 1$) across 35 Atari games.Comment: Accepted by NeurIPS 202

Theoretical model for the formation of caveolae and similar membrane invaginations

We study a physical model for the formation of bud-like invaginations on fluid lipid membranes under tension, and apply this model to caveolae formation. We demonstrate that budding can be driven by membrane-bound proteins, provided that they exert asymmetric forces on the membrane that give rise to bending moments. In particular, caveolae formation does not necessarily require forces to be applied by the cytoskeleton. Our theoretical model is able to explain several features observed experimentally in caveolae, where proteins in the caveolin family are known to play a crucial role in the formation of caveolae buds. These include 1), the formation of caveolae buds with sizes in the 100-nm range and 2), that certain N- and C-termini deletion mutants result in vesicles that are an order-of-magnitude larger. Finally, we discuss the possible origin of the morphological striations that are observed on the surfaces of the caveolae

Meta-descent for Online, Continual Prediction

This paper investigates different vector step-size adaptation approaches for non-stationary online, continual prediction problems. Vanilla stochastic gradient descent can be considerably improved by scaling the update with a vector of appropriately chosen step-sizes. Many methods, including AdaGrad, RMSProp, and AMSGrad, keep statistics about the learning process to approximate a second order update---a vector approximation of the inverse Hessian. Another family of approaches use meta-gradient descent to adapt the step-size parameters to minimize prediction error. These meta-descent strategies are promising for non-stationary problems, but have not been as extensively explored as quasi-second order methods. We first derive a general, incremental meta-descent algorithm, called AdaGain, designed to be applicable to a much broader range of algorithms, including those with semi-gradient updates or even those with accelerations, such as RMSProp. We provide an empirical comparison of methods from both families. We conclude that methods from both families can perform well, but in non-stationary prediction problems the meta-descent methods exhibit advantages. Our method is particularly robust across several prediction problems, and is competitive with the state-of-the-art method on a large-scale, time-series prediction problem on real data from a mobile robot.Comment: AAAI Conference on Artificial Intelligence 2019. v2: Correction to Baird's counterexample. A bug in the code lead to results being reported for AMSGrad in this experiment, when they were actually results for Ada

3D Printing of Bone Spurs Before Surgical Removal

Project Background: In the US alone, total knee arthroplasty is the most common performed orthopedic surgery, with over 700,000 cases per year.1 Overall, 21-25% underwent revision due to instability.2 The stability of a TKA depends largely on soft tissue balance for proper alignment in flexion and extension. Soft tissue balancing in TKA depends on posterior femoral condylar osteophytes, which prevent full extension of the knee and increase tension posteriorly. Pre-operative soft tissue visualization is difficult and this leaves an increased chance for implant failure and revision surgery. Having a 3D model of the bone spur before and during the surgery may decrease risk of complication and enhance soft tissue modification for proper knee balancing post TKA. Proposed Methods: We will 3D print a knee via MRI or CT that has been de-identified and provided by our mentor. The orthopedic surgeon will use this print out before or during surgery to address any concerns they have during the surgery in terms of proper balancing of the soft tissue of the knee. After approximately 10 uses within the OR, we will interview the surgeons and patients for their feedback. Patients undergoing total knee arthroplasty using the 3D printed knee will be matched with historic records of those who had undergone TKA. Rates of revision, patient satisfaction, and OR time will be compared across groups using chi-square and t-tests where appropriate. Results: We anticipate reduced OR time, increased patient satisfaction and decreased rates of revision. Conclusions: Our study demonstrates an initial use for 3D models as an aid or guide for total knee arthroplasty. Using a 3D model for TKA helps the surgeon visualize osteophytes and reduces the need for revision surgery. A larger study will need to be conducted in order to test the feasibility and practicality of 3D printing for surgery

3D Printing of Knee Models to Decrease OR Time and Reduce Revisional Surgery in Total Knee Arthroplasty (TKA)

Project Background: Osteophytes are a common problem, affecting 2% of the United States population. For many elderly people, these osteophytes will cause them to seek medical attention. Due to the 2-D nature of MRIs and CT scans, it can be difficult to gain a complete understanding of the complicated soft tissue structures surrounding the joint when performing a Total Knee Arthroplasty(TKA). Without proper removal of osteophytes and correct soft tissue balancing, there is an increased rate of revisional surgery. By utilizing a 3-D model preoperatively and within the OR, surgeons can visualize various aspects of the knee to determine what may be contributing to a soft tissue imbalance. Proposed Methods: We plan to conduct a prospective cohort study at the Rothman Institute. We will use CT images to create 3-D printed models of knees complicated with osteophytes. The surgeon will have the model to reference both preoperatively and during the surgery. We will measure the effectiveness of model by collecting data on the total procedural time, the rate of revisional surgery within the next year, and through feedback from the surgeons. Results: Although we do not currently have any results, we anticipate approval for our project shortly. We hope to have data collected within the next few months supporting our hypothesis that 3-D models will decrease both OR time and revisional surgeries. Conclusions: While the literature shows that these 3-D models may help with OR time, we have been unable to verify this yet. Additionally, revisional surgery can happen months later, so this data may be more difficult to collect in our time frame. After the pilot study has been completed, if we have promising results we hope to expand the project to include other types of surgeries affected by soft tissue balancing

3D Printing of Bone Spurs Before Surgical Removal During Total Knee Arthroplasty

Background: In the United States, total knee arthroplasty (TKA) is the most common performed orthopedic surgery, with over 700,000 cases per year. Overall, 21-25% underwent revision due to instability. Incorrect soft tissue balancing during the procedure can lead to improper alignment, flexion, and extension. Visualization of osteophytes in a 3D manner prior to removal is difficult and poses significant risks for improper balancing on TKA. The purpose of the study is to determine whether the utilization of 3D osteophyte models is beneficial to Orthopaedic surgeons in the course of care, specifically with regard to improving outcomes, decreasing complication rates, and decreasing OR time. Methods: A pilot study will be performed pending the approval of the IRB and research proposal by the Rothman Institute. Deidentified 3D models for upcoming procedures will be printed utilizing patient CT scans prior to date of operation. The models will be provided to Orthopaedic Surgeons at the Rothman Institute prior to the procedure. Data from at least 10 cases will be collected post-operatively, in which operating surgeons will be interviewed assess beliefs on utility of models, OR times, and rates of revision. Results: Direct interviews with Orthopaedic surgeons and residents of the Rothman Institute demonstrate early interest and support of the utilization of 3D models in the OR. Lack of IRB prevents the utilization of models in the OR, but we anticipate decrease OR time and increased satisfaction by involved surgeons. Conclusions: Initial feedback from Orthopaedic surgeons suggest a space for the utility of 3D models in the OR. A significant limitation may be accessing CT images of patients and they are often not performed prior to operations. Next steps include IRB approval and finalizing a logistical blueprint for utilization for the models in the OR, specifically whether the models will be use preoperatively or perioperatively

Histone acetylation increases in response to ferulic, gallic, and sinapic acids acting synergistically in vitro to inhibit \u3ci\u3eCandida albicans\u3c/i\u3e yeast‐to‐hyphae transition

Novel treatments are needed to prevent candidiasis/candidemia infection due to the emergence of Candida species resistant to current antifungals. Considering the yeast-to‐hyphae switch is a critical factor to Candida albicans virulence, phenols common in plant sources have been reported to demonstrating their ability to prevent dimorphism. Therefore, phenols present in many agricultural waste stress (ferulic (FA) and gallic (GA) acid) were initially screened in isolation for their yeast‐to‐hyphae inhibitory properties at times 3, 6, and 24 hr. Both FA and GA inhibited 50% of hyphae formation inhibitory concentration (IC50) but at a concentration of 8.0 ± 0.09 and 90.6 ± 1.05 mM, respectively, at 24 hr. However, the inhibitory effect of FA increased by 1.9–2.6 fold when combined with different GA concentrations. GA and FA values decreased even lower when sinapic acid (SA) was added as a third component. As evidenced by concave isobolograms and combination indexes less than 1, both GA:F A and GA:FA:SA combinations acted synergistically to inhibit 50% hyphae formation at 24 hr. Lastly, acetylation of histone H3 lysine 56 acetylation (H3K56) was higher in response to the triple phenolic cocktail (using the IC50 24 hr inhibitory concentration level) comparable with the nontreated samples, indicating that the phenols inhibited hyphal growth in part by targeting H3K56 acetylation