Learning to Modulate pre-trained Models in RL

Reinforcement Learning (RL) has been successful in various domains like robotics, game playing, and simulation. While RL agents have shown impressive capabilities in their specific tasks, they insufficiently adapt to new tasks. In supervised learning, this adaptation problem is addressed by large-scale pre-training followed by fine-tuning to new down-stream tasks. Recently, pre-training on multiple tasks has been gaining traction in RL. However, fine-tuning a pre-trained model often suffers from catastrophic forgetting. That is, the performance on the pre-training tasks deteriorates when fine-tuning on new tasks. To investigate the catastrophic forgetting phenomenon, we first jointly pre-train a model on datasets from two benchmark suites, namely Meta-World and DMControl. Then, we evaluate and compare a variety of fine-tuning methods prevalent in natural language processing, both in terms of performance on new tasks, and how well performance on pre-training tasks is retained. Our study shows that with most fine-tuning approaches, the performance on pre-training tasks deteriorates significantly. Therefore, we propose a novel method, Learning-to-Modulate (L2M), that avoids the degradation of learned skills by modulating the information flow of the frozen pre-trained model via a learnable modulation pool. Our method achieves state-of-the-art performance on the Continual-World benchmark, while retaining performance on the pre-training tasks. Finally, to aid future research in this area, we release a dataset encompassing 50 Meta-World and 16 DMControl tasks.Comment: 10 pages (+ references and appendix), Code: https://github.com/ml-jku/L2

Population-based SEER trend analysis of overall and cancer-specific survival in 5138 patients with gastrointestinal stromal tumor

Background: The objective of the present population-based analysis was to assess survival patterns in patients with resected and metastatic GIST. Methods: Patients with histologically proven GIST were extracted from the Surveillance, Epidemiology and End Results (SEER) database from 1998 through 2011. Survival was determined applying Kaplan-Meier-estimates and multivariable Cox-regression analyses. The impact of size and mitotic count on survival was assessed with a generalized receiver-operating characteristic-analysis. Results: Overall, 5138 patients were included. Median age was 62 years (range: 18–101 years), 47.3% were female, 68.8% Caucasians. GIST location was in the stomach in 58.7% and small bowel in 31.2%. Lymph node and distant metastases were found in 5.1 and 18.0%, respectively. For non-metastatic GIST, three-year overall survival increased from 68.5% (95% CI: 58.8–79.8%) in 1998 to 88.6% (95% CI: 85.3–92.0%) in 2008, cancer-specific survival from 75.3% (95% CI: 66.1–85.9%) in 1998 to 92.2% (95% CI: 89.4–95.1%) in 2008. For metastatic GIST, three-year overall survival increased from 15.0% (95% CI: 5.3–42.6%) in 1998 to 54.7% (95% CI: 44.4–67.3%) in 2008, cancer-specific survival from 15.0% (95% CI: 5.3–42.6%) in 1998 to 61.9% (95% CI: 51.4–74.5%) in 2008 (all PTrend < 0.05). Conclusions: This is the first SEER trend analysis assessing outcomes in a large cohort of GIST patients over a 11-year time period. The analysis provides compelling evidence of a statistically significant and clinically relevant increase in overall and cancer-specific survival from 1998 to 2008, both for resected as well as metastatic GIST