Learning Heuristic Selection with Dynamic Algorithm Configuration

A key challenge in satisficing planning is to use multiple heuristics within one heuristic search. An aggregation of multiple heuristic estimates, for example by taking the maximum, has the disadvantage that bad estimates of a single heuristic can negatively affect the whole search. Since the performance of a heuristic varies from instance to instance, approaches such as algorithm selection can be successfully applied. In addition, alternating between multiple heuristics during the search makes it possible to use all heuristics equally and improve performance. However, all these approaches ignore the internal search dynamics of a planning system, which can help to select the most useful heuristics for the current expansion step. We show that dynamic algorithm configuration can be used for dynamic heuristic selection which takes into account the internal search dynamics of a planning system. Furthermore, we prove that this approach generalizes over existing approaches and that it can exponentially improve the performance of the heuristic search. To learn dynamic heuristic selection, we propose an approach based on reinforcement learning and show empirically that domain-wise learned policies, which take the internal search dynamics of a planning system into account, can exceed existing approaches.Comment: Long version of the paper at the International Conference on Automated Planning and Scheduling (ICAPS) 202

Contextualize Me -- The Case for Context in Reinforcement Learning

While Reinforcement Learning ( RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes. Contextual Reinforcement Learning (cRL) provides a framework to model such changes in a principled manner, thereby enabling flexible, precise and interpretable task specification and generation. Our goal is to show how the framework of cRL contributes to improving zero-shot generalization in RL through meaningful benchmarks and structured reasoning about generalization tasks. We confirm the insight that optimal behavior in cRL requires context information, as in other related areas of partial observability. To empirically validate this in the cRL framework, we provide various context-extended versions of common RL environments. They are part of the first benchmark library, CARL, designed for generalization based on cRL extensions of popular benchmarks, which we propose as a testbed to further study general agents. We show that in the contextual setting, even simple RL environments become challenging - and that naive solutions are not enough to generalize across complex context spaces.Comment: arXiv admin note: substantial text overlap with arXiv:2110.0210

Expression of p89c-Mybex9b, an alternatively spliced form of c-Myb, is required for proliferation and survival of p210BCR/ABL-expressing cells

The c-Myb gene encodes the p75c-Myb isoform and less-abundant proteins generated by alternatively spliced transcripts. Among these, the best known is pc-Mybex9b, which contains 121 additional amino acids between exon 9 and 10, in a domain involved in protein–protein interactions and negative regulation. In hematopoietic cells, expression of pc-Mybex9b accounts for 10–15% of total c-Myb; these levels may be biologically relevant because modest changes in c-Myb expression affects proliferation and survival of leukemic cells and lineage choice and frequency of normal hematopoietic progenitors. In this study, we assessed biochemical activities of pc-Mybex9b and the consequences of perturbing its expression in K562 and primary chronic myeloid leukemia (CML) progenitor cells. Compared with p75c-Myb, pc-Mybex9b is more stable and more effective in transactivating Myb-regulated promoters. Ectopic expression of pc-Mybex9b enhanced proliferation and colony formation and reduced imatinib (IM) sensitivity of K562 cells; conversely, specific downregulation of pc-Mybex9b reduced proliferation and colony formation, enhanced IM sensitivity of K562 cells and markedly suppressed colony formation of CML CD34+ cells, without affecting the levels of p75c-Myb. Together, these studies indicate that expression of the low-abundance pc-Mybex9b isoform has an important role for the overall biological effects of c-Myb in BCR/ABL-transformed cells

Dramatic Repositioning of c-Myb to Different Promoters during the Cell Cycle Observed by Combining Cell Sorting with Chromatin Immunoprecipitation

The c-Myb transcription factor is a critical regulator of proliferation and stem cell differentiation, and mutated alleles of c-Myb are oncogenic, but little is known about changes in c-Myb activity during the cell cycle. To map the association of c-Myb with specific target genes during the cell cycle, we developed a novel Fix-Sort-ChIP approach, in which asynchronously growing cells were fixed with formaldehyde, stained with Hoechst 33342 and separated into different cell cycle fractions by flow sorting, then processed for chromatin immunoprecipitation (ChIP) assays. We found that c-Myb actively repositions, binding to some genes only in specific cell cycle phases. In addition, the specificity of c-Myb is dramatically different in small subpopulations of cells, for example cells in the G2/M phase of the cell cycle, than in the bulk population. The repositioning of c-Myb during the cell cycle is not due to changes in its expression and also occurs with ectopically expressed, epitope-tagged versions of c-Myb. The repositioning occurs in established cell lines, in primary human CD34+ hematopoietic progenitors and in primary human acute myeloid leukemia cells. The combination of fixation, sorting and ChIP analysis sheds new light on the dynamic nature of gene regulation during the cell cycle and provides a new type of tool for the analysis of gene regulation in small subsets of cells, such as cells in a specific phase of the cell cycle

GSK-3β Is Required for Memory Reconsolidation in Adult Brain

Activation of GSK-3β is presumed to be involved in various neurodegenerative diseases, including Alzheimer's disease (AD), which is characterized by memory disturbances during early stages of the disease. The normal function of GSK-3β in adult brain is not well understood. Here, we analyzed the ability of heterozygote GSK-3β knockout (GSK+/−) mice to form memories. In the Morris water maze (MWM), learning and memory performance of GSK+/− mice was no different from that of wild-type (WT) mice for the first 3 days of training. With continued learning on subsequent days, however, retrograde amnesia was induced in GSK+/− mice, suggesting that GSK+/− mice might be impaired in their ability to form long-term memories. In contextual fear conditioning (CFC), context memory was normally consolidated in GSK+/− mice, but once the original memory was reactivated, they showed reduced freezing, suggesting that GSK+/− mice had impaired memory reconsolidation. Biochemical analysis showed that GSK-3β was activated after memory reactivation in WT mice. Intraperitoneal injection of a GSK-3 inhibitor before memory reactivation impaired memory reconsolidation in WT mice. These results suggest that memory reconsolidation requires activation of GSK-3β in the adult brain