Maximum Entropy Heterogeneous-Agent Mirror Learning

Multi-agent reinforcement learning (MARL) has been shown effective for cooperative games in recent years. However, existing state-of-the-art methods face challenges related to sample inefficiency, brittleness regarding hyperparameters, and the risk of converging to a suboptimal Nash Equilibrium. To resolve these issues, in this paper, we propose a novel theoretical framework, named Maximum Entropy Heterogeneous-Agent Mirror Learning (MEHAML), that leverages the maximum entropy principle to design maximum entropy MARL actor-critic algorithms. We prove that algorithms derived from the MEHAML framework enjoy the desired properties of the monotonic improvement of the joint maximum entropy objective and the convergence to quantal response equilibrium (QRE). The practicality of MEHAML is demonstrated by developing a MEHAML extension of the widely used RL algorithm, HASAC (for soft actor-critic), which shows significant improvements in exploration and robustness on three challenging benchmarks: Multi-Agent MuJoCo, StarCraftII, and Google Research Football. Our results show that HASAC outperforms strong baseline methods such as HATD3, HAPPO, QMIX, and MAPPO, thereby establishing the new state of the art. See our project page at https://sites.google.com/view/mehaml

Hawkeye: Change-targeted Testing for Android Apps based on Deep Reinforcement Learning

Android Apps are frequently updated to keep up with changing user, hardware, and business demands. Ensuring the correctness of App updates through extensive testing is crucial to avoid potential bugs reaching the end user. Existing Android testing tools generate GUI events focussing on improving the test coverage of the entire App rather than prioritising updates and its impacted elements. Recent research has proposed change-focused testing but relies on random exploration to exercise the updates and impacted GUI elements that is ineffective and slow for large complex Apps with a huge input exploration space. We propose directed testing of App updates with Hawkeye that is able to prioritise executing GUI actions associated with code changes based on deep reinforcement learning from historical exploration data. Our empirical evaluation compares Hawkeye with state-of-the-art model-based and reinforcement learning-based testing tools FastBot2 and ARES using 10 popular open-source and 1 commercial App. We find that Hawkeye is able to generate GUI event sequences targeting changed functions more reliably than FastBot2 and ARES for the open source Apps and the large commercial App. Hawkeye achieves comparable performance on smaller open source Apps with a more tractable exploration space. The industrial deployment of Hawkeye in the development pipeline also shows that Hawkeye is ideal to perform smoke testing for merge requests of a complicated commercial App

Microtubule-mediated regulation of β2AR translation and unction in failing hearts

Background: Beta-1 adrenergic receptor (β 1 AR)- and Beta-2 adrenergic receptor (β 2 AR)-mediated cyclic adenosine monophosphate signaling has distinct effects on cardiac function and heart failure progression. However, the mechanism regulating spatial localization and functional compartmentation of cardiac β-ARs remains elusive. Emerging evidence suggests that microtubule-dependent trafficking of mRNP (messenger ribonucleoprotein) and localized protein translation modulates protein compartmentation in cardiomyocytes. We hypothesized that β-AR compartmentation in cardiomyocytes is accomplished by selective trafficking of its mRNAs and localized translation. Methods: The localization pattern of β-AR mRNA was investigated using single molecule fluorescence in situ hybridization and subcellular nanobiopsy in rat cardiomyocytes. The role of microtubule on β-AR mRNA localization was studied using vinblastine, and its effect on receptor localization and function was evaluated with immunofluorescent and high-throughput Förster resonance energy transfer microscopy. An mRNA protein co-detection assay identified plausible β-AR translation sites in cardiomyocytes. The mechanism by which β-AR mRNA is redistributed post–heart failure was elucidated by single molecule fluorescence in situ hybridization, nanobiopsy, and high-throughput Förster resonance energy transfer microscopy on 16 weeks post–myocardial infarction and detubulated cardiomyocytes. Results: β 1 AR and β 2 AR mRNAs show differential localization in cardiomyocytes, with β 1 AR found in the perinuclear region and β 2 AR showing diffuse distribution throughout the cell. Disruption of microtubules induces a shift of β 2 AR transcripts toward the perinuclear region. The close proximity between β 2 AR transcripts and translated proteins suggests that the translation process occurs in specialized, precisely defined cellular compartments. Redistribution of β 2 AR transcripts is microtubule-dependent, as microtubule depolymerization markedly reduces the number of functional receptors on the membrane. In failing hearts, both β 1 AR and β 2 AR mRNAs are redistributed toward the cell periphery, similar to what is seen in cardiomyocytes undergoing drug-induced detubulation. This suggests that t-tubule remodeling contributes to β-AR mRNA redistribution and impaired β 2 AR function in failing hearts. Conclusions: Asymmetrical microtubule-dependent trafficking dictates differential β 1 AR and β 2 AR localization in healthy cardiomyocyte microtubules, underlying the distinctive compartmentation of the 2 β-ARs on the plasma membrane. The localization pattern is altered post–myocardial infarction, resulting from t-tubule remodeling, leading to distorted β 2 AR-mediated cyclic adenosine monophosphate signaling