18 research outputs found
Modelling Behavioural Diversity for Learning in Open-Ended Games
Promoting behavioural diversity is critical for solving games with
non-transitive dynamics where strategic cycles exist, and there is no
consistent winner (e.g., Rock-Paper-Scissors). Yet, there is a lack of rigorous
treatment for defining diversity and constructing diversity-aware learning
dynamics. In this work, we offer a geometric interpretation of behavioural
diversity in games and introduce a novel diversity metric based on
determinantal point processes (DPP). By incorporating the diversity metric into
best-response dynamics, we develop diverse fictitious play and diverse
policy-space response oracle for solving normal-form games and open-ended
games. We prove the uniqueness of the diverse best response and the convergence
of our algorithms on two-player games. Importantly, we show that maximising the
DPP-based diversity metric guarantees to enlarge the gamescape -- convex
polytopes spanned by agents' mixtures of strategies. To validate our
diversity-aware solvers, we test on tens of games that show strong
non-transitivity. Results suggest that our methods achieve at least the same,
and in most games, lower exploitability than PSRO solvers by finding effective
and diverse strategies.Comment: corresponds to <[email protected]
Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Models
Large language models (LLMs) demonstrate their promise in tackling
complicated practical challenges by combining action-based policies with chain
of thought (CoT) reasoning. Having high-quality prompts on hand, however, is
vital to the framework's effectiveness. Currently, these prompts are
handcrafted utilizing extensive human labor, resulting in CoT policies that
frequently fail to generalize. Human intervention is also required in order to
develop grounding functions that ensure low-level controllers appropriately
process CoT reasoning. In this paper, we take the first step towards a fully
integrated end-to-end framework for task-solving in real settings employing
complicated reasoning. To that purpose, we offer a new leader-follower bilevel
framework capable of learning to ask relevant questions (prompts) and
subsequently undertaking reasoning to guide the learning of actions to be
performed in an environment. A good prompt should make introspective revisions
based on historical findings, leading the CoT to consider the anticipated
goals. A prompt-generator policy has its own aim in our system, allowing it to
adapt to the action policy and automatically root the CoT process towards
outputs that lead to decisive, high-performing actions. Meanwhile, the action
policy is learning how to use the CoT outputs to take specific actions. Our
empirical data reveal that our system outperforms leading methods in agent
learning benchmarks such as Overcooked and FourRoom
Online Double Oracle
Solving strategic games with huge action space is a critical yet
under-explored topic in economics, operations research and artificial
intelligence. This paper proposes new learning algorithms for solving
two-player zero-sum normal-form games where the number of pure strategies is
prohibitively large. Specifically, we combine no-regret analysis from online
learning with Double Oracle (DO) methods from game theory. Our method --
\emph{Online Double Oracle (ODO)} -- is provably convergent to a Nash
equilibrium (NE). Most importantly, unlike normal DO methods, ODO is
\emph{rationale} in the sense that each agent in ODO can exploit strategic
adversary with a regret bound of where is
not the total number of pure strategies, but rather the size of \emph{effective
strategy set} that is linearly dependent on the support size of the NE. On tens
of different real-world games, ODO outperforms DO, PSRO methods, and no-regret
algorithms such as Multiplicative Weight Update by a significant margin, both
in terms of convergence rate to a NE and average payoff against strategic
adversaries.Comment: [email protected]
LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learning
Efficient exploration is important for reinforcement learners to achieve high
rewards. In multi-agent systems, coordinated exploration and behaviour is
critical for agents to jointly achieve optimal outcomes. In this paper, we
introduce a new general framework for improving coordination and performance of
multi-agent reinforcement learners (MARL). Our framework, named Learnable
Intrinsic-Reward Generation Selection algorithm (LIGS) introduces an adaptive
learner, Generator that observes the agents and learns to construct intrinsic
rewards online that coordinate the agents' joint exploration and joint
behaviour. Using a novel combination of MARL and switching controls, LIGS
determines the best states to learn to add intrinsic rewards which leads to a
highly efficient learning process. LIGS can subdivide complex tasks making them
easier to solve and enables systems of MARL agents to quickly solve
environments with sparse rewards. LIGS can seamlessly adopt existing MARL
algorithms and, our theory shows that it ensures convergence to policies that
deliver higher system performance. We demonstrate its superior performance in
challenging tasks in Foraging and StarCraft II.Comment: arXiv admin note: text overlap with arXiv:2103.0915
Recommended from our members
Effect of Hydrocortisone on Mortality and Organ Support in Patients With Severe COVID-19: The REMAP-CAP COVID-19 Corticosteroid Domain Randomized Clinical Trial.
Importance: Evidence regarding corticosteroid use for severe coronavirus disease 2019 (COVID-19) is limited. Objective: To determine whether hydrocortisone improves outcome for patients with severe COVID-19. Design, Setting, and Participants: An ongoing adaptive platform trial testing multiple interventions within multiple therapeutic domains, for example, antiviral agents, corticosteroids, or immunoglobulin. Between March 9 and June 17, 2020, 614 adult patients with suspected or confirmed COVID-19 were enrolled and randomized within at least 1 domain following admission to an intensive care unit (ICU) for respiratory or cardiovascular organ support at 121 sites in 8 countries. Of these, 403 were randomized to open-label interventions within the corticosteroid domain. The domain was halted after results from another trial were released. Follow-up ended August 12, 2020. Interventions: The corticosteroid domain randomized participants to a fixed 7-day course of intravenous hydrocortisone (50 mg or 100 mg every 6 hours) (n = 143), a shock-dependent course (50 mg every 6 hours when shock was clinically evident) (n = 152), or no hydrocortisone (n = 108). Main Outcomes and Measures: The primary end point was organ support-free days (days alive and free of ICU-based respiratory or cardiovascular support) within 21 days, where patients who died were assigned -1 day. The primary analysis was a bayesian cumulative logistic model that included all patients enrolled with severe COVID-19, adjusting for age, sex, site, region, time, assignment to interventions within other domains, and domain and intervention eligibility. Superiority was defined as the posterior probability of an odds ratio greater than 1 (threshold for trial conclusion of superiority >99%). Results: After excluding 19 participants who withdrew consent, there were 384 patients (mean age, 60 years; 29% female) randomized to the fixed-dose (n = 137), shock-dependent (n = 146), and no (n = 101) hydrocortisone groups; 379 (99%) completed the study and were included in the analysis. The mean age for the 3 groups ranged between 59.5 and 60.4 years; most patients were male (range, 70.6%-71.5%); mean body mass index ranged between 29.7 and 30.9; and patients receiving mechanical ventilation ranged between 50.0% and 63.5%. For the fixed-dose, shock-dependent, and no hydrocortisone groups, respectively, the median organ support-free days were 0 (IQR, -1 to 15), 0 (IQR, -1 to 13), and 0 (-1 to 11) days (composed of 30%, 26%, and 33% mortality rates and 11.5, 9.5, and 6 median organ support-free days among survivors). The median adjusted odds ratio and bayesian probability of superiority were 1.43 (95% credible interval, 0.91-2.27) and 93% for fixed-dose hydrocortisone, respectively, and were 1.22 (95% credible interval, 0.76-1.94) and 80% for shock-dependent hydrocortisone compared with no hydrocortisone. Serious adverse events were reported in 4 (3%), 5 (3%), and 1 (1%) patients in the fixed-dose, shock-dependent, and no hydrocortisone groups, respectively. Conclusions and Relevance: Among patients with severe COVID-19, treatment with a 7-day fixed-dose course of hydrocortisone or shock-dependent dosing of hydrocortisone, compared with no hydrocortisone, resulted in 93% and 80% probabilities of superiority with regard to the odds of improvement in organ support-free days within 21 days. However, the trial was stopped early and no treatment strategy met prespecified criteria for statistical superiority, precluding definitive conclusions. Trial Registration: ClinicalTrials.gov Identifier: NCT02735707
Effect of angiotensin-converting enzyme inhibitor and angiotensin receptor blocker initiation on organ support-free days in patients hospitalized with COVID-19
IMPORTANCE Overactivation of the renin-angiotensin system (RAS) may contribute to poor clinical outcomes in patients with COVID-19.
Objective To determine whether angiotensin-converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) initiation improves outcomes in patients hospitalized for COVID-19.
DESIGN, SETTING, AND PARTICIPANTS In an ongoing, adaptive platform randomized clinical trial, 721 critically ill and 58 non–critically ill hospitalized adults were randomized to receive an RAS inhibitor or control between March 16, 2021, and February 25, 2022, at 69 sites in 7 countries (final follow-up on June 1, 2022).
INTERVENTIONS Patients were randomized to receive open-label initiation of an ACE inhibitor (n = 257), ARB (n = 248), ARB in combination with DMX-200 (a chemokine receptor-2 inhibitor; n = 10), or no RAS inhibitor (control; n = 264) for up to 10 days.
MAIN OUTCOMES AND MEASURES The primary outcome was organ support–free days, a composite of hospital survival and days alive without cardiovascular or respiratory organ support through 21 days. The primary analysis was a bayesian cumulative logistic model. Odds ratios (ORs) greater than 1 represent improved outcomes.
RESULTS On February 25, 2022, enrollment was discontinued due to safety concerns. Among 679 critically ill patients with available primary outcome data, the median age was 56 years and 239 participants (35.2%) were women. Median (IQR) organ support–free days among critically ill patients was 10 (–1 to 16) in the ACE inhibitor group (n = 231), 8 (–1 to 17) in the ARB group (n = 217), and 12 (0 to 17) in the control group (n = 231) (median adjusted odds ratios of 0.77 [95% bayesian credible interval, 0.58-1.06] for improvement for ACE inhibitor and 0.76 [95% credible interval, 0.56-1.05] for ARB compared with control). The posterior probabilities that ACE inhibitors and ARBs worsened organ support–free days compared with control were 94.9% and 95.4%, respectively. Hospital survival occurred in 166 of 231 critically ill participants (71.9%) in the ACE inhibitor group, 152 of 217 (70.0%) in the ARB group, and 182 of 231 (78.8%) in the control group (posterior probabilities that ACE inhibitor and ARB worsened hospital survival compared with control were 95.3% and 98.1%, respectively).
CONCLUSIONS AND RELEVANCE In this trial, among critically ill adults with COVID-19, initiation of an ACE inhibitor or ARB did not improve, and likely worsened, clinical outcomes.
TRIAL REGISTRATION ClinicalTrials.gov Identifier: NCT0273570
Dimethyl fumarate in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial
Dimethyl fumarate (DMF) inhibits inflammasome-mediated inflammation and has been proposed as a treatment for patients hospitalised with COVID-19. This randomised, controlled, open-label platform trial (Randomised Evaluation of COVID-19 Therapy [RECOVERY]), is assessing multiple treatments in patients hospitalised for COVID-19 (NCT04381936, ISRCTN50189673). In this assessment of DMF performed at 27 UK hospitals, adults were randomly allocated (1:1) to either usual standard of care alone or usual standard of care plus DMF. The primary outcome was clinical status on day 5 measured on a seven-point ordinal scale. Secondary outcomes were time to sustained improvement in clinical status, time to discharge, day 5 peripheral blood oxygenation, day 5 C-reactive protein, and improvement in day 10 clinical status. Between 2 March 2021 and 18 November 2021, 713 patients were enroled in the DMF evaluation, of whom 356 were randomly allocated to receive usual care plus DMF, and 357 to usual care alone. 95% of patients received corticosteroids as part of routine care. There was no evidence of a beneficial effect of DMF on clinical status at day 5 (common odds ratio of unfavourable outcome 1.12; 95% CI 0.86-1.47; p = 0.40). There was no significant effect of DMF on any secondary outcome
Learning Risk-Averse Equilibria in Multi-Agent Systems
In multi-agent systems, intelligent agents are tasked with making decisions
that have optimal outcomes when the actions of the other agents are as
expected, whilst also being prepared for unexpected behaviour. In this work, we
introduce a new risk-averse solution concept that allows the learner to
accommodate unexpected actions by finding the minimum variance strategy given
any level of expected return. We prove the existence of such a risk-averse
equilibrium, and propose one fictitious-play type learning algorithm for
smaller games that enjoys provable convergence guarantees in certain games
classes (e.g., zero-sum or potential). Furthermore, we propose an approximation
method for larger games based on iterative population-based training that
generates a population of risk-averse agents. Empirically, our equilibrium is
shown to be able to reduce the reward variance, specifically in the sense that
off-equilibrium behaviour has a far smaller impact on our risk-averse agents in
comparison to playing other equilibrium solutions. Importantly, we show that
our population of agents that approximate a risk-averse equilibrium is
particularly effective in the presence of unseen opposing populations,
especially in the case of guaranteeing a minimal level of performance which is
critical to safety-aware multi-agent systems
Online Markov decision processes with non-oblivious strategic adversary
We study a novel setting in Online Markov Decision Processes (OMDPs) where
the loss function is chosen by a non-oblivious strategic adversary who follows
a no-external regret algorithm. In this setting, we first demonstrate that
MDP-Expert, an existing algorithm that works well with oblivious adversaries
can still apply and achieve a policy regret bound of where is the size of adversary's pure
strategy set and denotes the size of agent's action space. Considering
real-world games where the support size of a NE is small, we further propose a
new algorithm: MDP-Online Oracle Expert (MDP-OOE), that achieves a policy
regret bound of where
depends only on the support size of the NE. MDP-OOE leverages the key
benefit of Double Oracle in game theory and thus can solve games with
prohibitively large action space. Finally, to better understand the learning
dynamics of no-regret methods, under the same setting of no-external regret
adversary in OMDPs, we introduce an algorithm that achieves last-round
convergence result to a NE. To our best knowledge, this is first work leading
to the last iteration result in OMDPs