Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation

We propose a new model, independent linear Markov game, for multi-agent reinforcement learning with a large state space and a large number of agents. This is a class of Markov games with independent linear function approximation, where each agent has its own function approximation for the state-action value functions that are marginalized by other players' policies. We design new algorithms for learning the Markov coarse correlated equilibria (CCE) and Markov correlated equilibria (CE) with sample complexity bounds that only scale polynomially with each agent's own function class complexity, thus breaking the curse of multiagents. In contrast, existing works for Markov games with function approximation have sample complexity bounds scale with the size of the \emph{joint action space} when specialized to the canonical tabular Markov game setting, which is exponentially large in the number of agents. Our algorithms rely on two key technical innovations: (1) utilizing policy replay to tackle non-stationarity incurred by multiple agents and the use of function approximation; (2) separating learning Markov equilibria and exploration in the Markov games, which allows us to use the full-information no-regret learning oracle instead of the stronger bandit-feedback no-regret learning oracle used in the tabular setting. Furthermore, we propose an iterative-best-response type algorithm that can learn pure Markov Nash equilibria in independent linear Markov potential games. In the tabular case, by adapting the policy replay mechanism for independent linear Markov games, we propose an algorithm with $\widetilde{O}(\epsilon^{-2})$ sample complexity to learn Markov CCE, which improves the state-of-the-art result $\widetilde{O}(\epsilon^{-3})$ in Daskalakis et al. 2022, where $\epsilon$ is the desired accuracy, and also significantly improves other problem parameters.Comment: 51 pages. Update: Accepted for presentation at the Conference on Learning Theory (COLT) 202

Influence and Optimization of Packet Loss on the Internet-Based Geographically Distributed Test Platform for Fuel Cell Electric Vehicle Powertrain Systems

In view of recent developments in fuel cell electric vehicle powertrain systems, Internet-based geographically distributed test platforms for fuel cell electric vehicle powertrain systems become a development and validation trend. Due to the involvement of remote connection and the Internet, simulation with connected models can suffer great uncertainty because of packet loss. Such a test platform, including packet loss characteristics, was built using MATLAB/Simulink for use in this paper. The simulation analysis results show that packet loss affects the stability of the whole test system. The impact on vehicle speed is mainly concentrated in the later stage of simulation. Aiming at reducing the effect of packet loss caused by Internet, a robust model predictive compensator was designed. Under this compensator, the stability of the system is greatly improved compared to the system without a compensator

Instrumental support in the physical activity community - premilinary results

Currently, we witness the growth of ICT-mediated solutions for chronic diseases management, especially to assist and support patients in lifestyle changes in order to improve their health condition. Being physically active is one the recommended lifestyle changes for patients with chronic diseases. The challenge within those ICT-mediated solutions for physical activity support is to allow patients to manage themselves their physical activity level (PAL) and provide them with the needed social support. One of those solutions available is the use of Virtual Community (VC)

Was Kaposi’s sarcoma-associated herpesvirus introduced into China via the ancient Silk Road? An evolutionary perspective

Kaposi’s sarcoma-associated herpesvirus (KSHV) has become widely dispersed worldwide since it was first reported in 1994, but the seroprevalence of KSHV varies geographically. KSHV is relatively ubiquitous in Mediterranean areas and the Xinjiang Uygur Autonomous Region, China. The origin of KSHV has long been puzzling. In the present study, we collected and analysed 154 KSHV ORF-K1 sequences obtained from samples originating from Xinjiang, Italy, Greece, Iran and southern Siberia using Bayesian evolutionary analysis in BEAST to test the hypothesis that KSHV was introduced into Xinjiang via the ancient Silk Road. According to the phylogenetic analysis, 72 sequences were subtype A and 82 subtype C, with C2 (n = 56) being the predominant subtype. The times to the most recent common ancestors (tMRCAs) of KSHV were 29,872 years (95% highest probability density [HPD], 26,851–32,760 years) for all analysed sequences and 2037 years (95% HPD, 1843–2229 years) for Xinjiang sequences in particular. The tMRCA of Xinjiang KSHV was exactly matched with the time period of the ancient Silk Road approximately two thousand years ago. This route began in Chang’an, the capital of the Han dynasty of China, and crossed Central Asia, ending in the Roman Empire. The evolution rate of KSHV was slow, with 3.44 × 10−6 substitutions per site per year (95% HPD, 2.26 × 10−6 to 4.71 × 10−6), although 11 codons were discovered to be under positive selection pressure. The geographic distances from Italy to Iran and Xinjiang are more than 4000 and 7000 kilometres, respectively, but no explicit relationship between genetic distance and geographic distance was detected

Laser-scribed graphene for sensors: preparation, modification, applications, and future prospects

Sensors are widely used to acquire biological and environmental information for medical diagnosis, and health and environmental monitoring. Graphene is a promising new sensor material that has been widely used in sensor fabrication in recent years. Compared with many other existing graphene preparation methods, laser-scribed graphene (LSG) is simple, low-cost, environmentally friendly, and has good conductivity and high thermal stability, making it widely used in the sensor field. This paper summarizes existing LSG methods for sensor fabrication. Primary LSG preparation methods and their variants are introduced first, followed by a summary of LSG modification methods designed explicitly for sensor fabrication. Subsequently, the applications of LSG in stress, bio, gas, temperature, and humidity sensors are summarized with a particular focus on multifunctional integrated sensors. Finally, the current challenges and prospects of LSG-based sensors are discussed

Oropharyngeal Muscle Exercise Therapy Improves Signs and Symptoms of Post-stroke Moderate Obstructive Sleep Apnea Syndrome

The primary aim of the current study was to assess the effects of oropharingeal muscle exercises in obstruction severity on stroke patients with OSAS. The secondary aims were to evaluate the effects of the exercises on rehabilitation of neurological function, sleeping, and morphology change of upper airway. An open-label, single-blind, parallel-group, randomized, controlled trial was designed. Fifty post-stroke patients with moderate OSAS were randomly assigned into 2 groups (25 in each group). For the therapy group, oropharyngeal muscle exercise was performed during the daytime for 20 min, twice a day, for 6 weeks. The control group was subjected to sham therapy of deep breathing. Primary outcomes were the obstruction severity by polysomnography. Secondary outcomes included recovery of motor and neurocognitive function, personal activities of daily living assessment (ADL), sleep quality and sleepiness scale. It also included upper airway magnetic resonance imaging (MRI) measurements. Assessments were made at baseline and after 6-week exercise. Finally, 49 patients completed the study. The apnea–hypopnea index, snore index, arousal index, and minimum oxygen saturation improved after exercise (P < 0.05). Oropharyngeal muscle exercises improved subjective measurements of sleep quality (P = 0.017), daily sleepiness (P = 0.005), and performance (both P < 0.05) except for neurocognition (P = 0.741). The changes in obstruction improvement, sleep characteristics and performance scale were also associated with training time, as detected by Pearson's correlation analysis. The anatomic structural remodeling of the pharyngeal airway was measured using MRI, including the lager retropalatal distance (P = 0.018) and shorter length of soft palate (P = 0.044) compared with the baseline. Hence, oropharyngeal muscle exercise is a promising alternative treatment strategy for stroke patients with moderate OSAS.Clinical Trial Registration:http://www.chictr.org.cn. Unique identifier: ChiCTR-IPR-1600997