LEMMA: Learning Language-Conditioned Multi-Robot Manipulation

Complex manipulation tasks often require robots with complementary capabilities to collaborate. We introduce a benchmark for LanguagE-Conditioned Multi-robot MAnipulation (LEMMA) focused on task allocation and long-horizon object manipulation based on human language instructions in a tabletop setting. LEMMA features 8 types of procedurally generated tasks with varying degree of complexity, some of which require the robots to use tools and pass tools to each other. For each task, we provide 800 expert demonstrations and human instructions for training and evaluations. LEMMA poses greater challenges compared to existing benchmarks, as it requires the system to identify each manipulator's limitations and assign sub-tasks accordingly while also handling strong temporal dependencies in each task. To address these challenges, we propose a modular hierarchical planning approach as a baseline. Our results highlight the potential of LEMMA for developing future language-conditioned multi-robot systems.Comment: 8 pages, 3 figure

OpenD: A Benchmark for Language-Driven Door and Drawer Opening

We introduce OPEND, a benchmark for learning how to use a hand to open cabinet doors or drawers in a photo-realistic and physics-reliable simulation environment driven by language instruction. To solve the task, we propose a multi-step planner composed of a deep neural network and rule-base controllers. The network is utilized to capture spatial relationships from images and understand semantic meaning from language instructions. Controllers efficiently execute the plan based on the spatial and semantic understanding. We evaluate our system by measuring its zero-shot performance in test data set. Experimental results demonstrate the effectiveness of decision planning by our multi-step planner for different hands, while suggesting that there is significant room for developing better models to address the challenge brought by language understanding, spatial reasoning, and long-term manipulation. We will release OPEND and host challenges to promote future research in this area

Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations

Offline reinforcement learning (RL) methods strike a balance between exploration and exploitation by conservative value estimation -- penalizing values of unseen states and actions. Model-free methods penalize values at all unseen actions, while model-based methods are able to further exploit unseen states via model rollouts. However, such methods are handicapped in their ability to find unseen states far away from the available offline data due to two factors -- (a) very short rollout horizons in models due to cascading model errors, and (b) model rollouts originating solely from states observed in offline data. We relax the second assumption and present a novel unseen state augmentation strategy to allow exploitation of unseen states where the learned model and value estimates generalize. Our strategy finds unseen states by value-informed perturbations of seen states followed by filtering out states with epistemic uncertainty estimates too high (high error) or too low (too similar to seen data). We observe improved performance in several offline RL tasks and find that our augmentation strategy consistently leads to overall lower average dataset Q-value estimates i.e. more conservative Q-value estimates than a baseline

Embodied BERT:A Transformer Model for Embodied, Language-guided Visual Task Completion

Language-guided robots performing home and office tasks must navigate in and interact with the world. Grounding language instructions against visual observations and actions to take in an environment is an open challenge. We present Embodied BERT (EmBERT), a transformer-based model which can attend to high-dimensional, multi-modal inputs across long temporal horizons for language-conditioned task completion. Additionally, we bridge the gap between successful object-centric navigation models used for non-interactive agents and the language-guided visual task completion benchmark, ALFRED, by introducing object navigation targets for EmBERT training. We achieve competitive performance on the ALFRED benchmark, and EmBERT marks the first transformer-based model to successfully handle the long-horizon, dense, multi-modal histories of ALFRED, and the first ALFRED model to utilize object-centric navigation targets.Comment: Accepted at Novel Ideas in Learning-to-Learn through Interaction (NILLI) workshop @ EMNLP 202

Hippo pathway-manipulating neutrophil-mimic hybrid nanoparticles for cardiac ischemic injury via modulation of local immunity and cardiac regeneration

The promise of regeneration therapy for restoration of damaged myocardium after cardiac ischemic injury relies on targeted delivery of proliferative molecules into cardiomyocytes whose healing benefits are still limited owing to severe immune microenvironment due to local high concentration of proinflammatory cytokines. Optimal therapeutic strategies are therefore in urgent need to both modulate local immunity and deliver proliferative molecules. Here, we addressed this unmet need by developing neutrophil-mimic nanoparticles NM@miR, fabricated by coating hybrid neutrophil membranes with artificial lipids onto mesoporous silica nanoparticles (MSNs) loaded with microRNA-10b. The hybrid membrane could endow nanoparticles with strong capacity to migrate into inflammatory sites and neutralize proinflammatory cytokines and increase the delivery efficiency of microRNA-10b into adult mammalian cardiomyocytes (CMs) by fusing with cell membranes and leading to the release of MSNs-miR into cytosol. Upon NM@miR administration, this nanoparticle could home to the injured myocardium, restore the local immunity, and efficiently deliver microRNA-10b to cardiomyocytes, which could reduce the activation of Hippo-YAP pathway mediated by excessive cytokines and exert the best proliferative effect of miR-10b. This combination therapy could finally improve cardiac function and mitigate ventricular remodeling. Consequently, this work offers a combination strategy of immunity modulation and proliferative molecule delivery to boost cardiac regeneration after injury