DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment

Large language models encode a vast amount of semantic knowledge and possess remarkable understanding and reasoning capabilities. Previous research has explored how to ground language models in robotic tasks to ensure that the sequences generated by the language model are both logically correct and practically executable. However, low-level execution may deviate from the high-level plan due to environmental perturbations or imperfect controller design. In this paper, we propose DoReMi, a novel language model grounding framework that enables immediate Detection and Recovery from Misalignments between plan and execution. Specifically, LLMs are leveraged for both planning and generating constraints for planned steps. These constraints can indicate plan-execution misalignments and we use a vision question answering (VQA) model to check constraints during low-level skill execution. If certain misalignment occurs, our method will call the language model to re-plan in order to recover from misalignments. Experiments on various complex tasks including robot arms and humanoid robots demonstrate that our method can lead to higher task success rates and shorter task completion times. Videos of DoReMi are available at https://sites.google.com/view/doremi-paper.Comment: 21 pages, 13 figure

Big-video mining of road appearances in full spectrums of weather and illuminations

Autonomous and safety driving require the control of vehicles within roads. Compared to lane mark tracking, road edge detection is more difficult because of the large variation in road and off-road materials and the influence from weather and illuminations. This work investigates visual appearances of roads under a spectrum of weather conditions. We use big-data mining on large scale naturalistic driving videos taken over a year through four seasons. Large video volumes are condensed to compact road profile images for analysis. Clusters are extracted from all samples with unsupervised learning. Typical views of a spectrum of weather/illuminations are generated from the clusters. Further, by changing the number of clusters we find a stable number for clustering. The learned data are used to classify driving videos into typical illumination types briefly. The surveyed data can also be used in the development of road edge detection algorithm and system as well as their testing

All weather road edge identification based on driving video mining

To avoid vehicle running off road, road edge detection is a fundamental function. Current work on road edge detection has not exhaustively tackled all weather and illumination conditions. We first sort the visual appearance of roads based on physical and optical properties under various illuminations. Then, data mining approach is applied to a large driving video set that contains the full spectrum of seasons and weathers to learn the statistical distribution of road edge appearances. The obtained parameters of road environment in color on road structure are used to classify weather in video briefly, and the corresponding algorithm and features are applied for robust road edge detection. To visualize the road appearance as well as evaluate the accuracy of detected road, a compact road profile image is generated to reduce the data to a small fraction of video. Through the exhaustive examination of all weather and illuminations, our road detection methods can locate road edges in good weather, reduce errors in dark illuminations, and report road invisibility in poor illuminations

Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning

Self-supervised audio-visual source localization aims to locate sound-source objects in video frames without extra annotations. Recent methods often approach this goal with the help of contrastive learning, which assumes only the audio and visual contents from the same video are positive samples for each other. However, this assumption would suffer from false negative samples in real-world training. For example, for an audio sample, treating the frames from the same audio class as negative samples may mislead the model and therefore harm the learned representations e.g., the audio of a siren wailing may reasonably correspond to the ambulances in multiple images). Based on this observation, we propose a new learning strategy named False Negative Aware Contrastive (FNAC) to mitigate the problem of misleading the training with such false negative samples. Specifically, we utilize the intra-modal similarities to identify potentially similar samples and construct corresponding adjacency matrices to guide contrastive learning. Further, we propose to strengthen the role of true negative samples by explicitly leveraging the visual features of sound sources to facilitate the differentiation of authentic sounding source regions. FNAC achieves state-of-the-art performances on Flickr-SoundNet, VGG-Sound, and AVSBench, which demonstrates the effectiveness of our method in mitigating the false negative issue. The code is available at \url{https://github.com/OpenNLPLab/FNAC_AVL}.Comment: CVPR202

Ligand-binding properties of XaffOBP9, a Minus-C odorant-binding protein from Xyleborus affinis (Coleoptera: Curculionidae: Scolytinae)

Xyleborus affinis, one of the most important pests of rubber trees, has caused severe damage to the natural rubber industry in Hainan province. The ability to detect host plants through a sensitive and specific olfactory system is crucial for Xyleborus affinis. Odorant binding proteins (OBPs) are believed to bind and carry hydrophobic active compounds from the environment to the surface of olfactory receptor neurons. To investigate the potential functional role of the highly expressed XaffOBP9 in binding with semiochemicals, we cloned and analyzed the cDNA sequence of XaffOBP9. The results showed that XaffOBP9 contains a 411bp open reading frame that encodes 136 amino acids. Then XaffOBP9 was expressed in Escherichia coli. The binding affinity of the recombinant OBP to 15 different ligands (14 host plant volatiles and 1 aggregation pheromone) was then examined using a fluorescence competitive binding approach. The results demonstrated that XaffOBP9 exhibited broad binding capabilities and strong affinities for 14 ligands. The structure of XaffOBP9 and its interactions with fourteen ligands were further analyzed by modeling and molecular docking, respectively. Based on the docking result, we found hydrophobic interactions are important between XaffOBP9 to these ligands and three amino acid residues (L71, Y106, and L114) were highly overlapped and contributed to the interaction with ligands. Mutation functional assays confirmed that the mutant L114A showed significantly reduced binding capacity to these ligands. This study suggested that XaffOBP9 may be involved in the chemoreception of semiochemicals and that it is helpful for the integrated management of X. affinis

Surface functionalization of Polymers of Intrinsic Microporosity (PIMs) membrane by polyphenol for efficient CO2 separation

Membrane separation technology offers a green, efficient and energy-saving approach for biogas upgrading. Membranes with high selectivity and high permeability are the key to achieve high performance. Polymers of Intrinsic Microporosity (PIMs) materials have shown excellent gas permeability but low selectivity which limits their practical application. Herein, a polyphenol, tannic acid, was coated on the PIM-1 membrane surface by a facile dipping method to fabricate composite membranes. Tannic acid containing a large number of polar oxygen-containing groups (quinone, phenolic hydroxyl) self-polymerized on the membrane surface to form a CO2-philic, defect-free and thin layer. The CO2/CH4 selectivity of the resultant composite membranes was increased after tannic acid coating while the permeability remained comparable to or even higher than pristine PIM-1 membrane, exceeding the reported 2008 upper bound

Identification of Novel Covalent XPO1 Inhibitors Based on a Hybrid Virtual Screening Strategy

Nuclear export protein 1 (XPO1), a member of the nuclear export protein-p (Karyopherin-P) superfamily, regulates the transport of “cargo” proteins. To facilitate this important process, which is essential for cellular homeostasis, XPO1 must first recognize and bind the cargo proteins. To inhibit this process, small molecule inhibitors have been designed that inhibit XPO1 activity through covalent binding. However, the scaffolds for these inhibitors are very limited. While virtual screening may be used to expand the diversity of the XPO1 inhibitor skeleton, enormous computational resources would be required to accomplish this using traditional screening methods. In the present study, we report the development of a hybrid virtual screening workflow and its application in XPO1 covalent inhibitor screening. After screening, several promising XPO1 covalent molecules were obtained. Of these, compound 8 performed well in both tumor cell proliferation assays and a nuclear export inhibition assay. In addition, molecular dynamics simulations were performed to provide information on the mode of interaction of compound 8 with XPO1. This research has identified a promising new scaffold for XPO1 inhibitors, and it demonstrates an effective and resource-saving workflow for identifying new covalent inhibitors