SANTA: Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition

Distantly-Supervised Named Entity Recognition effectively alleviates the burden of time-consuming and expensive annotation in the supervised setting. But the context-free matching process and the limited coverage of knowledge bases introduce inaccurate and incomplete annotation noise respectively. Previous studies either considered only incomplete annotation noise or indiscriminately handle two types of noise with the same strategy. In this paper, we argue that the different causes of two types of noise bring up the requirement of different strategies in model architecture. Therefore, we propose the SANTA to handle these two types of noise separately with (1) Memory-smoothed Focal Loss and Entity-aware KNN to relieve the entity ambiguity problem caused by inaccurate annotation, and (2) Boundary Mixup to alleviate decision boundary shifting problem caused by incomplete annotation and a noise-tolerant loss to improve the robustness. Benefiting from our separate tailored strategies, we confirm in the experiment that the two types of noise are well mitigated. SANTA also achieves a new state-of-the-art on five public datasets.Comment: Findings of ACL202

Distantly-Supervised Named Entity Recognition with Uncertainty-aware Teacher Learning and Student-student Collaborative Learning

Distantly-Supervised Named Entity Recognition (DS-NER) effectively alleviates the burden of annotation, but meanwhile suffers from the label noise. Recent works attempt to adopt the teacher-student framework to gradually refine the training labels and improve the overall robustness. However, we argue that these teacher-student methods achieve limited performance because poor network calibration produces incorrectly pseudo-labeled samples, leading to error propagation. Therefore, we attempt to mitigate this issue by proposing: (1) Uncertainty-aware Teacher Learning that leverages the prediction uncertainty to guide the selection of pseudo-labels, avoiding the number of incorrect pseudo-labels in the self-training stage. (2) Student-student Collaborative Learning that allows the transfer of reliable labels between two student networks instead of completely relying on all pseudo-labels from its teacher. Meanwhile, this approach allows a full exploration of mislabeled samples rather than simply filtering unreliable pseudo-labeled samples. Extensive experimental results on five DS-NER datasets demonstrate that our method is superior to state-of-the-art teacher-student methods

Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond

In this study, we explore the potential of Multimodal Large Language Models (MLLMs) in improving embodied decision-making processes for agents. While Large Language Models (LLMs) have been widely used due to their advanced reasoning skills and vast world knowledge, MLLMs like GPT4-Vision offer enhanced visual understanding and reasoning capabilities. We investigate whether state-of-the-art MLLMs can handle embodied decision-making in an end-to-end manner and whether collaborations between LLMs and MLLMs can enhance decision-making. To address these questions, we introduce a new benchmark called PCA-EVAL, which evaluates embodied decision-making from the perspectives of Perception, Cognition, and Action. Additionally, we propose HOLMES, a multi-agent cooperation framework that allows LLMs to leverage MLLMs and APIs to gather multimodal information for informed decision-making. We compare end-to-end embodied decision-making and HOLMES on our benchmark and find that the GPT4-Vision model demonstrates strong end-to-end embodied decision-making abilities, outperforming GPT4-HOLMES in terms of average decision accuracy (+3%). However, this performance is exclusive to the latest GPT4-Vision model, surpassing the open-source state-of-the-art MLLM by 26%. Our results indicate that powerful MLLMs like GPT4-Vision hold promise for decision-making in embodied agents, offering new avenues for MLLM research. Code and data are open at https://github.com/pkunlp-icler/PCA-EVAL/.Comment: FMDM@NeurIPS2023, Code and data: https://github.com/pkunlp-icler/PCA-EVAL

Human-in-the-Loop through Chain-of-Thought

While the emergence of powerful language models along with Chain-of-thought prompting has made automation more and more omnipresent, it sometimes demonstrates its weakness in long-term or multi-step logical reasoning. For example, users don't always get desirable answers for complex mathematical problems without human involvement. Against this background, we present the Manual Correction System (MCS) -- a human-in-the-loop system enhanced by Chain-of-Thought prompting, which explores how manual correction of sub-logics in rationales can improve LLM's reasoning performance. Moving one step forward, considering a system with human-in-the-loop involves more than having humans improve performance but also controlling the cost. Therefore, we post a Cost-utility Analysis Model for Human-in-the-Loop systems (CAMLOP) based on classical economics theory to analyze, quantify and balance the utility and the corresponding cost. We conduct experiments of MCS and CAMLOP with twelve datasets. A significant advantage w.r.t cost and utility proves its superiority over strong baselines

What you saw a while ago determines what you see now: The effect and temporal dynamics of awareness priming on implicit behavior

The perceptual content (e.g., seen a happy vs. seen a sad face) and also subjective visibility (e.g., whether the stimulus is visible or not) of a given (liminal) stimulus is influenced by the history of previously consciously experienced stimuli. This effect on subjective visibility, termed awareness priming, suggests that findings from a large body of literature on unconscious processing might be confounded by conscious awareness. However, those literatures on unconscious processing used implicit behavioral measures of unconscious processing. The challenge is only valid if previous visible stimuli (not just physically salient) also affect implicit behavior, e.g., response priming. Here, we used Continuous Flash Suppression (CFS) to probe the limits and temporal dynamics of awareness priming effect. We showed that prior conscious exposure to two Chinese words increases both visibility and discrimination accuracy, and also improves the response priming of words presented just at the visibility threshold. A correlation analysis revealed that this effect is only driven by the high visibility of the previous stimuli but not high physical saliency. Our results strongly validated the challenge from awareness priming to the literature on unconscious processing. Moreover, we found a different temporal dynamic for how previous visible exposure to a word affects current perception: previous short-term exposure (1-10 back trials) to a visible word only enhances discrimination accuracy of the same word in the current trial, whereas long-term exposure (10-30 back trials) exclusively elevates visibility. This novel finding suggests that areas higher in the processing hierarchy, with larger temporal receptive field, contribute to consciousness, while areas lower in the cortical hierarchy contribute to objective discrimination

Manganese Oxide/Carbon Yolk–Shell Nanorod Anodes for High Capacity Lithium Batteries

Transition metal oxides have attracted much interest for their high energy density in lithium batteries. However, the fast capacity fading and the low power density still limit their practical implementation. In order to overcome these challenges, one-dimensional yolk–shell nanorods have been successfully constructed using manganese oxide as an example through a facile two-step sol–gel coating method. Dopamine and tetraethoxysilane are used as precursors to obtain uniform polymer coating and silica layer followed by converting into carbon shell and hollow space, respectively. As anode material for lithium batteries, the manganese oxide/carbon yolk–shell nanorod electrode has a reversible capacity of 660 mAh/g for initial cycle at 100 mA/g and exhibits excellent cyclability with a capacity of 634 mAh/g after 900 cycles at a current density of 500 mA/g. An enhanced capacity is observed during the long-term cycling process, which may be attributed to the structural integrity, the stability of solid electrolyte interphase layer, and the electrochemical actuation of the yolk–shell nanorod structure. The results demonstrate that the manganese oxide is well utilized with the one-dimensional yolk–shell structure, which represents an efficient way to realize excellent performance for practical applications