Dual Attention Networks for Visual Reference Resolution in Visual Dialog

Visual dialog (VisDial) is a task which requires an AI agent to answer a series of questions grounded in an image. Unlike in visual question answering (VQA), the series of questions should be able to capture a temporal context from a dialog history and exploit visually-grounded information. A problem called visual reference resolution involves these challenges, requiring the agent to resolve ambiguous references in a given question and find the references in a given image. In this paper, we propose Dual Attention Networks (DAN) for visual reference resolution. DAN consists of two kinds of attention networks, REFER and FIND. Specifically, REFER module learns latent relationships between a given question and a dialog history by employing a self-attention mechanism. FIND module takes image features and reference-aware representations (i.e., the output of REFER module) as input, and performs visual grounding via bottom-up attention mechanism. We qualitatively and quantitatively evaluate our model on VisDial v1.0 and v0.9 datasets, showing that DAN outperforms the previous state-of-the-art model by a significant margin.Comment: EMNLP 201

PROGrasp: Pragmatic Human-Robot Communication for Object Grasping

Interactive Object Grasping (IOG) is the task of identifying and grasping the desired object via human-robot natural language interaction. Current IOG systems assume that a human user initially specifies the target object's category (e.g., bottle). Inspired by pragmatics, where humans often convey their intentions by relying on context to achieve goals, we introduce a new IOG task, Pragmatic-IOG, and the corresponding dataset, Intention-oriented Multi-modal Dialogue (IM-Dial). In our proposed task scenario, an intention-oriented utterance (e.g., "I am thirsty") is initially given to the robot. The robot should then identify the target object by interacting with a human user. Based on the task setup, we propose a new robotic system that can interpret the user's intention and pick up the target object, Pragmatic Object Grasping (PROGrasp). PROGrasp performs Pragmatic-IOG by incorporating modules for visual grounding, question asking, object grasping, and most importantly, answer interpretation for pragmatic inference. Experimental results show that PROGrasp is effective in offline (i.e., target object discovery) and online (i.e., IOG with a physical robot arm) settings.Comment: 7 pages, 6 figure

PGA: Personalizing Grasping Agents with Single Human-Robot Interaction

Language-Conditioned Robotic Grasping (LCRG) aims to develop robots that ground and grasp objects based on natural language instructions. While robots capable of recognizing personal objects like "my wallet" can interact more naturally with non-expert users, current LCRG systems primarily limit robots to understanding only generic expressions. To this end, we introduce a task scenario GraspMine with a novel dataset that aims to locate and grasp personal objects given personal indicators via learning from a single human-robot interaction. To address GraspMine, we propose Personalized Grasping Agent (PGA), that learns personal objects by propagating user-given information through a Reminiscence-a collection of raw images from the user's environment. Specifically, PGA acquires personal object information by a user presenting a personal object with its associated indicator, followed by PGA inspecting the object by rotating it. Based on the acquired information, PGA pseudo-labels objects in the Reminiscence by our proposed label propagation algorithm. Harnessing the information acquired from the interactions and the pseudo-labeled objects in the Reminiscence, PGA adapts the object grounding model to grasp personal objects. Experiments on GraspMine show that PGA significantly outperforms baseline methods both in offline and online settings, signifying its effectiveness and personalization applicability on real-world scenarios. Finally, qualitative analysis shows the effectiveness of PGA through a detailed investigation of results in each phase.Comment: 7 pages, under revie

Risk stratification of symptomatic brain metastases by clinical and FDG PET parameters for selective use of prophylactic cranial irradiation in patients with extensive disease of small cell lung cancer

Purpose: To identify risk factors for developing symptomatic brain metastases and evaluate the impact of prophylactic cranial irradiation (PCI) on brain metastasis-free survival (BMFS) and overall survival (OS) in extensive disease small cell lung cancer (ED-SCLC). Materials and methods: Among 190 patients diagnosed with ED-SCLC who underwent FDG PET/CT and brain Magnetic Resonance Imaging (MRI) prior to treatment, 53 (27.9%) received PCI while 137 (72.1%) did not. Prognostic index predicting a high risk of symptomatic brain metastases was calculated for the group without receiving PCI (observation group, n = 137) with Cox regression model. Results: Median follow-up time was 10.6 months. Multivariate Cox regression showed that the following three factors were associated with a high risk of symptomatic brain metastases: the presence of extrathoracic metastases (p = 0.004), hypermetabolism of bone marrow or spleen on FDG PET (p < 0.001), and high neutrophil-to-lymphocyte ratio (p = 0.018). PCI significantly improved BMFS in high-risk patients (1-year rate: 94.7% vs. 62.1%, p = 0.001), but not in low-risk patients (1-year rate: 100.0% vs. 87.7%, p = 0.943). However, PCI did not improve OS in patients at high risk for symptomatic brain metastases (1-year rate: 65.2% vs. 50.0%, p = 0.123). Conclusion: Three prognostic factors (the presence of extrathoracic metastases, hypermetabolism of bone marrow or spleen on FDG PET, and high neutrophil-to-lymphocyte ratio) were associated with a high risk of symptomatic brain metastases in ED-SCLC. PCI was beneficial for patients at a high risk of symptomatic brain metastases in terms of BMFS, but not OS. Thus, selective use of PCI in ED-SCLC according to the risk stratification is recommended. (C) 2020 Elsevier B.V. All rights reserved.

A pan-cancer analysis of the clinical and genetic portraits of somatostatin receptor expressing tumor as a potential target of peptide receptor imaging and therapy

Abstract Purpose Although somatostatin receptor (SST) is a promising theranostic target and is widely expressed in tumors of various organs, the indication for therapies targeting SST is limited to typical gastroenteropancreatic neuroendocrine tumors (NETs). Thus, broadening the scope of the current clinical application of peptide receptor radiotherapy (PRRT) can be supported by a better understanding of the landscape of SST-expressing tumors. Methods SST expression levels were assessed in data from The Cancer Genome Atlas across 10,701 subjects representing 32 cancer types. As the major target of PRRT is SST subtype 2 (SST2), correlation analyses between the pan-cancer profiles, including clinical and genetic features, and SST2 level were conducted. The median SST2 expression level of pheochromocytoma and paraganglioma (PCPG) samples was used as the threshold to define high-SST2 tumors. The prognostic value of SST2 in each cancer subtype was evaluated by using Cox proportional regression analysis. Results We constructed a resource of SST expression patterns associated with clinicopathologic features and genomic alterations. It provides an interactive tool to analyze SST expression patterns in various cancer types. As a result, eight of the 31 cancer subtypes other than PCPG had more than 5% of tumors with high-SST2 expression. Low-grade glioma (LGG) showed the highest proportion of high-SST2 tumors, followed by breast invasive carcinoma (BRCA). LGG showed different SST2 levels according to tumor grade and histology. IDH1 mutation was significantly associated with high-SST2 status. In BRCA, the SST2 level was different according to the hormone receptor status. High-SST2 status was significantly associated with good prognosis in LGG patients. High-SST2 status showed a trend for association with poor prognosis in triple-negative breast cancer subjects. Conclusion A broad range of SST2 expression was observed across diverse cancer subtypes. The SST2 expression level showed a significant association with genomic and clinical aspects across cancers, especially in LGG and BRCA. These findings extend our knowledge base to diversify the indications for PRRT as well as SST imaging