Recurrent Multimodal Interaction for Referring Image Segmentation

In this paper we are interested in the problem of image segmentation given natural language descriptions, i.e. referring expressions. Existing works tackle this problem by first modeling images and sentences independently and then segment images by combining these two types of representations. We argue that learning word-to-image interaction is more native in the sense of jointly modeling two modalities for the image segmentation task, and we propose convolutional multimodal LSTM to encode the sequential interactions between individual words, visual information, and spatial information. We show that our proposed model outperforms the baseline model on benchmark datasets. In addition, we analyze the intermediate output of the proposed multimodal LSTM approach and empirically explain how this approach enforces a more effective word-to-image interaction.Comment: To appear in ICCV 2017. See http://www.cs.jhu.edu/~cxliu/ for code and supplementary materia

A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks

Physics-informed neural networks (PINNs) have shown to be an effective tool for solving forward and inverse problems of partial differential equations (PDEs). PINNs embed the PDEs into the loss of the neural network, and this PDE loss is evaluated at a set of scattered residual points. The distribution of these points are highly important to the performance of PINNs. However, in the existing studies on PINNs, only a few simple residual point sampling methods have mainly been used. Here, we present a comprehensive study of two categories of sampling: non-adaptive uniform sampling and adaptive nonuniform sampling. We consider six uniform sampling, including (1) equispaced uniform grid, (2) uniformly random sampling, (3) Latin hypercube sampling, (4) Halton sequence, (5) Hammersley sequence, and (6) Sobol sequence. We also consider a resampling strategy for uniform sampling. To improve the sampling efficiency and the accuracy of PINNs, we propose two new residual-based adaptive sampling methods: residual-based adaptive distribution (RAD) and residual-based adaptive refinement with distribution (RAR-D), which dynamically improve the distribution of residual points based on the PDE residuals during training. Hence, we have considered a total of 10 different sampling methods, including six non-adaptive uniform sampling, uniform sampling with resampling, two proposed adaptive sampling, and an existing adaptive sampling. We extensively tested the performance of these sampling methods for four forward problems and two inverse problems in many setups. Our numerical results presented in this study are summarized from more than 6000 simulations of PINNs. We show that the proposed adaptive sampling methods of RAD and RAR-D significantly improve the accuracy of PINNs with fewer residual points. The results obtained in this study can also be used as a practical guideline in choosing sampling methods

High spatial-resolution imaging of label-free in vivo protein aggregates by VISTA

Amyloid aggregation, formed by aberrant proteins, is a pathological hallmark for neurodegenerative diseases, including Alzheimer's disease and Huntington's disease. High-resolution holistic mapping of the fine structures from these aggregates should facilitate our understanding of their pathological roles. Here, we achieved label-free high-resolution imaging of the polyQ and the amyloid-beta (Aβ) aggregates in cells and tissues utilizing a sample-expansion stimulated Raman strategy. We further focused on characterizing the Aβ plaques in 5XFAD mouse brain tissues. 3D volumetric imaging enabled visualization of the whole plaques, resolving both the fine protein filaments and the surrounding components. Coupling our expanded label-free Raman imaging with machine learning, we obtained specific segmentation of aggregate cores, peripheral filaments together with cell nuclei and blood vessels by pre-trained convolutional neural network models. Combining with 2-channel fluorescence imaging, we achieved a 6-color holistic view of the same sample. This ability for precise and multiplex high-resolution imaging of the protein aggregates and their micro-environment without the requirement of labeling would open new biomedical applications

Tool Selectivity in Left Occipitotemporal Cortex Develops without Vision

Previous studies have provided evidence for a tool-selective region in left lateral occipitotemporal cortex (LOTC). This region responds selectively to pictures of tools and to characteristic visual tool motion. The present human fMRI study tested whether visual experience is required for the development of tool-selective responses in left LOTC. Words referring to tools, animals, and nonmanipulable objects were presented auditorily to 14 congenitally blind and 16 sighted participants. Sighted participants additionally viewed pictures of these objects. In whole-brain group analyses, sighted participants showed tool-selective activity in left LOTC in both visual and auditory tasks. Importantly, virtually identical tool-selective LOTC activity was found in the congenitally blind group performing the auditory task. Furthermore, both groups showed equally strong tool-selective activity for auditory stimuli in a tool-selective LOTC region defined by the picture-viewing task in the sighted group. Detailed analyses in individual participants showed significant tool-selective LOTC activity in 13 of 14 blind participants and 14 of 16 sighted participants. The strength and anatomical location of this activity were indistinguishable across groups. Finally, both blind and sighted groups showed significant resting state functional connectivity between left LOTC and a bilateral frontoparietal network. Together, these results indicate that tool-selective activity in left LOTC develops without ever having seen a tool or its motion. This finding puts constraints on the possible role that this region could have in tool processing and, more generally, provides new insights into the principles shaping the functional organization of OTC.Psycholog

Clinical markers predict the efficacy of several immune checkpoint inhibitors in patients with non-small cell lung cancer in China

ObjectivesImmune checkpoint inhibitors (ICIs) are one of the most significant oncological treatment modalities as a result of the rapid advancement of immunotherapy. Programmed Cell Death-Ligand 1 (PD-L1) and tumor mutational burden (TMB) have emerged as key markers for predicting the efficacy and prognosis of ICIs in non-small cell lung cancer (NSCLC), and the predictive role of tumor-infiltrating lymphocytes (TILs) has also received significant attention. However, the prognosis of some individuals cannot be determined by these indicators; for instance, some patients with low PD-L1 expression also benefit from longer survival. Therefore, the purpose of this research was to investigate the connection between new haematological and pathological markers and clinical outcomes in NSCLC patients receiving ICIs.MethodsSeventy-six patients with stage III-IV NSCLC treated with ICIs were included in this study. We used the Mann-Whitney test, COX regression and Kaplan-Meier analysis to retrospectively analyze peripheral blood indicators and survival prognostic data of 76 patients in order to investigate the relationship between baseline neutrophil-to-lymphocyte ratio (NLR) and the efficacy of ICIs. To investigate the correlation between CXCL13, CXCR5, CD8 and the efficacy of ICIs, we assessed the expression levels of aforementioned indicators in biopsied tissues of 10 non-small cell lung tumors by immunohistochemistry (IHC) and immunofluorescence (IF) and performed statistical analysis.ResultsDisease control rate (DCR) was higher in patients with baseline NLR <3.4 (p=0.016) and neutrophil percentage <71% (P=0.015). Baseline NLR (HR=2.364, P=0.003) and neutrophil percentage (HR=2.824, P=0.013) had the greatest influence on patients’ survival prognosis, with baseline NLR exhibiting a stronger predictive value (AUC=0.717), according to univariate and multifactorial COX regression analyses of progression-free survival (PFS) and overall survival (OS). In NSCLC tissues, higher expression of CXCL13 was associated with better clinical outcomes (P=0.032) and higher expression of CD8 was associated with prolonged survival (P=0.022).ConclusionLow baseline NLR in peripheral blood and high expression of CD8 in tissues are associated with longer PFS and may have a potential predictive value for patients with stage III-IV NSCLC using ICIs