What-and-Where to Match: Deep Spatially Multiplicative Integration Networks for Person Re-identification

Matching pedestrians across disjoint camera views, known as person re-identification (re-id), is a challenging problem that is of importance to visual recognition and surveillance. Most existing methods exploit local regions within spatial manipulation to perform matching in local correspondence. However, they essentially extract \emph{fixed} representations from pre-divided regions for each image and perform matching based on the extracted representation subsequently. For models in this pipeline, local finer patterns that are crucial to distinguish positive pairs from negative ones cannot be captured, and thus making them underperformed. In this paper, we propose a novel deep multiplicative integration gating function, which answers the question of \emph{what-and-where to match} for effective person re-id. To address \emph{what} to match, our deep network emphasizes common local patterns by learning joint representations in a multiplicative way. The network comprises two Convolutional Neural Networks (CNNs) to extract convolutional activations, and generates relevant descriptors for pedestrian matching. This thus, leads to flexible representations for pair-wise images. To address \emph{where} to match, we combat the spatial misalignment by performing spatially recurrent pooling via a four-directional recurrent neural network to impose spatial dependency over all positions with respect to the entire image. The proposed network is designed to be end-to-end trainable to characterize local pairwise feature interactions in a spatially aligned manner. To demonstrate the superiority of our method, extensive experiments are conducted over three benchmark data sets: VIPeR, CUHK03 and Market-1501.Comment: Published at Pattern Recognition, Elsevie

Understanding Learned Models by Identifying Important Features at the Right Resolution

In many application domains, it is important to characterize how complex learned models make their decisions across the distribution of instances. One way to do this is to identify the features and interactions among them that contribute to a model's predictive accuracy. We present a model-agnostic approach to this task that makes the following specific contributions. Our approach (i) tests feature groups, in addition to base features, and tries to determine the level of resolution at which important features can be determined, (ii) uses hypothesis testing to rigorously assess the effect of each feature on the model's loss, (iii) employs a hierarchical approach to control the false discovery rate when testing feature groups and individual base features for importance, and (iv) uses hypothesis testing to identify important interactions among features and feature groups. We evaluate our approach by analyzing random forest and LSTM neural network models learned in two challenging biomedical applications.Comment: First two authors contributed equally to this work, Accepted for presentation at the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19

Learning a local-variable model of aromatic and conjugated systems

A collection of new approaches to building and training neural networks, collectively referred to as deep learning, are attracting attention in theoretical chemistry. Several groups aim to replace computationally expensive <i>ab initio</i> quantum mechanics calculations with learned estimators. This raises questions about the representability of complex quantum chemical systems with neural networks. Can local-variable models efficiently approximate nonlocal quantum chemical features? Here, we find that convolutional architectures, those that only aggregate information locally, cannot efficiently represent aromaticity and conjugation in large systems. They cannot represent long-range nonlocality known to be important in quantum chemistry. This study uses aromatic and conjugated systems computed from molecule graphs, though reproducing quantum simulations is the ultimate goal. This task, by definition, is both computable and known to be important to chemistry. The failure of convolutional architectures on this focused task calls into question their use in modeling quantum mechanics. To remedy this heretofore unrecognized deficiency, we introduce a new architecture that propagates information back and forth in waves of nonlinear computation. This architecture is still a local-variable model, and it is both computationally and representationally efficient, processing molecules in sublinear time with far fewer parameters than convolutional networks. Wave-like propagation models aromatic and conjugated systems with high accuracy, and even models the impact of small structural changes on large molecules. This new architecture demonstrates that some nonlocal features of quantum chemistry can be efficiently represented in local variable models

Do Deep Neural Networks Model Nonlinear Compositionality in the Neural Representation of Human-Object Interactions?

Visual scene understanding often requires the processing of human-object interactions. Here we seek to explore if and how well Deep Neural Network (DNN) models capture features similar to the brain's representation of humans, objects, and their interactions. We investigate brain regions which process human-, object-, or interaction-specific information, and establish correspondences between them and DNN features. Our results suggest that we can infer the selectivity of these regions to particular visual stimuli using DNN representations. We also map features from the DNN to the regions, thus linking the DNN representations to those found in specific parts of the visual cortex. In particular, our results suggest that a typical DNN representation contains encoding of compositional information for human-object interactions which goes beyond a linear combination of the encodings for the two components, thus suggesting that DNNs may be able to model this important property of biological vision.Comment: 4 pages, 2 figures; presented at CCN 201

Coupled surface to deep Earth processes: Perspectives from TOPO-EUROPE with an emphasis on climate- and energy-related societal challenges

Understanding the interactions between surface and deep Earth processes is important for research in many diverse scientific areas including climate, environment, energy, georesources and biosphere. The TOPO-EUROPE initiative of the International Lithosphere Program serves as a pan-European platform for integrated surface and deep Earth sciences, synergizing observational studies of the Earth structure and fluxes on all spatial and temporal scales with modelling of Earth processes. This review provides a survey of scientific developments in our quantitative understanding of coupled surface-deep Earth processes achieved through TOPO-EUROPE. The most notable innovations include (1) a process-based understanding of the connection of upper mantle dynamics and absolute plate motion frames; (2) integrated models for sediment source-to-sink dynamics, demonstrating the importance of mass transfer from mountains to basins and from basin to basin; (3) demonstration of the key role of polyphase evolution of sedimentary basins, the impact of pre-rift and pre-orogenic structures, and the evolution of subsequent lithosphere and landscape dynamics; (4) improved conceptual understanding of the temporal evolution from back-arc extension to tectonic inversion and onset of subduction; (5) models to explain the integrated strength of Europe's lithosphere; (6) concepts governing the interplay between thermal upper mantle processes and stress-induced intraplate deformation; (7) constraints on the record of vertical motions from high-resolution data sets obtained from geo-thermochronology for Europe's topographic evolution; (8) recognition and quantifications of the forcing by erosional and/or glacial-interglacial surface mass transfer on the regional magmatism, with major implications for our understanding of the carbon cycle on geological timescales and the emerging field of biogeodynamics; and (9) the transfer of insights obtained on the coupling of deep Earth and surface processes to the domain of geothermal energy exploration. Concerning the future research agenda of TOPO-EUROPE, we also discuss the rich potential for further advances, multidisciplinary research and community building across many scientific frontiers, including research on the biosphere, climate and energy. These will focus on obtaining a better insight into the initiation and evolution of subduction systems, the role of mantle plumes in continental rifting and (super)continent break-up, and the deformation and tectonic reactivation of cratons; the interaction between geodynamic, surface and climate processes, such as interactions between glaciation, sea level change and deep Earth processes; the sensitivity, tipping points, and spatio-temporal evolution of the interactions between climate and tectonics as well as the role of rock melting and outgassing in affecting such interactions; the emerging field of biogeodynamics, that is the impact of coupled deep Earth – surface processes on the evolution of life on Earth; and tightening the connection between societal challenges regarding renewable georesources, climate change, natural geohazards, and novel process-understanding of the Earth system

Deep Memory Networks for Attitude Identification

We consider the task of identifying attitudes towards a given set of entities from text. Conventionally, this task is decomposed into two separate subtasks: target detection that identifies whether each entity is mentioned in the text, either explicitly or implicitly, and polarity classification that classifies the exact sentiment towards an identified entity (the target) into positive, negative, or neutral. Instead, we show that attitude identification can be solved with an end-to-end machine learning architecture, in which the two subtasks are interleaved by a deep memory network. In this way, signals produced in target detection provide clues for polarity classification, and reversely, the predicted polarity provides feedback to the identification of targets. Moreover, the treatments for the set of targets also influence each other -- the learned representations may share the same semantics for some targets but vary for others. The proposed deep memory network, the AttNet, outperforms methods that do not consider the interactions between the subtasks or those among the targets, including conventional machine learning methods and the state-of-the-art deep learning models.Comment: Accepted to WSDM'1