Learning Two-Branch Neural Networks for Image-Text Matching Tasks

Image-language matching tasks have recently attracted a lot of attention in the computer vision field. These tasks include image-sentence matching, i.e., given an image query, retrieving relevant sentences and vice versa, and region-phrase matching or visual grounding, i.e., matching a phrase to relevant regions. This paper investigates two-branch neural networks for learning the similarity between these two data modalities. We propose two network structures that produce different output representations. The first one, referred to as an embedding network, learns an explicit shared latent embedding space with a maximum-margin ranking loss and novel neighborhood constraints. Compared to standard triplet sampling, we perform improved neighborhood sampling that takes neighborhood information into consideration while constructing mini-batches. The second network structure, referred to as a similarity network, fuses the two branches via element-wise product and is trained with regression loss to directly predict a similarity score. Extensive experiments show that our networks achieve high accuracies for phrase localization on the Flickr30K Entities dataset and for bi-directional image-sentence retrieval on Flickr30K and MSCOCO datasets.Comment: accepted version in TPAMI 201

Sub-Ohmic spin-boson model with off-diagonal coupling: Ground state properties

We have carried out analytical and numerical studies of the spin-boson model in the sub-ohmic regime with the influence of both the diagonal and off-diagonal coupling accounted for via the Davydov D1 variational ansatz. While a second-order phase transition is known to be exhibited by this model in the presence of diagonal coupling only, we demonstrate the emergence of a discontinuous first order phase transition upon incorporation of the off-diagonal coupling. A plot of the ground state energy versus magnetization highlights the discontinuous nature of the transition between the isotropic (zero magnetization) state and nematic (finite magnetization) phases. We have also calculated the entanglement entropy and a discontinuity found at a critical coupling strength further supports the discontinuous crossover in the spin-boson model in the presence of off-diagonal coupling. It is further revealed via a canonical transformation approach that for the special case of identical exponents for the spectral densities of the diagonal and the off-diagonal coupling, there exists a continuous crossover from a single localized phase to doubly degenerate localized phase with differing magnetizations.Comment: 11 pages, 7 figure

Dynamics of the sub-Ohmic spin-boson model: a time-dependent variational study

The Dirac-Frenkel time-dependent variation is employed to probe the dynamics of the zero temperature sub-Ohmic spin-boson model with strong friction utilizing the Davydov D1 ansatz. It is shown that initial conditions of the phonon bath have considerable influence on the dynamics. Counterintuitively, even in the very strong coupling regime, quantum coherence features still manage to survive under the polarized bath initial condition, while such features are absent under the factorized bath initial condition. In addition, a coherent-incoherent transition is found at a critical coupling strength alpha ~ 0.1 for s=0.25 under the factorized bath initial condition. We quantify how faithfully our ansatz follows the Schr\"{o}dinger equation, finding that the time-dependent variational approach is robust for strong dissipation and deep sub-Ohmic baths (s<<1).Comment: 8 pages, 6 figure

A Theoretical Analysis of NDCG Type Ranking Measures

A central problem in ranking is to design a ranking measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the widely used Normalized Discounted Cumulative Gain (NDCG)-type ranking measures. Although there are extensive empirical studies of NDCG, little is known about its theoretical properties. We first show that, whatever the ranking function is, the standard NDCG which adopts a logarithmic discount, converges to 1 as the number of items to rank goes to infinity. On the first sight, this result is very surprising. It seems to imply that NDCG cannot differentiate good and bad ranking functions, contradicting to the empirical success of NDCG in many applications. In order to have a deeper understanding of ranking measures in general, we propose a notion referred to as consistent distinguishability. This notion captures the intuition that a ranking measure should have such a property: For every pair of substantially different ranking functions, the ranking measure can decide which one is better in a consistent manner on almost all datasets. We show that NDCG with logarithmic discount has consistent distinguishability although it converges to the same limit for all ranking functions. We next characterize the set of all feasible discount functions for NDCG according to the concept of consistent distinguishability. Specifically we show that whether NDCG has consistent distinguishability depends on how fast the discount decays, and 1/r is a critical point. We then turn to the cut-off version of NDCG, i.e., NDCG@k. We analyze the distinguishability of NDCG@k for various choices of k and the discount functions. Experimental results on real Web search datasets agree well with the theory.Comment: COLT 201