Learning to Guide Decoding for Image Captioning

Recently, much advance has been made in image captioning, and an encoder-decoder framework has achieved outstanding performance for this task. In this paper, we propose an extension of the encoder-decoder framework by adding a component called guiding network. The guiding network models the attribute properties of input images, and its output is leveraged to compose the input of the decoder at each time step. The guiding network can be plugged into the current encoder-decoder framework and trained in an end-to-end manner. Hence, the guiding vector can be adaptively learned according to the signal from the decoder, making itself to embed information from both image and language. Additionally, discriminative supervision can be employed to further improve the quality of guidance. The advantages of our proposed approach are verified by experiments carried out on the MS COCO dataset.Comment: AAAI-1

Nonlinear Expectation Inference for Direct Uncertainty Quantification of Nonlinear Inverse Problems

Most existing inference methods for the uncertainty quantification of nonlinear inverse problems need repetitive runs of the forward model which is computationally expensive for high-dimensional problems, where the forward model is expensive and the inference need more iterations. These methods are generally based on the Bayes' rule and implicitly assume that the probability distribution is unique, which is not the case for scenarios with Knightian uncertainty. In the current study, we assume that the probability distribution is uncertain, and establish a new inference method based on the nonlinear expectation theory for 'direct' uncertainty quantification of nonlinear inverse problems. The uncertainty of random parameters is quantified using the sublinear expectation defined as the limits of an ensemble of linear expectations estimated on samples. Given noisy observed data, the posterior sublinear expectation is computed using posterior linear expectations with highest likelihoods. In contrary to iterative inference methods, the new nonlinear expectation inference method only needs forward model runs on the prior samples, while subsequent evaluations of linear and sublinear expectations requires no forward model runs, thus quantifying uncertainty directly which is more efficient than iterative inference methods. The new method is analysed and validated using 2D and 3D test cases of transient Darcy flows

A Generalized Flow-Based Method for Analysis of Implicit Relationships on Wikipedia

We focus on measuring relationships between pairs of objects in Wikipedia whose pages can be regarded as individual objects. Two kinds of relationships between two objects exist: in Wikipedia, an explicit relationship is represented by a single link between the two pages for the objects, and an implicit relationship is represented by a link structure containing the two pages. Some of the previously proposed methods for measuring relationships are cohesion-based methods, which underestimate objects having high degrees, although such objects could be important in constituting relationships in Wikipedia. The other methods are inadequate for measuring implicit relationships because they use only one or two of the following three important factors: distance, connectivity, and cocitation. We propose a new method using a generalized maximum flow which reflects all the three factors and does not underestimate objects having high degree. We confirm through experiments that our method can measure the strength of a relationship more appropriately than these previously proposed methods do. Another remarkable aspect of our method is mining elucidatory objects, that is, objects constituting a relationship. We explain that mining elucidatory objects would open a novel way to deeply understand a relationship

Watermarking Graph Neural Networks by Random Graphs

Many learning tasks require us to deal with graph data which contains rich relational information among elements, leading increasing graph neural network (GNN) models to be deployed in industrial products for improving the quality of service. However, they also raise challenges to model authentication. It is necessary to protect the ownership of the GNN models, which motivates us to present a watermarking method to GNN models in this paper. In the proposed method, an Erdos-Renyi (ER) random graph with random node feature vectors and labels is randomly generated as a trigger to train the GNN to be protected together with the normal samples. During model training, the secret watermark is embedded into the label predictions of the ER graph nodes. During model verification, by activating a marked GNN with the trigger ER graph, the watermark can be reconstructed from the output to verify the ownership. Since the ER graph was randomly generated, by feeding it to a non-marked GNN, the label predictions of the graph nodes are random, resulting in a low false alarm rate (of the proposed work). Experimental results have also shown that, the performance of a marked GNN on its original task will not be impaired. Moreover, it is robust against model compression and fine-tuning, which has shown the superiority and applicability.Comment: https://hzwu.github.io