Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval

Free-hand sketch-based image retrieval (SBIR) is a specific cross-view retrieval task, in which queries are abstract and ambiguous sketches while the retrieval database is formed with natural images. Work in this area mainly focuses on extracting representative and shared features for sketches and natural images. However, these can neither cope well with the geometric distortion between sketches and images nor be feasible for large-scale SBIR due to the heavy continuous-valued distance computation. In this paper, we speed up SBIR by introducing a novel binary coding method, named \textbf{Deep Sketch Hashing} (DSH), where a semi-heterogeneous deep architecture is proposed and incorporated into an end-to-end binary coding framework. Specifically, three convolutional neural networks are utilized to encode free-hand sketches, natural images and, especially, the auxiliary sketch-tokens which are adopted as bridges to mitigate the sketch-image geometric distortion. The learned DSH codes can effectively capture the cross-view similarities as well as the intrinsic semantic correlations between different categories. To the best of our knowledge, DSH is the first hashing work specifically designed for category-level SBIR with an end-to-end deep architecture. The proposed DSH is comprehensively evaluated on two large-scale datasets of TU-Berlin Extension and Sketchy, and the experiments consistently show DSH's superior SBIR accuracies over several state-of-the-art methods, while achieving significantly reduced retrieval time and memory footprint.Comment: This paper will appear as a spotlight paper in CVPR201

Why does the apparent mass of a coronal mass ejection increase?

Mass is one of the most fundamental parameters characterizing the dynamics of a coronal mass ejection (CME). It has been found that CME apparent mass measured from the brightness enhancement in coronagraph images shows an increasing trend during its evolution in the corona. However, the physics behind it is not clear. Does the apparent mass gain come from the mass outflow from the dimming regions in the low corona, or from the pileup of the solar wind plasma around the CME when it propagates outwards from the Sun? We analyzed the mass evolution of six CME events. Their mass can increase by a factor of 1.6 to 3.2 from 4 to 15 Rs in the field of view (FOV) of the coronagraph on board the Solar Terrestrial Relations Observatory (STEREO). Over the distance about 7 to 15 Rs, where the coronagraph occulting effect can be negligible, the mass can increase by a factor of 1.3 to 1.7. We adopted the `snow-plough' model to calculate the mass contribution of the piled-up solar wind in the height range from about 7 to 15 Rs. For 2/3 of the events, the solar wind pileup is not sufficient to explain the measured mass increase. In the height range from about 7 to 15 Rs, the ratio of the modeled to the measured mass increase is roughly larger than 0.55. Although the ratios are believed to be overestimated, the result gives evidence that the solar wind pileup probably makes a non-negligible contribution to the mass increase. It is not clear yet whether the solar wind pileup is a major contributor to the final mass derived from coronagraph observations. However, our study suggests that the solar wind pileup plays increasingly important role in the mass increase as a CME moves further away from the Sun.Comment: 27 pages, 2 tables, 9 figures, accepted by Ap

Zero-Shot Sketch-Image Hashing

Recent studies show that large-scale sketch-based image retrieval (SBIR) can be efficiently tackled by cross-modal binary representation learning methods, where Hamming distance matching significantly speeds up the process of similarity search. Providing training and test data subjected to a fixed set of pre-defined categories, the cutting-edge SBIR and cross-modal hashing works obtain acceptable retrieval performance. However, most of the existing methods fail when the categories of query sketches have never been seen during training. In this paper, the above problem is briefed as a novel but realistic zero-shot SBIR hashing task. We elaborate the challenges of this special task and accordingly propose a zero-shot sketch-image hashing (ZSIH) model. An end-to-end three-network architecture is built, two of which are treated as the binary encoders. The third network mitigates the sketch-image heterogeneity and enhances the semantic relations among data by utilizing the Kronecker fusion layer and graph convolution, respectively. As an important part of ZSIH, we formulate a generative hashing scheme in reconstructing semantic knowledge representations for zero-shot retrieval. To the best of our knowledge, ZSIH is the first zero-shot hashing work suitable for SBIR and cross-modal search. Comprehensive experiments are conducted on two extended datasets, i.e., Sketchy and TU-Berlin with a novel zero-shot train-test split. The proposed model remarkably outperforms related works.Comment: Accepted as spotlight at CVPR 201