Active Learning for Fine-Grained Sketch-Based Image Retrieval

The ability to retrieve a photo by mere free-hand sketching highlights the immense potential of Fine-grained sketch-based image retrieval (FG-SBIR). However, its rapid practical adoption, as well as scalability, is limited by the expense of acquiring faithful sketches for easily available photo counterparts. A solution to this problem is Active Learning, which could minimise the need for labeled sketches while maximising performance. Despite extensive studies in the field, there exists no work that utilises it for reducing sketching effort in FG-SBIR tasks. To this end, we propose a novel active learning sampling technique that drastically minimises the need for drawing photo sketches. Our proposed approach tackles the trade-off between uncertainty and diversity by utilising the relationship between the existing photo-sketch pair to a photo that does not have its sketch and augmenting this relation with its intermediate representations. Since our approach relies only on the underlying data distribution, it is agnostic of the modelling approach and hence is applicable to other cross-modal instance-level retrieval tasks as well. With experimentation over two publicly available fine-grained SBIR datasets ChairV2 and ShoeV2, we validate our approach and reveal its superiority over adapted baselines.Comment: Accepted at BMVC 202

Super-resolving Compressed Images via Parallel and Series Integration of Artifact Reduction and Resolution Enhancement

In this paper, we propose a novel compressed image super resolution (CISR) framework based on parallel and series integration of artifact removal and resolution enhancement. Based on maximum a posterior inference for estimating a clean low-resolution (LR) input image and a clean high resolution (HR) output image from down-sampled and compressed observations, we have designed a CISR architecture consisting of two deep neural network modules: the artifact reduction module (ARM) and resolution enhancement module (REM). ARM and REM work in parallel with both taking the compressed LR image as their inputs, while they also work in series with REM taking the output of ARM as one of its inputs and ARM taking the output of REM as its other input. A unique property of our CSIR system is that a single trained model is able to super-resolve LR images compressed by different methods to various qualities. This is achieved by exploiting deep neural net-works capacity for handling image degradations, and the parallel and series connections between ARM and REM to reduce the dependency on specific degradations. ARM and REM are trained simultaneously by the deep unfolding technique. Experiments are conducted on a mixture of JPEG and WebP compressed images without a priori knowledge of the compression type and com-pression factor. Visual and quantitative comparisons demonstrate the superiority of our method over state-of-the-art super resolu-tion methods.Code link: https://github.com/luohongming/CISR_PS

Investigation of Different Image Super Resolution Methods on Paired Electron Microscopic Images

This thesis is concerned with investigating super-resolution algorithms and solutions for handling electron microscopic images. Please note two main aspects differentiating the problem discussed here from those considered in the literature. The first difference is that in the electron imaging setting, a pair of physical high-resolution and low-resolution images is used, rather than a physical image with its downsampled counterpart. The high-resolution image covers about 25\% of the view field of the low-resolution image, and the objective is to enhance the area of the low-resolution image where there is no high-resolution counterpart. The second difference is that the physics behind electron imaging is different from that of optical (visible light) photos. The implication is that super-resolution models trained by optical photos are not effective when applied to electron images. Focusing on the unique properties, a global and local registration method is devised to match the high- and low-resolution image patches and different training strategies are discussed for applying deep learning and non deep learning based super-resolution methods to the paired electron images. This thesis investigates the uniqueness of the super-resolution problem on paired electron microscopic images. After extensive experimentation and comparison on 22 pairs of electron images, it is now believed that the self-training strategy, in which the training images come from the same image pair of the test set, leads to better super-resolution outcomes, despite the relatively small training data size. Deep learning-based super-resolution methods show the best performances, whereas a revised paired library-based non-local mean method shows advantage in training time and interpretability. Paired images super-resolution has important implications in many research areas. Paired electron images are rather common in scientific experiments, especially in material and medical research. Due to the destructive imaging process while using electron sources, researchers tend to use low-energy beams or subject the samples to a short duration of exposure to protect the sample. As a consequence, low-resolution images are generated. Super-resolution methods, which can subsequently boost these low-resolution images to a higher resolution, are much desired in scientific researches using electron imaging