Image quality assessment and multi-focus image fusion

Abstract

Ph.D.In the last decades, deep learning develops fast and achieves great success in many image processing tasks. The achievements prove that deep learning is a powerful tool. Therefore, I tried to adopt deep learning to solve two image processing tasks, image quality assessment and multi-focus image fusion.Image Quality Assessment (IQA) targets on objectively estimating the visual quality of an image, where the quality score is expected to be well correlated well with the human subjective visual quality score. Among all IQA metrics, the no-reference non-distortion specific ones are most challenging. They aim at accurately evaluating images distorted by any distortion types without the help of the reference image. Though different distortion types may lead to the different influence on Human Visual System(HVS), we focus on proposing non-distortion-specific models. During this process, the distortion information is a crucial clue to be utilized.An image quality evaluator, i.e. Deep Learning based Blind Image Quality Index(DL-BIQI), was first explored and designed to make full use of the distortion information. In DL-BIQI, several models are designed for several specific distortion types. Meanwhile, another deep classification model is proposed to estimate the presence of a set of distortions in the testing image. The final visual quality is obtained by a probability-weighted summation model. Experiments were conducted on the LIVE dataset to evaluate the effectiveness of DL-BIQA. The performance of the DL-BIQI achieves 0.951 for Spearman Rank-Order Correlation Coefficient (SROCC). It outper forms many state-of-the-art methods for comparison. Besides, the proposed distortion type classification model achieves 93.7% accuracy on the LIVE dataset.The success of DL-BIQI proves the effectiveness of distortion information in estimating visual quality. Inspired by this, another learning based IQA framework, Visual Importance and Distortion Guided Image Quality Assessment method (VIDGIQA), is proposed where the distortion information was adopted to guide the feature learning process. In addition, a regression method is designed in the framework to model and estimate the visual importance weights of all local regions. More importantly, VIDGIQA makes full use of deep learning by integrating all operations into one deep neural network, so that they can be jointly optimized and well cooperate with each other. Experiments were conducted to demonstrate the power of the VIDGIQA on several datasets, including the LIVE dataset, the TID 2013 dataset, the LIVE multiply distorted IQA dataset, the CSIQ dataset and the LIVE in the wild image quality database. VIDGIQA achieves 0.969 and 0.973 on the LIVE dataset in terms of SROCC and Pearson Linear Correlation Coefficient (LCC) respectively, which outperforms the state-of-the-art methods.In addition, deep learning is also tried to be adopted in the multi-focus image fusion task. In earlier image fusion works, supervised learning with large amount of data is rarely used because of the lack of a large amount of training data with labels. During my research, I found that the Gaussian blurring images in the IQA datasets are quite similar to the out-of-focus images. Inspired by this observation, I tried to introduce the highly related task, image quality assessment, to help measure focus levels and do image fusion in multi-focus image fusion. With a large amount of training data with labels provided in the IQA datasets, various deep neural networks could be explored.Thus two quality estimation based multi-focus image fusion methods are proposed. In these methods, the all-in-focus image is generated by pixel-wise summarizing the multi-focus source images with their estimated focus levels as weights. Since the visual quality of an image is highly correlated with its focus level, the visual quality is estimated to help pre-measure the focus levels. To smooth pre-measurement results and form the final-measurement, two edge-preserving smoothing filters were explored and result in the two fusion methods. The guided filter corresponds to Quality Estimation Based multi-focus Image Fusion method (QEBIF), while the fast bilateral solver corresponds to Bilateral Solver based Quality Estimation Based multi-focus Image Fusion method (BS-QEBIF). In these methods, the Confidence Map (CM) is proposed to measure the reliability of different local regions. Experimental results show that the proposed methods outperform the other fusion methods, and the fusion results can well maintain the detailed information in the multi-focus source images without suffering the ringing or blocking artifacts.In the meantime, another hand-crafted feature based image fusion method, the Edge Model And Multi-matting based multi-focus image fusion method (EMAM), is explored. The proposed method first estimates focus maps using a novel combination of edge models and a traditional block-based focus measure. Then a propagation process is conducted to obtain accurate weight maps based on a novel multi-matting model that makes full use of the spatial information. The fused all-in-focus image is also generated based on a weighted-summation model. Experimental results demonstrate that the proposed method achieved the state-of-the-art performance under various situations, even in cases with obvious mis-registration.Through the investigation of the above methods, it is found that deep learning is a very powerful tool. It can help these two traditional image processing tasks achieve state-of-the-art performance. During this process, it is not appropriate to regard deep learning as a black box. The key, in my opinion, is to add the deep understanding and discovery of the specific task and adopt these observations to designing the deep neural networks. Only in this way, deep learning can be utilized as a powerful tool and we achieve great success.深度學習的方法在近幾年取得了飛速的發展，並在很多圖像處理的領域取得了傲人的成績。這些成績證明瞭深度學習算法的高效性。因此，我積極探索深度學習算法並嘗試將其應用在圖像質量評估以及多聚焦圖像融合兩個領域。圖像質量評估致力於客觀得對於圖像的質量進行準確的評價。當用此種客觀方法得到的圖片質量，和用主觀測評時人對圖片評估的質量相似時，我們判斷此種客觀評價算法準確。在所有圖像質量評估的算法中，無參考非特定失真類型的圖像質量評估算法是最具挑戰性的一種。這種方法致力於在沒有參考圖的基礎上，對於圖片的質量進行準確的評估。因為不同的圖像失真類型對於人眼視覺系統有著不同的影響，因此我們在研究非特定失真圖像的處理算法時，圖像失真信息是我們要考慮的重要因素。為了能夠充分探究以及利用圖像失真信息，我們提出了一種圖像質量評估算法：基於深度學習的無參考質量評估標準DL-BIQI.在這種算法中，我們為每一種圖像失真類型設計了相應的網絡。同時，我們訓練了另外一個深度網絡來對輸入圖片屬於哪種失真類型的概率進行評判。輸入圖片的最終質量是由一個以概率為權重的組合加法模型確定的。為了證明該算法的有效性，我們將其在LIVE圖庫上進行了測試。DL-BIQI在SROCC指標上取得了0.951的好成績。這個結果超過了很多用來比較的最前沿的算法。同時，這個算法中對於圖像失真類型的判別達到了93.7% 的準確率。DL-BIQI算法的成功表明失真信息在評估圖片質量的過程中有著重大的指導意義。受此啟發，失真信息被嘗試應用在指導學習特徵的過程中，並衍生出另外一種更為複雜的基於機器學習的圖像質量評估算法VIDGIQA。在這個算法中，我們設計了一個回歸模型用來模擬並估計所有局部圖像塊的人眼視覺重要程度。更重要的是，該算法充分利用了深度學習網絡的優勢，把所有模塊整合到同一個深度網絡架構中。這樣，這個架構中所有的模塊都可以被聯合優化，達到最好的協同合作的效果。為了證明該算法的高效性，多個圖像質量圖庫參與了評測，包括LIVE圖庫，TID2013圖庫，LIVE多失真類型圖庫，CSIQ圖庫以及LIVE原創圖像集圖庫。VIDGIQA算法在LIVE圖庫上取得了SROCC 0.969 和 LCC 0.973的好成績，充分說明瞭該算法的高效性能。同時，我們也嘗試將深度學習的算法應用到多聚焦圖像融合的任務中。在早期多聚焦圖像融合的工作中，有監督的深度學習算法很少被成功應用。原因是在這個領域中，有標註的數據數量是非常有限的。我們在研究中發現，失焦的圖片和高斯模糊的圖片在原理和表現形式上有一些相似之處。受到這個想法的啟發，我們嘗試引入一個高度相似的任務，圖像質量評估，來幫助我們完成多聚焦圖像融合的工作。基於圖像質量評估領域提供的大量的有標注的數據，我們可以探索多種深度學習方法來幫助完成多聚焦圖像融合的工作。因此我們探索了兩種基於圖像質量的多聚焦圖像融合的方法。在這些方法中，全清晰的圖像是由一種像素級的權重模型累積生成的。由於圖像的視覺質量與其聚焦水平高度相關，因此圖像視覺質量可以幫助我們對其聚焦水平進行預測。之後，為了平滑預測量結果並形成最終聚焦水平測量結果，兩個可以保持邊緣的平滑濾波器被嘗試應用到算法中並構成兩種相應的算法。引導濾波器(the guided filter)對應基於質量估計的多焦點圖像融合方法`QEBIF’，而快速雙向求解器(the fast bilateral solver)對應基於雙邊求解器的多焦點圖像融合方法`BS-QEBIF’。同時，這兩種算法中都融合了信度圖來衡量不同位置的可靠性。實驗表明，以上兩種方法優於其他現有的先進的聚焦方法，融合結果可以很好地保持多焦點圖像中的詳細信息，而且不會產生振鈴或塊狀偽影。我們也同樣探索了一種基於手工特徵的圖像融合方法EMAM。該方法首先使用一種新穎的，融合了邊緣模型(edge model)和傳統的聚焦度量的組合來估計圖片聚焦程度。然後該算法基於一種可以充分利用空間信息的，新穎的模型(multi-matting model) 來獲得準確的權重。 全焦點圖像同樣基於加權和策略生成。 實驗結果表明，該方法在實際應用的各種情況下，包括在明顯失配 (mis-registration) 的情況下都具有非常先進的性能。通過對上述方法的研究，我們發現深度學習是一個非常強大的工具，可以幫助這兩個圖像處理的傳統任務取得非常先進的實驗效果。在此過程中，將深度學習視為黑匣子式的工具是很不恰當的。在我看來，此類研究的關鍵是增加對具體任務的深刻理解，並在設計網絡的過程中融會這些理解。這樣，深度學習才能作為一個種高效的工具，幫助我們完成更多的任務。Guan, Jingwei.Thesis Ph.D. Chinese University of Hong Kong 2017.Includes bibliographical references (leaves 87-94).Abstracts also in Chinese.Title from PDF title page (viewed on 15, October, 2019)