On Designing Deep Learning Approaches for Classification of Football Jersey Images in the Wild

Internet shopping has spread wide and into social networking. Someone may want to buy a shirt, accessories, etc., in a random picture or a streaming video. In this thesis, the problem of automatic classification was taken upon, constraining the target to jerseys in the wild, assuming the object is detected.;A dataset of 7,840 jersey images, namely the JerseyXIV is created, containing images of 14 categories of various football jersey types (Home and Alternate) belonging to 10 teams of 2015 Big 12 Conference football season. The quality of images varies in terms of pose, standoff distance, level of occlusion and illumination. Due to copyright restrictions on certain images, unaltered original images with appropriate credits can be provided upon request.;While various conventional and deep learning based classification approaches were empirically designed, optimized and tested, a solution that resulted in the highest accuracy in terms of classification was achieved by a train-time fused Convolutional Neural Network (CNN) architecture, namely CNN-F, with 92.61% accuracy. The final solution combines three different CNNs through score level average fusion achieving 96.90% test accuracy. To test these trained CNN models on a larger, application oriented scale, a video dataset is created, which may present an addition of higher rate of occlusion and elements of transmission noise. It consists of 14 videos, one for each class, totaling to 3,584 frames, with 2,188 frames containing the object of interest. With manual detection, the score level average fusion has achieved the highest classification accuracy of 81.31%.;In addition, three Image Quality Assessment techniques were tested to assess the drop in accuracy of the average-fusion method on the video dataset. The Natural Image Quality Evaluator (NIQE) index by Bovik et al. with a threshold of 0.40 on input images improved the test accuracy of the average fusion model on the video dataset to 86.36% by removing the low quality input images before it reaches the CNN.;The thesis concludes that the recommended solution for the classification is composed of data augmentation and fusion of networks, while for application of trained models on videos, an image quality metric would aid in performance increase with a trade-off in loss of input data

Pooling Faces: Template based Face Recognition with Pooled Face Images

We propose a novel approach to template based face recognition. Our dual goal is to both increase recognition accuracy and reduce the computational and storage costs of template matching. To do this, we leverage on an approach which was proven effective in many other domains, but, to our knowledge, never fully explored for face images: average pooling of face photos. We show how (and why!) the space of a template's images can be partitioned and then pooled based on image quality and head pose and the effect this has on accuracy and template size. We perform extensive tests on the IJB-A and Janus CS2 template based face identification and verification benchmarks. These show that not only does our approach outperform published state of the art despite requiring far fewer cross template comparisons, but also, surprisingly, that image pooling performs on par with deep feature pooling.Comment: Appeared in the IEEE Computer Society Workshop on Biometrics, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June, 201

Resilient Perception for Outdoor Unmanned Ground Vehicles

This thesis promotes the development of resilience for perception systems with a focus on Unmanned Ground Vehicles (UGVs) in adverse environmental conditions. Perception is the interpretation of sensor data to produce a representation of the environment that is necessary for subsequent decision making. Long-term autonomy requires perception systems that correctly function in unusual but realistic conditions that will eventually occur during extended missions. State-of-the-art UGV systems can fail when the sensor data are beyond the operational capacity of the perception models. The key to resilient perception system lies in the use of multiple sensor modalities and the pre-selection of appropriate sensor data to minimise the chance of failure. This thesis proposes a framework based on diagnostic principles to evaluate and preselect sensor data prior to interpretation by the perception system. Image-based quality metrics are explored and evaluated experimentally using infrared (IR) and visual cameras onboard a UGV in the presence of smoke and airborne dust. A novel quality metric, Spatial Entropy (SE), is introduced and evaluated. The proposed framework is applied to a state-of-the-art Visual-SLAM algorithm combining visual and IR imaging as a real-world example. An extensive experimental evaluation demonstrates that the framework allows for camera-based localisation that is resilient to a range of low-visibility conditions when compared to other methods that use a single sensor or combine sensor data without selection. The proposed framework allows for a resilient localisation in adverse conditions using image data but also has significant potential to benefit many perception applications. Employing multiple sensing modalities along with pre-selection of appropriate data is a powerful method to create resilient perception systems by anticipating and mitigating errors. The development of such resilient perception systems is a requirement for next-generation outdoor UGVs

PEA265: Perceptual Assessment of Video Compression Artifacts

The most widely used video encoders share a common hybrid coding framework that includes block-based motion estimation/compensation and block-based transform coding. Despite their high coding efficiency, the encoded videos often exhibit visually annoying artifacts, denoted as Perceivable Encoding Artifacts (PEAs), which significantly degrade the visual Qualityof- Experience (QoE) of end users. To monitor and improve visual QoE, it is crucial to develop subjective and objective measures that can identify and quantify various types of PEAs. In this work, we make the first attempt to build a large-scale subjectlabelled database composed of H.265/HEVC compressed videos containing various PEAs. The database, namely the PEA265 database, includes 4 types of spatial PEAs (i.e. blurring, blocking, ringing and color bleeding) and 2 types of temporal PEAs (i.e. flickering and floating). Each containing at least 60,000 image or video patches with positive and negative labels. To objectively identify these PEAs, we train Convolutional Neural Networks (CNNs) using the PEA265 database. It appears that state-of-theart ResNeXt is capable of identifying each type of PEAs with high accuracy. Furthermore, we define PEA pattern and PEA intensity measures to quantify PEA levels of compressed video sequence. We believe that the PEA265 database and our findings will benefit the future development of video quality assessment methods and perceptually motivated video encoders.Comment: 10 pages,15 figures,4 table

New Datasets, Models, and Optimization

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2021.8. 손현태.사진 촬영의 궁극적인 목표는 고품질의 깨끗한 영상을 얻는 것이다. 현실적으로, 일상의 사진은 자주 흔들린 카메라와 움직이는 물체가 있는 동적 환경에서 찍는다. 노출시간 중의 카메라와 피사체간의 상대적인 움직임은 사진과 동영상에서 모션 블러를 일으키며 시각적인 화질을 저하시킨다. 동적 환경에서 블러의 세기와 움직임의 모양은 매 이미지마다, 그리고 매 픽셀마다 다르다. 국지적으로 변화하는 블러의 성질은 사진과 동영상에서의 모션 블러 제거를 심각하게 풀기 어려우며 해답이 하나로 정해지지 않은, 잘 정의되지 않은 문제로 만든다. 물리적인 움직임 모델링을 통해 해석적인 접근법을 설계하기보다는 머신러닝 기반의 접근법은 이러한 잘 정의되지 않은 문제를 푸는 보다 현실적인 답이 될 수 있다. 특히 딥 러닝은 최근 컴퓨터 비전 학계에서 표준적인 기법이 되어 가고 있다. 본 학위논문은 사진 및 비디오 디블러링 문제에 대해 딥 러닝 기반 솔루션을 도입하며 여러 현실적인 문제를 다각적으로 다룬다. 첫 번째로, 디블러링 문제를 다루기 위한 데이터셋을 취득하는 새로운 방법을 제안한다. 모션 블러가 있는 이미지와 깨끗한 이미지를 시간적으로 정렬된 상태로 동시에 취득하는 것은 쉬운 일이 아니다. 데이터가 부족한 경우 디블러링 알고리즘들을 평가하는 것 뿐만 아니라 지도학습 기법을 개발하는 것도 불가능해진다. 그러나 고속 비디오를 사용하여 카메라 영상 취득 파이프라인을 모방하면 실제적인 모션 블러 이미지를 합성하는 것이 가능하다. 기존의 블러 합성 기법들과 달리 제안하는 방법은 여러 움직이는 피사체들과 다양한 영상 깊이, 움직임 경계에서의 가리워짐 등으로 인한 자연스러운 국소적 블러의 복잡도를 반영할 수 있다. 두 번째로, 제안된 데이터셋에 기반하여 새로운 단일영상 디블러링을 위한 뉴럴 네트워크 구조를 제안한다. 최적화기법 기반 이미지 디블러링 방식에서 널리 쓰이고 있는 점차적 미세화 접근법을 반영하여 다중규모 뉴럴 네트워크를 설계한다. 제안된 다중규모 모델은 비슷한 복잡도를 가진 단일규모 모델들보다 높은 복원 정확도를 보인다. 세 번째로, 비디오 디블러링을 위한 순환 뉴럴 네트워크 모델 구조를 제안한다. 디블러링을 통해 고품질의 비디오를 얻기 위해서는 각 프레임간의 시간적인 정보와 프레임 내부적인 정보를 모두 사용해야 한다. 제안하는 내부프레임 반복적 연산구조는 두 정보를 효과적으로 함께 사용함으로써 모델 파라미터 수를 증가시키지 않고도 디블러 정확도를 향상시킨다. 마지막으로, 새로운 디블러링 모델들을 보다 잘 최적화하기 위해 로스 함수를 제안한다. 깨끗하고 또렷한 사진 한 장으로부터 자연스러운 모션 블러를 만들어내는 것은 블러를 제거하는 것과 마찬가지로 어려운 문제이다. 그러나 통상 사용하는 로스 함수로 얻은 디블러링 방법들은 블러를 완전히 제거하지 못하며 디블러된 이미지의 남아있는 블러로부터 원래의 블러를 재건할 수 있다. 제안하는 리블러링 로스 함수는 디블러링 수행시 모션 블러를 보다 잘 제거하도록 설계되었다. 이에 나아가 제안한 자기지도학습 과정으로부터 테스트시 모델이 새로운 데이터에 적응하도록 할 수 있다. 이렇게 제안된 데이터셋, 모델 구조, 그리고 로스 함수를 통해 딥 러닝에 기반하여 단일 영상 및 비디오 디블러링 기법들을 제안한다. 광범위한 실험 결과로부터 정량적 및 정성적으로 최첨단 디블러링 성과를 증명한다.Obtaining a high-quality clean image is the ultimate goal of photography. In practice, daily photography is often taken in dynamic environments with moving objects as well as shaken cameras. The relative motion between the camera and the objects during the exposure causes motion blur in images and videos, degrading the visual quality. The degree of blur strength and the shape of motion trajectory varies by every image and every pixel in dynamic environments. The locally-varying property makes the removal of motion blur in images and videos severely ill-posed. Rather than designing analytic solutions with physical modelings, using machine learning-based approaches can serve as a practical solution for such a highly ill-posed problem. Especially, deep-learning has been the recent standard in computer vision literature. This dissertation introduces deep learning-based solutions for image and video deblurring by tackling practical issues in various aspects. First, a new way of constructing the datasets for dynamic scene deblurring task is proposed. It is nontrivial to simultaneously obtain a pair of the blurry and the sharp image that are temporally aligned. The lack of data prevents the supervised learning techniques to be developed as well as the evaluation of deblurring algorithms. By mimicking the camera image pipeline with high-speed videos, realistic blurry images could be synthesized. In contrast to the previous blur synthesis methods, the proposed approach can reflect the natural complex local blur from and multiple moving objects, varying depth, and occlusion at motion boundaries. Second, based on the proposed datasets, a novel neural network architecture for single-image deblurring task is presented. Adopting the coarse-to-fine approach that is widely used in energy optimization-based methods for image deblurring, a multi-scale neural network architecture is derived. Compared with the single-scale model with similar complexity, the multi-scale model exhibits higher accuracy and faster speed. Third, a light-weight recurrent neural network model architecture for video deblurring is proposed. In order to obtain a high-quality video from deblurring, it is important to exploit the intrinsic information in the target frame as well as the temporal relation between the neighboring frames. Taking benefits from both sides, the proposed intra-frame iterative scheme applied to the RNNs achieves accuracy improvements without increasing the number of model parameters. Lastly, a novel loss function is proposed to better optimize the deblurring models. Estimating a dynamic blur for a clean and sharp image without given motion information is another ill-posed problem. While the goal of deblurring is to completely get rid of motion blur, conventional loss functions fail to train neural networks to fulfill the goal, leaving the trace of blur in the deblurred images. The proposed reblurring loss functions are designed to better eliminate the motion blur and to produce sharper images. Furthermore, the self-supervised learning process facilitates the adaptation of the deblurring model at test-time. With the proposed datasets, model architectures, and the loss functions, the deep learning-based single-image and video deblurring methods are presented. Extensive experimental results demonstrate the state-of-the-art performance both quantitatively and qualitatively.1 Introduction 1 2 Generating Datasets for Dynamic Scene Deblurring 7 2.1 Introduction 7 2.2 GOPRO dataset 9 2.3 REDS dataset 11 2.4 Conclusion 18 3 Deep Multi-Scale Convolutional Neural Networks for Single Image Deblurring 19 3.1 Introduction 19 3.1.1 Related Works 21 3.1.2 Kernel-Free Learning for Dynamic Scene Deblurring 23 3.2 Proposed Method 23 3.2.1 Model Architecture 23 3.2.2 Training 26 3.3 Experiments 29 3.3.1 Comparison on GOPRO Dataset 29 3.3.2 Comparison on Kohler Dataset 33 3.3.3 Comparison on Lai et al. [54] dataset 33 3.3.4 Comparison on Real Dynamic Scenes 34 3.3.5 Effect of Adversarial Loss 34 3.4 Conclusion 41 4 Intra-Frame Iterative RNNs for Video Deblurring 43 4.1 Introduction 43 4.2 Related Works 46 4.3 Proposed Method 50 4.3.1 Recurrent Video Deblurring Networks 51 4.3.2 Intra-Frame Iteration Model 52 4.3.3 Regularization by Stochastic Training 56 4.4 Experiments 58 4.4.1 Datasets 58 4.4.2 Implementation details 59 4.4.3 Comparisons on GOPRO [72] dataset 59 4.4.4 Comparisons on [97] Dataset and Real Videos 60 4.5 Conclusion 61 5 Learning Loss Functions for Image Deblurring 67 5.1 Introduction 67 5.2 Related Works 71 5.3 Proposed Method 73 5.3.1 Clean Images are Hard to Reblur 73 5.3.2 Supervision from Reblurring Loss 75 5.3.3 Test-time Adaptation by Self-Supervision 76 5.4 Experiments 78 5.4.1 Effect of Reblurring Loss 78 5.4.2 Effect of Sharpness Preservation Loss 80 5.4.3 Comparison with Other Perceptual Losses 81 5.4.4 Effect of Test-time Adaptation 81 5.4.5 Comparison with State-of-The-Art Methods 82 5.4.6 Real World Image Deblurring 85 5.4.7 Combining Reblurring Loss with Other Perceptual Losses 86 5.4.8 Perception vs. Distortion Trade-Off 87 5.4.9 Visual Comparison of Loss Function 88 5.4.10 Implementation Details 89 5.4.11 Determining Reblurring Module Size 94 5.5 Conclusion 95 6 Conclusion 97 국문 초록 115 감사의 글 117박