Dominant Feature Pooling for Multi Camera Object Detection and Optimization of Retinex Algorithm

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2021.8. 이혁재.본 논문은 멀티 카메라 object detection CNN을 위한 detection 단계에서 활용하는 새로운 dominant feature pooling 방법을 제안한다. 멀티 카메라 시스템은 다양한 관점에서 물체의 이미지를 캡처하고, 물체의 더 많은 주요 feature를 detection에 활용할 수 있다. 따라서 여러 카메라에서 feature를 pooling하면 detection 정확도를 향상시킬 수 있다. 제안된 방법은 객체의 다양한 뷰포인트에서 얻은 feature vector 중에서 더 많은 정보를 제공하는 주요 feature을 선택하고 선택한 feature vector를 pooling하여 새로운 feature map을 구성한다. 제안된 방법은 단일 카메라에 대한 YOLOv3 네트워크를 기반으로 하며, 멀티 카메라 시스템에 대한 추가 학습 과정이 필요하지 않다. Dominant feature pooling의 효과를 주장하기 위해, 이 연구에서는 feature vector를 시각화하는 새로운 방법도 제안된다. 또한 object detection CNN은 저조도 환경에 대응이 취약하므로 이를 개선할 수 있는 Retinex 알고리즘의 활용 방법을 제안한다. 저조도 영상을 그대로 학습하여 개선을 할 수 있지만, 실 사용 환경에서 조도 정도를 예측할 수 없기 때문에 Retinex 개선이 필수적임을 실험을 통해 나타내었다. 또한 개선 효과가 뚜렷하지만 복잡도가 높은 Retinex 알고리즘을 HW 설계를 통해 최적화 하는 방법을 제안한다. Retinex 알고리즘 연산에 필수적인 exponentiation과 Gaussian filtering을 효율적으로 구현하는 방법을 제안하여 높은 해상도에서도 실시간으로 동작이 가능한 HW를 구현하였다.This paper proposes a novel dominant feature pooling method utilized in the detection phase for multi-camera object detection CNNs. Multi-camera systems can capture images of objects from various perspectives and utilize more of the important features of objects for detection. Thus, the detection accuracy can be improved by pooling the features of the multiple cameras. The proposed method constructs a new feature patch by selecting and pooling the dominant features that provides more information among the feature vectors obtained from various viewpoints of objects. The proposed method is based on the YOLOv3 network for a single camera, and does not require additional learning processes for multi-camera systems. To show the effectiveness of dominant feature pooling, a novel method of visualizing feature vectors is also proposed in this work. Furthermore, a method of utilizing Retinex algorithms that can improve response to low-light environments for object detection CNN is proposed. Although improvements can be made by learning low-light images as they are, experimental results show that Retinex improvements are essential because the degree of illumination cannot be predicted accurately to create new datasets in practical environments. This work proposes a method to optimize Retinex algorithms through HW designs. An efficient implementation of the exponentiation operation and the Gaussian filtering, which are essential for Retinex algorithm operations is proposed to implement HW that can operate in real time at high resolution.제 1 장 서 론 1 1.1 연구 배경 1 1.2 연구 내용 2 1.3 논문 구성 4 제 2 장 배경 이론 및 관련 연구 5 2.1 Object Detection CNN 5 2.2 Multi View CNN 6 2.3 Retinex 알고리즘 7 2.3.1 Retinex Algorithm using Gaussian Filter 8 2.3.2 Multiscale Retinex Algorithm 9 2.3.3 Efficient Naturalness Restoration 10 제 3 장 무인 판매대 시스템 12 3.1 무인 판매대 시스템 개요 12 3.2 Object Detection CNN을 활용한 상품 인식 16 3.3 Multi-Object Tracking을 활용한 상품 구매 판단 18 3.4 무인 판매대의 실시간 동작을 위한 최적화 방안 20 3.4.1 카메라 선택 알고리즘 20 3.4.2 Multithreading 24 3.4.3 Pruning 25 3.5 무인 판매대 시스템 성능 평가 27 3.5.1 Object Detection 성능 평가 27 3.5.2 무인 판매대 시스템 전체 결과 29 제 4 장 멀티 카메라 Dominant Feature Pooling 32 4.1 Object Detection CNN과 멀티 카메라 Object Clustering 33 4.1.1 Object Detection CNN 33 4.1.2 멀티 카메라 Object Clustring 35 4.2 Dominant Feature Pooling 방법 37 4.2.1 Dominant Feature Scoring 40 4.2.2 Dominant Feature Pooling 47 4.2.3 YOLOv3의 Detection Layer 재사용 50 4.3 Feature 시각화를 통한 제안 방법 분석 52 4.3.1 제안하는 Feature 시각화 방법 52 4.3.2 기존 단일 카메라 YOLOv3의 Feature 시각화 55 4.3.3 제안하는 방법의 멀티카메라 Feature 시각화 57 4.4 Dominant Feature Pooling 결과 및 분석 59 4.4.1 COCO Dataset에서의 결과 60 4.4.2 Custom Dataset에서의 결과 62 4.4.3 Scoring Method 별 결과 63 4.4.3 Dominant Feature Pooling의 수행시간 결과 64 제 5 장 Retinex Applied Object Detection 및 하드웨서 가속시스템 65 5.1 기존 Retinex 적용 연구 66 5.2 Retinex Applied Object Detection 68 5.2.1 Retinex Applied Object Detection 학습 68 5.2.2 Retinex Applied Object Detection 결과 72 5.3 Object Detection을 위한 Retinex 최적화 76 5.3.1 Gaussian Filter 크기에 따른 Retinex 효과 분석 76 5.3.2 Gaussain Filter 크기에 따른 Object Detection 결과 80 5.4 Retinex 하드웨어 시스템의 필요성 및 기존 연구 82 5.5 제안 하드웨어 시스템 구현 개요 85 5.6 제안 하드웨어 시스템 구현 특장점 89 5.6.1 Gaussian filter의 구현 89 5.6.2 Exponentiation의 구현 96 5.6.3 HDMI/DVI 지원 및 영상 latency 최소화 103 5.7 제안 하드웨어 시스템 구현 결과 및 분석 106 5.7.1 실시간 동작 및 낮은 latency에 대한 분석 106 5.7.2 제안한 시스템의 영상 처리 성능 결과 분석 109 5.7.3 제안한 시스템의 FPGA Resource Utilization 112 5.7.4 다른 시스템과의 Resource Utilization 비교 114 5.7.5 제안한 시스템의 영상 처리 성능 결과 분석 119 제 6 장 결론 120 참고문헌 121 Abstract 131박

A flexible FPGA implementation for illuminance–reflectance video enhancement

2Abstract: Digital cameras, new generation phones, commercial TV sets and, in general, all modern devices for image acquisition and visualization can benefit from algorithms for image enhancement suitable to work in real time and preferably with limited power consumption. Among the various methods described in the scientific literature, Retinex-based approaches are able to provide very good performances, but unfortunately they typically require a high computational effort. In this article, we propose a flexible and effective architecture for the real-time enhancement of video frames, suitable to be implemented in a single FPGA device. The video enhancement algorithm is based on a modified version of the Retinex approach. This method, developed to control the dynamic range of poorly illuminated images while preserving the visual details, has been improved by the adoption of a new model to perform illuminance estimation. The video enhancement parameters are controlled in real time through an embedded microprocessor which makes the system able to modify its behavior according to the characteristics of the input images, and using information about the surrounding light conditions.nonemixedMarsi S.;Ramponi G.Marsi, Stefano; Ramponi, Giovann

Towards the development of flexible, reliable, reconfigurable, and high-performance imaging systems

Current FPGAs can implement large systems because of the high density of reconfigurable logic resources in a single chip. FPGAs are comprehensive devices that combine flexibility and high performance in the same platform compared to other platform such as General-Purpose Processors (GPPs) and Application Specific Integrated Circuits (ASICs). The flexibility of modern FPGAs is further enhanced by introducing Dynamic Partial Reconfiguration (DPR) feature, which allows for changing the functionality of part of the system while other parts are functioning. FPGAs became an important platform for digital image processing applications because of the aforementioned features. They can fulfil the need of efficient and flexible platforms that execute imaging tasks efficiently as well as the reliably with low power, high performance and high flexibility. The use of FPGAs as accelerators for image processing outperforms most of the current solutions. Current FPGA solutions can to load part of the imaging application that needs high computational power on dedicated reconfigurable hardware accelerators while other parts are working on the traditional solution to increase the system performance. Moreover, the use of the DPR feature enhances the flexibility of image processing further by swapping accelerators in and out at run-time. The use of fault mitigation techniques in FPGAs enables imaging applications to operate in harsh environments following the fact that FPGAs are sensitive to radiation and extreme conditions. The aim of this thesis is to present a platform for efficient implementations of imaging tasks. The research uses FPGAs as the key component of this platform and uses the concept of DPR to increase the performance, flexibility, to reduce the power dissipation and to expand the cycle of possible imaging applications. In this context, it proposes the use of FPGAs to accelerate the Image Processing Pipeline (IPP) stages, the core part of most imaging devices. The thesis has a number of novel concepts. The first novel concept is the use of FPGA hardware environment and DPR feature to increase the parallelism and achieve high flexibility. The concept also increases the performance and reduces the power consumption and area utilisation. Based on this concept, the following implementations are presented in this thesis: An implementation of Adams Hamilton Demosaicing algorithm for camera colour interpolation, which exploits the FPGA parallelism to outperform other equivalents. In addition, an implementation of Automatic White Balance (AWB), another IPP stage that employs DPR feature to prove the mentioned novelty aspects. Another novel concept in this thesis is presented in chapter 6, which uses DPR feature to develop a novel flexible imaging system that requires less logic and can be implemented in small FPGAs. The system can be employed as a template for any imaging application with no limitation. Moreover, discussed in this thesis is a novel reliable version of the imaging system that adopts novel techniques including scrubbing, Built-In Self Test (BIST), and Triple Modular Redundancy (TMR) to detect and correct errors using the Internal Configuration Access Port (ICAP) primitive. These techniques exploit the datapath-based nature of the implemented imaging system to improve the system's overall reliability. The thesis presents a proposal for integrating the imaging system with the Robust Reliable Reconfigurable Real-Time Heterogeneous Operating System (R4THOS) to get the best out of the system. The proposal shows the suitability of the proposed DPR imaging system to be used as part of the core system of autonomous cars because of its unbounded flexibility. These novel works are presented in a number of publications as shown in section 1.3 later in this thesis