A Study on Image Registration between High Resolution Optical and SAR Images Using SAR-SIFT and DLSS

학위논문 (석사)-- 서울대학교 대학원 : 공과대학 건설환경공학부, 2018. 8. 김용일.최근 위성센서 기술의 발달로 다양한 센서를 탑재한 지구관측위성이 발사되면서, 다중센서 위성영상을 융합 분석하는 연구가 활발히 진행되고 있다. 특히, 광학영상과 SAR영상은 취급하는 파장대가 달라 동시에 활용할 경우 지표면에 대해 보다 구체적인 정보를 취득할 수 있으며, 더 나아가, 객체 추출, 변화탐지, 재난재해 모니터링 등 원격탐사 분야에 폭넓게 적용이 가능하다. 이를 위해서는 전처리 작업으로 두 영상 간 정합이 필수적으로 이루어져야 한다. 그러나, 광학영상과 SAR영상은 영상 취득시 위성센서 자세 및 취급하는 파장대의 상이함으로 기하 및 분광 정보 차이를 유발하여 영상 정합에 있어 유독 어려움이 존재한다. 이러한 차이는 건물이 밀집된 도심지역에서 부각되며, 중·저해상도 영상보다 고해상도 영상에서 두드러진다. 따라서, 본 연구에서는 도심지역에 대한 고해상도 광학영상과 SAR영상 간 정합에 효과적인 방법론을 제안하였다. 기존 광학영상과 SAR영상 간 정합 관련 연구는 크게 특징기반 정합기법과 강도기반 정합기법으로 진행되었다. 강도기반 정합기법은 분광 특성이 다른 영상 간 정합에 효과적이나, 영상 간 왜곡이 존재하지 않거나 기하학적 위치 차이가 적을 때에만 적용 가능하다. 고해상도 광학영상과 SAR영상은 지역적 왜곡이 존재하며, 두 영상 간 수십m 이상의 기하학적 위치 차이가 발생할 수 있다. 따라서, 고해상도 광학영상과 SAR영상 간 정합 연구는 강도기반 정합기법 보다 특징기반 정합기법이 중점적으로 진행되고 있다. 그러나, 특징기반 정합기법은 분광 특성이 다른 광학영상과 SAR영상에서 오정합쌍을 다수 추출하여 정합 성능이 떨어진다. 이를 해결하기 위해, 강도기반 정합기법과 특징기반 정합기법을 결합한 기법들이 제안되었으나, 도심지역에서 원 형상이 존재하는 지역이나 건물밀집지역을 제외한 지역 등과 같이 제한된 지역에서만 적용 가능하다는 한계점을 보였다. 이를 개선하기 위해, 본 연구에서는 특징기반 정합기법인 SAR-SIFT 기법과 강도기반 정합기법인 DLSS 기법을 결합한 정합기법을 제안하였다. 또한, 정합쌍을 추출하기 위해, 전처리 단계, 후보 정합쌍 추출 단계, 정밀 정합쌍 추출 단계인 총 세 단계를 추가하였다. 고해상도 광학영상과 SAR영상 간 정합을 위해서, SAR-SIFT 기법을 이용하여 특징점을 추출하고, 추출된 특징점에서 DLSS 기법을 이용하여 정합쌍을 추출하였다. 그러나, 추출된 정합쌍에 다수의 오정합쌍이 포함되는 문제점이 존재하였다. 이를 해결하기 위해, 추출된 정합쌍에서 임계치와 특징점 간 변위량을 이용한 전처리 단계와 후보 정합쌍 추출 단계를 통해 후보 정합쌍을 추출하고, 후보 정합쌍에 RANSAC 기법을 적용하여 정밀 정합쌍을 추출하는 방법을 제안하였다. 최종적으로 추출된 정밀 정합쌍을 이용하여 어핀 변환식(affine transformation)을 구성하고, 이를 적용하여 광학영상에 정합된 SAR영상을 생성하였다. 본 연구의 정확도를 검증하기 위하여, 대표적인 고해상도 광학영상인 KOMPSAT-2영상과 고해상도 SAR영상인 TerraSAR-X, Cosmo-SkyMed영상을 사용하였고, 시각적, 정량적 평가를 진행하였다. 시각적 평가를 위해서 모자이크 영상을 생성하였으며, 두 영상 간 경계에서 객체의 형상이 유지됨을 통해 정합이 우수하게 수행됨을 확인하였다. 정량적 평가를 위해서 수동 검사점을 통한 RMSE Ⅰ과 교차검증을 통한 RMSE Ⅱ를 사용하였으며, 모든 실험지역에 대해 RMSE Ⅰ은 1.51m에서 2.04m, RMSE Ⅱ는 1.34m에서 1.69m로 정확도가 도출되었다. 이는, 선행연구결과와 비교하였을 때 우수한 수준의 정확도로 확인되었다. 이를 통해, 제안 기법이 고해상도 광학영상과 SAR영상 간 정합에 효과적이며, 두 영상 간 융합 분석을 위해 효과적인 정합 기술로 활용될 것으로 사료된다.1. 서 론 1 1.1 연구배경 1 1.2 연구동향 4 1.3 연구의 목적 및 범위 7 2. 특징점 추출 10 2.1 영상 전처리 10 2.2 SAR-SIFT 기법을 통한 특징점 추출 11 2.2.1. SIFT 기법의 문제점 11 2.2.2. SAR-SIFT 기법 15 3. 정합쌍 추출 18 3.1 DLSS 기법을 통한 정합쌍 추출 18 3.1.1. 형상 서술자 LSS 19 3.1.2. 형상 서술자 벡터 DLSS 21 3.1.3. DLSS 기법의 문제점 22 3.2 제안된 정합쌍 추출 방법 24 3.2.1. 전처리 단계 24 3.2.2. 후보 정합쌍 추출 단계 26 3.2.3. 정밀 정합쌍 추출 단계 28 3.3 정합 및 정확도 평가 방법 29 3.3.1. 어핀 변환식 29 3.3.2. 정확도 평가 방법 31 4. 실험의 적용 및 평가 32 4.1 실험지역 및 자료 32 4.2 특징점 추출 결과 35 4.2.1. SIFT 기법을 통한 특징점 추출 결과 35 4.2.2. SAR-SIFT 기법을 통한 특징점 추출 결과 37 4.3 정합쌍 추출 결과 40 4.3.1. 기존 기법을 통한 정합쌍 추출 결과 40 4.3.2. 제안 기법을 통한 정합쌍 추출 결과 44 4.4 정합 결과 및 평가 49 5. 결론 55 Abstract 67Maste

STELLAR: A LARGE SATELLITE STEREO DATASET FOR DIGITAL SURFACE MODEL GENERATION

Stellar is a large, satellite stereo dataset. It contains rectified stereo pairs of the terrain captured by the satellite image sensors and corresponding true disparity maps and semantic segmentation. Unlike stereo vision in autonomous driving and mobile imaging, a satellite stereo pair is not captured simultaneously. Thus, the same object in a satellite stereo pair is more likely to have a varied visual appearance. Stellar provides flexible access to such stereo pairs to train methods to be robust to such appearance variation. We use publicly available data sources, and invented several techniques to perform data registration, rectification, and semantic segmentation on the data to build Stellar. In our preliminary experiment, we fine-tuned two deep-learning stereo methods on Stellar. The result demonstrates that most of the time, these methods generate denser and more accurate disparity maps for satellite stereo by fine-tuning on Stellar, compared to without fine-tuning on satellite stereo datasets, or fine-tuning on previous, smaller satellite stereo datasets. Stellar is available to download at https://github.com/guo-research-group/Stellar

Robust Modular Feature-Based Terrain-Aided Visual Navigation and Mapping

The visual feature-based Terrain-Aided Navigation (TAN) system presented in this thesis addresses the problem of constraining inertial drift introduced into the location estimate of Unmanned Aerial Vehicles (UAVs) in GPS-denied environment. The presented TAN system utilises salient visual features representing semantic or human-interpretable objects (roads, forest and water boundaries) from onboard aerial imagery and associates them to a database of reference features created a-priori, through application of the same feature detection algorithms to satellite imagery. Correlation of the detected features with the reference features via a series of the robust data association steps allows a localisation solution to be achieved with a finite absolute bound precision defined by the certainty of the reference dataset. The feature-based Visual Navigation System (VNS) presented in this thesis was originally developed for a navigation application using simulated multi-year satellite image datasets. The extension of the system application into the mapping domain, in turn, has been based on the real (not simulated) flight data and imagery. In the mapping study the full potential of the system, being a versatile tool for enhancing the accuracy of the information derived from the aerial imagery has been demonstrated. Not only have the visual features, such as road networks, shorelines and water bodies, been used to obtain a position ’fix’, they have also been used in reverse for accurate mapping of vehicles detected on the roads into an inertial space with improved precision. Combined correction of the geo-coding errors and improved aircraft localisation formed a robust solution to the defense mapping application. A system of the proposed design will provide a complete independent navigation solution to an autonomous UAV and additionally give it object tracking capability

Nonparametric image registration of airborne LiDAR, hyperspectral and photographic imagery of wooded landscapes

There is much current interest in using multisensor airborne remote sensing to monitor the structure and biodiversity of woodlands. This paper addresses the application of nonparametric (NP) image-registration techniques to precisely align images obtained from multisensor imaging, which is critical for the successful identification of individual trees using object recognition approaches. NP image registration, in particular, the technique of optimizing an objective function, containing similarity and regularization terms, provides a flexible approach for image registration. Here, we develop a NP registration approach, in which a normalized gradient field is used to quantify similarity, and curvature is used for regularization (NGF-Curv method). Using a survey of woodlands in southern Spain as an example, we show that NGF-Curv can be successful at fusing data sets when there is little prior knowledge about how the data sets are interrelated (i.e., in the absence of ground control points). The validity of NGF-Curv in airborne remote sensing is demonstrated by a series of experiments. We show that NGF-Curv is capable of aligning images precisely, making it a valuable component of algorithms designed to identify objects, such as trees, within multisensor data sets.This work was supported by the Airborne Research and Survey Facility of the U.K.’s Natural Environment Research Council (NERC) for collecting and preprocessing the data used in this research project [EU11/03/100], and by the grants supported from King Abdullah University of Science Technology and Wellcome Trust (BBSRC). D. Coomes was supported by a grant from NERC (NE/K016377/1) and funding from DEFRA and the BBSRC to develop methods for monitoring ash dieback from aircraft.This is the final version. It was first published by IEEE at http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7116541&sortType%3Dasc_p_Sequence%26filter%3DAND%28p_Publication_Number%3A36%29%26pageNumber%3D5

Automated Ortho-Rectification of UAV-Based Hyperspectral Data over an Agricultural Field Using Frame RGB Imagery

Low-cost Unmanned Airborne Vehicles (UAVs) equipped with consumer-grade imaging systems have emerged as a potential remote sensing platform that could satisfy the needs of a wide range of civilian applications. Among these applications, UAV-based agricultural mapping and monitoring have attracted significant attention from both the research and professional communities. The interest in UAV-based remote sensing for agricultural management is motivated by the need to maximize crop yield. Remote sensing-based crop yield prediction and estimation are primarily based on imaging systems with different spectral coverage and resolution (e.g., RGB and hyperspectral imaging systems). Due to the data volume, RGB imaging is based on frame cameras, while hyperspectral sensors are primarily push-broom scanners. To cope with the limited endurance and payload constraints of low-cost UAVs, the agricultural research and professional communities have to rely on consumer-grade and light-weight sensors. However, the geometric fidelity of derived information from push-broom hyperspectral scanners is quite sensitive to the available position and orientation established through a direct geo-referencing unit onboard the imaging platform (i.e., an integrated Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS). This paper presents an automated framework for the integration of frame RGB images, push-broom hyperspectral scanner data and consumer-grade GNSS/INS navigation data for accurate geometric rectification of the hyperspectral scenes. The approach relies on utilizing the navigation data, together with a modified Speeded-Up Robust Feature (SURF) detector and descriptor, for automating the identification of conjugate features in the RGB and hyperspectral imagery. The SURF modification takes into consideration the available direct geo-referencing information to improve the reliability of the matching procedure in the presence of repetitive texture within a mechanized agricultural field. Identified features are then used to improve the geometric fidelity of the previously ortho-rectified hyperspectral data. Experimental results from two real datasets show that the geometric rectification of the hyperspectral data was improved by almost one order of magnitude

Advances in Image Processing, Analysis and Recognition Technology

For many decades, researchers have been trying to make computers’ analysis of images as effective as the system of human vision is. For this purpose, many algorithms and systems have previously been created. The whole process covers various stages, including image processing, representation and recognition. The results of this work can be applied to many computer-assisted areas of everyday life. They improve particular activities and provide handy tools, which are sometimes only for entertainment, but quite often, they significantly increase our safety. In fact, the practical implementation of image processing algorithms is particularly wide. Moreover, the rapid growth of computational complexity and computer efficiency has allowed for the development of more sophisticated and effective algorithms and tools. Although significant progress has been made so far, many issues still remain, resulting in the need for the development of novel approaches