Exploiting Deep Features for Remote Sensing Image Retrieval: A Systematic Investigation

Remote sensing (RS) image retrieval is of great significant for geological information mining. Over the past two decades, a large amount of research on this task has been carried out, which mainly focuses on the following three core issues: feature extraction, similarity metric and relevance feedback. Due to the complexity and multiformity of ground objects in high-resolution remote sensing (HRRS) images, there is still room for improvement in the current retrieval approaches. In this paper, we analyze the three core issues of RS image retrieval and provide a comprehensive review on existing methods. Furthermore, for the goal to advance the state-of-the-art in HRRS image retrieval, we focus on the feature extraction issue and delve how to use powerful deep representations to address this task. We conduct systematic investigation on evaluating correlative factors that may affect the performance of deep features. By optimizing each factor, we acquire remarkable retrieval results on publicly available HRRS datasets. Finally, we explain the experimental phenomenon in detail and draw conclusions according to our analysis. Our work can serve as a guiding role for the research of content-based RS image retrieval

Aggregated Deep Local Features for Remote Sensing Image Retrieval

Remote Sensing Image Retrieval remains a challenging topic due to the special nature of Remote Sensing Imagery. Such images contain various different semantic objects, which clearly complicates the retrieval task. In this paper, we present an image retrieval pipeline that uses attentive, local convolutional features and aggregates them using the Vector of Locally Aggregated Descriptors (VLAD) to produce a global descriptor. We study various system parameters such as the multiplicative and additive attention mechanisms and descriptor dimensionality. We propose a query expansion method that requires no external inputs. Experiments demonstrate that even without training, the local convolutional features and global representation outperform other systems. After system tuning, we can achieve state-of-the-art or competitive results. Furthermore, we observe that our query expansion method increases overall system performance by about 3%, using only the top-three retrieved images. Finally, we show how dimensionality reduction produces compact descriptors with increased retrieval performance and fast retrieval computation times, e.g. 50% faster than the current systems.Comment: Published in Remote Sensing. The first two authors have equal contributio

Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

구글 포토 내 검색 기능을 통한 되찾기(Retrieval)의 학습을 중심으로

학위논문(석사)--서울대학교 대학원 :융합과학기술대학원 융합과학부(디지털정보융합전공),2019. 8. 이중식.The practices of photo retrieving on personal smartphones have extended. Not only pictures are browsed by scrolling up and down, but also a picture is easily come out as a result of typing a certain keyword. The technology of object recognition has changed how people look and browse personal photos; it not only classifies similar photos, but also assigns the labels that represent the referent of the classification. For instance, Google Photos has applied the object recognition and search system to allow users to manage personal photos. Eventually, the novel use of searching photos on personal album is expected to change the aspects of how people retrieve a particular photo out of thousands of accumulated ones in their cloud system. However, the novel technology is on its early stage and is at lack of leaving positive impression to the users. There is a gap between object recognition executed by the device and the user. When typing a query on its search bar for instance, the result is either none or countless number of results. The purpose of this study is to identify of the points of inconvenience in smartphone photo albums using object recognition and to deliver a better photo search user experience. In this paper, previous studies and preliminary research were thoroughly reviewed and held to fully grasp understanding of the inner workings of the object recognition, and to build a general frame of how people use the personal photo search system. In the main research, six photos search tasks were designed for a week with a total of 16 participants, aged between 20s and 30s. Search strategy tips were given to the experiment participants in order to collect particular strategy when searching photos. After collecting a total of 672 search tasks and used strategies, a post-questionnaire was followed. As a result of the analysis, the learning process of search system of the users has occurred. The study was able to identify how users learned the functions of photo search through strategy. As the number of 42 retrieval tasks, the average retrieval time of 16 users gradually decreased. As a result, the average retrieval time of the last day compared to the first day decreased by 31% from 51 seconds to 35 seconds. The average success rate of searches also rose by about 11% over 42 tasks performed each day for a week. The average number of search attempts of participants decreased by 28%. As the experience of photo retrieval is accumulated with the strategy provided, it is confirmed that the image of the learning and the improvement of the retrieval are improved in the photo retrieval using object recognition technology. In the learning style of individual users, 12 out of 16 participants participated in the appearance of learning, and 3 showed that learning did not take place. The other one showed no influence on learning. If learning is not or is not affected, it can be inferred that there is a difference in learning depending on which strategies are used in the initial search and how to adapt to the search function. Finally, 44.35% of the total search strategies were applied to the most commonly used strategies: 'use the correct name by using the high-level word (abstract concept) and the low-level word (concrete concept)'. Next was followed by using '(comma)', 'using search terms that appear on the screen like color', and 'using figures (women, men)'. The strategy that users individually formed was 12.20% of the total, and there was no difference in use over time. As for the strategies that the user made, 39.47% of the strategies were 'utilization of administrative area names', followed by 'automatic classification of people' and 'utilization of buildings'. Among the strategies that have been developed, 'Word usage tailored to Google Photos' has been found to recognize the characteristics of object recognition through experiences and form a search word by predicting it. In other words, as the experience of photo retrieval using object recognition accumulates, it shows the understanding of its characteristics. Through the above analysis, the study has examined the point where object recognition technology is difficult for user when it is used as a search in the smartphone photo album, and added a brief suggestion on how to supplement it. This study approached the user 's difficulty in using the object recognition technology as applied to the smartphone photo album. In addition, HCI (Human Computer Interaction) side has focused on the process of how the strategy made through the viewpoint of the device is accepted and transformed by the user. In addition, it is meaningful that the study tried to observe the interaction that occurred when the research of object recognition, which was concentrated only on improving the accuracy of recognition, was provided to actual users. Finally, it is meaningful that the study discussed the ways to utilize object recognition in order to utilize the medium of photography and sustainable use.지난 봄에 촬영한 벚꽃 사진이 찾고 싶을 때, 스마트폰의 사진첩에서 벚꽃을 검색해본다. 뒤이어 앨범 안에 있는 모든 벚꽃 사진이 나타난다. 이처럼, 스마트폰으로 촬영한 나의 사진도 이제 키워드를 통해 검색이 가능하다. 과거에는 앨범을 만들어서 관리하거나 수많은 사진의 홍수 속에서 스크롤 내려가며 브라우징했다면, 이제는 머리 속에 떠오르는 검색어를 활용해 원하는 사진을 찾을 수 있는 것이다. 이는 컴퓨터 비전 분야에서 객체 인식 기술이 비약적인 발전을 이루고, 비슷한 특징을 가지는 사진을 묶어낸 후 적합한 이름을 붙이는 데까지 가능해진 덕분이다. 그러나, 검색을 활용해보면 그 결과의 범위가 너무 넓어서 검색의 효과가 없거나, 사용자가 활용한 검색어에 부합하는 결과가 나오지 않는 경우도 있다. 거기에 정확지 않은 결과가 나타날 때도 있어 사진 검색을 활용하는 초기의 사용자에게 부정적인 경험을 남기게된다. 이에 본 연구는 객체 인식을 활용한 스마트폰 사진첩에서 나타나는 불편함의 지점을 밝혀내고, 사진 검색 경험을 개선하고자 하는 목적을 갖고 시작되었다. 객체 인식이 스마트폰 사진 검색과 같은 실생활에 적용되었을 때 나타나는 이러한 어색한 결과는 사진이라는 대상의 특성과 기기의 내부적 특성, 이로 인해 발생하는 기기와 사람 사이의 인식 차이로 인해 발생한다. 본문에서는 객체 인식의 내부적 특성을 문헌 연구와 사전 조사를 통해 수집한 후, 검색을 보다 효율적으로 할 수 있는 전략으로 일반화 하였다. 이후 사용자에게 사진 검색의 경험 중 이러한 특성을 검색의 전략으로 활용할 수 있도록 팁(Tip)을 작성한 후 이를 실험의 참여자에게 제공하였다. 이후 16 명의 20-30 대를 대상으로 일주일동안 매일 여섯번의 사진 찾기 과업을 수행하는 실험을 통해 연구 자료를 수집하였다. 이를 통해 총 672개의 검색 과업의 기록과 사용된 전략, 그리고 사용자가 형성한 전략을 수집했으며, 일주일간의 실험이 완료된 후에는 사후 설문 데이터를 얻을 수 있었다. 분석의 결과로, 먼저 전략을 통해 사용자가 사진 검색의 기능을 학습하는 모습을 확인할 수 있었다. 42 회의 검색 태스크가 누적됨에 따라 16 명의 평균 검색 소요 시간은 점차 감소하였으며, 이를 시간의 흐름에 따른 변화의 축에서 살펴보면 첫 번째 날에 비해 마지막 날의 평균 검색 소요 시간이 51 초에서 35 초로 31% 가량 감소했다. 검색의 평균 성공율 또한 일주일 동안 매일 진행된 42 회의 태스크에 걸져 약 11% 상승하는 모습을 보였다. 실험 참여자의 평균 검색 시도 횟수도 28% 감소하였다. 이를 통해 제공된 전략과 함께 사진 검색의 경험이 누적될수록 객체 인식 기술을 활용한 사진 검색에서 학습의 모습과 검색의 개선이 나타남을 확인할 수 있었다. 개별 사용자의 학습 형태에서는, 16명 중 12명의 참여자가 학습의 모습이 나타난 경우에 해당되었으며, 3 명은 학습이 이뤄지지 않은 모습을 보였다. 나머지 1 명은 학습에 영향을 받지 않는 모습을 나타내었다. 학습이 이뤄지지 않거나 영향을 받지 않는 경우, 초반의 검색에서 어떤 전략을 활용하고 검색 기능에 적응하는지에 따라 학습에서의 차이가 생기는 것으로 유추해볼 수 있었다. 마지막으로 제공된 전략 중 가장 많이 사용된 전략은 상위어(상위 개념의 단어), 하위어(구체적인 단어)를 활용하여 정확한 명칭 사용하기로 전체 검색의 44.35%이 해당 전략을 활용한 것으로 나타났다. 그 다음으로는 ,(콤마) 사용하기, 색상과 같이 화면에서 드러나는 검색어 활용하기, 인물(여자, 남자) 활용하기가 뒤를 이었다. 사용자가 개별적으로 형성한 전략은 전체의 12.20%로 나타났으며, 시간이 흐름에 따라 그 사용의 차이는 나타나지 않았다. 사용자가 직접 제작한 전략으로는 행정구역명의 활용이 개별 전략의 39.47%를 차지했으며, 그 다음으로는 자동 분류된 인물 활용, 그리고 건축물의 활용이 주로 활용되었다. 형성된 전략 중 구글 포토에 맞춘 단어 활용은 과거의 경험을 통해 객체 인식의 특성을 인지하고, 이를 예측하여 검색어를 형성하는 것으로 나타났다. 즉, 객체 인식을 활용한 사진 검색의 경험을 누적할수록 그 특성에 대한 이해가 생기는 모습을 보여준다. 이상의 분석을 통해 객체 인식 기술이 스마트폰 사진첩 내에서 검색으로 활용될 때 사용자에게 어려움을 가져오는 지점에 대해 살펴본 후, 이를 보완할 수 있는 방안에 대한 간략한 제언을 덧붙였다. 본 연구는 객체 인식 기술이 스마트폰 사진첩에 적용되면서 나타난 사용의 어려움을 사용자의 관점에서 접근하였다. 더하여 HCI(Human Computer Interaction)의 측변에서, 기기의 관점을 통해 제작된 전략이 어떻게 사용자에게 수용되고 변형되는지 그 과정에 집중했다는데 그 융합적 의의를 가진다. 더하여 인식의 정확도를 높이는데에만 집중되었던 객체 인식의 연구가 실제 사용자에게 제공되었을 때 발생하는 인터랙션을 실험을 통해 관찰을 시도했다는데 의의를 갖는다. 마지막으로는 사진이라는 매체의 활용성과 지속 가능한 활용을 위해 객체 인식을 활용할 수 있는 방식에 대한 논의를 펼쳤다는 데 의의가 있다.제 1 장 서론 1 제 1 절 연구의 배경 1 제 2 절 연구의 목적 2 제 2 장 관련 연구 4 제 1 절 이미지 검색에서 객체 인식의 활용 8 제 2 절 기기와 인간의 관점 차이 11 제 3 절 사진의 특성과 스마트폰 사진 찾기 14 제 3 장 연구 문제 19 제 1 절 연구 문제의 설정 19 제 2 절 측정을 위한 개념 정의 21 제 4 장 연구 방법 24 제 1 절 연구 방법 24 제 2 절 실험 방법 25 제 5 장 연구 결과 41 제 1 절 객체 인식을 활용한 사진 찾기의 학습 41 제 2 절 사용자 별 학습의 형태와 그 특성 48 제 3 절 제공된 전략의 활용과 변형 52 제 6 장 연구 논의 63 제 1 절 연구에서 나타난 기기와 인간의 차이 63 제 7 장 결론 및 연구의 의의 67 제 1 절 연구의 요약 67 제 2 절 연구의 한계 및 제언 70 제 3 절 연구의 의의 72 참고문헌 74 Abstract 78Maste

Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval

Learning powerful feature representations for image retrieval has always been a challenging task in the field of remote sensing. Traditional methods focus on extracting low-level hand-crafted features which are not only time-consuming but also tend to achieve unsatisfactory performance due to the complexity of remote sensing images. In this paper, we investigate how to extract deep feature representations based on convolutional neural networks (CNNs) for high-resolution remote sensing image retrieval (HRRSIR). To this end, several effective schemes are proposed to generate powerful feature representations for HRRSIR. In the first scheme, a CNN pre-trained on a different problem is treated as a feature extractor since there are no sufficiently-sized remote sensing datasets to train a CNN from scratch. In the second scheme, we investigate learning features that are specific to our problem by first fine-tuning the pre-trained CNN on a remote sensing dataset and then proposing a novel CNN architecture based on convolutional layers and a three-layer perceptron. The novel CNN has fewer parameters than the pre-trained and fine-tuned CNNs and can learn low dimensional features from limited labelled images. The schemes are evaluated on several challenging, publicly available datasets. The results indicate that the proposed schemes, particularly the novel CNN, achieve state-of-the-art performance

Toward Global Localization of Unmanned Aircraft Systems using Overhead Image Registration with Deep Learning Convolutional Neural Networks

Global localization, in which an unmanned aircraft system (UAS) estimates its unknown current location without access to its take-off location or other locational data from its flight path, is a challenging problem. This research brings together aspects from the remote sensing, geoinformatics, and machine learning disciplines by framing the global localization problem as a geospatial image registration problem in which overhead aerial and satellite imagery serve as a proxy for UAS imagery. A literature review is conducted covering the use of deep learning convolutional neural networks (DLCNN) with global localization and other related geospatial imagery applications. Differences between geospatial imagery taken from the overhead perspective and terrestrial imagery are discussed, as well as difficulties in using geospatial overhead imagery for image registration due to a lack of suitable machine learning datasets. Geospatial analysis is conducted to identify suitable areas for future UAS imagery collection. One of these areas, Jerusalem northeast (JNE) is selected as the area of interest (AOI) for this research. Multi-modal, multi-temporal, and multi-resolution geospatial overhead imagery is aggregated from a variety of publicly available sources and processed to create a controlled image dataset called Jerusalem northeast rural controlled imagery (JNE RCI). JNE RCI is tested with handcrafted feature-based methods SURF and SIFT and a non-handcrafted feature-based pre-trained fine-tuned VGG-16 DLCNN on coarse-grained image registration. Both handcrafted and non-handcrafted feature based methods had difficulty with the coarse-grained registration process. The format of JNE RCI is determined to be unsuitable for the coarse-grained registration process with DLCNNs and the process to create a new supervised machine learning dataset, Jerusalem northeast machine learning (JNE ML) is covered in detail. A multi-resolution grid based approach is used, where each grid cell ID is treated as the supervised training label for that respective resolution. Pre-trained fine-tuned VGG-16 DLCNNs, two custom architecture two-channel DLCNNs, and a custom chain DLCNN are trained on JNE ML for each spatial resolution of subimages in the dataset. All DLCNNs used could more accurately coarsely register the JNE ML subimages compared to the pre-trained fine-tuned VGG-16 DLCNN on JNE RCI. This shows the process for creating JNE ML is valid and is suitable for using machine learning with the coarse-grained registration problem. All custom architecture two-channel DLCNNs and the custom chain DLCNN were able to more accurately coarsely register the JNE ML subimages compared to the fine-tuned pre-trained VGG-16 approach. Both the two-channel custom DLCNNs and the chain DLCNN were able to generalize well to new imagery that these networks had not previously trained on. Through the contributions of this research, a foundation is laid for future work to be conducted on the UAS global localization problem within the rural forested JNE AOI