FPGA acceleration of a quantized neural network for remote-sensed cloud detection

The capture and transmission of remote-sensed imagery for Earth observation is both computationally and bandwidth expensive. In the analyses of remote-sensed imagery in the visual band, atmospheric cloud cover can obstruct up to two-thirds of observations, resulting in costly imagery being discarded. Mission objectives and satellite operational details vary; however, assuming a cloud-free observation requirement, a doubling of useful data downlinked with an associated halving of delivery cost is possible through effective cloud detection. A minimal-resource, real-time inference neural network is ideally suited to perform automatic cloud detection, both for pre-processing captured images prior to transmission and preventing unnecessary images being taken by larger payload sensors. Much of the hardware complexity of modern neural network implementations resides in high-precision floating-point calculation pipelines. In recent years, research has been conducted in identifying quantized, or low-integer precision equivalents to known deep learning models, which do not require the extensive resources of their floating-point, full-precision counterparts. Our work leverages existing research on binary and quantized neural networks to develop a real-time, remote-sensed cloud detection solution using a commodity field-programmable gate array. This follows on developments of the Forwards Looking Imager for predictive cloud detection developed by Craft Prospect, a space engineering practice based in Glasgow, UK. The synthesized cloud detection accelerator achieved an inference throughput of 358.1 images per second with a maximum power consumption of 2.4 W. This throughput is an order of magnitude faster than alternate algorithmic options for the Forwards Looking Imager at around one third reduction in classification accuracy, and approximately two orders of magnitude faster than the CloudScout deep neural network, deployed with HyperScout 2 on the European Space Agency PhiSat-1 mission. Strategies for incorporating fault tolerance mechanisms are expounded

Object Detection in 20 Years: A Survey

Object detection, as of one the most fundamental and challenging problems in computer vision, has received great attention in recent years. Its development in the past two decades can be regarded as an epitome of computer vision history. If we think of today's object detection as a technical aesthetics under the power of deep learning, then turning back the clock 20 years we would witness the wisdom of cold weapon era. This paper extensively reviews 400+ papers of object detection in the light of its technical evolution, spanning over a quarter-century's time (from the 1990s to 2019). A number of topics have been covered in this paper, including the milestone detectors in history, detection datasets, metrics, fundamental building blocks of the detection system, speed up techniques, and the recent state of the art detection methods. This paper also reviews some important detection applications, such as pedestrian detection, face detection, text detection, etc, and makes an in-deep analysis of their challenges as well as technical improvements in recent years.Comment: This work has been submitted to the IEEE TPAMI for possible publicatio

CNN기반의 FusionNet 신경망과 농지 경계추출 알고리즘을 이용한 토지피복분류모델 개발

학위논문 (석사) -- 서울대학교 대학원 : 농업생명과학대학 생태조경.지역시스템공학부(지역시스템공학전공), 2021. 2. 송인홍.토지이용이 빠르게 변화함에 따라, 토지 피복에 대한 공간정보를 담고 있는 토지 피복 지도의 신속한 최신화는 필수적이다. 하지만, 현 토지 피복 지도는 많은 시간과 노동력을 요구하는 manual digitizing 방법으로 제작됨에 따라, 토지 피복 지도의 업데이트 및 배포에 긴 시간 간격이 발생하는 실정이다. 본 연구에서는 convolutional neural network (CNN) 기반의 인공신경망을 이용하여 high-resolution remote sensing (HRRS) 영상으로부터 토지 피복을 분류하는 모델을 개발하고, 특히 농지 경계추출 알고리즘을 적용하여 농업지역에서 분류 정확도를 개선하고자 하였다. 개발된 토지 피복 분류모델은 전처리(pre-processing) 모듈, 토지 피복 분류(land cover classification) 모듈, 그리고 후처리(post-processing) 모듈의 세 모듈로 구성된다. 전처리 모듈은 입력된 HRRS 영상을 75%씩 중첩 분할하여 관점을 다양화하는 모듈로, 한 관점에서 토지 피복을 분류할 때 발생할 수 있는 오분류를 줄이고자 하였다. 토지 피복 분류 모듈은 FusionNet model 구조를 바탕으로 개발되었고, 이는 분할된 HRRS 이미지의 픽셀별로 최적 토지 피복을 부여하도록 설계되었다. 후처리 모듈은 픽셀별 최종 토지 피복을 결정하는 모듈로, 분할된 HRRS 이미지의 분류결과를 취합하여 최빈값을 최종 토지 피복으로 결정한다. 추가로 농지에서는 농지경계를 추출하고, 필지별 분류된 토지 피복을 집계하여 한 필지에 같은 토지 피복을 부여하였다. 개발된 토지 피복 분류모델은 전라남도 지역(면적: 547 km2)의 2018년 정사영상과 토지 피복 지도를 이용하여 학습되었다. 토지 피복 분류모델 검증은 학습지역과 시간, 공간적으로 구분된, 2018년 전라남도 수북면과 2016년 충청북도 대소면의 두 검증지역에서 수행되었다. 각 검증지역에서 overall accuracy는 0.81, 0.71로 집계되었고, kappa coefficients는 0.75, 0.64로 산정되어 substantial 수준의 토지 피복 분류 정확도를 확인하였다. 특히, 개발된 모델은 필지 경계를 고려한 농업지역에서 overall accuracy 0.89, kappa coefficient 0.81로 almost perfect 수준의 우수한 분류 정확도를 보였다. 이에 개발된 토지 피복 분류모델은 특히 농업지역에서 현 토지 피복 분류 방법을 지원하여 토지 피복 지도의 빠르고 정확한 최신화에 기여할 수 있을 것으로 생각된다.The rapid update of land cover maps is necessary because spatial information of land cover is widely used in various areas. However, these maps have been released or updated in the interval of several years primarily owing to the manual digitizing method, which is time-consuming and labor-intensive. This study was aimed to develop a land cover classification model using the concept of a convolutional neural network (CNN) that classifies land cover labels from high-resolution remote sensing (HRRS) images and to increase the classification accuracy in agricultural areas using the parcel boundary extraction algorithm. The developed model comprises three modules, namely the pre-processing, land cover classification, and post-processing modules. The pre-processing module diversifies the perspective of the HRRS images by separating images with 75% overlaps to reduce the misclassification that can occur in a single image. The land cover classification module was designed based on the FusionNet model structure, and the optimal land cover type was assigned for each pixel of the separated HRRS images. The post-processing module determines the ultimate land cover types for each pixel unit by summing up the several-perspective classification results and aggregating the pixel-classification result for the parcel-boundary unit in agricultural areas. The developed model was trained with land cover maps and orthographic images (area: 547 km2) from the Jeonnam province in Korea. Model validation was conducted with two spatially and temporally different sites including Subuk-myeon of Jeonnam province in 2018 and Daseo-myeon of Chungbuk province in 2016. In the respective validation sites, the models overall accuracies were 0.81 and 0.71, and kappa coefficients were 0.75 and 0.64, implying substantial model performance. The model performance was particularly better when considering parcel boundaries in agricultural areas, exhibiting an overall accuracy of 0.89 and kappa coefficient 0.81 (almost perfect). It was concluded that the developed model may help perform rapid and accurate land cover updates especially for agricultural areas.Chapter 1. Introduction 1 1.1. Study background 1 1.2. Objective of thesis 4 Chapter 2. Literature review 6 2.1. Development of remote sensing technique 6 2.2. Land cover segmentation 9 2.3. Land boundary extraction 13 Chapter 3. Development of the land cover classification model 15 3.1. Conceptual structure of the land cover classification model 15 3.2. Pre-processing module 16 3.3. CNN based land cover classification module 17 3.4. Post processing module 22 3.4.1 Determination of land cover in a pixel unit 22 3.4.2 Aggregation of land cover to parcel boundary 24 Chapter 4. Verification of the land cover classification model 30 4.1. Study area and data acquisition 31 4.1.1. Training area 31 4.1.2. Verification area 32 4.1.3. Data acquisition 33 4.2. Training the land cover classification model 36 4.3. Verification method 37 4.3.1. The performance measurement methods of land cover classification model 37 4.3.2. Accuracy estimation methods of agricultural parcel boundary 39 4.3.3. Comparison of boundary based classification result with ERDAS Imagine 41 4.4. Verification of land cover classification model 42 4.4.1. Performance of land cover classification at the child subcategory 42 4.4.2. Classification accuracy of the aggregated land cover to main category 46 4.4.3. Classification accuracy of boundary based aggregation in agricultural area 57 Chapter 5. Conclusions 71 Reference 73 국 문 초 록 83Maste

車載画像を用いた都市の4次元モデリング

Tohoku University岡谷貴之課

Tiny Machine Learning Environment: Enabling Intelligence on Constrained Devices

Running machine learning algorithms (ML) on constrained devices at the extreme edge of the network is problematic due to the computational overhead of ML algorithms, available resources on the embedded platform, and application budget (i.e., real-time requirements, power constraints, etc.). This required the development of specific solutions and development tools for what is now referred to as TinyML. In this dissertation, we focus on improving the deployment and performance of TinyML applications, taking into consideration the aforementioned challenges, especially memory requirements. This dissertation contributed to the construction of the Edge Learning Machine environment (ELM), a platform-independent open-source framework that provides three main TinyML services, namely shallow ML, self-supervised ML, and binary deep learning on constrained devices. In this context, this work includes the following steps, which are reflected in the thesis structure. First, we present the performance analysis of state-of-the-art shallow ML algorithms including dense neural networks, implemented on mainstream microcontrollers. The comprehensive analysis in terms of algorithms, hardware platforms, datasets, preprocessing techniques, and configurations shows similar performance results compared to a desktop machine and highlights the impact of these factors on overall performance. Second, despite the assumption that TinyML only permits models inference provided by the scarcity of resources, we have gone a step further and enabled self-supervised on-device training on microcontrollers and tiny IoT devices by developing the Autonomous Edge Pipeline (AEP) system. AEP achieves comparable accuracy compared to the typical TinyML paradigm, i.e., models trained on resource-abundant devices and then deployed on microcontrollers. Next, we present the development of a memory allocation strategy for convolutional neural networks (CNNs) layers, that optimizes memory requirements. This approach reduces the memory footprint without affecting accuracy nor latency. Moreover, e-skin systems share the main requirements of the TinyML fields: enabling intelligence with low memory, low power consumption, and low latency. Therefore, we designed an efficient Tiny CNN architecture for e-skin applications. The architecture leverages the memory allocation strategy presented earlier and provides better performance than existing solutions. A major contribution of the thesis is given by CBin-NN, a library of functions for implementing extremely efficient binary neural networks on constrained devices. The library outperforms state of the art NN deployment solutions by drastically reducing memory footprint and inference latency. All the solutions proposed in this thesis have been implemented on representative devices and tested in relevant applications, of which results are reported and discussed. The ELM framework is open source, and this work is clearly becoming a useful, versatile toolkit for the IoT and TinyML research and development community