Infrastructures and services for remote sensing data production management across multiple satellite data centers

With the number of satellite sensors and date centers being increased continuously, it is becoming a trend to manage and process massive remote sensing data from multiple distributed sources. However, the combination of multiple satellite data centers for massive remote sensing (RS) data collaborative processing still faces many challenges. In order to reduce the huge amounts of data migration and improve the efficiency of multi-datacenter collaborative process, this paper presents the infrastructures and services of the data management as well as workflow management for massive remote sensing data production. A dynamic data scheduling strategy was employed to reduce the duplication of data request and data processing. And by combining the remote sensing spatial metadata repositories and Gfarm grid file system, the unified management of the raw data, intermediate products and final products were achieved in the co-processing. In addition, multi-level task order repositories and workflow templates were used to construct the production workflow automatically. With the help of specific heuristic scheduling rules, the production tasks were executed quickly. Ultimately, the Multi-datacenter Collaborative Process System (MDCPS) were implemented for large-scale remote sensing data production based on the effective management of data and workflow. As a consequence, the performance of MDCPS in experiments environment showed that those strategies could significantly enhance the efficiency of co-processing across multiple data centers

A cloud-based remote sensing data production system

The data processing capability of existing remote sensing system has not kept pace with the amount of data typically received and need to be processed. Existing product services are not capable of providing users with a variety of remote sensing data sources for selection, either. Therefore, in this paper, we present a product generation programme using multisource remote sensing data, across distributed data centers in a cloud environment, so as to compensate for the low productive efficiency, less types and simple services of the existing system. The programme adopts “master–slave” architecture. Specifically, the master center is mainly responsible for the production order receiving and parsing, as well as task and data scheduling, results feedback, and so on; the slave centers are the distributed remote sensing data centers, which storage one or more types of remote sensing data, and mainly responsible for production task execution. In general, each production task only runs on one data center, and the data scheduling among centers adopts a “minimum data transferring” strategy. The logical workflow of each production task is organized based on knowledge base, and then turned into the actual executed workflow by Kepler. In addition, the scheduling strategy of each production task mainly depends on the Ganglia monitoring results, thus the computing resources can be allocated or expanded adaptively. Finally, we evaluated the proposed programme using test experiments performed at global, regional and local areas, and the results showed that our proposed cloud-based remote sensing production system could deal with massive remote sensing data and different products generating, as well as on-demand remote sensing computing and information service

The Overseeing Mother: Revisiting the Frontal-Pose Lady in the Wu Family Shrines in Second Century China

Located in present-day Jiaxiang in Shandong province, the Wu family shrines built during the second century in the Eastern Han dynasty (25–220) were among the best-known works in Chinese art history. Although for centuries scholars have exhaustively studied the pictorial programs, the frontal-pose female image situated on the second floor of the central pavilion carved at the rear wall of the shrines has remained a question. Beginning with the woman’s eyes, this article demonstrates that the image is more than a generic portrait (“hard motif ”), but rather represents “feminine overseeing from above” (“soft motif ”). This synthetic motif combines three different earlier motifs – the frontal-pose hostess enjoying entertainment, the elevated spectator, and the Queen Mother of the West. By creatively fusing the three motifs into one unity, the Jiaxiang artists lent to the frontal-pose lady a unique power: she not only dominated the center of the composition, but also, like a divine being, commanded a unified view of the surroundings on the lofty building, hence echoing the political reality of the empress mother’s “overseeing the court” in the second century during Eastern Han dynasty

Suitability Evaluation for Products Generation from Multisource Remote Sensing Data

With the arrival of the big data era in Earth observation, the remote sensing communities have accumulated a large amount of invaluable and irreplaceable data for global monitoring. These massive remote sensing data have enabled large-area and long-term series Earth observation, and have, in particular, made standard, automated product generation more popular. However, there is more than one type of data selection for producing a certain remote sensing product; no single remote sensor can cover such a large area at one time. Therefore, we should automatically select the best data source from redundant multisource remote sensing data, or select substitute data if data is lacking, during the generation of remote sensing products. However, the current data selection strategy mainly adopts the empirical model, and has a lack of theoretical support and quantitative analysis. Hence, comprehensively considering the spectral characteristics of ground objects and spectra differences of each remote sensor, by means of spectrum simulation and correlation analysis, we propose a suitability evaluation model for product generation. The model will enable us to obtain the Production Suitability Index (PSI) of each remote sensing data. In order to validate the proposed model, two typical value-added information products, NDVI and NDWI, and two similar or complementary remote sensors, Landsat-OLI and HJ1A-CCD1, were chosen, and the verification experiments were performed. Through qualitative and quantitative analysis, the experimental results were consistent with our model calculation results, and strongly proved the validity of the suitability evaluation model. The proposed production suitability evaluation model could assist with standard, automated, serialized product generation. It will play an important role in one-station, value-added information services during the big data era of Earth observation

Big Data Integration in Remote Sensing across a Distributed Metadata-Based Spatial Infrastructure

Since Landsat-1 first started to deliver volumes of pixels in 1972, the volumes of archived data in remote sensing data centers have increased continuously. Due to various satellite orbit parameters and the specifications of different sensors, the storage formats, projections, spatial resolutions, and revisit periods of these archived data are vastly different. In addition, the remote sensing data received continuously by each data center arrives at a faster code rate; it is best to ingest and archive the newly received data to ensure users have access to the latest data retrieval and distribution services. Hence, an excellent data integration, organization, and management program is urgently needed. However, the multi-source, massive, heterogeneous, and distributed storage features of remote sensing data have not only caused difficulties for integration across distributed data center spatial infrastructures, but have also resulted in the current modes of data organization and management being unable meet the rapid retrieval and access requirements of users. Hence, this paper proposes an object-oriented data technology (OODT) and SolrCloud-based remote sensing data integration and management framework across a distributed data center spatial infrastructure. In this framework, all of the remote sensing metadata in the distributed sub-centers are transformed into the International Standardization Organization (ISO) 19115-based unified format, and then ingested and transferred to the main center by OODT components, continuously or at regular intervals. In the main data center, in order to improve the efficiency of massive data retrieval, we proposed a logical segmentation indexing (LSI) model-based data organization approach, and took SolrCloud to realize the distributed index and retrieval of massive metadata. Finally, a series of distributed data integration, retrieval, and comparative experiments showed that our proposed distributed data integration and management program is effective and promises superior results. Specifically, the LSI model-based data organization and the SolrCloud-based distributed indexing schema was able to effectively improve the efficiency of massive data retrieval

pipsCloud: high performance cloud computing for remote sensing big data management and processing

With the increasing requirement of accurate and up-to-date resource & environmental information for regional and global monitoring, large-region covered multi-temporal, multi-spectral massive remote sensing (RS) datasets are exploited for processing. The remote sensing data processing generally follows a complex multi-stage processing chain, which consists of several independent processing steps subject to types of RS applications. In general the RS data processing for regional environmental and disaster monitoring are recognized as typical both compute-intensive and data-intensive applications.To solve the aforementioned issues efficiently, we propose pipsCloud which combine recent Cloud computing and HPC techniques to enable large-scale RS data processing system as on-demand real-time services. Benefiting from the ubiquity, elasticity and high-level of transparency of Cloud computing model, the massive RS data managing and data processing for dynamic environmental monitoring are all encapsulate as Cloud with Web interfaces. Where, a Hilbert-R+ based data indexing mechanism is employed for optimal query and access of RS imageries, RS data products as well as interim data. In the core platform beneath the Cloud services, we provide a parallel file system for massive high-dimensional RS data and offers interfaces for intensive irregular RS data accessing so as to provide improved data locality and optimized I/O performance. Moreover, we adopt an adaptive RS data analysis workflow manage system for on-demand workflow construction and collaborative execution of distributed complex chain of RS data processing, such as forest fire detection, mineral resources and coastline monitoring. Through the experimental analysis we have show the efficiency of the pipsCloud platform

An Assessment of Electric Power Consumption Using Random Forest and Transferable Deep Model with Multi-Source Data

Reliable and fine-resolution electric power consumption (EPC) is essential for effective urban electricity allocation and planning. Currently, EPC data exists mainly as statistics with low resolution. Many studies estimate fine-resolution EPC based on the positive correction between stable nighttime light and EPC distribution. However, EPC is related to various factors other than nighttime light and is spatially non-stationary. Yet this has been ignored in current research. This study developed a novel method to estimate EPC at 500 m resolution by considering spatially non-stationary through fusing geospatial data and high-resolution satellite images. Deep transfer learning and statistical methods were used to extract socio-economic, population density, and landscape features to describe EPC distribution from multi-source geospatial data. Finally, a random forest regression (RFR) model with features and EPC statistics is established to estimate fine-resolution EPC. A study area of Shenzhen city, China, is employed to evaluate the proposed method. The R2 between predicted EPC and statistical EPC is 0.82 at sub-district level in 2013, which is higher than an existing EPC product (Shi’s product) with R2=0.46, illustrating the effectiveness of the proposed method. Moreover, the EPC distribution for Shenzhen from 2013 to 2019 was estimated. Furthermore, the spatiotemporal dynamic of EPC was analyzed at the pixel and sub-district levels