Infrastructures and services for remote sensing data production management across multiple satellite data centers

Zhang, Jie, Yan, Jining, Ma, Yan, Xu, Dong, Li, Pengfei and Jie, Wei (2016) Infrastructures and services for remote sensing data production management across multiple satellite data centers. Cluster Computing, 19 (3). pp. 1243-1260. ISSN 1386-7857

[img]
Preview
PDF
MDCPS V3.6.pdf - Accepted Version

Download (12MB) | Preview

Abstract

With the number of satellite sensors and date centers being increased continuously, it is becoming a trend to manage and process massive remote sensing data from multiple distributed sources. However, the combination of multiple satellite data centers for massive remote sensing (RS) data collaborative processing still faces many challenges. In order to reduce the huge amounts of data migration and improve the efficiency of multi-datacenter collaborative process, this paper presents the infrastructures and services of the data management as well as workflow management for massive remote sensing data production. A dynamic data scheduling strategy was employed to reduce the duplication of data request and data processing. And by combining the remote sensing spatial metadata repositories and Gfarm grid file system, the unified management of the raw data, intermediate products and final products were achieved in the co-processing. In addition, multi-level task order repositories and workflow templates were used to construct the production workflow automatically. With the help of specific heuristic scheduling rules, the production tasks were executed quickly. Ultimately, the Multi-datacenter Collaborative Process System (MDCPS) were implemented for large-scale remote sensing data production based on the effective management of data and workflow. As a consequence, the performance of MDCPS in experiments environment showed that those strategies could significantly enhance the efficiency of co-processing across multiple data centers.

Item Type: Article
Additional Information: © Springer Verlag 2016. The final publication is available at Springer via http://dx.doi.org/10.1007/s10586-016-0577-6
Uncontrolled Keywords: Multi-datacenter infrastructure, Remote sensing data processing, Distributed computing, Big data computing, Data management, Workflow management
Subjects: Computing
Depositing User: WEI JIE
Date Deposited: 14 Sep 2016 09:40
Last Modified: 30 May 2017 02:45
URI: http://repository.uwl.ac.uk/id/eprint/2869

Downloads

Downloads per month over past year

Actions (login required)

View Item View Item

Menu