EGU21-1255, updated on 03 Mar 2021
https://doi.org/10.5194/egusphere-egu21-1255
EGU General Assembly 2021
© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Scientific workflow scheduling based on data transformation graph for remote sensing application

Zhuojing tian1, Zhenchun huang2, Yinong zhang3, Yanwei zhao4, En fu5, and Shuying wang6
Zhuojing tian et al.
  • 1Beijing Key Laboratory of Information Service Engineering, Beijing Union University, Beijing, China (tianzhuojing@foxmail.com)
  • 2Beijing National Research Center for Information Science and Technology, Beijing, China(huangzc@tsinghua.edu.cn)
  • 3School of Urban Rail Transit and Logistics, BeijingUnionUniversity, Beijing, China(zdhtyinong@buu.edu.cn)
  • 4School of Urban Rail Transit and Logistics, BeijingUnionUniversity, Beijing, China(1059889494@qq.com)
  • 5Beijing Key Laboratory of Information Service Engineering, Beijing Union University, Beijing, China (925378271@qq.com)
  • 6School of Urban Rail Transit and Logistics, BeijingUnionUniversity, Beijing, China(1020590018@qq.com)

Abstract: As the amount of data and computation of scientific workflow applications continue to grow, distributed and heterogeneous computing infrastructures such as inter-cloud environments provide this type of application with a great number of computing resources to meet corresponding needs. In the inter-cloud environment, how to effectively map tasks to cloud service providers to meet QoS(quality of service) constraints based on user requirements has become an important research direction. Remote sensing applications need to process terabytes of data each time, however frequent and huge data transmission across the cloud will bring huge performance bottlenecks for execution, and seriously affect the result of QoS constraints such as makespan and cost. Using a data transformation graph(DTG) to study the data transfer process of global drought detection application, the specific optimization strategy is obtained based on the characteristics of application and environment, and according to this, one inter-cloud workflow scheduling method based on genetic algorithm is proposed, under the condition of satisfying the user’s QoS constraints, the makespan the cost can be minimized. The experimental results show that compared with the standard genetic algorithm, random algorithm, random algorithm, and round-robin algorithm, the optimized genetic algorithm can greatly improve the scheduling performance of data computation-intensive scientific workflows such as remote sensing applications and reduce the impact of performance bottlenecks.

Keywords: scientific workflow scheduling; inter-cloud environment; remote sensing application; data transformation graph;

How to cite: tian, Z., huang, Z., zhang, Y., zhao, Y., fu, E., and wang, S.: Scientific workflow scheduling based on data transformation graph for remote sensing application, EGU General Assembly 2021, online, 19–30 Apr 2021, EGU21-1255, https://doi.org/10.5194/egusphere-egu21-1255, 2021.