Production Copernicus Data Dissemination via a consolidated Datahub
Abstract
A consortium of Australian state government and Federal government partners required an integrated approach to support governmental decision making, founded on the European Copernicus program’s satellite data products. These data products would be provided for a region covering most of south-east Asia, the Pacific Islands and Australia. A key outcome for the consortium was the setup and operation of a datahub, to greatly improve access to Copernicus data in a densely populated region of the planet, experiencing high rates of economic growth, and facing significant challenges in areas where earth observation (EO) can assist eg. environmental protection, sustainable natural resource use and risk reduction from natural disasters.
Geoscience Australia and the NCI have been running a production datahub (www.copernicus.org.au, copernicus.nci.org.au) presenting unified source data products from both ESA and EUMETSAT for Sentinel-1, 2 for the region and a global Sentinel-3 replica.
In this presentation, we will touch upon the following aspects for this virtual collaborative environment (VCE): a) the pipelines which have continually transferred over a petabyte of EO imagery; b) the QA/QC and data publication process for Copernicus data; c) production issues in hosting the VCE in a research computing setting (network, storage and compute).
Australia’s unique geographic location presents a number of challenges for delivering high performance data transfer services. The extended distances and consequential network latency requires extensive network tuning to ensure timely and accurate data movement.
NCI utilises a number of strategies in collaboration with our domestic National Research and Education Network (NREN) AARNet (Australia’s Academic and Research Network), our regional network partners TEIN (Trans Eurasian Information Network) and international network partners Internet2 (USA) and GEANT (Europe) to tune the end systems, networks and connecting exchanges to enable a reliable and timely transfer of ESA data products from Europe to the Regional Hub operated by NCI.
By locating the data closer to the consumers, end-users within the region are able to access the data stored in the datahub at high speed without needing to perform special tuning of the systems or applications used for data access.
Ongoing monitoring of the performance of the Europe to Australia transfers, the connecting networks and storage services is performed to ensure the highest levels of availability are maintained. All end-users are able to programmatically or via a web-interface, request a region of interest they require and download the data of interest. Users with accounts at the National Computational Infrastructure (NCI) are able to access the data in-situ from HPC jobs and interactive cloud computing desktop environments.
For future work, we will further open access to the repository using OGC services like WMS, WPS and WCS. WMS for rapid imagery inspection (True color, False color) in the native file format, as distributed by ESA and EUMETSAT, including basic biophysical parameter retrieval such as NDVI, NDWI and EVI. WPS for simple time series analysis and WCS for access to underlying raw data.
