Every day, satellites from the European Space Agency and European Meteorological Agency (EUMETSAT) circle the Earth, collecting images and observations from the planet below. Making the data available to a wide community of researchers, emergency services and government agencies in Australia involves the complex technical challenge of moving large volumes of data across the globe.
To provide rapid global access to the data, a number of regional storage hubs have been established. In Australia, the Federal Government through Geoscience Australia and NCI has developed the Australian Regional Copernicus Hub, to manage the data for the South East Asia and Pacific region.
Getting the data here starts when the images and observations are beamed from the satellites to the Copernicus earth-observation dissemination centre in Frankfurt, Germany, ready for distribution. Then, the data transfer from Frankfurt to Canberra moves 20 terabytes of data every day, the equivalent of streaming around 2400 movies. The 10,000-kilometre journey takes the data across continents and oceans before reaching NCI’s high-performance facility. The data follows a complex network of undersea fibre optic cables, and links together various National Research and Education Networks in Australia, Europe, Asia and the Americas.
Maintaining the integrity of the data across such a large distance is essential for researchers to be able to trust the accuracy of their datasets. NCI’s high-performance data team has developed a data transfer process using specially designed protocols and rigorous quality control methods to guarantee that when data arrives at NCI, it is suitable for use.
To ensure the highest reliability and to handle unexpected network failures, the datasets come to NCI over two geographically separate paths. The first via the USA, crossing the European Géant Network across the Atlantic Ocean from Amsterdam to New York, then transiting the Internet2 network to Seattle where they cross the Pacific Ocean to Sydney, ready for the final leg to Canberra via Australia’s Academic and Research Network (AARNet). The second path is via Asia, crossing the Géant network from London, transiting via the TEIN network to Singapore and then to Australia via Perth.
The data managed at NCI is provided by the European Space Agency and EUMETSAT and includes all satellite observations and images covering Australia, New Zealand, Indonesia and many countries in South Asia and the Pacific, as well as Australia’s marine reserves and sections of Antarctica.
The measurements include raw and processed datasets like ocean colour, ocean height, ground reflectance and ground moisture. From these datasets, scientists can analyse the environmental impact such as the build-up of fuel for bushfires, the change of flow rates in rivers and how land-use is affecting the environment. In addition, advanced radar imaging allows for the determination of ground subsidence with movements down to a few millimeters.
NCI has more than 10 petabytes of curated data collections of reference earth systems datasets, which include environmental data such as satellite imagery and data products, and many years of climate, weather and other geospatial data. These datasets have been organised at NCI in a way that allows for queries, combinations of data and new and innovative research. Working with various research communities, NCI has developed virtual laboratories and a vast software repository integrated together to support researchers, to make the best possible use of these data resources.
This research highlight was originally published in the 2016-2017 NCI Annual Report.
