Hub upgrade

With our Steering Committee agreeing to continue the Copernicus Australasia Regional Data Hub (the Hub) beyond June 2024, our focus has been to prepare for another five years of operation, particularly in light of the European Commission’s Copernicus Programme transitioning to the new Data Space Ecosystem platform (CDSE). The CDSE replaces the current EU public access (Open/Sci/API) hubs decommissioned October 2023.

Over the coming year, the Copernicus Australasia Regional Data Hub will be making significant upgrades to its systems to ensure it remains compatible with the changes occurring in Europe. The upgrades will also improve system robustness and reliability, provide faster and more guaranteed access to new data, while remaining compatible with other services. The system will provide for more flexible data management and allow the Hub to utilise the latest storage and network technologies.

The major component will be transitioning our infrastructure to the new GAEL Store Service (GSS) platform that has been developed by GAEL Systems. As part of this transition, our data acquisition processes will migrate to the CDSE (as well as maintaining data acquisition from EUMETSAT’s Datastore), and a new user portal will be deployed.

The new platform will change how the Hub’s data is stored at the National Computational Infrastructure (NCI). This means the APIs used to connect to the Hub’s data will change to align more closely with the CDSE APIs: https://documentation.dataspace.copernicus.eu/APIs/OData.html

We continue to work with GAEL Systems to test and resolve technical issues in the deployment of the new platform. Initially, the new system will only contain data collected from the go-live date onwards, however, it will be updated to hold all historic data over a period of 3-6 months. During this transition period, the current Hub portal will still be accessible, but will only contain historic data that has not yet been migrated to the new system. Once all the historic products have been migrated, we expect to decommission Sentinel Australasia Regional Access (SARA) and our legacy Hub. During this period of transition and decommissioning, we will provide a table on this webpage to allow users to track data locations.

Subscribe to our mailing list to receive more information as it becomes available subscribe to the Hub.

Frequently asked questions

What changes are being made to the Copernicus Australasia Regional Data Hub?

Over the next 6-12 months the Copernicus Australasia Regional Data Hub (the Hub) will be undergoing several software changes and updates to remain compatible with Copernicus EU’s new Copernicus Data Space Ecosystem platform, which replaces the public access EU hubs decommissioned during October 2023.
These updates include:
- Transitioning the Hub infrastructure to a new GAEL Store Service (GSS) platform.
- Migration of the Hub’s data acquisition processes to the ESA Copernicus Data Space Ecosystem and EUMETSAT Data Store.
- New user portal interface.
- Changes in the APIs used to access the Hub data.
- Changes to how data is stored at the National Computational Infrastructure.

Why are these changes occurring?

With ESA moving to the Copernicus Data Space Ecosystem, the current Hub infrastructure will no longer be compatible. Therefore, we need to migrate to a system that will maintain our ability to synchronise Sentinel data with Europe. The new system will improve robustness and reliability, and allow the Hub to utilise the latest storage and network technologies.

How will this affect me? Will it impact my access to the data?

Transitioning to the new system will allow the Hub to continue to provide open access to Sentinel data for the region. However, the new system will provide a new data portal interface and APIs. This means that access to the Hub will change and you may need to make modifications to scripts or programs in order to continue to download Sentinel data from the Hub.
In addition, it will take some time for us to migrate our current historic holdings (now over 6 petabytes) to the new system. During this transition phase, you may need to download products from both the old and new systems. Once the data migrations commence, we will provide a table to allow you to track data locations on this webpage.

Is there more information on the project timeline?

Yes, you can find more information on the upgrade project timeline

Phase	Timeframe	What is happening?	Impact to users
1. Implementation	Complete	- GAEL develops new software/platform - NCI deploys new platform - New platform configured to connect to ESA DataSpace & EUMETSAT DataStore - Testing	- None
2. Script migration	Nov 2024 - Jan 2025	- Adaption and development of partner (and expert user) auto download scripts to new platform	- Need to ensure user code/scripts are ready prior to new platform going live
3. Go live	Jan - Feb 2025	- New platform goes live, sync’ing new data from CDSE - Old hub stops syncing data (3 week overlap)	- New Sentinel data now only available from new portal - Must switch auto-download scripts (for new data) to new platform APIs.
4. Data Migration	Feb - May 2025	- Migrate and/or resync historic data from old Hub to new platform - NCI moves data to new data store	- Scripts to download historic data will need to be duplicated and modified to connect to new platform APIs - Historic data may be available from either old or new systems (see Data Transition table) - Users accessing data directly at NCI will need to change data access points
5. Decommissioning	Jun 2025	- Old hub will be turned off and decommissioned	- New and historic data only available via new portal and APIs

Is there any documentation on the new system?

A number of documents and guides are available to help transition to the new system:

API documentation – the new Hub will be using OData APIs that are compatible with the CDSE, more information can be found here: https://documentation.dataspace.copernicus.eu/APIs/OData.html
The new Hub web portal user guide
Data migration transition matrix (to come)
Python Library documentation (access: https://pypi.org/project/pygssearch/)
FAQ: how to modify old API scripts to new OData API’s – to come

When can we expect the Hub to be up and running fully with this new ecosystem?

We aim to have the new Hub online by early 2025. It will initially only contain new synchronised data, however, it will eventually replace the current Hub once all the historic data has been migrated to the new system.

Will the old system and the new system merge together so that the data is all in one place? If so, when?

The new GSS based system will eventually replace the current the Hub. Once all the historic data has been migrated to the new GSS system, the old system will be decommissioned and no longer available. The decommissioning date will be determined once the data migration is completed.

I access the Hub products directly from the file system at the National Computing Infrastructure (NCI), how will this change?

The new system will continue to use the file system at the NCI to store the Hub products, however due to the way the GSS software indexes files, products will not be stored in the same way or under the existing folder structure.
If you use the current file structure at NCI to “predict” where new products will be, this will no longer be available once products are migrated to the new GSS system. The only option to find products you need will be to search via the GSS portal or APIs.
Once you obtain the location of a product via the GSS portal or API, you may still be able to access the files directly from the file system, however we recommend where possible, to use the new command line tool, Python library, or GSS APIs to download any data from the Hub instead.
If you need help with accessing the new command line tool, Python library, or GSS APIs on the new system, please contact us via CopernicusAustralasia@ga.gov.au.

Who do I contact if I run into a technical issue?

If you have any problems or concerns about the new GSS system please contact us via CopernicusAustralasia@ga.gov.au