Hub upgrade

With our Steering Committee agreeing to continue the Copernicus Australasia Regional Data Hub (the Hub) beyond June 2024 when the current agreement ends, our focus has been to prepare for another five years of operation, particularly in light of the European Commission’s Copernicus Programme transitioning to the new Data Space Ecosystem platform (CDSE) this year. The CDSE will replace the current EU public access (Open/Sci/API) hubs due to be decommissioned at the end of September 2023.

Over the next 6-12 months, the Copernicus Australasia Regional Data Hub will be making significant upgrades to its systems to ensure it remains compatible with the changes occurring in Europe. The upgrades will also improve system robustness and reliability, provide faster and more guaranteed access to new data, while remaining compatible with other services. The system will provide for more flexible data management and allow the Hub to utilise the latest storage and network technologies.

The major component will be transitioning our infrastructure to the new GAEL Store Service (GSS) platform that has been developed by GAEL Systems. As part of this transition, our data acquisition processes will migrate to the CDSE (as well as maintaining data acquisition from EUMETSAT’s Datastore), and a new user portal will be deployed.

The GSS platform will change how the Hub’s data is stored at the National Computational Infrastructure (NCI). This means the APIs used to connect to the Hub’s data will change to align with the CDSE APIs: https://dataspace.copernicus.eu/analyse/apis

Our new Hub is due to go live at the end of September in line with the decommissioning of the legacy EU Hubs, and ESA’s transition to the CDSE. Initially, the GSS system will only contain data that is collected from September 2023 onwards. However, over a period of 3-6 months, it will be updated to hold all historic data. During this transition period, the current Hub portal will still be accessible, but will only contain pre-September 2023 historic data that has not yet been migrated to the new system. Once all the historic products have been migrated, we expect to decommission Sentinel Australasia Regional Access (SARA) and our legacy Hub. At this stage, we anticipate this will take place in early to mid-2024. Once the data migrations commence, we will provide a table to allow you to track data locations on this webpage.

Subscribe to our mailing list to receive more information as it becomes available subscribe to the Hub.

Frequently asked questions

What changes are being made to the Copernicus Australasia Regional Data Hub?

Over the next 6-12 months the Copernicus Australasia Regional Data Hub (the Hub) will be undergoing several software changes and updates to remain compatible with Copernicus EU’s new Copernicus Data Space Ecosystem platform, which will replace the current public access EU hubs due to be decommissioned around the end of September 2023.
These updates include:
- Transitioning the Hub infrastructure to a new GAEL Store Service (GSS) platform.
- Migration of the Hub’s data acquisition processes to the ESA Copernicus Data Space Ecosystem and EUMETSAT Data Store.
- New user portal interface.
- Changes in the APIs used to access the Hub data.
- Changes to how data is stored at the National Computational Infrastructure.

Why are these changes occurring?

With ESA moving to the Copernicus Data Space Ecosystem, the current Hub infrastructure will no longer be compatible. Therefore, we need to migrate to a system that will maintain our ability to synchronise Sentinel data with Europe. The new system will improve robustness and reliability, and allow the Hub to utilise the latest storage and network technologies.

How will this affect me? Will it impact my access to the data?

Transitioning to the new system will allow the Hub to continue to provide open access to Sentinel data for the region. However, the new system will provide a new data portal interface and APIs. This means that access to the Hub will change and you may need to make modifications to scripts or programs in order to continue to download Sentinel data from the Hub.
In addition, it will take some time for us to migrate our current historic holdings (now over 6 petabytes) to the new system. During this transition phase, you may need to download products from both the old and new systems. Once the data migrations commence, we will provide a table to allow you to track data locations on this webpage.

Is there more information on the project timeline?

Yes, you can find more information on the upgrade project timeline

Phase	Timeframe	What is happening?	Impact to users
1. Implementation	now – Sept 23 (early Oct 23)	- GAEL develops new software/platform - NCI deploys new platform - New platform configured to connect to ESA DataSpace & EUMETSAT DataStore - Testing	- None
2. Script migration	Aug – Sept 23	- Adaption and development of partner (and expert user) auto download scripts to new platform	- Need to ensure user code/scripts are ready prior to new platform going live
3. Go live	1 October 2023	- New platform goes live, sync’ing new data from CDSE - Old hub stops syncing data (may be short 1 week overlap)	- New Sentinel data now only available from new portal - Must switch auto-download scripts (for new data) to new platform APIs.
4. Data Migration	Sept 23 – Mar 24	- Migrate and/or resync historic data from old Hub to new platform - NCI moves data to new data store	- Scripts to download historic data will need to be duplicated and modified to connect to new platform APIs - Historic data may be available from either old or new systems (see Data Transition table) - Users accessing data directly at NCI will need to change data access points
5. Decommissioning	Mar 24 – June 24	- Old hub will be turned off and decommissioned	- New and historic data only available via new portal and APIs

Is there any documentation on the new system?

A number of documents and guides are available to help transition to the new system:

GSS API documentation – to come
GSS GUI/web portal user guide – to come
Data migration transition matrix
Python Library documentation (access: https://pypi.org/project/pygssearch/1.0.0/, User documentation)
FAQ: how to modify old API scripts to new GSS API’s – to come

When can we expect the Hub to be up and running fully with this new ecosystem?

We aim to have the new GSS based system online by October 2023. It will initially only contain new synchronised data, however, it will eventually replace the current the Hub once all the historic data has been migrated across by June 2024.

Will the old system and the new system merge together so that the data is all in one place? If so, when?

The new GSS based system will eventually replace the current the Hub. Once all the historic data has been migrated to the new GSS system, the old system will be decommissioned and no longer available. We expect the old system to be decommissioned by June 2024.

Who do I contact if I run into a technical issue?

If you have any problems or concerns about the new GSS system please contact us via CopernicusAustralasia@ga.gov.au