Hub upgrade

With our Steering Committee agreeing to continue the Copernicus Australasia Regional Data Hub (the Hub) beyond June 2024 when the current agreement ends, our focus has been to prepare for another five years of operation, particularly in light of the European Commission’s Copernicus Programme transitioning to the new Data Space Ecosystem platform (CDSE) this year. The CDSE will replace the current EU public access (Open/Sci/API) hubs due to be decommissioned during October 2023.

Over the next 6-12 months, the Copernicus Australasia Regional Data Hub will be making significant upgrades to its systems to ensure it remains compatible with the changes occurring in Europe. The upgrades will also improve system robustness and reliability, provide faster and more guaranteed access to new data, while remaining compatible with other services. The system will provide for more flexible data management and allow the Hub to utilise the latest storage and network technologies.

The major component will be transitioning our infrastructure to the new GAEL Store Service (GSS) platform that has been developed by GAEL Systems. As part of this transition, our data acquisition processes will migrate to the CDSE (as well as maintaining data acquisition from EUMETSAT’s Datastore), and a new user portal will be deployed.

The GSS platform will change how the Hub’s data is stored at the National Computational Infrastructure (NCI). This means the APIs used to connect to the Hub’s data will change to align with the CDSE APIs: https://dataspace.copernicus.eu/analyse/apis

Our new Hub is due to go live during October 2023 in line with the decommissioning of the legacy EU Hubs, and ESA’s transition to the CDSE. Initially, the GSS system will only contain data that is collected from October 2023 onwards. However, over a period of 3-6 months, it will be updated to hold all historic data. During this transition period, the current Hub portal will still be accessible, but will only contain pre-October 2023 historic data that has not yet been migrated to the new system. Once all the historic products have been migrated, we expect to decommission Sentinel Australasia Regional Access (SARA) and our legacy Hub. At this stage, we anticipate this will take place in early to mid-2024. Once the data migrations commence, we will provide a table to allow you to track data locations on this webpage.

Subscribe to our mailing list to receive more information as it becomes available subscribe to the Hub.

Frequently asked questions

  • Over the next 6-12 months the Copernicus Australasia Regional Data Hub (the Hub) will be undergoing several software changes and updates to remain compatible with Copernicus EU’s new Copernicus Data Space Ecosystem platform, which will replace the current public access EU hubs due to be decommissioned during October 2023.
  • These updates include:
    • Transitioning the Hub infrastructure to a new GAEL Store Service (GSS) platform.
    • Migration of the Hub’s data acquisition processes to the ESA Copernicus Data Space Ecosystem and EUMETSAT Data Store.
    • New user portal interface.
    • Changes in the APIs used to access the Hub data.
    • Changes to how data is stored at the National Computational Infrastructure.
  • With ESA moving to the Copernicus Data Space Ecosystem, the current Hub infrastructure will no longer be compatible. Therefore, we need to migrate to a system that will maintain our ability to synchronise Sentinel data with Europe. The new system will improve robustness and reliability, and allow the Hub to utilise the latest storage and network technologies.
  • Transitioning to the new system will allow the Hub to continue to provide open access to Sentinel data for the region. However, the new system will provide a new data portal interface and APIs. This means that access to the Hub will change and you may need to make modifications to scripts or programs in order to continue to download Sentinel data from the Hub.
  • In addition, it will take some time for us to migrate our current historic holdings (now over 6 petabytes) to the new system. During this transition phase, you may need to download products from both the old and new systems. Once the data migrations commence, we will provide a table to allow you to track data locations on this webpage.

Yes, you can find more information on the upgrade project timeline

PhaseTimeframe What is happening?Impact to users
1. Implementation now – Sept 23 (early Oct 23) - GAEL develops new software/platform
-  NCI deploys new platform
- New platform configured to connect to ESA DataSpace & EUMETSAT DataStore 
- Testing
- None
2. Script migration Aug – Sept 23 - Adaption and development of partner (and expert user) auto download scripts to new platform - Need to ensure user code/scripts are ready prior to new platform going live
3. Go live October 2023 - New platform goes live, sync’ing new data from CDSE
- Old hub stops syncing data (may be short 1 week overlap)
- New Sentinel data now only available from new portal
- Must switch auto-download scripts (for new data) to new platform APIs.
4. Data Migration Sept 23 – Mar 24 - Migrate and/or resync historic data from old Hub to new platform
- NCI moves data to new data store
- Scripts to download historic data will need to be duplicated and modified to connect to new platform APIs
- Historic data may be available from either old or new systems (see Data Transition table)
- Users accessing data directly at NCI will need to change data access points
5. Decommissioning Mar 24 – June 24 - Old hub will be turned off and decommissioned - New and historic data only available via new portal and APIs
  • A number of documents and guides are available to help transition to the new system:
    • GSS API documentation – to come
    • GSS GUI/web portal user guide – to come
    • Data migration transition matrix
    • Python Library documentation (access: https://pypi.org/project/pygssearch/1.0.0/, User documentation)
    • FAQ: how to modify old API scripts to new GSS API’s – to come
  • We aim to have the new GSS based system online by October 2023. It will initially only contain new synchronised data, however, it will eventually replace the current the Hub once all the historic data has been migrated across by June 2024.
  • The new GSS based system will eventually replace the current the Hub. Once all the historic data has been migrated to the new GSS system, the old system will be decommissioned and no longer available. We expect the old system to be decommissioned by June 2024.