Hub upgrade
With our Steering Committee agreeing to continue the Copernicus Australasia Regional Data Hub (the Hub) beyond June 2024, our focus has been to prepare for another five years of operation, particularly in light of the European Commission’s Copernicus Programme transition to the new Data Space Ecosystem platform (CDSE). The CDSE replaces the original EU public access (Open/Sci/API) hubs decommissioned in October 2023.
Over the past two years, the Copernicus Australasia Regional Data Hub has made significant upgrades to its systems to ensure it remains compatible with the changes occurring in Europe. The upgrades improve system robustness and reliability, provide faster and more guaranteed access to new data, while remaining compatible with other services. The system also provides for more flexible data management and allow the Hub to utilise the latest storage and network technologies.
The major component was transitioning our infrastructure to the new GAEL Store Service (GSS) platform that has been developed by GAEL Systems, and migrating our data acquisition processes to the CDSE (as well as maintaining data acquisition from EUMETSAT’s Datastore), and a new user portal, Discover, has been deployed.
The new platform will change how the Hub’s data is stored at the National Computational Infrastructure (NCI). This means the APIs used to connect to the Hub’s data will change to align more closely with the CDSE APIs: https://documentation.dataspace.copernicus.eu/APIs/OData.html.
We continue to work with GAEL Systems to test and resolve technical issues in the deployment of the new platform. Initially, the new system will only contain data collected from the go-live date onwards, however, it will be updated to hold all historic data over the next 6 months. During this transition period, the current Hub portal (<link to SARA>) will still be accessible, but will only contain historic data that has not yet been migrated to the new system. Once all the historic products have been migrated, we expect to decommission Sentinel Australasia Regional Access (SARA) and our legacy Hub. During this period of transition awe will provide a timetable in the FAQ below to allow users to track data locations.
Subscribe to our mailing list to receive more information as it becomes available subscribe to the Hub.
Frequently asked questions
- The Copernicus Australasia Regional Data Hub (the Hub) has undergone several software changes and updates to remain compatible with Copernicus EU’s new Copernicus Data Space Ecosystem platform.
- These updates include:
- Transitioning the Hub infrastructure to a new GAEL Store Service (GSS) platform.
- Migration of the Hub’s data acquisition processes to the ESA Copernicus Data Space Ecosystem and EUMETSAT Data Store.
- New user portal interface, Discover.
- Changes in the APIs used to access the Hub data.
- Changes to how data is stored at the National Computational Infrastructure.
- With ESA moving to the Copernicus Data Space Ecosystem, the current Hub infrastructure will no longer be compatible. Therefore, we need to migrate to a system that will maintain our ability to synchronise Sentinel data with Europe. The new system improves robustness and reliability, and allows the Hub to utilise the latest storage and network technologies.
- Transitioning to the new system allows the Hub to continue to provide open access to Sentinel data for the region. However, the new system provides a new data portal interface and APIs. This means that access to the Hub has changed and you may need to make modifications to scripts or programs in order to continue to download Sentinel data from the Hub.
- In addition, it will take some time for us to migrate our current historic holdings (now over 6 petabytes) to the new system. During this transition phase, you may need to download products from both the old and new systems. Once the data migrations commence, we will provide a table to allow you to track data locations on this webpage.
- A number of documents and guides are available to help transition to the new system:
- API documentation – the new Hub will be using OData APIs that are compatible with the CDSE, more information can be found here: https://documentation.dataspace.copernicus.eu/APIs/OData.html.
- The new Hub’s Discover portal user guide.
- Python Library documentation (access: https://pypi.org/project/pygssearch/).
- Now that the new system has been released, all the historic data will be migrated across to the new system. The old system will remain in place for the next 6 months to ensure access to the historic data, but once the data migration is complete the old Hub and SARA portal will be decommissioned and no longer available.
- For now, the new system will continue to use the file system at the NCI to store the Hub products, however due to the way the GSS platform indexes files, products will not be stored in the same way or under the existing folder structure.
- If you used the current file structure at NCI to “predict” where products will be, this will no longer be available once products are migrated to the new system. The only option to find products you need, will be to search via Discover or APIs. For more information, please see the Discover portal user guide.
- Once you obtain the location of a product via the Discover portal or API, you may still be able to access the files directly from the file system, however for future compatibility, we recommend where possible, to use the new command line tool, Python library, or APIs to download any data from the Hub instead.
- If you need help with accessing data on the new system, please contact us via CopernicusAustralasia@ga.gov.au.
- If you have any problems or concerns about the new system please contact us via CopernicusAustralasia@ga.gov.au