PID Services for CMIP6: Implementation and Deployment Overview
Explore the comprehensive overview of the PID services for CMIP6 presented at the ESGF F2F conference in 2015. The detailed information covers aspects such as object properties, handle records, implementation plan, operative deployment, consumer library, and Rabbit queue specifics, shedding light on the technical components and tools involved.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
ESGF PID Services for CMIP6 ESGF F2F conference, Dec 2015 Monterey, CA Tobias Weigel, Merret Buurman, Katharina Berger Deutsches Klimarechenzentrum (DKRZ) Weigel, Buurman, Berger (DKRZ)
What is this all about? 21.14100/<UUID> IDENTIFIER Object Properties URL creation_date tracking_ID checksum checksum_method parent DRS_ID ... Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 2 19.03.2025
Properties stored in Handle records Files URL creation_date tracking_ID checksum checksum_method parent DRS_ID Datasets URL replaced_by preceded_by aggregation_level Tombstone flag children DRS_ID + Errata information Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 3 19.03.2025
Implementation plan Agreements as per WIP white paper (Weigel, Lautenschlager, Juckes) All CMIP6 files bear Handle-compatible tracking_IDs Additional Handles for aggregation levels New component designed to be not a blocker Technical components Modifications to CMOR RabbitMQ-based deployment Additional tools and web services Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 4 19.03.2025
Operative deployment ESGF nodes, publisher with local PID library Queue Consumer services Central Rabbit exchange Handle Server Highly Available, not located at one of the nodes solr Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 5 19.03.2025
Consumer library Python library, used by the publisher Interface agreements Some checks/decisions can be done locally Actions are devised and sent to the Rabbit queue If queue is unreachable, the process will not be blocked Mechanism to dump actions to local queue file First version ready by end of December Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 6 19.03.2025
Rabbit queue Specific details on queue implementation still depend on High Availability deployment n Nodes talk to 1 Rabbit exchange with 1 consumer service Queue is one way only no feedback Performance trade-off Message consumption may take time; publisher may be already finished when feedback is sent Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 7 19.03.2025
Queue consumer and Handle server Consumer talks directly to Handle server through its native Java interface Increase performance avoid network communication If consumer is broken, messages will not be lost, but will be held at the central exchange Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 8 19.03.2025
Direct end-user value: Checking tool Check file(s) for new versions in user s workspace (command line tool) Report on metadata Have metadata or data changed? If available: Reported errata Possible direct download of new file(s) Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 9 19.03.2025
Thank you for your attention weigel@dkrz.de buurman@dkrz.de berger@dkrz.de Weigel, Buurman, Berger (DKRZ) ESGF F2F 2015: PID Services for CMIP6 10 19.03.2025
