Accounting Repository Changes and Data Retention Policies
Explore the latest changes in accounting repository structures and data retention policies for MPI accounting. Learn about the migration from EMI2 to EMI3, data summarization processes, and evolving schema support. Discover how CPU accounting data is managed, including user data deletion policies and summary data retention. Stay informed about external client support and the transition to new schema formats in the accounting portal.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
EGI-InSPIRE Accounting Status Stuart Pullinger John Gordon 1 6/18/2025 www.egi.eu www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE RI-261323
Overview MPI Accounting Accounting Repository Changes Cloud Accounting Application Accounting EMI2 to EMI3 Migration Message Broker Changes Storage Accounting Pay for use Accounting www.egi.eu EGI-InSPIRE RI-261323
A quick note CPU Accounting data deletion Data Retention Policy Delete user data after 18 months Keep summaries Data deleted last week 2 days Back to normal after weekend Will use different method in future www.egi.eu EGI-InSPIRE RI-261323
MPI Accounting Collect data on Number of cores Processors field Number of worker nodes NodeCount field Apel EMI3 Parser support LSF PBS SLURM SGE (Processors only) www.egi.eu EGI-InSPIRE RI-261323
MPI Accounting External client support ARC QCG EDGI Data is being collected (new schema) But Summary data sent to accounting portal in old schema www.egi.eu EGI-InSPIRE RI-261323
Accounting Repository Changes EMI2 APEL Client EMI2 APEL System Monthly Summary Created Publishing Host Portal Convert to old schema EMI3 APEL Client/SSM2 EMI3 APEL System Monthly Summary Created www.egi.eu EGI-InSPIRE RI-261323
Accounting Repository Changes EMI3 data summarised on EMI3 repository Including Processors & NodeCount Summaries transferred to EMI2 repository Changed to old schema Loses Processors & NodeCount (masks data for same month/site from EMI2) Transferred to other machine for transfer to portal www.egi.eu EGI-InSPIRE RI-261323
Accounting Repository Changes EMI2 APEL Client EMI2 APEL System Monthly Summary Created Convert to new schema EMI3 APEL Client/SSM2 EMI3 APEL System Monthly Summary Created Portal SSM2 www.egi.eu EGI-InSPIRE RI-261323
Accounting Repository Changes EMI3/MPI summaries from EMI3 repository to portal Easy 1 extra cron job to run ssmsend EMI3/MPI summaries + EMI2 summaries More complicated Convert old schema to new Insert default values Several different options Hardware considerations www.egi.eu EGI-InSPIRE RI-261323
Cloud Accounting 15 sites sending data OpenStack: 10 OpenNebula: 4 Stratuslab: 1 20,729 VMs accounted for since 15th March 2013 www.egi.eu EGI-InSPIRE RI-261323
Cloud Accounting Externally developed scripts extract accounting data Older OpenStack: osssm Newer OpenStack: ceilometer2ssm OpenNebula: cloudacc All use SSM2 to send to repository www.egi.eu EGI-InSPIRE RI-261323
Cloud Accounting to Production By End April Repository Extra cron jobs and monitoring Additional database and backup Schema compliance Some work needed for externally developed scripts In progress www.egi.eu EGI-InSPIRE RI-261323
Application Accounting Account for application usage Uses LD_PRELOAD Loads altered fork() implementation Logs all calls to fork() Parser code written & integrated with APEL code repo TODO: database schema Hopefully have prototype for EGI CF www.egi.eu EGI-InSPIRE RI-261323
EMI2 to EMI3 Migration EMI2 Security updates end April 74 Sites publishing to EMI3/SSM2 Additionally 4 sites publishing to SSM1.2 156 Sites publishing via EMI2 New database schema Migrate data or fresh start Cannot publish via both routes simultaneously Cannot publish partial months via both routes www.egi.eu EGI-InSPIRE RI-261323
EMI2 to EMI3 Migration Migration document and script 3 sites experienced problems Duplicate entries in client database Causes Nagios Sync errors Fix being investigated Choose to either Migrate data from old database to new Clean accounting log directory Reparse log files with EMI3 parser Delete partial months www.egi.eu EGI-InSPIRE RI-261323
Message Broker Changes Test message broker ending TESTNWOB network CPU testing Cloud testing Storage testing New queues on PROD Network /queue/global.accounting.cputest.central /queue/global.accounting.cloudtest.central /queue/global.accounting.storagetest.central Or similar www.egi.eu EGI-InSPIRE RI-261323
Message Broker Changes Sites will need host to be in GOCDB Including certificate DN Not monitored Not production Use SSL or not? www.egi.eu EGI-InSPIRE RI-261323
