ARC Middleware for Computing Grids
"Learn about the ARC Middleware, a tool designed to enable computing grids for LHC experiments. Developed by NorduGrid, ARC provides interfaces to computing resources, job control, data staging, and more. Discover the key components and interfaces of ARC, optimized for HPC deployment and open-source collaboration."
Uploaded on | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
ARC overview Oxana Smirnova ISGC 2019, Taipei
What is ARC Middleware to enable computing grids Motivated by the needs of LHC experiments Main goal: common interface to disparate computing facilities First public preview in 2002, first formal release in 2004 Developed by NorduGrid since 2001 Designed with a distributed Nordic Tier1 in mind Optimised for HPC deployment Built-in data caching Open Source, mostly volunteer contributors Coordinated by the NorduGrid Collaboration Supported by EU in past, NeIC now (partially) 1/4/2019 www.nordugrid.org/arc 2
Key ARC components Key components: ARC CE a Compute Element, providing interfaces to computing resources Modular, consists of several sub-components (services and utilities) Interface for job control Interface for exposing resource and job status info Data staging and shared cache management utilities Jobs do not need to stage data in or out CLI client tools to interact with ARC CE and relevant third-party services CLI for jobs management CLI for X509 proxy management (client to VOMS) CLI for file transfer (a wide range of protocols) API: C++ and Python, for interfacing to full software stack Enables custom services and clients, including arcControlTower (aCT) 1/4/2019 www.nordugrid.org/arc 3
ARC CE components and interfaces 1/4/2019 www.nordugrid.org/arc 4
ARC-CE components on a (SLURM) cluster Supported batch systems and back-ends: SLURM HTCondor SGE PBS/Torque LoadLeveler LSF BOINC Job forking Local submission Remote submission via ssh NB: only SLURM is currently available to testers, community help with other systems is much welcomed! Head Node Session directory Runtime env. ARC6 Control directory CA infoproviders DTR Cache certificates SLURM controller A-REX gridftp/http Runtime env. ARIS JURA users user mapping Shared file system Legend Worker Node SLURM daemon (ARC components in orange): files Worker Node SLURM daemon processes 1/4/2019 www.nordugrid.org/arc 5
Customisation, optimisation for data-intensive jobs Cluster Worker Node Worker Node Worker Node Worker Node ARC-CE is a complex service Cluster or HPC Highly customisable, works on any system cache Storage Other CE ARC-CE Many sub-components => careful configuration ARC-CE can do all the data transfers Works best with high-end storage for cache Allows to cache frequently used files Minimizes bandwidth Maximizes worker nodes usage efficiency 1/4/2019 www.nordugrid.org/arc 6
Supported workflows and deployment modes Out of the box: ARC client discovers matching resources and pushes jobs Classical HPC workflow Suitable for smaller communities and testing ARC-CE + ARC Control Tower (aCT): aCT acts as a centralised client, still a push model Used for the Nordic Tier1 and some other HPC sites Dry ARC-CE deployment with pilot jobs No use of ARC-CE data management and cache Most ARC-CE deployments at WLCG are like this Closest analogy to CREAM-CE 1/4/2019 www.nordugrid.org/arc 7
Integration with WLCG and EGI ARC-CE is fully integrated into WLCG and EGI operations Registered service in GOCDB Accounting reports sent to APEL by ARC s JURA module GLUE2 info Glue1 is possible but strongly discouraged and not supported Part of UMD releases Undergoes standard EGI certification User support via GGUS ARC unit Widely used by ATLAS sites, also by LHCb, CMS, ALICE and smaller VOs that are supported by respective WLCG sites 1/4/2019 www.nordugrid.org/arc 8
ARC-CE instances in GOCDB ARC-CE in EGI 100 90 80 70 60 50 40 30 20 10 0 Oct-12 Mar-13 Jun-13 Sep-13 Dec-13 Mar-14 Jun-14 cf. CREAM-CE: 371 instances as of today 1/4/2019 9 www.nordugrid.org/arc
ARC-CE geography Data as of February 2019 1/4/2019 www.nordugrid.org/arc 10
Next Big Thing Teaser: ARC6 All details in the next presentations! Same interfaces, mostly the same design Backwards compatible with clients REST interface added Completely different configuration Everything in a single config file, like in ARC5 But a totally different structure, no automatic conversion Some new services and cleaned-up internals New code repository (NeiC Coderefinery gitlab) and release procedures 1/4/2019 www.nordugrid.org/arc 11
ARC release cycle Overview: http://www.nordugrid.org/arc/releases/ Current stable release: ARC 15.03 update 20 A.k.a. ARC5 Contains ARC core packages tagged 5.4.4 Includes documents (tag 2.0.21), Nagios probes (tag 1.9.1) etc After ARC6 comes out, only critical bug fix releases Apart of that, only the current release is supported, no backporting of fixes to old tags Coming release: ARC6 Will include only ARC packages themselves Documentation moves online, separate releases for Nagios probes Tagging scheme: 6.m.0 for backwards compatibility No distinction between feature and bugfix releases Releases triggered by demand, no fixed schedule 1/4/2019 www.nordugrid.org/arc 12
Documentation, support, availability Documentation: ARC5: documentation is distributed with the software ARC CE sysadmin guide is the must ARC6: revamped documentation online at http://www.nordugrid.org Still in works Support: For those familiar with GGUS, submit tickets to ARC unit For community support, subscribe to either: nordugrid-discuss@nordugrid.org generic CERN e-group wlcg-arc-ce-discuss@cern.ch WLCG-specific For bug reports and feature requests, submit tickets to: https://bugzilla.nordugrid.org Code: https://source.coderefinery.org/nordugrid/arc Linux packages: Global Linux repositories (CentOS, Debian, EPEL) Upstream: http://download.nordugrid.org/repos.html 1/4/2019 www.nordugrid.org/arc 13
