Efficient Data Transfer Service: FTS Overview & Key Features
Learn about FTS (File Transfer Service), a bulk data movement service that efficiently transfers files between sites, distributing LHC data across the WLCG infrastructure. Explore its key features, architecture, types of transfers supported, optimizer capabilities, and archive monitoring. Discover how FTS makes data transfers reliable, optimized, and easily manageable, benefiting users in various scenarios.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
EGI Data Transfer Service - FTS Service Owners: Rose Cooper (rose.cooper@stfc.ac.uk) Brian Davies (brian.davies@stfc.ac.uk) 1 23-09-2022
What is FTS? Bulk data movement service Responsible for moving files from one site to another efficiently Distributes the majority of LHC data across the WLCG infrastructure 23-09-2022 2
Key features of FTS Easy to use: Session reuse Web monitoring of transfers Intelligent Scheduler: Balances transfers between many sites Endpoint centric Runtime optimization Robust service: Checksums and retries provided per transfer Can deploy and run zero config Flexible service: Multiprotocol support Interact via REST interface & WebFTS Real-time monitoring 23-09-2022 3
FTS architecture 23-09-2022 4
Types of transfers FTS Third party copy GridFTP XrootD Webdav (HTTP) Tape storage Data Centre A Data Centre B Data transfer Staging Data transfer Data transfer Data Centre C Multihop transfers Staging of data from tapes & archiving data to tape Improved with the new QoS daemon 23-09-2022 5
FTS optimizer Makes it possible to run transfers between any two end points with good reliability and performance Makes it possible to improve throughput and avoid overload Uses a working range: maximum to allow for to other processors/users using the resources minimum stop FTS being too conservative with its estimates Also uses parallel streams 23-09-2022 6
Archive monitoring Feature that follows a file transfer from disc buffer to tape when enabled, a transfer will only show as complete once it has successfully arrived onto tape. Enabled on SRM and XRootD (CERN CTA) protocols Only possible when using the REST interface (e.g. fts- rest-transfer-submit) Can also produces a destination file report additional option that can be set when submitting the transfer request produces a report if the transfer fails because the destination file already exists 23-09-2022 7
WebFTS Web based interface hosted by CERN Able to submit and monitor transfers between endpoints Authenticates via X509 certificates Maintains main functionality of the service with quicker and simpler accessibility 23-09-2022 8
Usecases for WebFTS vs FTS How much data do you need to move across the network? 10 s of GB? 100 s of GB PB?? How often are you likely to need FTS? Single user or many? Technical set up options? 23-09-2022 9
Latest release features Many changes in terms of the software and its deployment: migration from Python2 (+ Pylons framework) to Python3 (+Flask framework) compatibility with MySQL8 (or equivalent) OIDC token refactoring implementation of OC11/GDPR changes Improved view on transfer limits in the FTS web monitoring Overwrite now enabled for intermediary hops in multihop transfers Full support in the Gfal2 clients for the Tape REST API Support FTS in Archiveinfo and release options via the Tape REST API 23-09-2022 10
Current status of FTS at RAL Now have a FTS instance dedicated to the EGI community Accessible via fts3egi.scd.rl.ac.uk will be able to provide dedicated monitoring to ascertain accurate usage metrics by the community Access to FTS via X.509 certificates and VOMS proxy 23-09-2022 11
Future plans for RAL instance Upgrade all production instances of FTS at RAL to the latest version of FTS 3.12.1. provide new functionality to all users of our instances, most significantly the token authentication refactoring Large update with lots of new features! Enable token integration for FTS instances Make EGI-Checkin an available authentication method for the RAL EGI-FTS instance. additionally integrate with IAM Improve user accessibility & integration with the wider EGI community 23-09-2022 12
Using FTS Submit a transfer request via the command line: fts-transfer-submit <url> <source> <destination> Submit a transfer request through WebFTS: Manage data transfer requests with Rucio 23-09-2022 13
Monitoring transfers 23-09-2022 14
Demo 23-09-2022 15
FTS Use cases Experiments which produce large volumes of data Need to move the data to different sites for analysis or storage Able to move data in bulk Can move data to tape as well as disc Manage transfers from multiple VOs Either use built in FTS optimisation or can set own limits 23-09-2022 16
Setting up FTS pre requisits Client software is available for RHEL 6 and 7 derivatives Can be installed from the EPEL repositories for RHEL 7 with the following package: yum install fts-rest-cli y To access the service you will need: an EGI Check-in account An X509 certificate Enrolment to a VO that has access to the services and resources you need 23-09-2022 17
Setting up FTS - configuration Verify configuration In general the commands can be used by specifying the public & private key When performing numerous commands this isn t the most efficient method! 23-09-2022 18
Setting up FTS proxy creation Creating a proxy certificate for the user removes the need to specify the location of the public & private key in each FTS command. voms-proxy init Also need to specify the VO being used for the proxy 23-09-2022 19
Starting a transfer Transfer should also be visible on the online monitoring https://fts3egi.scd.rl.ac.uk:8449/fts3/ftsmon/#/ can search for specific transfers using the ID generated by the fts-transfer-submit 23-09-2022 20
Documentation FTS documentation: https://fts3-docs.web.cern.ch/fts3-docs/ WebFTS: https://webfts.cern.ch/index.php 23-09-2022 21
www.egi.eu/projects/egi-ace Website: www.egi.eu/projects/egi-ace Afbeelding met bijl, vectorafbeeldingen Automatisch gegenereerde beschrijving EGI Foundation EGI Foundation @EGI_eInfra @EGI_eInfra 23-09-2022 22
