Challenges and Approaches for Providing a Pipeline as a Service
Explore the challenges and approaches in offering a pipeline as a service, including issues such as flexibility vs. responsibility, resource allocation, and failure handling. Learn about the experiences shared at ELIXIR All Hands 2017 in Rome, Italy.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Challenges and approaches for providing a pipeline as a service Lars Ailo Bongo (ELIXIR-NO) ELIXIR All Hands 2017, 21-23 March, Rome, Italy www.elixir-europe.org
META-pipe: marine metagenomics analysis pipeline 2 as-a-service interfaces: Galaxy interface for Norwegian users (since 2015) Web app interface (to be released in 2017) Compute intensive Small team Academic setting ELIXIR All Hands 2017, 21-23 March, Rome, Italy
META-pipe backend architecture ELIXIR All Hands 2017, 21-23 March, Rome, Italy
Challenges What should be exposed to users? Trade-off: flexibility vs responsibility as-a-VM = more flexible, but users must install, allocate resources, handle failures as-a-service = less flexible, but service provider takes more responsibility Issue: What to do when the analysis fails? Who should do it? ELIXIR All Hands 2017, 21-23 March, Rome, Italy
Challenges -examples Number of pipeline parameters More => better analysis results Fewer => easier to predict resource needs Resource allocation By user => can use national resources By service provider => easier to maintain service Failure handling By user => difficult to detect, understand and resolve By service provider => more complicated backend ELIXIR All Hands 2017, 21-23 March, Rome, Italy
META-pipe: Galaxy pipeline Standardized Galaxy interface: + flexible, extensible, powerful - big and complicated software stack Norwegian research infrastructure resources are pre-allocated + Not visible for users and easy to administer - Supercomputers not ideal for workload, industry and non-Norwegian users need to pay Failure handling is mostly left to users (red boxes) + Simpler backend - Difficult to understand failures, wasted resources due to restarts, more support work ELIXIR All Hands 2017, 21-23 March, Rome, Italy
META-pipe: Web app Web app interface: + Easier resource allocation, simpler software stack - Just META-pipe, limited configuration possibilities Bring your own cloud resources + Can use cheapest (/free) resources - Failure handling and support on external resources Failure handling is mostly hidden for users + Easy to understand job status - Backend must group together jobs, restart parts of failed jobs, and may need manual intervention ELIXIR All Hands 2017, 21-23 March, Rome, Italy
Summary Our goal: provide a service where resource allocation and failure handling is hidden from user Our approach: Backend designed to hide failures Abstract, layer, and simplify where possible Our challenges: Resources outside of our control (failure detection, handling, and support) Job structure (mix of high-memory and compute intensive) 3rdparty tools (software quality issues, security issues) ELIXIR All Hands 2017, 21-23 March, Rome, Italy
Acknowledgments META-pipe team: ELIXIR-FI and ELIXIR-CZ Nils P. Willassen, Lars Ailo Bongo, Erik Hjerde, Espen M. Robertsen, Inge Alexander Raknes, Aleksandr Agafonov, Terje Klemetsen, Giacomo Tartari AAI, cloud setup EXCELERATE WP6 ELIXIR-NO NeLS ELIXIR All Hands 2017, 21-23 March, Rome, Italy
