High-Throughput Computing in Python with HTCondor Week 2019
Explore the power of High-Throughput Computing in Python using HTCondor. Learn about common problems in scientific computing, simulation analysis, job submission, and management with HTMap. Discover who HTMap is for, and who it is not for. Delve into independent and interdependent execution strategies, and find relevant links to GitHub for more resources.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
1 High-Throughput Computing in Python, Powered by HTCondor Josh Karpel HTMap - HTCondor Week 2019
Common Problems in Scientific Computing 2 What happens to Y as we change X? 500 Simulations 1 Hour Each HTMap - HTCondor Week 2019
3 What happens to Y as we change X? Simulation Y1 X1 Analysis Setup Simulation Y2 X2 Simulation Y3 X3 HTMap - HTCondor Week 2019
4 X = [0, 1, 2, ] Y = map(simulation, X) make_plot(X, Y) HTMap - HTCondor Week 2019
5 HTMap is a Pure-Python, Seamless, Lightweight, Focused HTCondor Job Submitter & Manager HTMap - HTCondor Week 2019
Who is HTMap for? 6 Has a working computation written in Python Knows Scientific Python, but not necessarily anythingelse Wants to use high- throughput computing but is very busy Users Facilitators Well, it works on my laptop Users have similar workflows, but dissimilarskills HTMap - HTCondor Week 2019
Who is HTMap NOT for? 7 The output of my analysis is a file My function takes a millisecond to run But I don t use Python HTMap - HTCondor Week 2019
8 Independent Execution multiprocessing htmap condor_submit concurrent.futures Short-Running Long-Running dask condor_submit_dag Interdependent Execution HTMap - HTCondor Week 2019
9 HTMap https://github.com/htcondor/htmap Slides and Code https://github.com/JoshKarpel/htcondor-week-htmap-talk HTMap - HTCondor Week 2019
