Advanced User Interfaces and APIs for High-Performance Computing
Explore how to run models on HPC, manage data, and organize research on patient data integrated with PLGrid infrastructure. Learn about tools like Prometheus, Rimrock, PLGData, and GitLab for efficient model execution and data management in a secure environment.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
T2.5 (M1-M48) (HPC) User Interfaces and APIs Marek Kasztelnik, Tomasz Guba a, Marian Bubak | Academic Computer Centre Cyfronet AGH, Krak w, Poland 12/12/2018 http://dice.cyfronet.pl/ H2020 EU PROJECT | Topic SC1-DTH-07-2018 | GA: 826494
T2.5 User Interfaces and APIs Goal: run models on HPC, browse models, inputs and outputs 5 main elements: 1. Infrastructure -> Prometheus (T2.3) 2. Run jobs on Prometheus using a REST API -> Rimrock (https://submit.plgrid.pl) 3. Manage data stored on Prometheus using Web and/or a REST API -> PLGData (https://data.plgrid.pl) 4. Models repository and versioning -> GitLab 5. Organize model execution on patient data -> Model Execution Environment (MEE) 2 5 1 4 3 2
T2.5 Model Execution Environment Organize research on patient data Integrated with PLGrid infrastructure (automatic execution on the HPC cluster and data management) Give possibilities to upload and download files to/from Prometheus storage But can be integrated for PRIMAGE with other file storage infrastructure (cloud) Connected with GitLab repositories for model versioning Simulations are organised in pipelines (more on them in the WP5 presentation) Patient and relevant clinical data Run Browse inputs and outputs Select model version 3
T2.5 Rimrock Robust Remote Process Controller Submit Prometheus Computations using REST API Possibility to submit and monitor Slurm jobs executing on the Prometheus supercomputer using a REST API Secured by PLGrid authentication and authorisation framework Example - submit a job to Prometheus using the curl tool: proxy="`cat {path-to-proxy-file} | base64 | tr -d '\n'` curl -k -X POST --data '{"host":"prometheus.cyfronet.pl", "script":"#!/bin/bash\n#SBATCH -A {grantid}\n echo hello\n exit 0"}' \--header "Content-Type:application/json" --header "PROXY:$proxy" https://submit.plgrid.pl/api/jobs In PRIMAGE the MEE environment will do this on your behalf 4
T2.5 PLGData Browse Prometheus files Prometheus web file browser and a REST API Possibility to create, delete, rename and change access rights to files on the HPC cluster file system using a Web UI and a REST API Secured by PLGrid authentication and authorisation framework Example - download a file from Prometheus: curl -X GET https://data.plgrid.pl/download/people/plguserlogin/graph.png --data-urlencode proxy="`cat grid_proxy`" 5
T2.5 Graphical User Interface for Prometheus Jobs Additional Option: using a GUI inside a running Prometheus job Prometheus supports starting Graphical User Interfaces from inside of a running Slurm job (e.g. Matlab UI or Ansys RSM): 1. Load the pro-viz module 2. Start a new job using pro-viz command (not supported yet by Rimrock) 3. Use VNC to start Graphical User Interface on job resources Supported applications: Ansys RSM Ansys Electronic Desktop (Maxwell) Matlab Mathematica 6
http://dice.cyfronet.pl/ Marek Kasztelnik | m.kasztelnik@cyfronet.pl
