High Performance Computing Overview at UC Davis

Slide Note

Experience a brief look into High Performance Computing resources offered by the HPC Core Facility at UC Davis. Gain insight into cluster terminology, node functionalities, and cluster anatomy. Discover the benefits of utilizing a cluster for computational tasks.

kristyan Follow

Uploaded on Feb 23, 2025 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

Download Presentation

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

E N D

Presentation Transcript

Introduction High Performance Computing at UC Davis 1

Introduction What this talk is Brief overview of the resources offered by the High Performance Computing Core Facility How to get access Very brief introduction to cluster terminology and concepts 2

Introduction What this talk isn t In-depth discussion of UNIX, clusters, or computing in general Finished! It s a living document, and your questions will help improve future versions! 3

Introduction Who am I? Michael Casper Lewis Systems Architect, HPC Core Facility 25 years of experience managing UNIX at scale, 20 years of HPC cluster design and administration. 4

Introduction If you were plowing a field, which would you rather use: Two strong oxen or 1024 chickens? Seymour Cray 5

Introduction What is a cluster? A collection of machines that work together Why use a cluster? Chickens are cheaper than oxen There is a limit on the size of a single ox but try getting 1024 chickens to move in the same direction 6

Cluster Terminology Node: An individual computer in the cluster Login (or head) node: The node users log in to and use to submit jobs. Scheduler (and queue): The entity that coordinates which jobs run at what time on which node(s) CPU Core: A single processing unit Thread: Processing unit that shares resources with other threads Interconnect: The network connecting the nodes to each other and to the storage Core (memory): RAM Swap: If RAM is exceeded, virtual memory will be used, using disk instead of RAM. (Much slower) Thrash: Competing processes swapping at the same time. 7

Anatomy of a cluster 8

Head/login node connects to the Internet and to the cluster interconnect Nodes connect only to the interconnect 9

The cluster cafe You can think of the cluster as a restaurant: Jobs are parties coming to eat Tables are the nodes of the cluster Not all tables will be the same size, or have the same attributes. Your queue wait times will depend on what resources you are asking for. A single free seat at a shared table is quicker to free up than a table for 10. 10

The cluster cafe It is possible to request other resources when you submit your job "I'd like a table by the window" Or even request a specific node "I want to sit at table 2. I'll wait." Normally the scheduler will seat as many jobs at the same table as it can, but you can request a table by yourself with --exclusive The scheduler will wait to run your job until it can fulfill your requirements. It is important to remember that you are only requesting resources. Just like requesting a table for 10 doesn t mean 10 people will show up to lunch, requesting 10 CPUs doesn t mean your job will magically use all 10. 11

Slurm We use Slurm as the scheduler for all of the clusters on campus. If you are familiar with another scheduler, (SGE, LSF, ), you can use the Scheduler Rosetta Stone to translate: https://slurm.schedmd.com/rosetta.pdf 12

Basic Slurm Concepts Nodes are grouped into partitions (which can overlap) Your access will be determined by QOS (Quality of Service) or fairshare, depending on the cluster. For QOS-based clusters, you will have high priority access to your group s resources, and lower priority access to others . Low priority access can suspend or terminate and requeue if higher priority jobs need to run. Fairshare clusters use dynamic priority to achieve your lab s buy-in over time 13

Using the scheduler sbatch/srun/salloc: submit jobs scontrol show {job|node}: get job/node status squeue: show jobs in the queue sprio: show job priorities sinfo: report state of partitions and nodes scancel: cancel/stop a job scontrol update: change a job after submission sacct: view accounting information sshare: view fairshare information 14

Example job script: #!/bin/bash # options for sbatch #SBATCH --job-name=name # Job name #SBATCH --nodes=1 # will usually be 1 unless you are doing MPI #SBATCH --ntasks=1 # number of cpus to use #SBATCH --time=30 # Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days- hours:minutes" and "days-hours:minutes:seconds". #SBATCH --mem=500 # Memory pool for all cores (see also --mem-per-cpu) #SBATCH --partition=production # cluster partition 15

Example continued # for calculating the amount of time the job takes and echo the hostname begin=`date +%s` echo $HOSTNAME # Sleep for 5 minutes sleep 300 # getting end time to calculate time elapsed end=`date +%s` elapsed=`expr $end - $begin` echo Time taken: $elapsed 16

srun An interactive session can be requested with srun. Any options given in the script with #SBATCH are equivalent to using the same options on the command line when submitting, so make sure you copy the important ones into your srun command. Note that all of these will be treated as comments by bash, so it is still possible to run the script standalone. An interactive session can be requested with srun: srun --time 30 --mem 500 --pty /bin/bash Once you are allocated resources, you can run your script interactively for debugging purposes. Note that all of the SLURM options will be treated as comments by the shell, so running the script as-is is fine: node$ ./job.sh 17

sbatch Once you have verified that your job is working, you can submit it to run in batch mode: sbatch job.sh 18

Querying Slurm squeue - list your currently queued/running jobs sprio - list job priorities sinfo - show available partitions 19

scancel - cancel job (running or in queue) To cancel a single job: scancel <jobid> To cancel all your jobs, you can use -u with your username: scancel -u <username> 20

sacct - view resources used by your job sacct --format=JobID,JobName,MaxRSS,Elapsed,State There are lots of parameters collected. sacct -l -j jobid will show many of them. sacct -e will display a list of metrics you can query. Use -S and -E for the start and end of the query. Default is today. 21

More Slurm Gres (Generic Resource) - things like a GPU that are special to a node Task arrays -w to specify a node (should not need to do in most circumstances) -x to exclude nodes Should use node features instead in most cases Reservations 22

Slurm environment variables SLURM_NNODES=1 SLURM_JOBID=56107292 SLURM_NTASKS=1 SLURM_TASKS_PER_NODE=1 SLURM_CPUS_PER_TASK=4 ... 23

Priority You can view your lab s current priority on the computing portal under the slurm accounts tab. You can also view your priority on the command line line using the sshare command: Account User RawShares NormShares RawUsage EffectvUsage FairShare rcc 712954 0.004252 1183660 0.000097 rcc adam parent 0.004252 0 0.000000 0.789897 rcc buduchin parent 0.004252 0 0.000000 0.789897 rcc mclewis 1 0.111111 1697 0.001434 0.763490 24

sprio To view where your jobs are in the queue, you can use the sprio command. 25

Data transfer How to transfer data to and from the cluster scp - fine for individual files, GUI clients available for OSX/Windows rsync - better for directory structures git - great for scripts globus - non-free software, can be very difficult to use ftp - very old protocol, can be problimatic on modern networks, but sometimes you have no choice. avoid transfers in the first place! Work in-place on the login node. 26

scp Old classic. Quick and easy to use from the command line, and GUI clients are available for common platforms. mobaXterm has built-in file transfer winscp cyberduck filezilla 27

scp example Example: scp local_file.txt username@cluster.hpc.ucdavis.edu:remote_file.txt If you leave off the remote_file.txt, the remote file will have the same name as the local file: scp local_file.txt username@cluster.hpc.ucdavis.edu: If you forget the colon, scp will revert to a local cp and you will create a file called username@cluster in your current directory! 28

rsync Used to synchronize (can go both ways) entire directory structures. Gold standard for transfers of large collections of files, but performance can suffer when transferring large numbers of small files. Copy only differences, don t re-copy files that have not changed (enables resumption of inturrupted transfers) rsync -a local_dir username@cluster.hpc.ucdavis.edu: a for archive . Will try to preserve permissions, timestamps, etc. 29

git Push/pull from a remote repository. Allows development/testing of code on your local workstation, push that code to repo, pull it on cluster. And you get version control as a bonus! 30

ftp Recommended clients lftp ncftp It is possible to transfer data in/out of box with ftp. (See HPC core website FAQ) 31

Avoid transfers in the first place Use wget/curl to retrieve data sets directly to cluster storage Edit files in-place on a login node Possible to use IDEs like visual studio 32

Use the transfer node Some clusters have a dedicated transfer node 33

Cluster Etiquette Login Node 34

Things to do Use the scheduler! (i.e. do not ssh to nodes to run) Give good estimates of your job s needs (sacct is your friend!) If your job can be split into smaller sections, you may get scheduled sooner, as the scheduler will try to backfill shorter jobs. Limit use of interactive sessions Task arrays are your friend Checkpoint your job if possible Make use of local storage (/scratch/) on the nodes for intermediate results 35

Things NOT to do Run jobs on the head node Many simultaneous writes to network filesystem- use local scratch if you can! Go around scheduler and run directly on the nodes 36

Getting access to compute resources Farm, Franklin, Peloton https://hippo.ucdavis.edu/ HPC2 https://hpc.ucdavis.edu/form/account-request-form LSSC0 - compute portal https://computing.genomecenter.ucdavis.edu/ Legacy clusters (crick, hpc1, ) https://wiki.cse.ucdavis.edu/cgi-bin/index2.pl 37