Exploring Scalability Issues in High-Performance Computing
Delve into key concepts and discussions on scalability challenges in high-performance computing, as presented at the "Scaling to New Heights" workshop. Explore factors influencing application scalability, performance goals, and strategies for addressing issues such as granularity, latencies, synchronization, and load balancing.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
NSF Home Page NCSA Home Page San Diego Supercomputer Center Home Page Scaling to New Heights Retrospective IEEE/ACM SC2002 Conference Baltimore, MD [Trimmed & Distilled for SOS7 by M. Levine 4-March-2003, Durango] SOS7, Durango CO, 4-Mar-2003
Contacts and References David O Neal oneal@ncsa.uiuc.edu John Urbanic urbanic@psc.edu Sergiu Sanielevici sergiu@psc.edu Workshop materials: www.psc.edu/training/scaling/workshop.html SOS7, Durango CO, 4-Mar-2003
Introduction More than 80 researchers from universities, research centers, and corporations around the country attended the first "Scaling to New Heights" workshop, May 20 and 21, 2002, at the PSC, Pittsburgh. Sponsored by the NSF leading-edge centers (NCSA, PSC, SDSC) together with the Center for Computational Sciences (ORNL) and NERSC, the workshop included a poster session, invited and contributed talks, and a panel. Participants examined issues involved in adapting and developing research software to effectively exploit systems comprised of thousands of processors. [Fred/Neil s Q1.] The following slides represent a collection of ideas from the workshop SOS7, Durango CO, 4-Mar-2003
Basic Concepts All application components must scale Control granularity; Virtualize Incorporate latency tolerance Reduce dependency on synchronization Maintain per-process load; Facilitate balance Only new aspect, at larger scale, is the degree to which these things matter SOS7, Durango CO, 4-Mar-2003
Poor Scalability? (Keep your eye on the ball) Speedup Processors SOS7, Durango CO, 4-Mar-2003
Good Scalability? (Keep your eye on the ball) Speedup Processors SOS7, Durango CO, 4-Mar-2003
Performance is the Goal! (Keep your eye on the ball) Speedup Processors SOS7, Durango CO, 4-Mar-2003
Issues and Remedies Granularity Latencies Synchronization Load Balancing Heterogeneous Considerations [Q2a] [Q2b] [Q2c] SOS7, Durango CO, 4-Mar-2003
Granularity Define problem in terms of a large number of small objects independent of the process count [Q2a] Object design considerations Caching and other local effects Communication-to-computation ratio Control granularity through virtualization Maintain per-process load level Manage comms within virtual blocks, e.g. Converse Facilitate dynamic load balancing SOS7, Durango CO, 4-Mar-2003
Latencies Network Latency reduction lags improvement in flop rates; Much easier to grow bandwidth Overlap communications and computations; Pipeline larger messages Don t wait Speculate! [Q2b] Software Overheads Can be more significant than network delays NUMA architectures Scalable designs must accommodate latencies SOS7, Durango CO, 4-Mar-2003
Synchronization Cost increases with the process count Synchronization doesn t scale well Latencies come into play here too Distributed resource exacerbates problems Heterogeneity another significant obstacle Regular communication patterns are often characterized by many synchronizations Best suited to homogeneous co-located clusters Transition to asynchronous models? SOS7, Durango CO, 4-Mar-2003
Load Balancing Static load balancing Reduces to granularity problem Differences between processors and network segments are determined a priori Dynamic process management requiring distributed monitoring capabilities [Q2c] Must be scalable System maps objects to processes SOS7, Durango CO, 4-Mar-2003
Heterogeneous Considerations Similar but different processors or network components configured within a single cluster Different clock rates, NICs, etc. Distinct processors, networking segments, and operating systems operating at a distance Grid resources Elevates significance of dynamic load balancing; Data-driven objects immediately adaptable SOS7, Durango CO, 4-Mar-2003
Tools [Q2d?] Automated algorithm selection and performance tuning by empirical means, e.g. ATLAS Generate space of algorithms and search for fastest implementations by running them Scalability prediction, e.g. PMaC Lab Develop performance models (machine profiles; application signatures) and trending patterns Identify/fix bottlenecks; choose new methods? SOS7, Durango CO, 4-Mar-2003
Topics for Discussion How should large, scalable computational science problems be posed? Should existing algorithms and codes be modified or should new ones be developed? Should agencies explicitly fund collaborations to develop industrial-strength, efficient, scalable codes? What should cyber-infrastructure builders and operators do to help scientists develop and run good applications? SOS7, Durango CO, 4-Mar-2003
Summary Comments (MJL) Substantial progress, with scientific payoff, is being made. It is hard work without magic bullets. >>> Dynamic load balancing <<< Big payoff, homogeneous and heterogeneous Requires considerable people work to implement Runtime overhead very small. SOS7, Durango CO, 4-Mar-2003
