Benchmarking for Exascale Computing: Overview & Importance
Exascale computing, performing at the 10^18 FLOPS level, presents challenges in leveraging its capabilities. Benchmarking plays a crucial role in measuring application performance and advancing High Performance Computing. Learn about the ExCALIBUR Benchmarking project and its significance in preparing research codes for exascale hardware. Explore the framework, issues with traditional benchmarking, and how automation enhances reproducibility.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
ADVANCED RESEARCH COMPUTING Benchmarking for the Exascale ExCALIBUR Benchmarking Overview Ilektra Christidi1, Tuomas Koskela1, Mose Giordano1, Emily Dubrovska1 Tom Deakin2, Kaan Olgu2, Chris Maynard3,4, David Case3, Jamie Quinn1 1 - Advanced Research Computing Centre, UCL 2 - Advanced Computer Systems, University of Bristol 3 - Computer Science, University of Reading 4 - Met Office 11th June 2024 Festival of Digital Research & Scholarship
Motivation Why Is Benchmarking Important? Exascale computing refers to supercomputers that can perform at the exascale level 1018 floating point operations per second (FLOPS). Enabling applications to exploit the capabilities of exascale hardware is not straightforward, but benchmarking is a crucial piece of the puzzle. Benchmarking measures the performance of an application in a controlled environment. Assessing performance data is necessary for advancements in High Performance Computing (HPC).
Exascale Computing ALgorithms & Infrastructures Benefitting UK Research Outline What Is ExCALIBUR Benchmarking? Issues with traditional benchmarking: time-consuming (manual process) error prone difficult to reproduce complicated by heterogeneous HPC systems Using a benchmarking framework and suite helps automate benchmarking, which reduces errors and improves reproducibility. ExCALIBUR is a UK-wide project that aims to prepare research codes for the exascale. This benchmarking project is part of their Hardware and Enabling Software (H&ES) theme.
What It Is What It Isn t An authoritative collection of benchmarks. Contributions of benchmark applications are welcome from everyone. A benchmarking campaign. Results aren t compared and featured data is only for illustrative purposes. Curated results. This project doesn t provide a platform for storing data. A framework for automated and reproducible benchmarking. Installable as a Python package excalibur-tests1. Designed to simplify the process of running benchmarks and collecting their outputs. Spreading awareness of good benchmarking practices. 1https://github.com/ukri-excalibur/excalibur-tests
Current Status v1.0.0 2 deployed on most HPC systems. Benchmark apps. System configurations. Documentation + examples. Active contributions. 2 https://zenodo.org/records/11144872
Aspects of a Benchmarking Framework Post- processing Spack ReFrame Analyse Output Benchmark Capture Output Build Benchmark Run Visualise Output Benchmark
Spack Is not itself a build system but manages dependencies and automates builds, ensuring portability across systems. Maintains a repository of build recipes. Supports a variety of build systems. Supports customisable environments.
ReFrame Framework for regression tests and benchmarks tailored for HPC systems. Abstracts interactions with the scheduler and build system. Separates test functionality (and configuration) from low-level implementation (and system configuration). Tests are easy to parametrise and tag. Once Spack and ReFrame are set up, you can execute your benchmark: reframe -c benchmarks/apps/BENCH_NAME -r Benchmark outputs are stored in a performance log (perflog) file.
Post-Processing python post_processing.py log_path config_path Perflog (original) (transformed) Parsing + Unpacking Filtering + Transformation DataFrame Perflog Plotting DataFrame Perflog Plot CSV file Configuration
Streamlit Post-processing can be run as a web app via Streamlit. Plot config is now an optional argument because it can be created from scratch in the UI. streamlit run streamlit_post_processing.py log_path -- [-c config_path]
Future Plans What Comes after v1.0.0? Establish a service guarantee maintenance of the framework and system configurations. Systematic deployment include framework as CI on key HPC systems. Platform for hosting data curate and store benchmark results. Further collaborations and more! Maintain the suite and support benchmark and system additions. Post-processing development. More plot types (scaling, regression). Usability improvements.
Bonus: Performance Portability Assessment of how a code s performance changes when moved to different platforms, environments, or coding paradigms. An appropriate benchmark is required for all these combinations. The plot on the right took 18 months to produce. Tom Deakin, Andrei Poenaru, Tom Lin, and Simon McIntosh-Smith. "Tracking Performance Portability on the Yellow Brick Road to Exascale." In International Workshop on Performance, Portability and Productivity in HPC Held in Conjunction with Supercomputing (P3HPC). IEEE, 2020. https://doi.org/10.1109/P3HPC51967.2020.00006.
Case Study: BabelStream Benchmark of achievable memory bandwidth. BabelStream automated with ExCALIBUR benchmarking framework. The new plot on the right took a few days to produce. T. Koskela et al. "Principles for Automated and Reproducible Benchmarking" In First International Workshop on HPC Testing and Evaluation of Systems, Tools, and Software (HPCTESTS 2023) Held in Conjunction with Supercomputing 2023. https://dl.acm.org/doi/10.1145/3624062.3624133
