Data Challenges and Farm Expansion Plans for 12 GeV User Group Board of Directors Meeting
"Explore the data challenges and growth plans for a 12 GeV farm discussed at the User Group Board of Directors Meeting. Learn about capacity upgrades, workflow enhancements, and overcoming data scale goals in advance."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Computing Update Data Analysis (farm) for 12 GeV User Group Board of Directors Meeting Chip Watson Scientific Computing, Deputy CIO Outline Data challenges, farm capacity growth Plans for petabytes Workflow & related topics
Quick Overview of Expansions FY14: Not much happening. Improve software & operations. FY15: First major 12 GeV farm upgrade (5K-6K cores) FY16: Major LQCD upgrade Second major 12 GeV farm upgrade (tbd) Add second tape library
Data Challenges for 12 GeV Goal: 10% scale 24 months in advance 25% scale 18 months in advance 50% scale 12 months in advance 100% scale 6 months in advance Test everything downstream of data acquisition transfer of data from hall to data center near-live analysis (data buffer on disk) push to tape pull from tape + offline analysis
Data Challenges for 12 GeV Farm / LQCD node sharing: move nodes Hall D: online at 5000 cores May 2015 10% done 25% Feb 2014, will loan 1K+ cores, so farm is at 2.2-2.5K, with Hall D using half, so simulating real competing load 50% late summer 2014, will loan 2K 2 K cores, and might allow ongoing use of 1000 cores until FY15 cluster comes online 100% January 2015, new FY15 farm nodes go online, support final data challenge
Offline 2014 Evolution Workflow tools define & track a workflow , consisting of many jobs, tasks, file I/O operations auto-retry on failed jobs way to query (or see online) how much progress the workflow has achieved add / remove tasks from workflow as it is running Write through disk cache never fills, overflows to tape can be used by Globus Online WAN file transfers to write to Jlab tape library Stage-out unused work disks
