VLASS Coarse Cube Imaging Challenges and Solutions
Explore the complexities of VLASS coarse cube imaging, from data rates to workflow issues. Learn about proposed changes to streamline the imaging pipeline and enhance efficiency. Discover the impact of creating separate data staging and imaging pipelines.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
VLASS Coarse Cube Imaging Case James Robnett August 26,27 CHTC/NRAO Site Visit 1
VLASS Coarse Cubes Description VLASS observes 20 square degrees / hour at a data rate of ~80GB/hour. Single square degree is ~4GB of raw data. Calibration results in 3 copies, raw, model and corrected giving ~12GB/square degree aggregate. Observations are typically 4 or 8 hours in length 80 to 160 square degrees Results in full calibrated data sets of either 960GB or 1920GB (bit less in practice due to flagging) Imaging is performed across 2 square degrees with a 1 square degree cut out for the final image Input data for imaging is therefore either 8GB (corrected column only) or 24GB (raw, model, corrected columns) Coarse cube imaging is performed per spectral window (SPW) 16 SPWS of 128 1MHz channels each (2GHz total band width). August 26,27 CHTC/NRAO Site Visit 2
VLASS Coarse Cubes Cube imaging flow diagram August 26,27 CHTC/NRAO Site Visit 3
VLASS Coarse Cubes Workflow issues Imaging pipeline of single square degree requires 60 to 120x more input data than is actually used for imaging 2 square degrees of corrected data vs 40 or 80 square degrees of raw, model and corrected data Hif_importdata() requires ~5 hours to scan input data Imaging proceeds vs all 16 SPW Results in 72 to 96 hour executions Requires large(ish) input data (8GB) Issues within a single square degree require retention of full calibrated MS August 26,27 CHTC/NRAO Site Visit 4
VLASS Coarse Cubes Proposed changes Create separate data staging and imaging pipelines Reduces imaging pipeline input data to only the relevant data Requires adding split calibrated product tracking to workflow manager Allows per square degree clean up of input data, currently if one square degree needs reprocessing we require full calibrated data. Replace hif_importdata() with hif_restore() Faster context creation though less of an issue if importing 8GB. Image per SPW Reduces input data to 500MB Reduces per job runtime to many ones of hours Per square degree run time reduced via concurrency Requires weblog merge function if we re to avoid Data Analyst impact August 26,27 CHTC/NRAO Site Visit 5
VLASS Coarse Cubes Data split flow diagram Prestage split data Allows rapid clean up of calibrated MS August 26,27 CHTC/NRAO Site Visit 6
VLASS Coarse Cubes Revised Imaging flow diagram Pipeline executed per SPW Much smaller data transport Complicated QA step by Analysts with 16 disjoint weblogs August 26,27 CHTC/NRAO Site Visit 7
chtc.cs.wisc.edu www.nrao.edu science.nrao.edu public.nrao.edu The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc. August 26,27 CHTC/NRAO Site Visit 8
