ET-PP WP8 Computing and Data Model Slides Barcelona 17 June 2024

et pp wp8 n.w
1 / 15
Embed
Share

"Explore the computing and data model slides prepared by Alba Gonzalvez for the ET-PP Meeting in Barcelona on June 17, 2024. Dive into the division of tasks, workshops, milestones, deliverables, and computing requirements for the ET project."

  • ET-PP
  • Computing
  • Data Model
  • Barcelona
  • Workshop
rayaanacharya
ferrick_e Follow

Uploaded on | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

Download Presentation

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

E N D

Presentation Transcript

  1. ET-PP WP8 Computing and data model Slides prepared by Alba Gonzalvez together with EIB chairs and WP8 work package leaders ET-PP Meeting Barcelona 17 June 2024

  2. EiB-WP8 organization Chairs: Stefano Bagnasco (INFN Torino) , Patrice Verdier (IP2I-IN2P3) ET-PP WP8 ET EiB Division 1: Software, frameworks, and data challenge support Andres Andres Tanasijczuk Tanasijczuk(UC Louvain) Chairs: Achim Stahl, Nadia Tonello Task 8.1 Task 8.1 T0 data center Patrice Patrice Verdier Division 2: Services and Collaboration Support Antonella Antonella Bozzi Bozzi (EGO) Verdier (IP2I-IN2P3) Task 8.2 Task 8.2 Computing and Data Model Anastasios Anastasios Fragkos Fragkos (Geneva) Division 3: Computing and data model, Resource Estimation Gonzalo Merino Gonzalo Merino (PIC) Task 8.3 Task 8.3 Resources Silvio Pardi Silvio Pardi (INFN Napoli) Division 4: Multimessenger alerts infrastructure Steven Schramm Steven Schramm (Universite de Gene ve) Task 8.4 Task 8.4 Data Access Implementation Nadia Tonello Nadia Tonello (BSC) TTG: Technology Tracking working Group Sara Sara Vallero Vallero(INFN Torino) ET-PP Meeting Barcelona 17 June 2024 2

  3. ET-PP WP8 Computing and Data models ET-PP WP8 ET EiB Workshops Workshops WPs WPs Due date Due date M8.1 M8.1 Workflows Requirements collection and constraints: computing and data Workflows Requirements collection and constraints: computing and data Sep 2023 Sep 2023 Oct 2023 Oct 2023 Computing Infrastructures availability for ET workflows, characteristics and sustainability M8.2 WP9 Aug 2024 milestones milestones WP8 WP8 M8.3 On site infrastructure, computing and data model WP6 Aug 2025 M8.4 Low latency and offline workflows and computing model WP6 Dec 2025 M8.5 Data management, access, policy and implementation WP2, WP6 July 2026 Lead Lead Due date Due date Deliverables Deliverables . D8.1 D8.1 Computing and Data Requirements Computing and Data Requirements UniGe UniGe Feb 2024 Feb 2024 Feb 2024 Feb 2024 WP8 WP8 D8.2 Computing and Data Model UniGe Feb 2026 D8.3 Data Access Implementation Guidelines IFAE July 2026 ET-PP Meeting Barcelona 17 June 2024 3

  4. Computing and data requirements (D8.1) 1. Introduction ET-PP WP8 ET EiB 1.1 Scope 1.1.1 Low latency pre-merger alerts 1.2 Future ET Computing Model 1.3 Template for requirements collection 2. ET Science drivers of computing requirements 2.1 Low latency science drivers 3. Computing requirements 3.1 Computing and data processing requirements 3.1.1 Online requirements 3.1.2 Low latency requirements 3.1.3 Offline requirements 3.2 Collaboration Algorithms 3.3 Collaboration Support 3.4 Data Management 4. Computing requirements summary tables 4.1.1 Online requirements tables 4.1.2 Low latency requirements tables 4.1.3 Offline requirements tables 5. Computing and software expertise 6. Technological assessment 7. Conclusions Taken Taken from resources expected for ET is given in section 4. These expected expected to to evolve evolve and and will will be accurately predict its computing needs. Nevertheless, the baseline scenario presented here represents a significant but modest computing challenge compared to that faced by the HL-LHC experiments. The greatest challenge for ET will be training and retaining sufficient software and computing expertise to be able to leverage common tools, and create and maintain the bespoke solutions needed by ET over the long lifetimeof the experiment. from the the D D8 8. .1 1 summary summary: : The summary for the computing These numbers numbers are with MDCs MDCs so that ET can are be tracked tracked with ET-PP Meeting Barcelona 17 June 2024 4

  5. Computing and data requirements (D8.1) ET-PP WP8 ET EiB ONLINE LOW-LATENCY Low-lat. search pipelines Full bandwidth raw DAQ Monitoring Detector control Automation h(t) reconstruction Alert generation system Sky localization Parameter estimation Bulk data raw Event DB Uncalibrated aggregated h(t) Alerts OFFLINE Calibrated aggregated h(t) Derived data Offline data processing Reduced datasets Deep searches Precise parameter estimation Frequency-domain searches Figure credit: Stefano Bagnasco ET-PP Meeting Barcelona 17 June 2024 5

  6. Computing and data requirements (D8.1) ET-PP WP8 ET EiB ONLINE Full bandwidth raw DAQ Monitoring Detector control Automation h(t) reconstruction Bulk data raw Uncalibrated aggregated h(t) Calibrated aggregated h(t) Reduced datasets Figure credit: Stefano Bagnasco ET-PP Meeting Barcelona 17 June 2024 6

  7. Computing and data requirements (D8.1) ET-PP WP8 ET EiB LOW-LATENCY Low-lat. search pipelines Alert generation system Sky localization Parameter estimation Event DB Alerts ET-PP Meeting Barcelona 17 June 2024 7

  8. Computing and data requirements (D8.1) ET-PP WP8 ET EiB OFFLINE Derived data Offline data processing Deep searches Precise parameter estimation Frequency-domain searches ET-PP Meeting Barcelona 17 June 2024 8

  9. ET-PP WP8 Computing and Data models ET-PP Workshops Workshops WPs WPs Due date Due date M8.1 M8.1 Workflows Requirements collection and constraints: computing and data Workflows Requirements collection and constraints: computing and data Sep 2023 Sep 2023 Oct 2023 Oct 2023 WP8 milestones WP8 milestones M8.2 M8.2 Computing Infrastructures availability for ET workflows, characteristics Computing Infrastructures availability for ET workflows, characteristics WP9 WP9 Aug 2024 Aug 2024 July July 2024 2024 M8.3 On site infrastructure, computing and data model WP6 Aug 2025 M8.4 Low latency and offline workflows and computing model WP6 Dec 2025 M8.5 Data management, access, policy and implementation WP2, WP6 July 2026 Deliverables Deliverables Lead Lead Due date Due date Documents Documents . D8.1 D8.1 Computing and Data Requirements Computing and Data Requirements UniGe UniGe Feb 2024 Feb 2024 Feb 2024 Feb 2024 WP8 WP8 D8.2 Computing and Data Model UniGe Feb 2026 D8.3 Data Access Implementation Guidelines IFAE July 2026 ET-PP Meeting Barcelona 17 June 2024 9

  10. EiB-WP8 second workshop in preparation ET-PP WP8 ET EiB M M8 8. .2 2Computing ComputingInfrastructures Host: University of Napoli (Silvio Pardi, Task Leader 8.3 Resources) Infrastructuresavailability availabilityfor for ET ET workflows, workflows, characteristics characteristics Date Date: 8-9-10 July 2024 - Remote connection option available for those unable to attend in person - Proposed schedule: Monday noon to Wednesday afternoon Agenda Agendadraft Indico link: https WP8+EiBDiv3, with the contribution of WP9. - Computing resources (following the requirements coming from MDCs etc. ) - Sustainability of the different options - Inviting external experts from WLCG (LHC computing) and technology tracking draft https: ://indico //indico. .ego ego- -gw gw. .it/event/ it/event/723 723/ / Please register ASAP if interested ET-PP Meeting Barcelona 17 June 2024 10

  11. OSCARS proposals ET-PP WP8 ET EiB First call Submission deadline: 15thMay It was a coordinated, team effort from EIB and WP8. ETAP: PI Paul Laycock, MADDEN: PI Federica Legger, Co-I: Andres Tanasijczuk ; also including Stefano Bagnasco and Lia Lavezzi in the writing team ETAP ETAP(Universite de Gene ve) Access to multiple ESCAPE DataLakes. Rich metadata service integration Access to multiple rich metadata instances A lightweight CRM service monitoring the VRE, useful for monitoring and measuring MDCs MADDEN MADDEN(INFN-TO & Universite Catholique de Louvain) Multi-RI DataLake managed with Rucio, simulating data exchange with CE Development and test of RucioFS (POSIX FUSE mount of the Rucio catalogue) Extend RucioFS to support advanced metadata Strong synergy with Virgo O5 Second OSCARS call (November?) Streaming data for LL? Something IVOA-related? ET-PP Meeting Barcelona 17 June 2024 11

  12. coming soon MDCs closer collaboration ET-PP WP8 ET EiB Feedback Feedbackon on ET ET Symposium Symposium Establish better communication between OSB Div10-EiB for MDCs. Agreement on the need to set clear goals and deliverables for future MDCs. Discussion on establishing common resources for benchmarking different approaches and best practice guidelines within groups. Might consider a task force to manage MDCs and gather feedback on resources and tools. Proposal for code reviews to expedite progress. Plans to draft guidelines for software engineering and code management Plans to circulate a document describing activities and solicit input from interested parties. ET-PP Meeting Barcelona 17 June 2024 12

  13. coming soon Tool evaluation kits ET-PP WP8 ET EiB We will create a matrix to identify urgent functionalities and tools needed. Independent packaged parts of the final architecture Provide limited functionalities, possibly some as mere demonstrators To be released to users (i.e., they MUST be functional) Different implementations may exist, with different tools/technologies used to provide same functionality Integration of existing tools, no excessive bespoke developments, to map kits onto smallish projects Examples: ESCAPE Datalake + RucioFS for data distribution IAM-basedAAI ESCAPE Datalake + VRE interactive data analysis OSDF + INFNCloud interactive data analysis Packaged and quality-tested MDC data generation tool Paul s rich metadata tool ET-PP Meeting Barcelona 17 June 2024 13

  14. WP9 session: Sustainability aspects of computing ET-PP WP8 ET EiB Sustainability aspects of computing will be a major theme of the July workshop in Napoli Hardware sustainability Hardware sustainability - Consider the carbon pollution emitted during the creation and disposal of ET on-premise computing devices. - Maximize the lifespan of the ET-fully-dedicated hardware - Maximize the utilization time of the shared devices (cloud) Software efficiency Software efficiency - Expertise for code optimization, parallelization and scalability - Profiling and maintenance of code - Adaptation to heterogenous technology and its evolution Energy awareness Energy awareness - Power Usage Effectiveness of computing centers - Energy proportionality: energy consumption and effective work done (i.e. inefficient code) - Baseline (Idle states) consumption awareness (i.e. disk data usage) - Transparency regarding the sources of energy employed (clean, green, renewable). ET-PP Meeting Barcelona 17 June 2024 14

  15. Thank you! ET-PP Meeting Barcelona 17 June 2024 15

Related


No related presentations.

More Related Content