Common Software Stack Insights: Workshop Motivations and Opportunities
Discover the motivations and opportunities surrounding the development of a common software stack within the scientific community. Delve into the challenges and scope of implementing such a stack, emphasizing the need for collaboration and innovation in this specialized field.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Towards a Common Software Stack Weidong Li (IHEP), Dario Menasce (INFN), Pere Mato Vila (CERN) 2019-06-13
Motivations of the workshop A common software framework would have to be versatile as to encompass the different needs of the various experiments It should take into account the use of new processors (parallel processing, GPUs, FPGAs, etc.) It should allow to simply define different detector geometries Could we have a common event data model? We are here to discuss these things and more and possibly come out of the workshop with a working plan on how to develop together a common solution
Opportunities vs obstacles towards a CSS Opportunities are pretty obvious and almost needless to mention: Avoid the reinvent-the-wheel syndrome Software reaches maturity faster (larger user base for debugging after deployment) Economy of scale: larger impact for the same amount of funding Avoid duplication efforts Foster the development of interoperability standards 3
Opportunities vs obstacles towards a CSS The idea, therefore, sounds good, but its implementation is fraught with obstacles: Political: who s going to take charge/responsibility of projects? (Experiments? Funding Agencies?...). Easier to accomplish for US (a single country) than for Europe or the world wide community at large. Sociological: the base of developers is rather varied. Some are physicists developing code for experiments (eg. frameworks or DAQ), others are more infrastructure oriented (Grid/Cloud managers), others concerned with specific packages (ROOT, GEANT, simulators etcc..). Mixing these communities, with their own languages has been proven challenging in the past. What about careers? Pressure from industry is growing: can we keep up with it? Technical: who decides for a project? Its inception, its development, its maintenance, its funding 4
Scope of a CSS What is meaningful as a CSS and what is unreasonable to expect MonteCarlo generators, analysis tools (a la ROOT, R, etc ) are obvious candidates Analysis frameworks? Maybe (some caveats exists) An ecosystem of different solutions is always a healthy thing (not too many): it promotes novel ideas and approaches How to accomplish this? Design in advance by exhaustive listing of requirements is difficult but possible (not the most widely adopted approach in our field) DAQ systems? Same kind of considerations as frameworks Unreasonable is to expect customized solutions: projects should strike the right balance between sufficient generality and efficiency 5
Questions: Do we want a common [turnkey] software stack? YES/NO If YES, What should be included in the stack? Should we include package XYX? YES/NO If YES, Who develops and maintains it? Who is responsible of adding and integrate it with the rest of the stack? Who manages the stack as a coherent entity? Who does provide the infrastructure and the user support? If NO, We continue as now, N different stacks with some components that are shared 6
Content of a CSS List of components: - HEP de-facto standard: ROOT, Geant4, HepMC, New HEP libraries: VecCore, VecMath, VecGeom,... Externals: Boost, GSL, Eigen, ... EDM and Geo libraries: DD4Hep, PODIO, Rec/Tracking libraries: ACTS, ... Framework: Gaudi/Marlin - - - CSS - - 7
Event Data Model Yes, sharing a concrete EDM is a huge plus for reusing algorithms and applications Does everybody agree? Defining the EDM with a tool like PODIO is another big plus Full flexibility on the implementation (the what versus the how) We could have a base EDM which is mandatory to be understood by all common components and extensions for experiment special cases Do we agree that we need to define a common EDM? podFCEDM (POD future collider EDM) 8
Framework The stack should have one and only one data processing framework Without this condition we cannot provide a turnkey stack What do we need from a framework? Are we able to take a decision? Less dependencies Reduce the number and sizes of external libraries Replace inactive and unpopular libraries Make it lightweight and convenient to be distributed and deployed Be portable on various computing resources Cloud, supercomputer, volunteer computing, and etc. Non-x86 hosts, such as ARM Integration with other software and tools General purpose software integration, such as DD4HEP Experiment-specific software integration, such as database Event Store and Data I/O services Flexible data types other than DataObject in TES Unified data structure for transient and persistent data Management of condition data Parallel Computing How to implement a reentrant algorithm Migration cost of existing codes Smooth switching between parallel and serial mode 9
Stakeholders 10
