INDIGO Project Overview: Virtualised Resource Infrastructure Goals
"Learn about the INDIGO project aiming to develop a data/computing platform for scientific communities on diverse e-infrastructures, fostering cloud competence in Europe and open-source software development."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
INDIGO WP4 Virtualised Resource Infrastructure Marcus hardt@kit.edu on behalf of WP4: Patrick Fuhrmann (DESY) Alvaro Garcia Lopez (IFCA) Zdenek Sustr (CESNET)
INDIGO DATACLOUD Project ID 653549 26 Partners: INFN (Italy), CIRMMP (Italy), CNRS (France), REPLY (Italy), STFC (UK), CESNET (Czech Republic) Req. EU Contribution: 11.138.114 , 30 Months, 1580 PM (52,6 FTE) INDIGO will allow application development and execution on Cloud and Grid based infrastructures, as well as on HPC clusters PaaS for e-Infrastructures Start: April 1 2015 ;) CSIC (Spain), INAF (Italy), CERN (Switzerland), CMCC (Italy), AGH (Poland), INGV (Italy), DESY (Germany), LIP (Portugal), UPV (Spain), KIT (Germany), ICCU (Italy), INDRA (Spain), CEA (France), ATOS (Spain), UU (Netherlands), T-Systems (Germany), CNR (Italy), EGI.eu (Netherlands), PSNC (Poland), RBI (Croatia),
High-level Description INDIGO aims at developing a data/computing platform targeted at scientific communities, deployable on multiple hardware, and provisioned over hybrid (private or public) e-infrastructures. In INDIGO, key European developers, resource providers, e- infrastructures and scientific communities have joined to ensure the successful exploitation and sustainability of the project outcome. A key strength and benefit of the INDIGO consortium, for both the scientific and industrial sectors, is the potential to create a new sustainable Cloud competence in Europe for PaaS, similar to what OpenNebula or OpenStack have done for IaaS. 3 3 Davide Salomoni, 15/12/2014 INDIGO
INDIGO General Objectives Objective 1: Development of a Platform based on open source software, without restrictions on the e-Infrastructure to be accessed (public or commercial, GRID/Cloud/HPC) or its underlying software. Objective 2: Providing the interface between e-Infrastructures and Platforms. Objective 3: Provide high-level access to the platform services in the form of science gateways and access libraries. Objective 4: Streamline the adoption of the software products. INDIG 4 4 Davide Salomoni, 15/12/2014 O
Objectives 5 INDIGO-DataCloud 19/05/2015
Indigo Work Packages (WP) NA WP1: admin and financial management, project quality assurance, global oversight. WP2: this is where Research Communities express requirements, provide feedback, review deployed services. This WP also includes dissemination and communication activities. SA WP3: software management, deployment of pilot services. JRA WP4: resource virtualization. WP5: PaaS framework. WP6: APIs and portals, data-driven workflows. 6 6 Davide Salomoni, 15/12/2014 INDIGO
WP4 Virtualised Resource Infrastructure Overall Goals Virtualizing local compute, storage and networking resources (IaaS) Enable existing e-Infrastructures to be used within INDIGO PaaS Integrate with coming Indigo AAI (i.e. not using robot certificates) 7 INDIGO-DataCloud 05/06/2025
WP4 Virtualised Resource Infrastructure Division of work within WP4 4.1 Computing Support for Containers Improve on-demand compute capabilities / improved orchestration and scheduling 4.2 Storage Defining interfaces and implementing QoS support for storage systems Providing access to the same storage through various standard access protocols 4.3 Network Evaluation SDN to configure local networks and meet PaaS needs Manage local virtual Networks Common Subtasks: 4.4 Authentication, Authorization and Identity Management (AAI) 4.5 Service Discovery and Monitoring 8 INDIGO-DataCloud 05/06/2025
T4.1 Cloud Computing Virtualisation Task global objectives Enable the use of containers in CMFs. Provide local site orchestration. Improve the CMF schedulers. Integrate novel AAI. Provide a discovery, availability and accounting services 9 INDIGO-DataCloud 19/05/2015
T4.1 Cloud Computing Virtualisation Subtasks Support for PaaS developments on top of IaaS. Develop or extend container support for ONE and OpenStack. Trusted repositories for containers. Extend relevant IaaS standard interfaces if needed. Improving IaaS schedulers and integrate non-cloud resources. Improve the existing cloud schedulers for OpenStack and ONE (spot instances and fair-share scheduling). Integration of container execution in batch systems. Access to Infiniband and GPGPUs. Provide local site orchestration and extend the existing orchestrators and standards (e.g. TOSCA). Integrate Indigo's AAI into the CMFs. Provide monitoring data to higher level services. 10 INDIGO-DataCloud 19/05/2015
T4.2 Cloud Storage Virtualisation Objectives Enable users via WP5 and WP6 to store and manage their data Support users in their overall data lifecycle management Enable cross protocol access Integrate novel AAI solutions Flexible to ensure flexibility when choosing protocols Interfaces Pragmatic approach Sites have to provide Protocols of the categories of: Block, File Object Sites are free to choose which protocol in each category: CDMI / S3 / RADOS? WebDAV / sFTP / gFTP / CIFS RBD / iSCSI 11 INDIGO-DataCloud 05/06/2025
T4.2 Cloud Storage Virtualisation Subtasks: Storage QoS Unification of ontology and policies to define terms for retention-policies and access- latency Contribution/collaboration with RDA groups Definition and implementation of interface for specification of r/p and a/l for storage Implementation of backend storage support for r/p and a/l interface Cross Protocol + Cross AAI Cross Protocol Access same file with different protocol For file-based protocols (sFTP + gFTP + WebDAV) likely trivial For cross file / object storage: more challenging 1. Object-Storage backed filesystem <=> filesystem-backed object storage 2. Namespace mapping 3. Support for concurrent access Cross AAI Map so same local UID+GIDs for different credentials and AAIs Includes different CNs in X.509 or X.509 + SAML + OIDC 12 INDIGO-DataCloud 19/05/2015
T4.3 Network Virtualisation Recommendations Networking setup inside images to achieve uniform behavior in different cloud management frameworks Cloud site networking setup Standards Connecting VMs to networks with OCCI Creating [private] networks with OCCI Security group definition over OCCI (a new OCCI Specification to be developed) Tools Adequate functionality in OpenNebula (through rOCCI and, optionally, other tools (local management, neutron) Adequate functionality on Open Stack through occi-os Appliances Ideas for users and cloud site administrators: networking elements (DHCP , NAT, VPN, FlowMon) 13 INDIGO-DataCloud 19/05/2015
T4.4 AAI Management To ensure the integration of federated AAI technologies into OpenStack, OpenNebula, CEPH, dCache and other supported INDIGO products, allowing users to access infrastructure resources using their home or guest IdP account. Pre-cap: INDIGO AAI Currently under discussion and development Users in SAML world Services in REST / X.509 world New authorisation mechanisms based on well elaborated schema: Virtual Organisations => Summary: Support for multiple AAI systems embedded in architecture Implement support for INDIGO AAI Site-local + service-local credential translation for: Authentication translation Incoming authentication (+authorisation) token maps to site(+service)-local accounts File-based example: UID (+ GIDs) Authorisation translation Incoming authentication token is complemented by additional authorisation tokens and mapped File-based example: more GIDs We are aware of ARGUS We are not sure if we have enough manpower for full support, but the spirit will be taken. 14 INDIGO-DataCloud 19/05/2015
T4.5 Service Discovery and Monitoring To extend existing local-site monitoring services for all INDIGO products to provide to higher-level services of WP5 and WP6 with monitoring and accounting information through a query- API 15 INDIGO-DataCloud 19/05/2015
Conclusion We only just started, but we have plans Requirements and Architecture Deliverables due until M6 (Sep 30th) Plan is to do many things right this time Starting from AAI All the way through to user experience Let s see each other in 28.5 months ;) 16 INDIGO-DataCloud 19/05/2015
