Innovative Scientific Data Storage Solutions
Explore the evolution of data storage from disk-based systems to distributed cloud-like services tailored for scientific data access. Learn about cutting-edge technologies like Ceph@M-SoCentre and the economic models supporting these advanced storage infrastructures.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Some history Back in a day where Storage was disk based : a experience can store all their data in a disk And after A disk was not enough, we need a server with a bunch of disk to store data But now A entier datacenter is not enough LHC : 1 000 PB LSST: 20 TB / day for 10 years Ska Observatory: 10 PB / day But also A long tail of science need a small part need to be addressed 2 AG ing nierie 12/11/2020
ceph@msocentre GOAL to provide a distributed, resilient, and high performance infrastructure dedicated to scientific data acces from different computing platform of the m socentre standard interfaces (S3, ) Infrastructure distributed over 3 geographic sites Fault tolerante Up to 8 PB of raw data 3 AG ing nierie 12/11/2020
Adapt for your needs A storage cloud like service (onedrive, dropbox equivalent) Based on a high scalable backend A s3 acces point to storage massive dataset Accessible from cloud@vd and ruche Experience based acces, not user based In progess: A galaxy of services that can have direct access to data (jupyterhub ) A part of national storage federation (MesoNet) 4 AG ing nierie 12/11/2020
Some possible workflow 5 AG ing nierie 12/11/2020
Economic model (draft) A small quota is available freely for experiences 1 TB After that, experiences must buy the resources they want to have ~ 80 /TB for 7 years University pay the hosting for UPSaclay labs Outside UPSaclay, the hosting is invoiced ~ 15 /TB/year 6 AG ing nierie 12/11/2020
