Istat SDMX Toolkit for Standardisation of Statistical Processes
"Explore the Istat SDMX Toolkit designed to standardize and industrialize statistical business processes. Learn about its features, implementation principles, and open-source nature, benefiting from easy installation, configuration, and usage with optimal performance."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
SDMX Global Conference 2021 SDMX Istat Toolkit to support the standardisation and industrialisation of the statistical business processes FRANCESCO RIZZO, ALESSIO CARDACINO Istat
Summary The Toolkit as cornerstone of Istat SDMX strategy Design and implementation principles of the toolkit Free and open source toolkit Transformation Excel2CSV tool Meta and Data Manager: main features, architectures Data Browser: main features, architectures Conclusions 2
The Toolkit as cornerstone of Istat SDMX strategy SDMX API (SEP) CARICOM, Macedonia, Tunisia, Serbia, Lebanon, Laos RDF Data Cube DCAT SDMX formats 3
Design and implementation principles of the toolkit Reengineering by experience Old version used as prototype Feedbacks collected from +500 users (trainings and national and international projects) Easy to install (Plug and Play) Not more than 20 min Technical skills easy to find in statistical organizations Easy to configure Default configuration (Play) Extended configuration through GUI Easy to branding (including different languages handling) CSS files resource files Adequate performance Table with 100.000 cells in 10 seconds Benchmark with the current .Stat V.7 and other architectures Easy to use (GUI design driven by users) 4
Free and open source Toolkit Reporting Dissemination Transformation Collection - - GUI (Data Browser) M2M (APIs) - SDMX - CKAN (DCAT) - Ad-hoc utilities - - - - Excel CSV (Custom and SDMX) SDMX-ML PC-AXIS - Excel2CSV Data Manager - - - Star-scheme builder Data cubes management Data flows management Metadata Manager - - - SDMX Registry Nomenclature server DCAT data catalogs - - Structural metadata management Reference metadata management 5
Transformation Excel2CSV tool SDMX DSDs 6
Meta Manager main features SDMX artefacts CRUD: Concept Schemes, Code Lists, Category Schemes, Agency Schemes, Data Structure Definitions, Data Flows, Categorisations Hierarchical CL, Metadata Structure Definitions, Metadata Flows can be used but not created Structural metadata utilities: Derive item schemes (Code Lists, Concept Schemes, Category Schemes) Add items preserving hierarchy Include Parents , Children , Descendants Merge item schemes (Code Lists, Concept Schemes, Category Schemes) Compare item schemes (Code Lists, Concept Schemes, Category Schemes) Compare DSDs Upgrade DSD Overcome SDMX 2.1 deficiencies: add items to final item schemes, modify/add names, descriptions and annotations to final artefacts Manage annotations for presentation purpose (e.g. Order, Layout, etc.) Reference metadata editor DCAT Data catalog editor 8
Data Manager main features Mapping Loading Publishing Building Building: Multidimensional data cubes based on SDMX DSDs (advanced database star-schemas synchronized with the structural metadata repository are created within the dissemination database) Mapping: Custom CSV files Excel files (multidimensional statistical tables) PC-Axis files Loading: Excel, CSV, PC-Axis and SDMX-ML data files Publishing: Data flows As sub cube of a data cube As sub cube with less dimensions (a new DSD is generated automatically) Download: SDMX-ML (2.0 & 2.1), SDMX-CSV, SDMX-JSON, Custom CSV, RDF/XML, DCAT-JSON 9
Meta and Data Manager architectures Internet Intranet Global Registry MDM Client + Back-end Eurostat Registry IMF Registry Node N Node 1 Node 2 MDM Back-office MDM Back-office MDM Back-office .. UNdata SDMX API 10
Data Browser architecture SDMX-ML (2.1) Json-Stat (extended) SDMX-Json (2.1) SDMX 3.0 or Others Cache 12
Conclusions SDMX implementations must be supported by a long running strategy The adoption of SDMX tools should always be preceded by the evaluation of the following dimensions: available features, usability, learning curve, software installation and upgrade easiness, required capabilities SDMX Toolkit has become the reference architecture in Istat for streamlining dissemination, reporting and metadata management SDMX Istat Toolkit is available as free and open source, therefore other Organisations can contribute in enhancing the Toolkit with suitable software developments 13
Thanks Francesco RIZZO | rizzo@istat.it Alessio CARDACINO | alcardac@istat.it SDMX Istat toolkit download: https://sdmxistattoolkit.github.io/index.html Data Browser in action: Permanent census of population and housing https://esploradati.censimentopopolazione.istat.it/databrowser/#/en
