Istat Modernization Programs: Implementing SDMX Strategy
Istat has been modernizing its operations since 2010, focusing on quality improvement, cost reduction, and sustainability. By implementing the SDMX strategy, the organization aims to standardize and industrialize its processes, utilizing use cases, enablers, and IT solutions for successful implementation.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
SDMX Istat strategy in Istat: state of art and perspectives 12thSDMX Experts workshop 7-11 October 2024 Amsterdam, the Netherlands
Summary The modernization programs in ISTAT The ISTAT s SDMX implementation strategy: Use cases Enablers Maturity model Software: the SDMX ISTAT Toolkit IT Architectures Perspectives: Experimenting with the generative AI Lessons learnt: needs for the SDMX implementation strategy
The modernization programs in ISTAT Starting from 2010 Istat has been running several modernisation programs (2010, 2016,2019,2024) with the following main aims: Increase of quality in production, dissemination and exchange of statistical data, according to international regulations,guidelines,standards and best practices Costs reduction in the middle terms (reuse as much as possible of methodology,meta-information,software tools and libraries) Sustainable solutions in the long terms SDMX was chosen by the top management as the main driver for facilitating the standardization and industrialisation of the business processes
The SDMX implementation strategy in Istat: use cases Oriented to get concrete achievements about the following use cases: Migration of dissemination systems to modern platforms (e.g. IstatData) Eurostat SDMX data collections SDDS-PLUS Data Data Reporting Dissemination SDMX Toolkit Twinnings funded by EU Project funded by AICS IPAs and Grants by EU candidates countries Hub of Public Statistics Support to other countries in modernization Improving quality in the NSS
Enablers of the SDMX implementation strategy Reengineering of statistical business processes and increasing the usage of metadata within the data life cycle European regulations and agreements with international and national organizations Moving out from legacy dissemination systems towards plug & play IT architectures and tools Top management Production Units IT Dept.
The IT solution to support the SDMX implementation: Istat Toolkit Structural Metadata Management Data Management Reference Metadata Editing DCAT Catalogues Editing Business driven and SDMX based Installation wizard and Configuration Console Optimized UI s objects, wizards and walk-throughs Features for different users skill (Basic, Intermediate, Advanced) Plug & Play and High usability Functionalities for different use cases Stand-alone architecture (deployed in one or more servers) Hub architectures Flexible and scalable Cache system Empirical techniques for slicing Performance
SDMX implementation timeline in Istat Legacy dissemination data warehouse 2022 2024 2018 2006 2016 2009 2014 AI Dissemination Reporting Eurostat SODI project Hub of the public statistics SDMX Istat Toolkit
Data reporting in SDMX All the statistics collected in SDMX by Eurostat are prepared by the domain managers using a centralised infrastructure and pushed using eDamis Census data are exposed via an SDMX web service to be consumed by the European Census Hub Most of the statistics concerning the European High Value Datasets are accessible in M2M through an SDMX API All the data categories requested by the IMF SDDS Plus projects are offered through the National Summary Data Page
SDMX within the National Statistical System The Hub of the official statistics is a project that Istat has been running within the NSS: Facilitate the modernization of the dissemination information systems in the Public Administrations Improve quality as detailed by the European Code of Practice / Quality Assurance Framework Support the semantic interoperability as detailed by the Digital Agenda at European and National level Implement a distributed IT architecture with a single access point for the end users
Data dissemination in SDMX Most of the legacy dissemination systems were already migrated in a unique SDMX-based platform: Corporate data warehouse (IstatData): 2 billions of data points spread on 450 data cubes, disseminated through 2900 data flows External trade data warehouse (CoeWeb): 10 billions of data points spread on 45 data cubes, disseminated through 78 data flows Permanent Population and household Census: 700 millions of data points spread on 25 data cubes, disseminated through 564 data flows
Supporting other countries inn implementing SDMX Transformation from other format (e.g. from Excel or not-coded data in legacy database) Avoid data redundancy for data reporting to different organizations (e.g. Eurostat, SDDS) Build capabilities starting from basic explanation on how manage data using standards Get practical results in a short time frame
Short-time Perspectives: Experimenting with the generative AI Strategic orientation of the ISTAT management towards the new frontieres of data search and discovery for empowering the user experience: Scope: integrate natural language allowing non expert end-users in searching statistics through questions like: What are the statistics that explain the trend of Italian economy in 2024? What are the statistics about households? What is the population of the major Italian cities? Jan 2025 Jul 2025 Sep 2024 Jun 2024 Implementation (*) PoC Pilot
Lesson learnt Before starting an SDMX implementation strategy the following complexities must be carefully considered IT solutions selection (in-house Vs market solution) Effort and Budget Priorities identification Changes in the Data Life Cycle Staff and capabilities Learning Curve Implementation plan
Thank you Alessio Cardacino (alcardac@istat.it) SDMX Istat toolkit: https://sdmxistattoolkit.github.io Corporate dissemination data warehouse (IstatData): https://esploradati.istat.it/databrowser/#/en Permanent census of population and housing: https://esploradati.censimentopopolazione.istat.it/databrowser/#/en External trade dissemination system (CoeWeb): https://esploradati.istat.it/coeweb/databrowser/#/en
