Introduction to Data Management and Engineering Frameworks

Data Management & FAIR NDBI046 NDBI046 - - Introduction to Data Engineering Introduction to Data Engineering 202 2024/2025 4/2025 Petr Petr koda koda https://github.com/skodapetr https://github.com/skodapetr https://www.ksi.mff.cuni.cz https://www.ksi.mff.cuni.cz

Data Asset An asset is an economic resource, that can be owned or controlled, and that holds or produces value. Assets can be converted to money. -- Data Management Body of Knowledge Value is the difference between the cost of and the benefit: Cost of obtaining and storing data Cost of replacing data if it were lost Impact to the organization if data were missing Cost of improving data Benefits of higher quality data What the data could be sold for Expected revenue from innovative uses of data . 2

Data Management Data Management is the development, execution, and supervision of plans, policies, programs, and practices that deliver, control, protect, and enhance the value of data and information assets throughout their lifecycles. -- Data Management Body of Knowledge Data management professionals (technical): Data management professionals (business): Database Administrators Data Stewards Network Administrators Data Strategists Developers Chief Data Officers Data Governance Council A strategy is a set of choices and decisions that together chart a high-level course of action to achieve high-level goals. -- Data Management Body of Knowledge 3

Data Management Frameworks Strategic Alignment Model (Henderson and Venkatraman, 1999) Amsterdam Information Model The DAMA DMBOK Framework (The DAMA Wheel) Hexagon, and Context Diagram 4

Aiken's pyramid Phase 1 Application with database capabilities. Phase 2 There are issues with quality. Phase 3 Manage management ~ governance. Phase 4 Profit. 5

Data Life Cycle Manage data flow from the first contact to the last. Phases (incomplete list): Data capture / creation Personal to global: Data storage Panel on Climate Change (IPCC) expected > 80 PB Data maintenance Large Synoptic Survey Telescope expected > 500 PB Data synthesis Data usage Image of a supermassive black hole 4.5 PB Data publication Data archival Data destruction / purge 6

Data Governance All organizations make decisions about data, regardless of whether they have a formal Data Governance function. 7

Data Governance Framework of people, policies, processes and technologies that define how you manage your organization s data. -- SAS The act or process of governing or overseeing the control and direction of something. -- Merriam-webster the exercise of authority and control (planning, monitoring, and enforcement) over the management of data assets. The Data Governance function guides all other data management functions. -- Data Management Body of Knowledge Strategy Compliance Policy (e.g., metadata management) Issue management Standards and quality Data management projects (sponsoring) Oversight Data asset valuation 8

Data Governance Decisions that must be made to ensure effective data management. Areas: Data principles Data quality Metadata Data access Data lifecycle 9

Data Governance Organization Functions: Members: Legislative-like Defining policies, standards, and the Enterprise Data Architecture. Steering Committee Council Stewardship Teams Data Stewardship is the most common label to describe accountability and responsibility for data and processes that ensure effective control and use of data assets. Judicial-like Issue management and escalation. Executive Protecting and serving, administrative responsibilities. Operating Model Types Centralized Federated 10