Statistical Production Process at ISTAT: Experiences and Challenges
This content highlights the statistical production process at ISTAT, including data entry, outlier detection, estimation procedures, and more. The process involves standardizing data, revising outliers, and estimating biases in customs data to ensure accurate trade statistics. Explore the complexities and strategies involved in compiling external trade data.
Uploaded on Feb 19, 2025 | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Data Processing and Database Management Markie Muryawan United Nations Statistics Division 1
Statistical production process: Experiences from ISTAT (a) Automatic upload or manual data entry of customs data; (b) Exclusion of trade flows not relevant for the compilation of external trade statistics; (c) Standardization of customs data according to statistical standards, including both classification and analytical variables;
Statistical production process: ISTAT (d) Rapid detection and revision of major outliers, having significant impact on the aggregate trade figures published as flash estimates or preliminary data; (e) Thorough analysis and revision of outliers at the product/country level, including mis-classification problems at the product, country or other statistical variables level; (f) Estimation of a possible random item or unit (non-response problems);
Statistical production process: ISTAT (g) Estimation procedures related to structural biases in customs data, such as systematic delays in data transmission, under coverage due to the adoption of exemption thresholds, etc.; (h) Estimation of peculiar external trade flows not covered, or poorly covered, by custom data.
Data pipeline Bronze Silver Gold Datasets Aggregate to World, Total HS Conversion to other classification Custom partner country aggregates IMTS Framework Non-customs Data Estimation Customs Data
Data pipeline Need a database ? Relational database can enforce certain validation automatically Such as verifying accepted codes (against standard list) It can also facilitate data query and analysis Using SQL to extract data And it supports large amount of data Any database is big enough for IMTS The good part is that any specialized Trade Data Tools have a built-in database
Data pipeline Need the data tools? Excel is no longer sufficient for implementing high quality data processing pipeline Lack of integrity code checking Limited automation reproducibility No support for modern data processing or analysis language (i.e., R, Python, SQL) But it can still be use as data source if size permitted Eurotrace developed by Eurostat can be used to generate a gold trade dataset But the lifecycle is ending, and there is no longer development (only maintenance Trade Data Tools (TDT) developed jointly by UNSD and Eurostat aims to replace Eurotrace But it is still on development, and limited country testing Developed your own pipeline using (such as) R or Python Required data skills
Data pipeline Need a Cloud ? Modern data tools allow to utilize the Cloud for computing and storing Of course, the data will be private and only accessible to you Just like your gmail account or enterprise email using outlook.com With the Cloud, it is normally no installation needed in your PC or Laptop A better cybersecurity because it is backup and updated regularly However, there may be a restriction from legal point of view No official data can be stored in a Cloud Cost? Yes, if you use your own account Or UN Global Platform can offer a complementary access to compute and store (for TDT)
Quality related to Data Processing Coverage Do you include everything? Share of electronic declarations in data collected by Customs (% of total trade value) Do you still use paper declarations? How are they being processed? When are they added? What is their quality? Share of trade data collected via Customs declarations (% of total trade value) How much data is collected from other sources such Oil Ministry, Electricity companies, etc? Estimation of low value trade not requiring Customs declarations Is this done? How is it done?
Quality related to Data Processing Availability of quantity information Number of 8-digit product codes without quantity Share of records without quantity (% from total trade value) Share of estimated quantity (% of total trade value)
Quality related to Data Processing Revision policy How often do you revise data? Do you archive all revisions? Estimation of low value trade not requiring Customs declarations
Quality related to Data Processing Data Validation Validation of input Does the declaration system of Customs contain mandatory fields, relevant for trade statistics? Does the Customs system check immediately for validity of codes (country, commodity, quantity, currency)? Does the Customs system immediately do credibility checks of commodity prices? Validation by Statistical unit What is the percentage of records with coding errors? Do you do checks on values and unit-values (outliers, time series, checks against other data sources)?
Quality related to Data Processing Correction Management Do you keep logs of all corrections? Which corrections do you keep? Do you keep the original data as well? Is everything time-stamped. Do you publish meta-data which each data release? Which metadata do you publish?
Quality related to Data Processing Documentation of the Data Processing System Procedures for code validation Procedures for value and quantity checking Editing procedures Estimation procedures Procedures for unit value checking, time series checking, checking of plausibility of vale for commodity by trading partner
