Scanning Technologies in Pacific Island Census Planning
Utilizing scanning technologies in population and housing censuses across Pacific Island countries is discussed in this document. It highlights the benefits, challenges, and comparisons with traditional manual methods. The adoption of scanning has shown improvements in data processing speed and quality, offering insights into the scanning process, technologies used, and the experiences of various countries. The document also presents a comparison between manual and automated processes, showcasing the time and efficiency advantages of scanning.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Part 3 - B: Scanning 2020 World Round of Population and Housing Censuses Pacific Island countries census planning meeting International recommendations/standards, contemporary technologies and regional cooperation Noumea, New Caledonia, 27 July 31 July 2015
Introduction Filling-in of forms is almost the same as KDE, tabulation is identical. 2010 was 1stcensus round in which scanning has been used in the Pacific - Fiji (2007), Vanuatu / Solomon Islands (2009), Samoa / Cook Islands (2011). Few teething issues and lessons to take into account for the 2020 census round If scanning is correctly prepared for, and well managed, it can significantly speed up data processing and improve data quality by avoiding manual data entry errors
Background Technology is changing, OMR / OCR = automation. Only user involvement to confirm important data + fix validation errors. Scanned questionnaires can be: Linked to a database allowing efficient retrieval of forms when errors are detected, Effectively archived for future reference Used for on-screen coding
Background (cont.) Scanning isn t the answer for all countries / applications: Initial investment in scanning technology is high for small countries, More significant requirement to have good technical skills within the Statistics office to manage the network used for scanning and also to program and manage validation checks.
Number of Forms Scanned Year Country # Forms Scanned (approx) 2009 Solomon Is. HH Listing 8,000 2009 Vanuatu HH Listing 4,000 2009 Solomon Is. Pilot Census 1,500 2009 Vanuatu Pilot Census 1,500 2010 Vanuatu Census 55,000 2010 Solomon Is. Census 115,000 Samoa HH Listing 1,040 2011 Samoa Pilot Census 500 2012 Samoa Census 28,000 HH, 80,000 Person 2012 Cook Islands Census 5,000 HH 15,000 Individual
KDE vs Scanning Cleaning/ Double-checking Automated Entry Model Questionnaire Batch Editing / Tabulation Manual Entry Samoa Example DURATION Manual (KDE) Automated (Scanning) Questionnaire Design Few weeks Little bit longer than KDE make scannable Data Capture 16 weeks 6 weeks Cleaning /Batch Editing 3 weeks 7 weeks Tabulation Same Same 19 weeks 13 weeks
KDE vs Scanning (cont.) Samoa Example COSTS Manual (KDE) Automated (Scanning) Staff costs + USD$1500 Equipment Costs Readsoft Forms Scanners (2) Server +USD$20,000 +USD$14,000 +USD$10,000 Printing Costs +USD$15,000 Total extra costs Roughly +USD$57,500
What did not work so well (cont.) Edit specifications not developed by subject matter specialists. Cleaning of data shouldn't be trial and error should be rule based and tested. Some countries had too many open-ended questions = lots of coding and slow data entry. Reliance on external expertise
What did not work so well (cont.) Verifiers sometimes gave the impression that they could simply push buttons without thinking the software assists the verifier to determine the problem but doesn t always give the answer. Verifiers need stop and think. Pencil used in scanning Poor fieldwork Poor office controls
Recommendations Preparation Prepare questionnaires, edit specifications, tabulation plans and data dictionaries well in advance Subject matter specialists need to be involved in the creation of the edit specifications and tabulation plans. All validation and edit rules developed must be based on these specs. Guidelines in U.N. Editing Handbook Prepare all of the above in unison so that iterative changes can be made
Recommendations (cont.) Processing Double-check ID fields in the office Order forms prior to data processing As a post-processing exercise, link households up with GPS data if it exists to find out if forms may need to be located and processed if they have been missed.
Recommendations (cont.) Scanning When deciding between KDE and Scanning, countries should think carefully of whether or not they will have the technical expertise in-house. Pay closer attention to evaluation of scanning/verifying in real time. If scanning, ensure questionnaires are correctly designed with pure drop-out colours, correctly sized mark boxes and clear recognition marks. (Readsoft FORMS detailed instructions, VNSO, FIBOS, SBS, SPC) Pencil should not be used to fill in scan questionnaires only black pen.
Recommendations (cont.) Additional Assistance South-South collaboration works well. Vanuatu National Statistics Office shared their scanning/data entry specialist with the Solomon islands, Samoa and recently the Cook Island NSOs. SPC intends to use this much more. Training!!!!!
Recommended Training Produce a sample set of questionnaires which clearly have TRAINING ONLY written on them so they can t be confused at a later date with the real questionnaires. Carry out the same detailed explanations for each question as with manual entry. Verifiers need to fill out many forms to properly understand the flow and structure. All verifiers need to have a go of scanning and all scan operators a go at verifying so they better understand the whole process.
Recommended Training (cont.) Run many tests of sample datasets through the scanner, deleting the sets and restarting over. Verify the test datasets many times they can be reused without having to rescan every time. Once the real scanning starts, process a few EA s then stop and get verifiers to check each others work in Excel. If there are problems work through them in detail with the verifiers, deleting whole EAs if necessary to redo them.
Conclusion Countries decision which process/technology to adopt will depend on the complexity of their questionnaires, existing in- house skills, desired results. Poor fieldwork, resulting in messy, incomplete forms contributes to many of the problems encountered during data processing later on. Maybe we need to look at new technologies to manage this too? One big lesson learnt over the last few years is that scanning is not magic, and cannot fix everything. It speeds up data entry, but if not managed correctly can introduce many other complexities.
Conclusion (cont.) Tablets and mobile phones were not used in the 2010 round of censuses (only tested) though are now more and more viable with increases in technology and dropping costs. Data validation at the point of entry is the best way to increase the quality of a census.
Questions?? 1. During the 2010 census round several new data collection technologies were tested. What were the advantages of the technology tested? (will be country dependant) 2. Disadvantages / problems encountered? 3. Do you consider this is the right way forward, and if so, then what needs to be done to alleviate the issues raised in (b)?
