Survey Development and Process Insights in Ireland
Explore the comprehensive journey of the Programme for the International Assessment of Adult Competencies (PIAAC) survey in Ireland, from development to dissemination. Follow the timeline, see the background, survey process, and understand the meticulous approach taken in instrument development and data collection.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
PIAAC: Delivering the Survey in Ireland Kevin McCormack Senior Statistician - Social Analysis www.PIAAC.ie PIAAC@cso.ie
Overview Background Development Survey Process Content Results Dissemination Handouts PIAAC Brochure Background Questionnaire PIAAC Factsheet Outline of international report Proposed thematic reports
Background SLA 2010 - 2013 DES (and DETE originally) CSO Statistics Act 1993 Data collection Dissemination Timeline 2009 2013 Publication Oct 2nd 2013
PIAAC Timeline October 2013 Publication International National & Databases June 2010 July 2011 Main Study preparation: Testing Developing IT systems Commissioning laptops Training interviewers Sampling, maps, addresses Questionnaire testing August 2011 March 2012 Main Study 5,983 completed interviews 72% Overall response rate 70% Computer 30% Paper May 2012 May2013 Data processing, verification, cleaning Table generation National report May 2010 March 2010 Field Trial Interviewer training Translation and Localisation tasks began SLA signed CSO, DES, DETE September 2009 CSO commenced formal engagement 2009 2009 2010 2010 2011 2012 2013 2013
Survey Development Instrument development Translation and Localisation Scoring testing and re-testing IT development Virtual Machine Screener and Case Management System development Field Test
Survey Development OECD oversight The Consortium Sampling plans Interviewer training Quality Control Weighting - Westat Data Adjudication
Survey Process Randomly Randomly selected !!! selected !!! Main Study Aug 2011 March 2012 Sampling Interviewer training Data collection Data processing 15 700 Blocks 1 Person Households
Survey Process Main Study Sampling Interviewer training Data collection Data processing Mix of new and experienced interviewers 8 day training programme OECD specification Focus on : Gaining co-operation Use of survey instrument Other important elements: Nature of survey Quality control Confidentiality
Survey Process Main Study Sampling Interviewer training Data collection Data processing Encrypted data - file transfer & laptops Screener - random adult (16-65) Targets and bonus payments Incentive payment Management information system Unlimited call backs Strict weekly monitoring Quality control Taped interviews 10% of all interviews verified Telephone calls, field visits,
Survey Process Main Study Sampling Interviewer training Data collection Data processing Industry and occupation coding Manual Automatic Verification against LFS Paper booklet scoring Inter-coder reliability design Anchor booklets (international) Data cleaning Screener vs PIAAC questionnaire
Survey Content Background Questionnaire Core IT competency Basic Skills assessment Direct Assessment Computer Based Assessment Paper Based Assessment
Background Questionnaire A. General Information Age & Gender B. Education and Training Formal Education & Training C. Current status and work history D. Current work Occupation & Earnings E. Last job - Occupation & Earnings F. Skills used at work Frequency of use of workplace skills G. Literacy, numeracy and ICT skills used at work H. Literacy, numeracy and ICT skills used in everyday life I. About yourself Respondent attitudes and activities J. Background information including parents occupations
PIAAC Assessment Design Everyone takes the BQ BQ Aged 16-65 6 tasks: 4 of 6 to PASS Scrolling, clicking, highlighting, Basic IT Skills Basic Skills (Paper) Basic Skills (Computer) FAIL FAIL <4 of 8 <4 of 8 All those who fail the simple literacy or numeracy task on FAIL FAIL <3 of 6 <3 of 6 (same as computer tasks) 4 of 8 to PASS 3 of 6 to PASS Selected at random Components booklet 4 literacy tasks 4 numeracy tasks 3 numeracy tasks Numeracy and Literacy Numeracy and Problem-solving automatically to the Reading 3 literacy tasks Literacy and Numeracy Literacy OR Numeracy booklet computer or paper are routed Assessment (Computer) Assessment (Paper) Reading Components
PIAAC Assessment Design BQ Basic IT Skills Basic Skills (Paper) Basic Skills (Computer) Assessment (Computer) Assessment (Paper) Reading Components
Results Background Questionnaire Economic Education Social Assessments Literacy Numeracy Problem-solving in Technology Rich Environments (PS-TRE) Age Gender Skills use (work & home)
PIAAC Proficiency Scales Proficiency Item difficulty Item Response Theory & scale scores Cut-points (levels 1 to 5) IRELAND: Trends from IALS to PIAAC (linked items)
Trends (IALS vs PIAAC) IALS ALL PIAAC Prose Literacy Prose Literacy Document Literacy Document Literacy Literacy (rescaled for trend) Literacy (rescaled for trend) Literacy Quantitative Literacy Numeracy Numeracy Problem solving Problem solving in TRE
PIAAC Proficiency Scales Level 1 Level 2 Level 3 Level 4 Level 5 0-225 226-275 276-325 326-375 376-500
Key findings of IALS 25% of Irish Adults at lowest levels of literacy Gender: Males more likely to be at higher levels in Quantitative Literacy otherwise little difference between males and females Age: Older people more likely to be at lower levels of ability related in IALS to educational attainment Education: Higher performance related to higher levels of educational achievement.
Prose Literacy (IALS 1997) Document Literacy (IALS 1997) Quantitative Literacy (IALS 1997) Scales 40% 35% 34% 30% 32% 32% 31% 30% 28% 25% 25% 25% 23% 20% 15% 16% 14% 10% 12% 5% 0% Level 1 Level 2 Level 3 Levels 4/5 IALS 1997
International comparison IALS 1997 Documents scale 100% 5.8 11.5 16 16.1 18.9 19 90% 20 25.1 35.5 80% 70% 60% 50% 40% 30% 45.4 20% 25.3 23.7 10% 18.2 18.1 16.2 10.1 9 6.2 0%
International comparison IALS 1997 Quantitative scale
Some data! Percentage of those 20-24 with Leaving Certificate or equivalent Both sexes Male Female 89.8 87.6 85.3 82.8 77.3 72.0 1996 2011 Source: CSO (Census 1996 & QNHS Q2 2011)
Highest level of Education Attained (Aged 25-64) 31% 1996 2011 28% 25% 24% 23% 22% 15% 12% 10% 10% Primary (incl. no formal education) Lower secondary Upper secondary Third level: Non-degree Third level: Degree or higher Source: CSO (Census 1996 & QNHS Q2 2011)
Population change 1996 to 2011 Age profile of Population 1996 vs 2011 450000 400000 +1m (26%) 350000 300000 250000 200000 150000 100000 50000 0 0 - 4 years 5 - 9 years 10 - 14 years 15 - 19 years 20 - 24 years 25 - 29 years 30 - 34 years 35 - 39 years 40 - 44 years 45 - 49 years 50 - 54 years 55 - 59 years 60 - 64 years 65 - 69 years 70 - 74 years 75 - 79 years 80 - 84 years 85 years and over 2011 1996
Country of Birth 1996 2011 Outside Ireland, 7% Outside Ireland, 20% Ireland, 80% Ireland, 93% Source: CSO (Census 1996 & 2011)
PIAAC results International report 2nd October 2013 Two volumes Results (I) and Technical (II) Distribution of skills (mean scores) International comparisons Main focus on LITERACY Other domains on the web Problem-Solving in TRE for later thematic releases National report Focus on Irish performance Trends from IALS Relevant international comparisons
Profile 9 Census 2011 A Study of Education & Skills in Ireland November 22nd 2012
Thank you Questions?
