M&E Data Collection Tools and Planning Overview
This content covers developing data collection tools, planning strategies, and the importance of data collection in project cycles. Learn about designing tools, setting baseline values, and engaging stakeholders for effective monitoring and evaluation.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
M&E Concepts: Developing Data Collection Tools 2022 BUREAU OF INTERNATIONAL LABOR AFFAIRS
M&E Training Course Series Six Courses 1. Introduction to Results-Based Management 2. Designing Results Frameworks 3. Designing and Defining Performance Indicators 4. Developing Data Collection Tools 5. Setting Baseline Values and Indicator Targets 6. Using Data For Project Improvement https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Training Objectives At the end the training, participants will have increased their understanding of: How to design data collection tools to support project monitoring. Key considerations for planning data collection. https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Agenda Overview of data collection Developing data collection tools Planning for data Collection https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Overview of Data Collection BUREAU OF INTERNATIONAL LABOR AFFAIRS
Data Collection in the RBM Cycle 1. 2. 3. Define Results (RF) Select Develop a Performance Monitoring Plan Performance Indicators 6. 5. 4. Use Analysis to Inform Mgmt Decisions Analyze Performance Data Collect Performance Data https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Data Collection in the Project Cycle Data collection happens throughout the life of a project. During Implementation (periodic) (e.g., indicator data, midterm evaluation) Endline: as projects ending (e.g., last data collection of project or final evaluation) Baseline: before activities begin (or the first collection of data) https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Why Collect data? To promote learning - using evidence and reporting findings to help internal and external learning. To demonstrate accountability- delivering on commitments. To document - recording and creating an institutional memory.- Reporting on performance To explore - seeing what works, what does not, and why. To involve - engaging stakeholders through a participatory process. To gain support - demonstratingresults to help gain support among stakeholders. https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Data Collection Tools and Methods Primary (Project collected data) Interviews Key informant, group (e.g., community) Surveys and questionnaires Household level surveys (Intake forms) Satisfaction surveys Online surveys Direct Observation (structured or unstructured) Site visit logs Service delivery checklists (e.g., job sites, schools, etc.) Knowledge or skill tests Project records (e.g., attendance records, meeting logs, registration forms) Secondary (Second-hand data) Accessing government data Census data, inspection reports or data Administrative data from partners School records, company records, etc. Desk or literature review (e.g., qualitative review) Publicly available data analysis Internet search for existing data, reports and studies https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Types of Data: Quantitative and Qualitative Quantitative Qualitative Number based, countable or measurable Tells you how many, how much, how often Methods are measuring and counting (e.g., close-ended survey questions) Analyzed through statistical analysis Interpretation-based and descriptive Helps understand why, how or what happened Subjective and unique Encourages discussion and exploration of attitudes and behaviors Methods are interviewing and observing Analyzed by grouping data into categories and themes (e.g., content analysis) Examples: Feedback, problem-solving strategies used, changes or explanation of behaviors, observations or perspectives of individuals or groups. Examples: Age, height, income, attendance, group size, test score, feedback/opinion based on fixed scale. https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Roles and Responsibilities for Data Collection & Tools Grantee Draft instruments or identify access to sources of reliable secondary data Align instruments (including question #s) with indicator data needed Revise instruments or data collection procedures based on ILAB feedback Establish data collection roles and responsibilities among staff and partners Conduct data collection, management and reporting ILAB Staff Ensure grantee has instruments/data collection sources for all indicators Provide feedback and approve instruments and data sources (if secondary) Approve instruments as part of Comprehensive Monitoring and Evaluation Plan (CMEP) Ensure roles and responsibilities for collecting data are clear in CMEP Approve start of direct services to adults/children after intake forms are finalized https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Developing Data Collection Tools BUREAU OF INTERNATIONAL LABOR AFFAIRS
Steps for Developing Data Collection Tools 1. Define purpose and needed data 2. Identify target respondents 3. Choose the methods and approach 4. Select and develop the data collection tool 5. Pre-test the data collection tool 6. Prepare for data collection https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Step 1: Define Purpose or Data Needs Capture reliable data for each performance indicator in the CMEP Have a clear concept of what information you want to collect. What type of information do you need to collect to report on the indicator(s)? What do you want to learn or know? How is the data going to be used? https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Example: Determining the Information Needs What type of information do you need to collect? Indicator Needed Information # of children engaged in or at high risk of entering child labor provided education or training services (E1) Disaggregated by sex, location, type of education service Age of children Engaged in child labor At high risk Type of education/training service provided Sex Location Labor practices implemented Type of business Location of business % of targeted businesses that implemented improved labor practices Disaggregated by type of business and location https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Step 2: Identify Target Respondents Who will the data be collected from? Once you know what information you need - determine the target population and who can provide the information. Example: If project wants to know about the relevance of its training the best people to interview would be the participants. If a project wants to know about the application of skills in the workplace the best people to interview might be participants supervisors. https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Example: Determining the Respondent Who will provide the information? Indicator Needed Information Respondent Age of child Engaged in child labor At high risk Type of education/training service provided Sex Location # of children engaged in or at high risk of entering child labor provided education or training services (E1) Disaggregated by sex, location, type education Head of household Teacher or trainer Labor practices implemented Type of business Location of business % of targeted businesses that implemented improved labor practices Disaggregated by type of business and location Business owner or employee Inspector https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Step 3: Choose Method and Approach Find the best way to reach the respondent What methods will be used? Interviews, surveys, observations, project records, etc. How will the data be collected? In-person, mail, telephone, electronic through email or web based https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Types of Data Collection Methods Common types of methods used by ILAB grantees: Interviews: Key informants (e.g., beneficiaries, partners) Focus Groups: 6-8 with common experience Surveys: Behavior, Perception, Satisfaction, Awareness, etc. Testing: Knowledge, skills, certification Observations: Practices, behaviors, skills/techniques Reviewing Project Records: Collecting data from project administrative forms Capacity Assessments: Measure organizational capacity (e.g., NGOs, CBOs) https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Cost Speed Determining Methods Geographic Diversity Level of Accuracy Frequency Reliability https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Determining Methods Pros and Cons Method Pros Cons Can gather in-depth information Opportunity to clarify response through follow-up questions Personalized approach Requires trained interviewers Smaller sample within timeframe Interviewer's presence or characteristics may result in bias. Interviews Allow you to interview multiple people at one time. Identify new ideas and issues Clarify responses to follow-up questions Difficulties with analyzing data Requires skilled facilitators or moderators Can be expensive (e.g., if renting space) Focus Groups Natural setting Ability to choose interaction level of data collector Requires skilled observer Difficulty with generalizing large populations Data collectors' interactions could cause problems with objectivity Inability to gather further information Difficulty in determining sample Can be expensive Observations Easy to do with large number of people Easy to conduct statistical analysis Online surveys are cost effective Surveys https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Example: Determining Methods How will you get the information? Indicator Needed Information Respondent Methods Age of child Engaged in child labor At high risk Type of education/training service provided Sex Location Head of household Teacher or trainer # of children engaged in or at high risk of entering child labor provided education or training services (E1) Disaggregated by sex, location, type education Survey Review project records Interviews Survey Interviews Observation Review business records Labor practices implemented Type of business Location of business Business owner or employee Inspector % of targeted businesses that implemented improved labor practices Disaggregated by type of business and location https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Step 4: Selecting and Developing the Data Collection Tool Primary Data Collection Tools Surveys (e.g., Household intake forms) Interview or focus group guides Tests (e.g., knowledge or skills) Capacity assessment tools Observation checklists Project service delivery forms (e.g., Attendance sheets, inventory forms, activity logs, registration forms) Streamline data collection process (e.g., when possible, use one tool to collect data for multiple indicators) https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Step 4: Developing the Data Collection Tool Structure the Tool Identify & organize the topics to be covered (e.g., demographics, attitudes, behavior, feedback) Include instructions on how to complete tool Include information on purpose of the instrument, how data will be used, confidentiality, ask for Consent Draft the Questions 1. Question Purpose 2. Types of questions 3. Wording of questions 4.Order of questions 5.Length of tool 6.Relevant to respondent https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Types of Questions Closed ended questions - offer a limited range of possible answers, either one or multiple responses per question, or a range that respondents can choose from (e.g., multiple choice, matrix, scale) Open ended questions - allow respondent to provide their response in their own words. https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Question Types: Examples 1. Closed/Multiple Choice: Where do you live? a) DC b) Virginia c) Maryland d) other 2. Closed: Was this program useful? Yes _____ No_________ 3. Open-ended: What program activities were most useful and why? __________________ 4. Agreement Scale (Likert scale): How much do you agree with the statement The children s rights campaign was effective in changing public opinion about child labor? a.) Strongly Agree b.) Agree C.) Disagree D.) Strongly Disagree https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Questions Types: Examples Closed-ended Project Records: 1.a Organizations from Which Your Staff Has Receive Grant Management Training 1.b Number of People Trained 1.c Month/ Year Of Training 2. How trainees described the Utility of the Training Very Useful Moderately Useful Not Useful https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Benefits of Question Types Open-ended questions Closed-ended questions Useful in getting at why and how answers Allows details & clarifications Provides unexpected insights Generates insightful quotes Get answers to questions for which response categories are difficult to predict Easy to ask and record Allows quicker responses Helps recall by providing response categories Allows more error-free coding and analysis https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Avoid Double Negatives: Do you disagree that there should not be a change in labor laws? Avoid Double- barreled: Do you think your job is interesting and stressful or boring and relaxed? Avoid Leading Questions and Use Neutral Words: Should government officials be sent to stop you company s bad labor practices? Make questions and answers specific - not vague: Be clear on what you are asking and be careful about overlapping response categories or unbalanced response scales. Tip for Developing Good Questions https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Make questions as simple as possible. Ask the question in the way it might normally be asked if it was a spoken question. Tip for Developing Good Questions (cont.) Avoid using jargon or abbreviations. This can be confusing to respondent and effect the quality of their answer. Avoid too many open questions. Closed questions are much easier to analyze than open questions. Don't ask for written information unless prepared to spend the time analyzing, otherwise it's a waste of the respondent's time. https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Example: Determining Tool What tool will you use to collect data? Indicator Needed Information Respondent Methods Tool Intake form Attendance sheet Interview guide Age of child Engaged in child labor At high risk Type of education/training service provided Sex Location Head of household Teacher or trainer Survey Review project records Interviews # of children engaged in or at high risk of entering child labor provided education or training services (E1) Disaggregated by sex, location, type education Labor practices implemented Type of business Location of business Business owner or employee Inspector Survey Interview Observation Review business records % of targeted businesses that implemented improved labor practices Disaggregated by type of business and location Survey Interview guide Observation check-list Policy manual https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Step 5: Pre-test Data Collection Tool Pre-testing is important! Ensures the quality of the tool and decreases the chance of error. Should be conducted systematically, using the same method of administration. Should be administered with a small group of respondents with similar demographics/characteristics as target population, but who are not part of the sample. https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Following the Pre-testing Some questions that might be helpful to answer: Were the instructions easy to follow and understand? Were any questions confusing? redundant? Were questions and answer choices clear? Were they able to answer all of the questions? How long did it take to complete? Too long? Ask respondents for feedback or reflect as a group on its usability of the tool. Revise the tool incorporating the feedback from the pre-test. If Necessary, do another pre-test, if not, prepare for implementation. For online surveys: Test for functionality Review skip sequences Check mandatory questions Make sure easy to navigate through https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Step 6: Planning for Data Collection When will data be collected? How often? During enrollment, baseline/midline/endline, quarterly Who will collect it? Project staff, hired enumerators Training data collectors Role and responsibilities Data collection in District A Data collection in District C Data collection in District B https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Step 6: Planning for Data Collection (cont.) Where will data be collected (e.g., location)? Households, project activity sites, businesses, etc. Logistics: transportation, time of day, etc. Any instructions or processes that need to be followed to collect data? Access or permission issues (e.g., letter from government) Notifying or setting up appointments with respondents Planning for data collection quality checks and oversight CMEP Section IV: Implementation and Management of CMEP https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Roles and Responsibilities Data collection (e.g., project staff, enumerators) Data collection management (e.g., logistics, field monitors) Quality Control Data Analysis Reporting https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Ensuring Data Quality Several Things Can Go Wrong Mistakes in methodology or data collection (e.g., poor data collection tools) Poor data collection planning Mis-interpretation or mis-analysis of data Poor management of data or databases with duplicate records, missing records, partial records Need to scrub data exported from databases prior to using Mitigate Data Quality Issues Pre-testing Training of data collectors Triangulation Reviewing data including during data collection. Informal and formal audits (during and after data collection) https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Knowledge Check Knowledge Check
Question 1: Each performance indicator should be linked to at least one data collection tool in the CMEP. True False https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Question 2: Data collection instruments are developed after the final CMEP is approved. True False https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Question 3: Data collected about beneficiaries' age, test scores, employment status and ethnicity would be considered what type of data? Quantitative Qualitative https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Question 4: Qualitative data is typically easier to aggregate, analyze and interpret in comparison to quantitative data. True False https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Question 5: Which of the following data collection methods is best suited for collecting information from a large population? Focus group discussion Key informant interview Observation Survey https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Question 6: Grantees should only pre-test their instruments if conducting a survey. True False https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Conclusion BUREAU OF INTERNATIONAL LABOR AFFAIRS
Data Tools, Methods, Sources and Types Data Benefits Accountability, Learning, Exploration, Involvement and buy-in Types of data collection tools and methods Surveys, project documents, interview guides, knowledge tests, etc. Primary data sources: Beneficiaries, project partners, government officials, etc. Types of Data: Primary versus secondary data Qualitative versus quantitative data https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Data Collection Steps 1. Define purpose and needed data 2. Identify target respondents 3. Choose the methods and approach 4. Select and develop the data collection tool 5. Pre-test the data collection tool(s) 6. Prepare for data collection https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
Resources Monitoring and Evaluation (M&E) Resource Guide for OCFT Projects: Monitoring and Evaluation (M&E) Resource Guide for OCFT Projects (dol.gov) USAID Data Collection Methods and Tools for Performance Monitoring: Data Collection Methods and Tools for Performance Monitoring (usaidlearninglab.org) The Ultimate Guide to Effective Data Collection: The-Ultimate-Guide-to-Effective-Data- Collection.pdf (orange.ngo) Methodologies for Data Collection and Analysis for Monitoring and Evaluation (IOM): IOM- Monitoring-and-Evaluation-Guidelines-Chapter-4_0.pdf Tips for Developing Survey Instruments and Questionnaires (Bureau of educational and Cultural Affairs Evaluation Division): eca_survey_best_practices_final_formatted.pdf (state.gov) https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
THANK YOU https://www.dol.gov/agencies/ilab Office of Child Labor, Forced Labor, and Human Trafficking @ILAB_DOL
