Data-Rich Life: Best Practices for Collection & Management

Field Guide to Living a Data-Rich Life Webinar 9: Best Practices for Data Collection and Management Hosted by Durham s Partnership for Children, with support from the North Carolina Partnership for Children Presented by Compass Evaluation and Research

Field Guide to Living a Data-Rich Life Welcome Compass Evaluation and Research Webinar 9 in our year long data quality series Last time (webinar 8), we discussed Approaches to sampling Types of samples Using sampling in your evaluations Today, we will: Review three next steps once data are collected Discuss practices for ensuring secure collection and management of data Best Practices for Data Collection and Management

Field Guide to Living a Data-Rich Life Overview Compass Evaluation and Research Overview of data collection approaches and next steps Identify Personally Identifiable Information Overview of data security basics Strategies and tips for securely managing data Tips for creating an analysis framework Best Practices for Data Collection and Management

Field Guide to Living a Data-Rich Life Data Collection Approaches Compass Evaluation and Research Approach Approach Type(s) Type(s) of Data of Data Next Steps Next Steps Use of spreadsheets to enter and manage raw data Securely maintain both electronic and raw data Create an analysis framework to organize electronic data for analysis Survey Mix of quantitative and qualitative Primarily qualitative Focus Group Interview Primarily qualitative Standardized Assessment Primarily quantitative Observation Mix of quantitative and qualitative Primarily qualitative Audio/Video Recording Document review Mix of quantitative and qualitative Best Practices for Data Collection and Management

Three Next Steps Field Guide to Living a Data-Rich Life Compass Evaluation and Research Now that you have your data-- Use spreadsheets (or other electronic format) to enter and manage raw data Securely maintain both electronic and raw data Create an analysis framework to organize electronic data for analysis Best Practices for Data Collection and Management

Moving from Data to Analyses Field Guide to Living a Data-Rich Life Compass Evaluation and Research Raw data hard to work with for analysis purposes Plan to enter raw data into an electronic format Use electronic format for both quantitative and qualitative data management and analysis Quantitative data Spreadsheet Statistical package such as SAS or SPSS Qualitative data Spreadsheet Word processing format Audio-or visual-recordings Use a rubric to capture key elements Transfer rubric data into spreadsheet Best Practices for Data Collection and Management

Securely Maintain and Manage Data Field Guide to Living a Data-Rich Life Compass Evaluation and Research Must protect personally identifiable information (PII) PII includes data such as: Name Mother s maiden name Address Date of birth Place of birth Social Security Number Names of parents and family members Best Practices for Data Collection and Management

Strategies for Securely Maintaining and Managing Data Field Guide to Living a Data-Rich Life Compass Evaluation and Research Adhere to requirements in documents such as Health Insurance Portability and Accountability Act (HIPAA) and Family Educational Rights and Privacy Act (FERPA) Adhere to agency policies and procedures for data security Adhere to funder requirements Develop new policies and procedures (as needed) to ensure data are securely collected and managed Participate in trainings targeting data security Best Practices for Data Collection and Management

HIPAA: Privacy and Security Rules Field Guide to Living a Data-Rich Life Compass Evaluation and Research Privacy Rule Conditions of releasing or sharing identifiable health and health services information Security Rule Conditions for securing and protecting identifiable health and health services information Best Practices for Data Collection and Management

Privacy Rule Field Guide to Living a Data-Rich Life Individually identifiable health information = protected health information Includes any information that can be used to establish identity. Examples include: Name Address Date of Birth Social Security Number Protected health information can be secured by making it de-identified Data from which identifying information has been removed No restrictions on use Can substitute in unique identifiers that cannot be linked to identity Compass Evaluation and Research Best Practices for Data Collection and Management

HIPAA Security Rule Field Guide to Living a Data-Rich Life Compass Evaluation and Research Facilitates the use of Electronic Health Records (E-PHI) and other technology while still protecting identifiable information and identity Use of technology increases risks of unintentional or malicious release of information Ensure confidentiality and integrity of data created, received, maintained, or transmitted Anticipate, identify, and protect against threats Anticipate and protect against non-required or non-permitted use and disclosure Ensure workplace compliance Best Practices for Data Collection and Management

Implications for Data Collection and Management Make clear to participants whether their data are ANONYMOUS ANONYMOUS or CONFIDENTIAL CONFIDENTIAL Anonymous: Anonymous: impossible to link information with identity, even by researcher No names No address or contact information No Date of Birth or Social Security Number No use of an ID number that a partnering agency or non-project staff may recognize No unique demographic features No questions for which the response might identify the individual No ISP or computer address information (electronic survey or web-based data) Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Implications for Data Collection and Management Make clear to participants whether their data are ANONYMOUS ANONYMOUS or CONFIDENTIAL CONFIDENTIAL Confidential Confidential: : identification is possible but researchers pledge to safeguard Researchers may be able to establish identity Because identity is known to the research team, have to protect the information Locked cabinets and passwords Use of IDs that are unique to the project Separate analysis files from master list of subjects Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Implications for Data Collection and Management Make clear to participants how personal and private data will be secured Who will have access? How will the data be stored and kept safe? How will the data be destroyed? When will the data be destroyed? Does an individual have to provide information that they don t want to share? Obtain informed and written consent for the use of data Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Informed Written Consent: Basic Elements A statement that the program involves the collection of data for program implementation, evaluation, or research purposes An explanation of the purposes of the data collections The expected duration of the subject's participation A description of the data procedures to be followed (including data security) A description of any reasonably foreseeable risks associated with collecting the data A description of any benefits to the subject or to others which may reasonably be expected Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Informed Written Consent: Basic Elements A statement describing the extent, if any, to which confidentiality of records identifying the subject will be maintained Explanation of any compensation to be offered An explanation of whom to contact for answers about rights and protections A statement that participation is voluntary, refusal to participate will involve no penalty or loss of benefits to which the subject is otherwise entitled, and the subject may discontinue participation at any time without penalty or loss of benefits, to which the subject is otherwise entitled Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Implications for Data Collection and Management Administrative Protocols Physical Treatment of Data Technical Systems for Collecting, Managing, and Storing Data Field Guide to Living a Data-Rich Life Compass Evaluation and Research Also important: Procedures for Collecting, Managing, and Storing Data Organizational Culture around Data Security and Confidentiality Best Practices for Data Collection and Management

Administrative Protocols Field Guide to Living a Data-Rich Life Compass Evaluation and Research Ongoing and regular assessment of risks related to PII Example: annual assessment of data security protocols and technology Ongoing implementation of security measures to reduce risks and vulnerabilities Example: require visitors to sign-in and be escorted Identifying a staff person or administrator for developing, implementing, and managing data security Example: identify a data security coordinator to manage and stay updated on protocols and legal requirements Best Practices for Data Collection and Management

Physical Treatment of Data Limit access to client data and information Especially for data and information covered by Privacy Rule Example: define separate levels of users in a program database Limit access to computers, personal digital assistants (PDAs), workstations, and all other electronic media that may contain identifiable data Example: use of password protections, encryption, computer time outs Develop, implement, and manage policies, procedures, and protocols for the appropriate transfer, removal, and destruction of electronic media Includes re-use of electronic media Example: National Institute of Standards and Technology (NIST) guidance and best practices for data destruction; Guidelines for Media Sanitation. Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Technical Systems Field Guide to Living a Data-Rich Life Compass Evaluation and Research Ensure only authorized users can access data kept electronically Ensure management and oversight of use Includes hardware, software, and procedures for tracking and reviewing who is accessing and using electronic data Ensure electronic data cannot be altered or destroyed Ensure that the transmission or transfer of electronic data prevents unauthorized access and use of electronic data Best Practices for Data Collection and Management

Procedural Safeguards Field Guide to Living a Data-Rich Life Compass Evaluation and Research Differentiate between Required and Addressable Requirements Required: have to implement the protocol Addressable: have to implement the protocol if it applies to your situation or context Required security standards must be met Addressable security standards must be met if reasonable and appropriate Ensure agency policies and procedures are updated to meet required and addressable standards Best Practices for Data Collection and Management

Organizational Culture Field Guide to Living a Data-Rich Life Compass Evaluation and Research Policies, procedures, and protocols for use within the organization and by business associates Business Associates also are liable for secure collection and management of data Provide training and refreshers for Board and Committee members and staff Develop a culture of ensuring data security and confidentiality Daily practices of communicating about programs Daily practices of securing participant files and records Remember that organizations are responsible for Acts of business associations violate the Privacy or Security Rules Addressing a breach or violation by a business associate Best Practices for Data Collection and Management

Steps to Greater Security Field Guide to Living a Data-Rich Life Compass Evaluation and Research Review resources at PTAC PTAC: Privacy Technical Assistance Center Look for an agency or data security handbook If a handbook exists: review specific procedures against recommended data security steps If a handbook doesn t exist, consider developing one Trainings in protecting personally identifiable information Upcoming from the North Carolina Partnership for Children Protection of Human Subjects: https://phrp.nihtraining.com/users/login.php PTAC guidance videos Best Practices for Data Collection and Management

Analysis Framework Field Guide to Living a Data-Rich Life Compass Evaluation and Research Playbook for linking data to evaluation or research questions Focus is on how to answer each evaluation question, using available data Details the steps to be followed to analyze data for each evaluation question Remember: each evaluation question treated separately Plan for linking variables across one or more data collection instruments, as needed Best Practices for Data Collection and Management

Analytic Framework Example, Part 1 Field Guide to Living a Data-Rich Life Compass Evaluation and Research Evaluation question: To what extent do parents exhibit improvements in positive discipline practices after completing a Positive Parenting training series? Hypotheses: Parents improve, do not improve (stay the same), or exhibit worse discipline practices after completing the training series. Hypotheses: Parents with different primary languages will exhibit different levels of gains. Best Practices for Data Collection and Management

Analytic Framework Example, Part 2 Data collected: pre and post Parenting Practices Interview surveys, enrollment and attendance data, parent demographics Specific variables: PPI subscales: Harsh Discipline, Consistent Discipline, Appropriate Expectations Analytic approach: Calculate the percent of participants making gains Compare average pre and post scores on the three PPI subscales Examine the extent of any gains Regroup parents/parent data by primary language and re-examine gains Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Analytic Framework Example, Part 3 Field Guide to Living a Data-Rich Life Compass Evaluation and Research Step 1: enter all raw survey data into a spreadsheet Step 2: following publisher guidelines, calculate sub-scale scores and enter into spreadsheet Step 3: identify participants with both a pre and a post test score Step 4: identify the percent of participants with both a pre and a post who experienced positive change Step 5: calculate average pre test score and average post test score Step 6: as possible, apply statistical tests to test for the significance of gains Best Practices for Data Collection and Management

Analytic Framework Example, Part 4 Field Guide to Living a Data-Rich Life Compass Evaluation and Research Step 7: divide the sample into two groups, by primary language Step 8: repeat steps 3-6 for each language group and compare findings Step 9: respond to hypotheses These are your findings for this evaluation question Best Practices for Data Collection and Management

Advanced Topic: Missing Data Field Guide to Living a Data-Rich Life Compass Evaluation and Research May occur in several ways Low response rate Participants who only supply a pre or a post but not both Participants who leave some questions or survey items blank Analyses typically conducted with complete data Consult with statistician or methodologist on how to fill in missing data Best Practices for Data Collection and Management

Questions? Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Next Steps The webinar series is designed to help you create a plan to capture, manage, analyze, and use high quality data: September 28, 2016: Finding the Value in Evaluation: Cultural Relativity and Bias. October 25, 2016: Using Data: Effective Reporting and Grant Writing. November 30, 2016: Thinking Beyond Your Program: Evaluating Systems and Collaborations. Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management

Additional questions? Taylor Webber-Fields Durham s Partnership for Children Phone: 919-403-6960 Email: taylor@dpfc.net Field Guide to Living a Data-Rich Life Compass Evaluation and Research Taylor will receive and direct your questions or requests to the appropriate Partnership staff members Best Practices for Data Collection and Management

Additional questions? Feel free to contact me, too! Sarah Heinemeier Compass Evaluation and Research Phone: 919-308-5019 Email: sarahhei@compasseval.com Field Guide to Living a Data-Rich Life Compass Evaluation and Research Best Practices for Data Collection and Management