Annual Strategies for Success Data Collection Training Overview

Annual Strategies for Success data Collection Training August 1, 2019

Agenda YRRS versus ASFS - IMPORTANT NOTES Refresh our memories The Annual SFS Instrument (middle school, high school; English, Spanish) Review the protocol document Selecting schools Selecting classrooms Approaching school administrators Parental consent & Student assent Collecting data Tools and templates Important Reminders

NM YRRS PLEASE MAKE NOTE!!!!!!! The YRRS will be implemented this Fall in the schools. The Annual Strategies for Success DOES NOT REPLACE THE YRRS. If a school is deciding between the YRRS and the ASFS- They NEED TO DO THE YRRS. THE YRRS TAKES PRIORITY OVER THE ASFS IN ALL CASES!!!!! The ASFS should be done in addition to the YRRS, not in place of the YRRS. Please take time to tell the school(s) why the ASFS is a great addition but it not a replacement for the YRRS. If schools are considering not doing the YRRS because of the ASFS, please discourage this as strongly as possible and put them in touch with Dan Green immediately (or with Liz or Martha). We do not want to in anyway undermine the amazing work that DOH is doing with the YRRS. This is very, very, very important. The surveys serve different purposes and therefore both are very important and valuable. If you are unable to administer the ASFS in schools because of the YRRS, please let your Program Manager at OSAP and PIRE know this.

To refresh our memories The Annual Strategies for Success is administered once a year only. Ideally it is administered around the same time each year using the same methodology as previous years. There is a middle school version 6th-8thgrade assessing ATOD use & intervening variables There is a high school version 9th 12thgrade assessing ATOD use & intervening variables All instruments are available English and in Spanish. You may implement the survey using paper questionnaires or on-line. PFS 2015 communities are required to collect the Annual SFS (particularly if the YRRS is not being collected). Other communities doing environmental prevention strategies targeting youth are also encouraged to use the Annual SFS. All instruments & related documents are located on the NM Prevention website- NMPrevention.org

Revisions to the Survey Instruments and Protocol Revisions have been made to the Annual Strategies for Success instruments for this Fiscal Year. 1. adding Juul as a descriptor in the e-cigarette questions 2. defining binge drinking separate for boys and girls (4 drinks in a row/5 drinks in a row) 3. changing the heroin question on the high school survey to reflect the past 30-day rather than lifetime use 4. no incentive for implementing the gambling module this year

The Qualtrics Option Both Strategies for Success (B/P & A) instruments are available in paper form in English and in Spanish You have the option to collect data using tablets, laptops, or computers if this is a reasonable option in a school. The advantage of collecting data on-line include: Students will be skipped out of questions that do not apply to them, shortening the length of the survey for many students You do not have to do data entry or cleaning. A disadvantage of collecting on-line is that the provider must prep each tablet or computer ahead of time, understand first-hand how the on-line survey works, and trouble shoot problems ahead of time by pilot testing.

Reviewing the Protocol Document The protocol is posted on the NM Prevention website (www.NMPrevention.org) under Prevention Program Evaluation Instruments. The provider and the local evaluator must review this document thoroughly and complete the data collection protocol and other documents listed in the protocol together. When you can conduct the survey is likely dictated by the school(s) you end up working with. If you survey after a holiday or school event that might coincide with increased substance use, you should aim to collect around the same time each year so that you are not comparing high use times against low use times (e.g., post spring break estimates with pre spring break estimates). Selecting your survey sample involves selecting schools and then possibly selecting classrooms within the schools. There is the ideal and then the real . Aim for ideal and do what you can to prevent and avoid the pitfalls that inevitably occur by planning ahead as much as possible.

Selecting schools Do you have more than one high school and/or middle school in your community? If no, survey both schools. If yes, you probably need to select a subset of schools to survey. If you have a large community (e.g. Do a Ana County) this becomes tricky (and takes some effort). List all the middle schools & high schools in your community and student size ideally as well and the grades included at each school. (e.g., 9- 12, 10-12, 5-8, 6-8, etc.) Are these public or private schools? Religious? Charter? Do they cater to at-risk students? Gifted students? http://ped.state.nm.us/ped/districtwebsites.html If appropriate, remove any schools that are not representative and may skew your sample favorably or unfavorably (e.g., juvenile detention schools). Decide how many schools you d like to work with. That is the number of schools you will sample. Separately for middle and high schools, categorize them by size, e.g., large, medium and small. Then assign each school a number, (i.e.,1-10) within each category. Using a random number generator, randomly select 1 or more schools to survey. (https://www.random.org/; http://stattrek.com/statistics/random-number-generator.aspx) Ideally, you want at least one school from each category. You may find that a school you select is unwilling to allow you to survey the students. Then you have to go back and select another school randomly.

Example- Middle School Other On-line data collection an option? McKinley County- Middle schools Student Population Classes per grade characteristics of possible importance Location Grades Type of School Chief Manuelito Middle School Gallup 6,7,8 350 Gallup Middle School Gallup 6,7,8 500 Kennedy Middle School Gallup 6,7,8 225 Uplift School Gallup 6,7,8 72 Charter 1 per grade Crownpoint Middle School Crownpoint 6,7,8 500 Navajo Middle School Navajo 6,7,8 225 Ramah Middle/High School Ramah 6,7,8 75 Thoreau Middle School Thoreau 6,7,8 150 Tohatchi Middle School Tohatchi 6,7,8 95

Example- Middle School Other Classes per grade On-line data collection an option? characteristi cs of possible importance Assign a number (s) McKinley County- Middle schools Student Population Type of School Size Randomly select Location Grades category 1 per grade Uplift School Gallup 6,7,8 72 Charter Small NA Ramah Middle/High School Ramah 6,7,8 75 Small 1 Tohatchi Middle School Tohatchi 6,7,8 95 Small 2 * Thoreau Middle School Thoreau 6,7,8 150 Small 3 Kennedy Middle School Gallup 6,7,8 225 Medium 1 Navajo Middle School Navajo 6,7,8 225 Medium 2 Chief Manuelito Middle School Crownpoint Middle School Crownpoi Gallup 6,7,8 350 Medium 3 * 6,7,8 500 Large 1 * nt Gallup Middle School Gallup 6,7,8 500 Large 2

Example- High School On-line data collection an option? Other Student Population Classes per grade McKinley County- High schools Location Grades Type of School characteristics of possible importance Gallup High School Gallup 9,10,11,12 2000 public Hiroshi Miyamura High School Gallup 9,10,11,12 150 public Gallup Central High School Gallup 9,10,11,12 500 public college prep/college credits earned Middle College High School Gallup 10,11,12 100 (68?) Charter Crownpoint High School Crownpoint 9,10,11,12 500 public Navajo Pine High School Ramah Middle/High School Thoreau High School Tohatchi High School Navajo Ramah Thoreau Tohatchi 9,10,11,12 9,10,11,12 9,10,11,12 9,10,11,12 500 120 450 200 public public public public 100% free/reduced lunch, 50% Navajo spoken at home Pueblo Pintado Yes; all students have iPads Tse Yi Gai High School 8,9,10,11,12 125 public 1 per grade Twin Buttes High School Zuni High School Zuni Zuni 9,10,11,12 9,10,11,12 85 100 Tribal? Tribal?

Example- High School Other Classes per grade On-line data collection an option? Student Population Type of School characteristics of possible importance Size Assign a number Randomly select McKinley County- High schools Location Grades category Twin Buttes High School Zuni High School Zuni Zuni 9,10,11,12 9,10,11,12 85 100 Tribal? Tribal? Small-1 Small-1 1 2 * college prep/college credits earned Middle College High School Gallup 10,11,12 100 (68?) Charter Small-1 3 Ramah Middle/High School Ramah 9,10,11,12 120 public Small-2 1 Yes; all students have iPads 100% free/reduced lunch, 50% Navajo spoken at home Pueblo Pintado 1 per grade Tse Yi Gai High School 8,9,10,11,12 125 public Small-2 2 Hiroshi Miyamura High School Gallup 9,10,11,12 150 public Small-2 3 * Tohatchi High School Thoreau High School Gallup Central High School Tohatchi Thoreau Gallup 9,10,11,12 9,10,11,12 9,10,11,12 200 450 500 public public public Small-2 Medium Medium 4 1 2 Crownpoint High School Crownpoint 9,10,11,12 500 public Medium 3 Navajo Pine High School Gallup High School Navajo Gallup 9,10,11,12 9,10,11,12 500 2000 public public Medium Large 4 1 * *

Selecting classrooms/Students Ideally, 100% of the classrooms in the school will participate. More likely, you may get to survey 9thgrade homeroom classrooms or health classes, or something along those lines. You and the school administrators will have to negotiate what will work for them and for you. If you are unable to survey all the students, then ask if you can randomly select 2 to 3 classrooms in each grade and survey them. Then using the same approach to randomly selecting school, randomly select classrooms. There is no need to sort classrooms by size.

Example: Selecting Classrooms Crownpoint High School Randomly Choose 2 Randomly Choose 2 Grade Teachers Assign a number Grade Teachers Assign a number 9th Jones 1 * 11th Campos 1 Silver 2 Catton 2 Garcia 3 Chavez 3 * Abieta 4 Chapo 4 * Agnew 5 * Dawes 5 10th Alderete 1 * 12th Denison 1 * Barriga 2 Draper 2 Becenti 3 * Dooley 3 Bond 4 D'Orazio 4 * Blackgoat 5 Esparza 5

Calculating a response rate Before you go in to collect data, you need to know how many students are going to be given the questionnaire. This means knowing the number of students in the classroom, and the number with parental permission. You will need to calculate your response rate based on these two numbers. The Feds are requiring a 70% or better response rate on the Annual SFS survey Therefore, you are expected to calculate a response rate to your Annual SFS survey and to report your response rate in your Annual Summary Report at the end of the fiscal year or whenever you completed your summary report. To calculate a response rate you must know the Total number of students in school or classrooms selected; Total number of students you have permission to survey in the school/classrooms; Total number of students who actually complete the survey. (at least of the questions) Let s work through 2 examples

Calculating a Response Rate: Example 1 Scenario: You have been given permission to survey 4 classrooms in grades 6, 7 and 8 in a given middle school and in each of those classrooms, there are 30 to 35 students.

Example 1: Response rate calculations Response rate 1: The # of completed surveys divided by the # expected (# with parental permission) total # of students # with parental permission # completed surveys 6th grade Classroom 1 Classroom 2 Classroom 3 Classroom 4 30 31 33 30 30 29 30 30 25 29 30 30 341/368 = .93 7th grade 93% Response rate (in this school) Classroom 1 Classroom 2 Classroom 3 Classroom 4 32 33 30 31 32 30 28 29 30 30 28 28 Response rate 2: The # of completed surveys divided by the # expected (total # of students) 8th grade Classroom 1 Classroom 2 Classroom 3 Classroom 4 30 34 35 31 380 30 34 35 31 368 26 25 30 30 341 341/380 = .90 90% Response rate (in this school) TOTALS

Calculating a Response Rate: Example 2 Scenario: You have been given permission to survey all the students in grades 9 through 12 in a given high school in your community.

Example 2: Response rate calculations Response rate 1: The # of completed surveys by the # expected (with parental permission) total # of students # with parental permission # completed surveys 1160/1214 = .96 9th grade 350 250 248 96% Response rate (in this school) 10th grade 429 336 333 Response rate 2: The # of completed surveys by the # expected (# total students) 11th grade 388 360 325 1160/1571 = .74 12th grade 404 268 254 74% Response rate (in this school) TOTALS 1571 1214 1160

Approaching School Administrators Be prepared! Read the entire protocol and have your request ready when you meet with them. Prepare your elevator speech ahead of time. Bring a copy of the survey, the parental consent documents, and Memorandum of Agreement (MOA) to aid the discussion. Know what incentive(s) you intend to offer to the school and/or teachers for participating. Of course, be open to requests. Be able to explain how this is different/similar to the YRRS and how you intend to use the data and share your findings with them. Be able to explain how this benefits them!!!! Be sure to discuss how parental consent will be gathered by the school Create an MOU or MOA ideally so the school will be committed to participating each year.

Parental Consent When collecting data from minors, you must have parental consent. This can be done actively or passively You and the school must decide the best approach. Active consent, requires the parent or guardian to return a signed consent form in order for the student to participate. They must actively opt in. Passive consent, requires the parent or guardian to return a signed form in order to the student to not participate. They must actively opt out (or passively opt in). Discuss the best ways to gather parental consent in the school.

Once you have permission to collect data you still have to collect and analyze the data. You cannot collect data from any student without parental consent, AND You must get youth assent to participate. Youth are allowed to refuse to participate. To protect the students privacy, ideally teachers would not administer nor collect the questionnaires. (This is not always practical, in which case, you need to have a way to collect the completed surveys that protects the student responses.) The Annual SFS is anonymous so students should not put their names or any identifying information on the survey. The assent documents should be kept separate from the questionnaires.

Two options for collecting data Paper Questionnaire Qualtrics App Fairly simple to set up and administer Allows for data collection on IOS or Android based mobile devices. Must complete data entry Can collect data when not connected to internet and upload later. Does not require any fancy technology Can collect data directly on-line Does not have automatic skip patterns to it may take longer to complete. No data entry required Requires practice beforehand and working out all the contingencies

Data Collection: Paper Questionnaires For many of you, collecting data using a paper survey and allowing students to complete it and hand it back is likely to be the easiest. If you use paper surveys, you will be responsible for completing the first page of the survey. You must include your: Site ID: http://www.nmprevention.org/Service-Providers.html School ID (2 digits # - 01, 02, 03, etc.) School Name Date you are administering the survey This survey is conducted by the (Coalition Name) on behalf of the NM Office of Substance Abuse Prevention (NM OSAP). If you would like to know more about why we are doing this survey, please contact (First and Last name of contact person at Coalition) at (Provide a phone number) or (Provide an email address). If you have questions or concerns about this study or your rights as a survey participant in the study, please contact (First and Last name of contact person at School) at (Provide a phone number) or (Provide an email address). A Survey ID (4 digits- 0001, 0002, 0003, etc. or 1111, 1112, 1113, etc.) VERY IMPORTANT!!!!!!!!!

Data Collection: Paper Questionnaires You must read the student assent document to the students and have them sign and turn in if they agree to participate. You must store these documents for 5-7 years. You must read the first page of the survey, that explains the survey, to the students. Students must understand why they are being asked to complete the survey. Allow time for students to ask any questions they may have prior to completing the survey. Remind them to not provide any identifying information on the instrument itself. Provide a safe way for them to turn in their completed questionnaires so that teachers and/or providers can t see their responses. Examples: Provide every student with a large manila envelope to place survey in prior to turning it in. You can ask them to seal it even. Provide a large box with a lid for students to place the completed survey in after completing it. You must provide each student a copy of the Take Home Information document.

To Do: Reminder List for Paper Data Collection You must: Prepare the first page of every paper survey with the required information. You must collect student assent on paper and store it for 5 7 years. You must read the cover page information to the students. You must provide a safe, secure and private means to collecting the completed questionnaires You must provide every student with the take home information You must complete data entry as soon as possible.

Data Collection: On-line Questionnaires In some schools, it may be possible to collect data on-line using the Qualtrics Survey option. If you use on-line questionnaires, students ideally should be able to select these two pieces of information (with assistance & monitoring from you) County (drop down list) School Name (drop down list) This hopefully will eliminate the problem of there being incorrectly entered information and typos.

Data Collection: On-line Questionnaires Some very important considerations if collecting using Qualtrics on-line survey. Will you have a strong wi-fi connection at the school? If not, you must download the Qualtrics App onto any computer you are taking into the school. This will allow students to complete the survey off-line. PLEASE NOTE: The offline app only works on IOS devices and not Android. Alternatively, if you have wi-fi access, you can access the survey on-line with a browser. But you must test this ahead of time to make sure it will work. If you are borrowing i-Pads from Coop Consulting with the Qualtrics App on it (as opposed to going directly to the survey on-line on their own tablets or laptops) and having students complete the survey off-line, then you must upload every survey from every tablet or else they will not be included in your final tally. Example: A school allows you to bring 30 iPads to each classroom to collect data. However, there is no wi-fi access in the school. You must set up each iPad to collect data off-line. You may collect data in room 1, and then go to the next room, set up all the tablets again, collect the data, again, and go to the next room, etc. All the surveys completed on the tablet will be stored on the tablet in the app. But they are not loaded to the on-line database. After collecting all the data, when you go back to your office, you must load all the surveys completed on each tablet up to the on-line version of Qualtrics. In other words, you will need to link all 30 tablets to the web and upload all the surveys. After you upload all the surveys, you will then need to log out of the Qualtrics app on every iPad.

To Do Reminder List for On-line data collection You must: Download the Qualtrics App onto any tablets you are taking into the school.* The exception to this, is if you have wi-fi or internet access at the school can you can then use the on-line browser to access the survey on-line. You must collect student assent on paper and store for 5-7 years. You must read the information/consent page of the survey and answer any questions. You must provide the take-home information to all students. If tablets, laptops, computers are not able to get on-line, you must upload all the completed surveys from all the tablets, laptops, computers when you are able to get on-line. You must log out of Qualtrics after uploading the data.

Specific to on-line data collection After you have completed on-line data collection in a school, please let us know the exact dates when data were collected. The easiest way to do this is probably to track in a spreadsheet the school and the dates data collection occurred. The reason for this is because on occasion folks may go back on a computer at a later time, the URL is still in the browser, and someone completes the survey. We want to remove any illegitimate surveys when we clean the data and this will help us do that.

PLEASE NOTE: PIRE will provide an in person/detailed training on the on-line use of the survey if this would be helpful to you or your staff. Please send a request to: Kim Zamarin at: kzamarin@pire.org or to Martha Waller at mwaller@pire.org I strongly recommend this if this is the first time collecting data on-line.

If you want to use Tablets If you think you will want to use iPads from Coop Consulting to collect ASFS data then . Contact Tina Ruiz, Andrea Niehaus, Liz Lilliott and Kim Zamarin ASAP when you know when you want to collect data and approximately how many tablets you would need.

A Note About Adding Extra Questions You may choose to add additional questions that are specific to your program and will be asked of students only in your community. If you choose to add questions, they must be reviewed and approved by PIRE ahead of time. We will need time to review, approve, translate and program the questions into the on-line survey. For paper surveys, we need time to review, approve, and translate the questions. Please allow more than two weeks if at all possible for this process to take place.

A Note About Incentives You do not have to use incentives. If you choose to have incentives, they should be not be any greater than $1.50 per student. You may choose to incentivize the school, the teachers, or the students but not all of them. If you choose to use incentives, please talk with your program manager ahead of time to make sure they are okay with your approach.

Data Collection Budget for the ASFS Similarly to the recent NMCS, you are now asked to complete a budget for collecting the ASFS. Ideally this exercise will help you think through the many things that have to happen in order to collect the ASFS data including needed staffing, printing, travel, incentives, etc. This is not intended to be horribly burdensome, but rather a way to help you plan better real costs involved.

Tools and Templates Available to Help You Active parental consent forms in English and in Spanish Passive parental consent forms in English and in Spanish Student assent forms in English and in Spanish Memorandum of agreement between coalition and school (defines roles, responsibilities, expectations) Data collection planning template Budget Template Data collection protocol (to be submitted to PIRE for review by SEOW prior to any data collection) Step-by step guide to collecting data using tablets or computers

Important Reminders Your data collection protocol MUST be reviewed by PIRE. Turn your protocol into PIRE at least 2 weeks prior to when you want to start collecting data. Do not wait to start contacting school administrators. This is the hardest and most time consuming part. Consider how you might complement the YRRS data collection. Always consider how you will protect the privacy of the students and the anonymity of their data. Read the student assent form to the students prior to their signing. Provide all students with take home information about who to contact should concerns arise from taking the survey. All necessary documents are on the NM Prevention website: NMPrevention.org PIRE would like work with each community as needed to help develop or revise the data collection protocol to make sure that approval is straightforward.

What & When to submit to PIRE for review WHAT? WHEN? Data Collection protocol Submit all the preparation documents at least 2 weeks (preferably more) prior to when you intend to collect data Completed cover page for survey Completed take home information form Completed parent and student consent/assent documents Submit to PIRE, with the exception of the budget which should go to OSAP. Completed Budget (submit to OSAP)

Questions and concerns? The PIRE Team Martha Waller Liz Lilliott Dave Currey mwaller@pire.org lilliott@pire.org dcurrey@pire.org 919-265-2631 575-765-2330 (919) 265-2622 Lei Zhang Kim Zamarin Ashley Simons-Rudolph lzhang@pire.org KZamarin@pire.org ASimons-Rudolph@PIRE.org 919-265-2624 505-907-0801 919-265-2620 Christina L pez-Guti rrez Marissa Elias CLopez@pire.org melias@pire.org 575-650-4111 505-765-2329