Assessment & Evaluation at NSF CISE REU Site PI Meeting
This content showcases the participants, supporters, and tools used in the NSF CISE REU Site PI Meeting held in February 2012. It includes information on project supporters from different universities, assessment tools, and future directions for assessment and evaluation in the research program.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Assessment & Evaluation NSF CISE REU Site PI Meeting February, 2012
Project Supporters 2009 Working Group Members Guy Alain Amousou Chris Aberson Andy Fagg Stephen Gilbert Sanjay Madrias Joan Peckham Kevin Zeng 2010 Implementation Members Brooklyn College, CUNY Iowa State University Colorado Springs Jackson State University Dakota State University Louisiana State University Depaul University Montclaire State University Depauw University Marshall University Hope College Oklahoma University 2010 Participants: 13 Common Application 20 CISE REU Pre-Post Survey Wendy Cooper Manfred Huber Eric Wong Teresa Dahlberg Niels Lobo Yu-Dong Yoa University of Alabama University of Central Florida UNC Charlotte University of Texas Austin University of South Carolina University of Houston University of Illinois Urbana-Champaign University of Massachusetts Amherst University of Wisconsin Oshkosh NSF CISE REU PI Meeting, Monterey, February 2012
Project Supporters 2011 Implementation Members Auburn University University of Alabama Dakota State University of California- Berkley Iowa State University University of Central Arkansas Louisiana State University of Central Florida Virginia Tech Marquette University of Houston Marshall University University of Illinois at Urbana-Champaign Rutgers University Montclaire University University of Massachusetts Amherst University of Texas San Antonio North Dakota State University University of Missouri Texas State San Marcos UNC Charlotte University of South Florida University of Texas Austin University of Wisconsin Oshkosh Washington State University University of Maryland 2011 Participants: 20 Common Application 18 CISE REU Pre-Post Survey NSF CISE REU PI Meeting, Monterey, February 2012
Assessment Project Common Application a la carte Survey Evaluation Toolkit Coitweb.uncc.edu/reu/toolkit Future Directions Toolkit and Survey Refinements Continued Longitudinal Follow up Explore Common Application tools NSF CISE REU PI Meeting, Monterey, February 2012
Common Application 2010 (13 sites) Most applied to 1 site Application Range: 29-152 Average Number of Applications per Site: 77 ~ 130 accepted = 18% approximate acceptance rate 2011 (19 sites) (only 14 with apps) Most applied to 1 site Application Range: 4-176 Average Number of Applications per Site: 79 697 309 Unique Applicants Applied to 1+Sites NSF CISE REU PI Meeting, Monterey, February 2012
Unique Applicant Gender 1% 27% 26% 2% 72% 72% 2010 2011 Males 615 Males 500 Females 220 Females 187 Didn't Specify 16 Didn't Specify 10
Unique Applicant Ethnicity 1% 2% 10% 1%7% 14% 18% 16% 7% 10% 61% 52% 1% 1% 2010 2011 Asian 117 Asian 68 African American 133 African American 124 Hispanic/Latino 81 Hispanic/Latino 52 Pacific Islander 4 Pacific Islander 4 Caucasian 440 Caucasian 426 Native American 7 Native American 14 Multi-Ethnic/Other 63 Multi-ethnic/Other 9
Applicant Level in School 12% 4% 18% 16% 28% 50% 22% 42% 8% 2010 2011 Freshman 64 Freshman 86 Sophomore 264 Sophomore 191 Junior 294 Junior 273 Senior 123 Senior 142
Degrees Considering 5% 5% 0% 1% 26% 28% 66% 69% Master's Programs 479 2010 2011 Master's Programs 552 PhD Programs 180 PhD Programs 235 Undecided 3 Undecided 9 Not Considering Grad Program 5 Not Considering Grad Program 45
Post-Graduation Plans - Industry 33% 34% 36% 33% 2% 6% 1% 27% 23% 5% 2010 2011 Agree 157 Agree 281 Strongly Agree 104 Strongly Agree 228 Disagree 25 Disagree 45 Strongly Disagree 9 Strongly Disagree 6 Uncertain 146 Uncertain 287
Post-Graduation Plans - Academia 24% 24% 48% 50% 13% 11% 15% 2% 2010 12% 1% 2011 Agree 104 Agree 201 Strongly Agree 57 Strongly Agree 128 Disagree 49 Disagree 100 Strongly Disagree 9 Strongly Disagree 11 Uncertain 222 Uncertain 407
First Person in Family to Attend College 1% <1% 19% 19% 81% 80% 2011 2010 No 679 No 356 Yes 165 Yes 85 No selection 5 No selection 1
Immediate Family Involved in Research 14%<1% 1% 14% 85% 86% 2010 2011 No 380 No 725 Yes 61 Yes 117 No selection 1 No selection 7
CISE REU Pre-Post Survey Pre survey: May-June and Post survey: July-August Total N: 339 REU Students; 20 Sites in 2010 + 18 in 2011 What is measured: Self Efficacy I can formulate a research problem Intent to attend graduate school I plan to apply to graduate school in a computing discipline Attitudes towards computing I like to use computer science to solve problems Help seeking/coping skills When I do poorly on an exam, typically I .skip class Positive Impact has been demonstrated NSF CISE REU PI Meeting, Monterey, February 2012
Methodology Items 4 point Likert type scale, 4 being positive Some items were reverse scored Collapsed into construct means representing 4 variables Ethnicity collapsed into URM status Reliability Coefficient alphas above .547 MANOVA To test hypothesis that there would be differences between means based on time, gender, URM status NSF CISE REU PI Meeting, Monterey, February 2012
Survey Participant Profiles Pre Demographics (N=219) Gender Ethnicity Asian African American Hispanic/Latino Native Hawaiian/Pacific Islander Female 32% Caucasian Native American Prefer not to specify Multi-ethnic Male 68% 2%3% 3% 9% 10% Level in school Level in school in Fall 14% 58% 1% freshman sophomore junior senior 0% 6% 14% (28) reported having had 1 prior REU 33% 61% NSF CISE REU PI Meeting, Monterey, February 2012
Survey Participant Profiles Gender Post Demographics (N=160) Male Female Prefer not to specify Ethnicity at Post Survey 4% Asian African American Hispanic/Latino Native Hawaiian/Pacific Islander Caucasian Native American 37% Prefer not to specify Multi-ethnic 59% 4% 7% 7% 10% Level in School in Fall 2011 4% sophomore junior senior 11% 5% 2% 39% 55% 56% NSF CISE REU PI Meeting, Monterey, February 2012
Combined Outcomes 2010+2011 2010 Increases at Post Assessment were not significant No main effects, no interactions 2011 Significant increase at Post for Self-Efficacy NSF CISE REU PI Meeting, Monterey, February 2012
Outcomes 2010 No significant increases 2011 *Significant increase (p=.00) Construct Self- Efficacy Time pre Mean 3.05 SD 0.47 Construct Self- Efficacy Time pre Mean 3.11 SD 0.47 post pre post pre post pre 0.40 0.46 0.47 0.32 0.29 0.33 post pre post pre post pre 0.39 0.59 0.66 0.36 0.38 0.39 3.43 3.10 3.19 3.59 3.65 2.94 3.48* 3.22 3.23 3.67 3.70 2.98 Intent Intent Attitude Attitude Help Seeking- Coping Help Seeking- Coping post 0.37 post 2.98 0.36 2.98 NSF CISE REU PI Meeting, Monterey, February 2012
Post Program Evaluation Majority of REU Students were satisfied with their program Faculty advisor 89% Housing 90% Program in general 90% Research experience 91% Interaction with staff 88% Interaction with students 90% NSF CISE REU PI Meeting, Monterey, February 2012
What does this mean? We know there is positive impact from the REU experience Increases between pre and post surveys Slightly higher interest in graduate school among applicants that did participate to those who didn t Individual sites and students have powerful stories The surprise factor can negatively impact attitudes in the short term Look at individual stories Consider the learning outcomes NSF CISE REU PI Meeting, Monterey, February 2012
Implications & Limitations Effects are small given the time frame Effects may be delayed Self-report Possible ceiling effect Consider time series design What else should we measure? Mentoring Ethics Knowledge gains NSF CISE REU PI Meeting, Monterey, February 2012
Next Steps Common Application Track offers Follow up with applicants Compare accepted/declined on key indicators Survey Deeper analysis Larger sample More variables, or different variables Control groups? Learning outcomes? NSF CISE REU PI Meeting, Monterey, February 2012
Discussion Collaborations for Research & Writing Circles Case Studies Develop and pilot new survey modules (i.e. ethics) CISE REU Evaluation Toolkit http://www.coitweb.uncc.edu/reu/toolkit/ NSF CISE REU PI Meeting, Monterey, February 2012
