MSBS Program Quality Improvement Overview
The MSBS Program focuses on fostering critical analysis skills, promoting research abilities in behavioral sciences, and enhancing knowledge of theories and methods. The program aligns its objectives with the university's mission to provide a quality education for diverse students, preparing them for professional success and lifelong learning in the behavioral sciences field.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Department of Psychology & Human Ecology School of Education and Behavioral Sciences Masters of Science in Behavioral Sciences 421601 610 Program Quality Improvement Report 2009-2010 1
MSBS Program Objectives & Student-Learning Objectives 1. To foster the ability to synthesize and critically analyze information. 1. Students will synthesize and analyze information critically. To promote research skills in the behavioral sciences. 2. Students will demonstrate basic research skills in the behavioral sciences. To promote basic knowledge of theories, methods, and findings in the behavioral sciences. 3. Students will demonstrate knowledge of theories, methods, and findings in the behavioral sciences. To promote competence in the application of basic knowledge within the field of the behavioral sciences. 4. Students will apply basic knowledge within the field of the behavioral sciences. 2. 3. 4. 2 Program Quality Improvement Report 2009-2010
Holistic Alignment of Program Objectives to University, School, Departmental, & Program Mission The MSBS Program is dedicated to providing a quality education for a diverse and dynamic student body. As a consequence, our program is designed to serve that segment of the population employed in or interested in a broad spectrum of professions in the behavioral sciences. Our students are educated to be and have proven to be a dynamic force in the behavioral sciences in our community. Program Quality Improvement Report 2009-2010 3
Holistic Alignment of Program Objectives to University, School, Departmental, & Program Mission The MSBS Program is committed to fostering a student- centered academic environment that combines innovative classroom teaching with experiential learning. We have listened to our students and have restructured our program to meet their career and educational goals. We have provided them an education anchored by dynamic professionals actively working in the field of the behavioral sciences and have collaborated with the behavioral sciences community to ensure an experiential learning environment. Program Quality Improvement Report 2009-2010 4
Holistic Alignment of Program Objectives to University, School, Departmental, & Program Mission The MSBS Program is dedicated to preparing students for professional success, responsible citizenship, life-long learning, and meaningful contributions to a rapidly changing world. We have structured our program to prepare students for successful professional licensure in the behavioral sciences with a strong ethical underpinning which will ensure significant and dynamic contributions to a rapidly changing community. Program Quality Improvement Report 2009-2010 5
Holistic Alignment of Program Objectives to University, School, Departmental, & Program Mission With a strong emphasis on the study of psychological cultural diversity and a zeal for a strong work ethic, the MSBS Program is a driving force in the cultural life and economic development of our community. Focused on promoting theoretical & methodological cultural competencies, our program challenges our students to be a driving force for compassionate care in our community. Program Quality Improvement Report 2009-2010 6
Measures of Learning Outcomes 1. Direct Measures: a. Locally developed tests i. Within course tests ii. End of program comprehensive examination b. Literature reviews scored by rubrics Indirect Measures: a. Graduates exit interviews b. Employer/supervisor evaluation of practicum experience. 2. 7 Program Quality Improvement Report 2009-2010
Measures of Learning and Service Outcomes 3. Strategies for addressing shortfalls in student learning prior to graduation: a. A Student Data Base has been created: i. Demographic Information ii. Undergraduate performance iii. Educational/Career goals iv. Recruitment & Retention Information v. Advisement Information vi. Academic Tracking Information Program Quality Improvement Report 2009-2010 8
Measures of Learning and Service Outcomes facilitate identification of shortfalls, online learning modules will be developed in order to help student remediation. i. Synthesis & Analysis ii. Research Skills iii. Theories, methods & findings iv. Applications b. Using the Student Data Base to Program Quality Improvement Report 2009-2010 9
ACTION PLAN 09-10 PQIR Objective 1: Students will synthesize and analyze information critically 1. Create embedded questions in Research Methods related to synthesis and critical analysis. A. Develop validity and reliability for embedded questions. B. Action Taken: ABANDONED Create rubric to assess synthesis and critical analysis in Research Methods final/applied project. A. Develop validity and reliability for the rubric. B. Action Taken: MODIFICATION: Create a rubric to assess synthesis and critical analysis in within the first 18 credit hours and then again in the last 18 credit hours. 2. 10 10 Program Quality Improvement Report 2009-2010
ACTION PLAN 09-10 PQIR Objective 1: Students will synthesize and analyze information critically 3. Assess synthesis and critical analysis in first course post 21 credit hours. A. Action Taken: 2.B. 4. Continue to indirectly assess synthesis and critical analysis in the exit interview process. A. Action Taken: Exit interview data collection on synthesis and critical analysis continues. Program Quality Improvement Report 2009-2010 11
ACTION PLAN 09-10 PQIR Objective 1: Students will synthesize and analyze information critically Program Quality Improvement Report 2009-2010 12
ACTION PLAN 09-10 PQIR Objective 1: Students will synthesize and analyze information critically Program Quality Improvement Report 2009-2010 13
ACTION PLAN 09-10 PQIR Obj. 2: Students will demonstrate basic research skills in the behavioral sciences. 1. Operationally Define Research Skills. A. Action Taken: Operational Definition of Research Skills. 1. Students will demonstrate knowledge in research ethics. 2. Students will analyze and synthesize information in the existing professional literature. 3. Students will demonstrate the ability to form a logical and testable hypothesis. Program Quality Improvement Report 2009-2010 14
ACTION PLAN 09-10 PQIR Obj. 2: Students will demonstrate basic research skills in the behavioral sciences. 1. Operationally Define Research Skills. A. Action Taken: Operational Definition of Research Skills 4. Students will demonstrate the ability to develop research designs appropriate to test the hypothesis. 5. Students will demonstrate knowledge of statistics. a. Students will demonstrate the ability to select the appropriate statistics for data analysis. b. Students will demonstrate the ability to analyze data and interpret results. 6. Students will demonstrate the ability to present findings. Program Quality Improvement Report 2009-2010 15
ACTION PLAN 09-10 PQIR Obj. 2: Students will demonstrate basic research skills in the behavioral sciences. 2. Develop embedded questions in Research Methods, Ethics and Statistics in order to directly measure the students knowledge of research skills. Develop reliability and validity for embedded questions. 1. Action Taken: Embedded questions within three main courses were developed in Spring 2010 and data collection will begin Fall 2010. Action Taken: Question were disseminated to topic matter experts to determine Content Validity. Experts were asked to sort questions by subject content. Crombach Alpha coefficient were performed and the results indicate a r = .92 2. 1. Program Quality Improvement Report 2009-2010 16
ACTION PLAN 09-10 PQIR Obj. 2: Students will demonstrate basic research skills in the behavioral sciences. 3. Develop a rubric to evaluate the students ability to apply Research Skills in an end of course project. A. Develop reliability and validity for applied Research Skills rubric. 4. Action Taken: Initial development of the rubric began Spring 2010. Developmental work continues. A. Action Taken: Pending finalization of rubric. Program Quality Improvement Report 2009-2010 17
ACTION PLAN 09-10 PQIR Obj. 2: Students will demonstrate basic research skills in the behavioral sciences. 4. Assess research skills in the behavioral sciences in first course post 21 credit hours. A. Action Take: AMENDENED Assess research skills in the behavioral sciences within the first 18 credit hours. New programmatic changes have been submitted for Regent consideration. 5. Continue to indirectly assess synthesis and critical analysis in the exit interview process. A. Action Taken: Continues to be assessed. Program Quality Improvement Report 2009-2010 18
Learning Objective 1 Outcome Data Students will synthesis and analyze information critically. 1. 2. Rubric Measure - Direct Measure Exit Interview Questions pertaining to synthesis and analysis Indirect Measure. 19 19 Program Quality Improvement Report 2009-2010
Learning Objective 1: Students will synthesize and analyze information critically MEASUREMENTS OF STUDENT LEARNING Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements CURRICULUM AREA OR TARGET AUDIENCE PROGRAM OUTCOME Schedule for measurements Measurements Learning Objective 1 Career & Lifestyle Development Locally developed rubric for synthesis & critical analysis. Criterion Validity was determined by requesting topic matter experts perform a card sort exercise by categories and descriptions . r = .89 Interater Reliability was performed. r = .68 Every semester so every student will be measured. 20 20 Program Quality Improvement Report 2009-2010
Assignment With the exception of Graduate Statistics, every graduate course has literary assignment meant to test the students ability to synthesize and analyze information. These assignments all include a review of the professional literature with an emphases on the students ability to pull information from different sources to create a cohesive analysis of the information. Program Quality Improvement Report 2009-2010 21
Learning Obj. 1 Direct Measure Rubric Outcome CATEGORIES: 1. Introduction & Summary 2. Thesis 3. Organization 4. Supporting Information 5. Evidence & Examples 6. Paragraphs 7. Transitions 8. Conclusion 9. Mechanics Program Quality Improvement Report 2009-2010 22
Learning Obj. 1 Direct Measure Rubric Outcome 1 = Below Standard 2 = Approaching Standard 3 = Meets Standard 4 = Above Standard has not met programmatic expectations Spring 2010 Synthesis & Critical Analysis Rubric Mean & Standard Deviations n = 23 Intro & Summ Thesis Organ Supp Info Evid & Examp Parag Trans Concl Mech Total 1.87 2.17 2.26 3.57 2.43 1.83 1.74 1.87 1.61 2.15 0.81 0.83 0.86 0.51 0.66 0.83 0.69 0.76 0.58 0.24 Program Quality Improvement Report 2009-2010 23
Learning Obj. 1 Direct Measure Rubric Outcome 1. Trend Analysis: First semester to use this measure. No trend analysis. 2. Action Plan: A. Continue to collect Interrater Analysis data. a. Timeline: Spring 2011 Program Quality Improvement Report 2009-2010 24
Learning Obj. 1 Direct Measure Rubric Outcome 2. Action Plan: B. Intervention to improve performance: a. Determine the source of the problem. Is it a poor measure, poor instructor preparation, or poor student performance? b. Give students the rubric with syllabus. i. Timeline: Each semester c. Develop an online module to aid in understanding and usage of the dimensions of Synthesis & Critical Analysis. i. Timeline: Start development Spg 2011; Finished by Spg 2012 Program Quality Improvement Report 2009-2010 25
Learning Objective 1: Students will synthesize and analyze information critically MEASUREMENTS OF STUDENT LEARNING OR SERVICE OUTCOME Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements CURRICULUM AREA OR TARGET AUDIENCE PROGRAM OUTCOME Schedule for measurements Measurements Learning Objective 1 Graduating students Graduate Exit Interview (indirect) No validity measure available. No reliability measure available. Every semester so every student will be measured. 26 26 Program Quality Improvement Report 2009-2010
Learning Objective 1: Indirect Measure Exit Interview Results 2009-2010 1. Please rate how well the program prepared you to synthesize information on a scale of 1 (low) to 7 (high)? a. n = 38 b. Mean = 5.72 c. Standard Deviation = 0.92 2. Please rat how well the program prepared you to critically analyze information on a scale from 1 (low) to 7 (high)? a. N = 38 b. Mean = 5.78 c. Standard Deviation = 1.01 27 Program Quality Improvement Report 2009-2010
Learning Obj. 1: Indirect Measure Exit Interview Program Quality Improvement Report 2009-2010 28
Learning Obj. 1: Indirect Measure Exit Interview Action Plan: 1. Continue to collect data on the Exit Interview. 2. It is expected that the corrective plan for improving performance on the Rubric measure will improve on Exit Interview reporting. Program Quality Improvement Report 2009-2010 29
Learning Objective 2 Outcome Data Students will demonstrate basic research skills in the behavioral sciences. 1. Direct Measure: Research Proposal graded by Rubric. 2. Indirect Measure: Exit Interview Questions pertaining to attainment of research skills in the behavioral sciences. Program Quality Improvement Report 2009-2010 30
Learning Objective 2: Students will demonstrate research skills in the behavioral sciences. Rubric Outcome Direct Measure MEASUREMENTS OF STUDENT LEARNING Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements CURRICULUM AREA OR TARGET AUDIENCE PROGRAM OUTCOME Schedule for measurements Measurements Learning Objective Ethics & Professional Development Locally developed rubric for research skills will be used to determine how well a student has demonstrated research skills in a research proposal assignment. Criterion Validity was determined by requesting topic matter experts perform a card sort exercise by categories and descriptions . r = .92 None available Every semester so every student will be measured. Graduate Statistics Research Methods 31 31 Program Quality Improvement Report 2009-2010
Learning Obj. 2 Direct Measure Rubric Measure 1. Students will demonstrate knowledge in research ethics. 2. Students will analyze and synthesize information in the existing professional literature. 3. Students will demonstrate the ability to form a logical and testable hypothesis. Program Quality Improvement Report 2009-2010 32
Learning Obj. 2 Direct Measure Rubric Measure 4. Students will demonstrate the ability to develop research designs appropriate to test the hypothesis. 5. Students will demonstrate knowledge of statistics. a. Students will demonstrate the ability to select the appropriate statistics for data analysis. b. Students will demonstrate the ability to analyze data and interpret results. 6. Students will demonstrate the ability to present findings. Program Quality Improvement Report 2009-2010 33
Learning Obj. 2 Direct Measure Rubric Outcome 1 = Below Standard 3 = Approaching Standard 4 = Meets Standard 5 = Above Standard Does not meet standard Research Skills Means n = 10 Present Total Ethics Rev Liture Hypo Design Appro Stat Inter Result 3.70 4.24 3.15 3.62 4.40 4.24 4.00 3.91 Program Quality Improvement Report 2009-2010 34
Learning Obj. 2 Direct Measure Rubric Outcome 1. Trend Analysis: First semester to use this measure. No trend analysis. 2. Action Plan: A. Continue Development of Rubric: a. At this point the rubric is just a likert scale without descriptions. b. Timeline: Sprg 2011 c. Needs to create a statistics & ethics rubric. Needs to develop specific description of each standard. B. Collect Interrater Reliability data. a. Timeline: Each semester for graduating students. d. Program Quality Improvement Report 2009-2010 35
Learning Obj. 2 Direct Measure Rubric Outcome 2. Action Plan: C. Shortfalls in student performance. 1. Determine the source of the problem. a. Poor rubric? b. Poor rater training? c. Poor student performance? Program Quality Improvement Report 2009-2010 36
Learning Objective 2: Students will demonstrate research skills in the behavioral sciences. Exit Interview Indirect Measure MEASUREMENTS OF STUDENT LEARNING Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements CURRICULUM AREA OR TARGET AUDIENCE PROGRAM OUTCOME Schedule for measurements Measurements Learning Objective Graduating Seniors Locally developed questions on Exit Interview pertaining to program s ability to prepare students to do research. None available None available Every semester so every student will be measured. 37 37 Program Quality Improvement Report 2009-2010
Learning Obj. 2 Indirect Measure Exit Interview Outcome Rate how well the program promoted competence in research skills on a scale from 1 (low) to 7 (high) ? n = 38 Mean = 5.89 Standard Deviation = 1.02 Program Quality Improvement Report 2009-2010 38
Learning Obj. 2: Indirect Measure Exit Interview Outcome Program Quality Improvement Report 2009-2010 39
Learning Obj. 2: Indirect Measure Exit Interview Outcome Action Plan: 1. Continue to collect data on the Exit Interview. 2. It is expected that the corrective plan for improving performance on the Rubric measure will improve rating on Exit Interview reporting. Program Quality Improvement Report 2009-2010 40
Learning Objective 4 Students will apply basic knowledge within the field of the behavioral sciences. 1. Rubric Outcome Direct Measure 2. Exit Interview Question pertaining to the application of knowledge within the field of the behavioral sciences Indirect Measure Program Quality Improvement Report 2009-2010 41
Learning Objective 4: Students will apply basic knowledge within the field of the behavioral sciences. Rubric Outcome Direct Measure MEASUREMENTS OF STUDENT LEARNING Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements CURRICULUM AREA OR TARGET AUDIENCE PROGRAM OUTCOME Schedule for measurements Measurements Learning Objective 4 Practicum Rubric used by the Oklahoma State Department of Health Evaluation of Supervised Experience. None available None available. Every semester so every student will be measured. 42 42 Program Quality Improvement Report 2009-2010
Learning Obj. 4 Direct Measure Rubric Measure Areas rated: Individual counseling Group counseling Marital, Family, Couples counseling Child counseling Assessment/diagnosis Treatment planning Makes appropriate referrals Program Quality Improvement Report 2009-2010 43
Learning Obj. 4 Direct Measure Rubric Measure Continued: Consults with other professionals Conducts research Knows licensing law and rules Conforms to Rules of Professional Conduct Uses disclosure statement Is prompt/current on paperwork and records Cares for own mental health Program Quality Improvement Report 2009-2010 44
Learning Obj. 4 Direct Measure Rubric Measure Continued: Utilizes supervision sessions effectively Maintains professional boundaries Stays within limits of competence level Keeps current with professional literature Program Quality Improvement Report 2009-2010 45
Learning Obj. 4 Direct Measure Rubric Measure Levels of appraisal: 0 = No observation 1 = Needs Improvement 2 = Acceptable 3 = Above average skill Note: No observations were not used in calculation of proficiency of application Program Quality Improvement Report 2009-2010 46
Learning Obj. 4 Direct Measure Rubric Measure Application of knowledge within the field of the behavioral sciences. n = 7 Ind Grp M&F Child Dx Tx Ref Conslt Res Fall 2009 2.10 No Obs 2.00 2.33 2.20 2.00 2.14 2.30 2.50 Sprg 2010 2.50 No Obs 2.33 2.67 2.60 2.50 2.57 2.50 2.50 Program Quality Improvement Report 2009-2010 47
Learning Obj. 4 Direct Measure Rubric Measure Application of knowledge within the field of the behavioral sciences. n = 7 Law PC Discl PPW MH Sup Bound Comp PLit Fall 2009 2.20 3.00 3.00 2.70 2.20 1.70 3.00 2.50 1.50 Sprg 2010 2.30 3.00 3.00 2.70 2.20 2.10 3.00 2.00 2.00 Program Quality Improvement Report 2009-2010 48
Learning Obj. 4 Direct Measure Rubric Measure Action Plan: 1. Continue to collect data this upcoming year. 2. Attempt to collect state-wide data to use as a comparison group. 3. Make the rubric more detailed in order to help practicum supervisors further understand the standard. 4. Use supervisor rating to make curriculum changes. Program Quality Improvement Report 2009-2010 49
Learning Objective 4: Students will apply basic knowledge within the field of the behavioral sciences. Exit Interview Indirect Measure MEASUREMENTS OF STUDENT LEARNING Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements CURRICULUM AREA OR TARGET AUDIENCE PROGRAM OUTCOME Schedule for measurements Measurements Learning Objective Graduating Seniors Locally developed Exit Interview questions pertaining to the programs ability to prepare students in professional competency. None available None available Every semester so every student will be measured. 50 50 Program Quality Improvement Report 2009-2010
