Innovative Training Model for Research Transformation

Transforming Practice through an Innovative Training Model: Transforming Practice through an Innovative Training Model: MLA Research Training Institute MLA Research Training Institute Jodi L. Philbrick, MSLS, PhD, AHIP Senior Lecturer, Department of Information Science, University of North Texas Lorie Kloda, MLIS, PhD, AHIP Associate University Librarian, Planning & Community Relations, Concordia University Susan Lessick, MA, MLS, AHIP, FMLA Project Director, MLA Research Training Institute Librarian Emerita, University of California, Irvine Emily Vardell, MLS, PhD, AHIP Assistant Professor, School of Library and Information Management, Emporia State University

Goals of MLA Research Training Institute Goals of MLA Research Training Institute Increase research competencies Increase research quality, quantity, and dissemination Build research capacity to contribute to health and library improvements

Features of the Research Training Institute Features of the Research Training Institute (2018, 2019, & 2020 institutes) (2018, 2019, & 2020 institutes) One-year institute that includes: 1. Immersive training workshop 2. Mentoring and monitoring 3. Online community of practice 4. Capstone presentation at MLA Annual meeting

RTI Cohorts & Assessments RTI Cohorts & Assessments Cohort 1 Cohort 2 2018 2019 Cohort 3 2019 2020 20 fellows Cohort 4 2020 2021 20 fellows 2021 - 2022 20 fellows 32 fellows (includes 6 LIS grad students) Pre- and post-test assessment data Research progress RTI impact Pre- and post- test assessment data

Research Questions Research Questions 1. Was the RTI program effective in improving fellows perceived research confidence? 2. Was the RTI effective in improving fellows research output? 3. What have been the self-reported impacts of the RTI on fellows and their institutions?

RTI Fellows Prior Research Experience & Research Education Activities Prior research experience since obtaining LIS master s degree Cohort 1 (N=19) Cohort 2 (N=20) Cohort 3 (N=20) Have conducted research since master s degree 12 11 5 Prior research education activities of participants Cohort 1 (N=20) Cohort 2 (N=20) Cohort 3 (N=20) Continuing education programs 13 13 9 Formal master s degree and information science course 11 9 13 Staff development programs provided by your institution 6 1 3 Formal degree non-LIS course 4 6 6 Self-education activities 2 12 11 None 2 1 1 Formal doctoral degree LIS course 1 0 1

Fellows Reasons for Participating in the RTI (highest ranked reasons) REASONS (N=20) GROUP % IN AGREEMENT C1 100% Will help me contribute to research and scholarship C2 100% C3 100% Will advance the profession C1 95% C2 100% C3 95% Will provide opportunity to partner with and understand the needs of researchers C1 95% C2 95% C3 90%

Fellows Reasons for Participating in the RTI (Cont) REASONS (N=20) GROU P % IN AGREEMENT Will increase likelihood I will conduct program evaluations and assessments C1 100% C2 95% C3 70%% Will increase likelihood I will engage in evidence-based decision making C1 95% C2 95% C3 75% Will help demonstrate the value of my library to my administration and users C1 85% C2 100% C3 80% Will support my tenure and/or promotion efforts C1 75% C2 50% C3 70%

Determining Confidence Levels of Participants Determining Confidence Levels of Participants A pre- and post- assessment instrument based on Librarian Research Confidence Scale (LRCS-10) (Brancolini & Kennedy, 2017) Pre-assessment survey deployment: Cohort 1: May 3-31, 2018 Cohort 2: May 15-31, 2019 Cohort 3: June 22-26, 2020 Post-assessment survey deployment: Cohort 1: August 14-31, 2018 Cohort 2: August 22-29, 2019 Cohort 3: September 3- 11, 2020

Analyzing Confidence Levels of Participants Analyzing Confidence Levels of Participants Used the Wilcoxon Signed Ranks Test to determine if there was statistically significant difference (p-value 0.05) in the self-reported research confidence of the Cohort 1, Cohort 2, & Cohort 3 fellows before and after the RTI workshop Used the Friedman Test to determine if there was a statistically significant difference (p-value 0.05) in the self- reported research confidence of the Cohort 1 & Cohort 2 fellows before, after, and one year after the RTI workshop

Research Confidence Levels Before and After Workshop Research Confidence Levels Before and After Workshop Wilcoxon Signed Ranks Test Results Wilcoxon Signed Ranks Test Results

Fellows Research Confidence Levels Before and After Workshop (1) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 COHORT 2 COHORT 3 Median Rating (Pre) Median Rating (Post) Z score & p- value Median Rating (Pre) Median Rating (Post) Z score & p- value Median Rating (Pre) Median Rating (Post) Z score & p- value Questions about skills needed for a research project 1. Turning my topic into a question. 3 4 -3.087 .002 3 4 -3.491 .000 3 4 -2.358 .018 2. Designing a project to answer my question. 2.5 4 -3.630 .000 3 4 -3.815 .000 2 4 -3.619 .000 3. Selecting methods and procedures for my question. 2 3 -3.352 .001 2 4 -3.971 .000 2 3.5 -3.720 .001 4. Developing plan and timeline for my study. 2 4 -3.534 .000 3 4 -1.973 .049 2 4 -3.769 .001 5. Identifying appropriate information sources in which to conduct my literature search. 4 5 -2.221 .026 2.5 5 -3.769 .000 4 4 -1.098 .272

Fellows Research Confidence Levels Before and After Workshop (2) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 COHORT 2 COHORT 3 Median Rating (Pre) Median Rating (Post) Z score & p- value Median Rating (Pre) Median Rating (Post) Z score & p- value Median Rating (Pre) Median Rating (Post) Z score & p- value Questions about skills needed for a research project 6. Using relevant keywords and search strategies to discover literature about the research topic. 4 5 -2.804 .005 4 5 -2.299 .022 4 4.5 -2.310 .021 7. Assessing and synthesizing literature that is relevant to your research question. 3 4 -2.984 .003 4 4 -0.758 .448 3 4 -3.087 .002 8. Using a theoretical framework to inform the research design of your study. 1 3 -3.022 .003 1.5 3 -2.702 .007 2 3 -3.487 .002 9. Identifying sources of research funding and funding agency requirements. 2 3 -3.570 .000 2 3 -3.384 .001 10. Choosing an appropriate data gathering procedure. 2 3.5 -4.011 .000 2 4 -3.787 .000 2 4 -3.901 .000

Fellows Research Confidence Levels Before and After Workshop (3) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 COHORT 2 COHORT 3 Median Rating (Pre) Median Rating (Post) Z score & p-value Median Rating (Pre) Median Rating (Post) Z score & p-value Median Rating (Pre) Median Rating (Post) Z score & p-value Questions about skills needed for a research project 11. Determining which members of a population to include in your study. 2 4 -3.672 .000 2 4 -3.676 .000 2.5 4 -3.758 .000 12. Knowing how to design a focus group. 2 3 -3.804 .000 2 3.5 -3.903 .000 1.5 3 -3.209 .001 13. Knowing how to run a focus group. 2 3 -3.682 .000 2 3 -3.677 .000 1 3 -3.410 .001 14. Knowing how to design an interview. 2 4 -3.685 .000 2 4 -3.794 .000 2 4 -3.630 .000 15. Knowing how to conduct an interview. 2 4 -3.499 .000 2 4 -3.903 .000 2 3 -3.517 .000

Fellows Research Confidence Levels Before and After Workshop (4) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 COHORT 2 COHORT 3 Median Rating (Pre) Median Rating (Post) Z score & p-value Median Rating (Pre) Median Rating (Post) Z score & p-value Median Rating (Pre) Median Rating (Post) Z score & p- value Questions about skills needed for a research project 16. Knowing how to design a survey. 2 4 -3.839 .000 2 4 -3.938 .000 17. Knowing how to administer a survey. 2.5 4 -3.250 .001 2.5 4 -3.703 .000 2 4 -3.926 .000 18. Knowing institutional processes and standards to ensure that your study is conducted ethically. 3 4 -3.274 .001 3 4 -3.469 .001 3 4 -2.961 .000 19. Knowing what method of data analysis you would use for your study. 1 3 -3.668 .000 1.5 4 -3.872 .000 1 3 -3.835 .000 20. Knowing what type of assistance you might need to undertake data analysis (e.g., data/statistics consulting, transcription, software). 2 4 -3.809 .000 1 4 -3.864 .000 2 3 -3.602 .000

Fellows Research Confidence Levels Before and After Workshop (5) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 COHORT 2 COHORT 3 Median Rating (Pre) Median Rating (Post) Z score & p- value Median Rating (Pre) Median Rating (Post) Z score & p- value Median Rating (Pre) Median Rating (Post) Z score & p- value Questions about skills needed for a research project 21. Knowing how to manage the data you have gathered. 2 3.5 -3.668 .000 4 4 -.924 .356 2 3 -3.575 .000 22. Knowing how to code qualitative data to identify themes and sub-themes. 1 3 -3.660 .000 2 4 -3.560 .000 1 3 -3.469 .001 23. Reporting results in written format. 2 3 -3.486 .000 3 4 -3.787 .000 3 4 -3.473 .001 24. Reporting results verbally. 2 3 -3.463 .001 2.5 4.5 -3.677 .000 3 4 -2.795 .005 25. Identifying appropriate places to disseminate results. 3 4 -3.640 .000 3 4 -3.405 .001 3 4 -3.094 .002 26. Tracking the dissemination and impact of your research. 3 4 -3.458 .001 2.5 4 -3.072 .002 3 3.5 -1.458 .145

Research Confidence Levels Research Confidence Levels Before, After, & One Year After Workshop Before, After, & One Year After Workshop Friedman Test Results Friedman Test Results

Cohort 1 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (1) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 1. Turning my topic into a question. 3 4 4 14.392 .001 2. Designing a project to answer my question. 2.5 4 4 28.836 .000 3. Selecting methods and procedures for my question. 2 3 4 20.985 .000 4. Developing plan and timeline for my study. 2 4 4 18.000 .000 5. Identifying appropriate information sources in which to conduct my literature search. 4 5 5 7.190 .027

Cohort 1 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (2) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 6. Using relevant keywords and search strategies to discover literature about the research topic. 4 5 5 12.884 .002 7. Assessing and synthesizing literature that is relevant to your research question. 3 4 5 14.982 .001 8. Using a theoretical framework to inform the research design of your study. 1 3 3 25.581 .000 9. Identifying sources of research funding and funding agency requirements. 2 3 3 19.500 .000 10. Choosing an appropriate data gathering procedure. 2 3.5 4 32.794 .000

Cohort 1 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (3) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 11. Determining which members of a population to include in your study. 2 4 4 22.164 .000 12. Knowing how to design a focus group. 2 3 4 30.207 .000 13. Knowing how to run a focus group. 2 3 4 30.145 .000 14. Knowing how to design an interview. 2 4 4 28.444 .000 15. Knowing how to conduct an interview. 2 4 4 25.240 .000

Cohort 1 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (4) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 16. Knowing how to design a survey. 2 4 4 27.263 .000 17. Knowing how to administer a survey. 2.5 4 4 19.478 .000 18. Knowing institutional processes and standards to ensure that your study is conducted ethically. 3 4 5 20.491 .000 19. Knowing what method of data analysis you would use for your study. 1 3 3 26.517 .000 20. Knowing what type of assistance you might need to undertake data analysis (e.g., data/statistics consulting, transcription, software). 2 4 3 25.200 .000

Cohort 1 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (5) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 1 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 21. Knowing how to manage the data you have gathered. 2 3.5 3 22.800 .000 22. Knowing how to code qualitative data to identify themes and sub-themes. 1 3 3 26.821 .000 23. Reporting results in written format. 2 3 4 20.481 .000 24. Reporting results verbally. 2 3 4 21.893 .000 25. Identifying appropriate places to disseminate results. 3 4 4 27.345 .000 26. Tracking the dissemination and impact of your research. 3 4 4 19.538 .000

Cohort 2 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (1) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 2 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 1. Turning my topic into a question. 3 4 4 15.125 .001 2. Designing a project to answer my question. 3 4 4 26.793 .000 3. Selecting methods and procedures for my question. 2 4 4 28.557 .000 4. Developing plan and timeline for my study. 3 4 4 24.295 .000 5. Identifying appropriate information sources in which to conduct my literature search. 2.5 5 5 14.649 .001

Cohort 2 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (2) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 2 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 6. Using relevant keywords and search strategies to discover literature about the research topic. 4 5 5 11.302 .004 7. Assessing and synthesizing literature that is relevant to your research question. 4 4 4.5 12.667 .002 8. Using a theoretical framework to inform the research design of your study. 1.5 3 3 14.880 .001 9. Identifying sources of research funding and funding agency requirements. 10. Choosing an appropriate data gathering procedure. 2 4 4 30.145 .000

Cohort 2 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (3) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 2 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 11. Determining which members of a population to include in your study. 2 4 4 20.698 .000 12. Knowing how to design a focus group. 2 3.5 4 23.639 .000 13. Knowing how to run a focus group. 2 3 4 20.109 .000 14. Knowing how to design an interview. 2 4 4 26.655 .000 15. Knowing how to conduct an interview. 2 4 4 27.898 .000

Cohort 2 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (4) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 2 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 16. Knowing how to design a survey. 17. Knowing how to administer a survey. 2.5 4 4 26.308 .000 18. Knowing institutional processes and standards to ensure that your study is conducted ethically. 3 4 5 22.291 .000 19. Knowing what method of data analysis you would use for your study. 1.5 4 3 27.594 .000 20. Knowing what type of assistance you might need to undertake data analysis (e.g., data/statistics consulting, transcription, software). 1 4 4 25.733 .000

Cohort 2 Fellows Research Confidence Levels Before, Immediately After, and One Year After Workshop (5) Rated with Likert scale: 5: Very Confident; 4 Confident; 3 Moderately Confident; 2 Slightly Confident; 1 Not At All Confident COHORT 2 Median Rating (Pre) Median Rating (Post) Median Rating (One Yr Later) Chi-square value p-value Questions about skills needed for a research project 21. Knowing how to manage the data you have gathered. 4 4 4 5.434 .066 22. Knowing how to code qualitative data to identify themes and sub-themes. 2 4 4 26.333 .000 23. Reporting results in written format. 3 4 4.5 23.705 .000 24. Reporting results verbally. 2.5 4.5 4.5 23.186 .000 25. Identifying appropriate places to disseminate results. 3 4 4.5 17.273 .000 26. Tracking the dissemination and impact of your research. 2.5 4 4 19.283 .000

Research Progress of Cohort 1 Research Progress of Cohort 1 (as of May 2021, 3-year post-program) 6 7 3 2 Published Write-Up Data Analysis Data Collection IRB Approval (if applicable) Research Design/Methods Literature Review 18/20 RTI 18 fellows presented posters on their research projects at MLA 19 and 20 Annual Meetings. Research Questions Cohort 1 research projects generated 6 publications so far! * 2 Fellows Withdrew

Research Progress of Cohort 2 Research Progress of Cohort 2 (as of May 2021, 2-year post program) 4 5 3 1 Published 1* Write-Up Data Analysis 1* Data Collection IRB Approval (if applicable) Research Design/Methods Literature Review 13/20 RTI 19 fellows presented posters on their research projects at MLA 20 and 21 Annual Meetings. Research Questions Cohort 2 research projects generated 4 publications so far! *Had delays/withdrawals due to COVID-19 pandemic, job changes, and/or other issues. *5 Fellows Withdrew

Impact of RTI on Cohort 1 & Cohort 2 & their Institutions Type of Impact C1 Frequency C2 Frequency Formed internal and external research collaborations 13 12 Shared RTI experience with colleagues through informal and formal venues 12 3 Provided leadership to strengthen research capacity 9 2 Strengthened relationships with individuals outside of the library 8 6 Increased visibility of the library and its services 7 3 Improved or initiated new library services 6 3 Gained a better understanding of the users served 5 4 Received recognition for research 5 1 Decided to pursue more research education 4 1 Participated in other research activities 4 7 Increased confidence in conducting research 3 7 Developed research support materials 2 2 Impacted way research is conducted at institution 1 2 Gained institution s interest in study findings 1 5 Increased the research culture at institution 1 n/a Guided and educated users about the research process n/a 8 Received research funding/support n/a 3 Communicated about research with others n/a 2

Conclusions: Conclusions: Was RTI program effective in improving fellows research confidence? RTI training has significantly increased the research confidence of fellows in all three cohorts. RTI training has significantly increased the research confidence of fellows in Cohort 1 and Cohort 2 from prior to the training workshop to the two subsequent points in time (after workshop and one year later). RTI research project experience has improved the research confidence of most fellows in Cohort 1 and Cohort 2.

Conclusions: ( Conclusions: (Cont Cont) ) Was the RTI effective in improving fellows research output? COVID-19 pandemic impacted the research progress of fellows in Cohort 2. 70% (28/40) of Cohort 1 and Cohort 2 fellows completed research projects. 78% (31/40) of Cohort 1 and Cohort 2 fellows presented research posters at MLA annual conferences Research projects of fellows in Cohort 1 and Cohort 2 generated 11 research publications so far. What has been the self-reported impacts of the RTI on fellows and their institutions? The most frequently reported institutional impact of RTI training in Cohort 1 and Cohort 2 is forming internal and external research partnerships.

Reference Reference Brancolini, K.R., & Kennedy, M.R. (2017). The development and use of a research self-efficacy scale to assess the effectiveness of a research training program for academic librarians. Library and Information Research, 41(24), 44-84. https://doi.org/10.29173/lirg760

Comments/Questions? Comments/Questions? For additional information about RTI: RTI web site MLANET, under Professional Development link at top of page http://www.mlanet.org/p/cm/ld/fid=1333 More RTI assessment findings https://www.mlanet.org/p/cm/ld/fid=1634 Contact Us: Jodi Philbrick (Jodi.Philbrick@unt.edu) Lorie Kloda (lorie.kloda@concordia.ca) Susan Lessick (slessick@uci.edu) Emily Vardell (emily.vardell@gmail.com) This project was made possible in part by the Institute of Museum and Library Services (RE- 95-17-0025-17).