Impact Scoring Trends in AHRQ Peer Review Study Sections
Explore the analysis of impact scoring trends within AHRQ peer review study sections conducted by experts from the Office of Extramural Research, Education, and Priority Populations. The study aims to identify patterns in research grant application scoring and assess differences between various study sections.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Analysis of Overall Impact Scoring Trends within AHRQ Peer Review Study Sections Gabrielle Quiggle, MPH; Rebecca Trocki, MSHAI; Kishena Wadhwani, PhD, MPH; Francis Chesley, MD Office of Extramural Research, Education, and Priority Populations, Agency for Healthcare Research and Quality
Background: Peer Review The Agency for Healthcare Research and Quality (AHRQ) has a chartered health services research Initial Review Group (IRG) responsible for the peer review of grant applications submitted for funding opportunity. This IRG is comprised of five subcommittees or study sections: o Healthcare Systems & Value Research (HSVR) o Healthcare Safety & Quality Improvement Research (HSQR) o Healthcare Information Technology Research (HITR) o Health Care Effectiveness and Outcomes Research (HEOR) o Health Care Research Training (HCRT)
Background: Peer Review Research grant applications submitted to AHRQ are reviewed by one of five standing study section committees. Applications are submitted in response to a Program Announcement (PA) or Request for Application (RFA). General research grant mechanisms of interest: R01 Research project grants (Independent) R03 Small research project grants R18 Research demonstration and dissemination project grants
Background: Scoring AHRQ uses a 9-point overall impact score system to evaluate the scientific/technical merit of research grant applications submitted to AHRQ for funding opportunities. AHRQ Peer Review Scoring Descriptor Table Additional Guidance on Strengths/Weaknesses Exceptionally strong with essentially no weaknesses Extremely strong with negligible weaknesses Very strong with only some minor weaknesses Strong but with numerous minor weaknesses Strong but with at least one moderate weakness Some strengths but also some moderate weaknesses Some strengths but with at least one major weakness A few strengths and a few major weaknesses Very few strengths and numerous major weaknesses Non-numeric score options: DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND = Not Discussed Score 1 2 3 4 5 6 7 8 9 Descriptor Exceptional Outstanding Excellent Very Good Good Satisfactory Fair Marginal Poor High Medium Low
Background: Scoring The final overall impact score reflects the average of the impact scores provided by the study section members as a whole (x10). Percentiles are calculated to rank applications relative to each other. Preliminary Score 9 9 Poor 1 1 Exceptional Final Overall Impact Score 10 10 90 90 Low impact High impact
Study Objectives 1. To determine whether trends exist in the scoring of research grant applications submitted for funding by AHRQ 2. To assess potential differences in scoring trends between the five (5) AHRQ study sections
Methods: Data Collection Final impact scores were obtained from the following applications: o First-time applications o Received from October 2009 to June 2014 (15 review cycles) o Submitted to one of five AHRQ study sections o Not withdrawn Resubmitted applications and applications not discussed (ND) were excluded from sample Data collection using NIH eRA Commons and NIH Query View & Report Database (QVR system)
Methods: Analysis Means (SD) and medians (range) were calculated for each quarterly review meeting and fiscal year. Score trends were assessed by council meeting for each study section, using all application mechanisms. o Subgroup analysis was conducted on applications considered under general research mechanisms R01, R03, and R18 in FY 2011-2014 Percentile standardized scores were used to compare score trends between study sections. Descriptive statistics and linear regression conducted using MS Excel and SAS 9.3.
Results AHRQ received 3,370 applications between Fiscal Year 2010 and 2014. 1,752 (52%) applications were discussed and received a final overall impact score. Slight trends towards lower (better) median scores were found in four out of five AHRQ study sections: Study section Study section HEOR HSQR HSVR HITR HCRT Trend line Trend line -0.67x + 47 -0.98x + 50 +0.45x + 36 -0.23x + 35 -0.41x + 37 R R2 2 0.203 0.233 0.110 0.081 0.107
Results HEOR (FY 2011 HEOR (FY 2011 - - 2014) 2014) 90 y = -0.6713x + 47.114 R = 0.2028 80 70 60 Impact Score Impact Score 50 40 30 20 10 2010-10 2011-01 2011-05 2011-10 2012-01 2012-05 AHRQ Review Cycle AHRQ Review Cycle Scored = 21 ND = 15 % ND = 42% 2012-10 2013-01 2013-05 2013-10 2014-01 2014-05 Scored = 20 ND = 5 % ND = 20% Scored = 42 ND = 39 % ND = 48% Scored = 45 ND = 13 % ND = 22% Scored = 27 ND = 26 % ND = 49% Scored = 32 ND = 12 % ND = 27% Scored = 30 ND = 24 % ND = 44% Scored = 21 ND = 11 % ND = 34% Scored = 22 ND = 19 % ND = 46% Scored = 30 ND = 32 % ND = 52% Scored = 13 ND = 8 % ND = 38% Scored = 25 ND = 15 % ND = 38%
Results HSQR (FY 2011 HSQR (FY 2011 - - 2014) 2014) 90 y = -0.9833x + 49.583 R = 0.2325 80 70 60 Impact Score Impact Score 50 40 30 20 10 2011-10 2012-01 2012-05 2012-10 2013-01 2013-05 2013-10 2014-01 2014-05 AHRQ Review Cycle AHRQ Review Cycle Scored = 15 ND = 7 % ND = 32% Scored = 31 ND = 25 % ND = 45% Scored = 35 ND = 34 % ND = 49% Scored = 25 ND = 27 % ND = 52% Scored = 21 ND = 22 % ND = 51% Scored = 28 ND = 13 % ND = 32% Scored = 27 ND = 29 % ND = 52% Scored = 22 ND = 19 % ND = 46% Scored = 23 ND = 27 % ND = 54%
Results HSVR (FY 2011 HSVR (FY 2011 - - 2014) 2014) 90 y = 0.4458x + 35.644 R = 0.1096 80 70 60 Impact Score Impact Score 50 40 30 20 10 2010-10 2011-01 2011-05 2011-10 2012-01 2012-05 AHRQ Review Cycle AHRQ Review Cycle Scored = 10 ND = 20 % ND = 67% 2012-10 2013-01 2013-05 2013-10 2014-01 2014-05 Scored = 9 ND = 9 % ND = 50% Scored = 35 ND = 36 % ND = 51% Scored = 30 ND = 27 % ND = 47% Scored = 27 ND = 36 % ND = 57% Scored = 32 ND = 27 % ND = 46% Scored = 27 ND = 16 % ND = 37% Scored = 14 ND = 11 % ND = 44% Scored = 11 ND = 9 % ND = 45% Scored = 26 ND = 18 % ND = 41% Scored = 14 ND = 9 % ND = 39% Scored = 12 ND = 9 % ND = 43%
Results HITR (FY 2011 HITR (FY 2011 - - 2014) 2014) 90 y = -0.2343x + 35.106 R = 0.081 80 70 60 Impact Score Impact Score 50 40 30 20 10 2010-10 2011-01 2011-05 2011-10 2012-01 2012-05 AHRQ Funding Cycle AHRQ Funding Cycle Scored = 28 ND = 17 % ND = 38% 2012-10 2013-01 2013-05 2013-10 2014-01 2014-05 Scored = 17 ND = 14 % ND = 45% Scored = 48 ND = 50 % ND = 51% Scored = 35 ND = 29 % ND = 45% Scored = 44 ND = 28 % ND = 39% Scored = 32 ND = 27 % ND = 46% Scored = 29 ND = 31 % ND = 52% Scored = 28 ND = 25 % ND = 47% Scored = 21 ND = 21 % ND = 50% Scored = 44 ND = 41 % ND = 48% Scored = 26 ND = 24 % ND = 48% Scored = 32 ND = 24 % ND = 43%
Results HCRT (FY 2011 HCRT (FY 2011 - - 2014) 2014) 90 y = -0.4091x + 36.992 R = 0.1072 80 70 60 Impact Score Impact Score 50 40 30 20 10 2010-10 2011-01 2011-05 2011-10 2012-01 2012-05 AHRQ Review Cycle AHRQ Review Cycle Scored = 14 ND = 8 % ND = 36% 2012-10 2013-01 2013-05 2013-10 2014-01 2014-05 Scored = 17 ND = 16 % ND = 48% Scored = 26 ND = 21 % ND = 45% Scored = 36 ND = 17 % ND = 32% Scored = 22 ND = 29 % ND = 57% Scored = 27 ND = 14 % ND = 34% Scored = 24 ND = 8 % ND = 25% Scored = 18 ND = 7 % ND = 28% Scored = 11 ND = 13 % ND = 54% Scored = 8 ND = 13 % ND = 62% Scored = 20 ND = 11 % ND = 35% Scored = 26 ND = 21 % ND = 45%
Results Subgroup analysis included 1,086 applications (57% discussed) considered under general research mechanisms R01, R03, and R18. Triaging of applications was high among R03 (54.8%) and R18 (57.5%) applications across all study sections, compared to R01 applications (11.4%). SCORED VS. UNSCORED APPLICATIONS UNSCORED SCORED 203 177 154 167 86 159 103 37 HQER HSVR HSQR HTDS Mean scores: R01 = 34.2 13.3 to 42.8 12.5 R03 = 36.4 15.2 to 41.9 12.8 R18 = 38.0 13.6 to 43.1 18.0
Results Comparison of score distribution between study sections: Percentile scores did not differ by study section, adjusting for FY, for R01 (F=0.74, p=0.53), R03 (F=0.31, p=0.82), and R18 (F=0.22, p=0.88).
Conclusions The analysis of impact scores among study sections as a function of time revealed no statistically significant differences. AHRQ study sections perform consistently over time, reflecting both the assessments of the reviewers and the quality of the applications. These results show that careful selection of subject-matter experts, and consistency and uniformity in conducting the evaluation of research grant applications, are the best practices for peer review.
Acknowledgements FIRST AUTHOR: Gabrielle Quiggle, MPH AHRQ/OEREP staff: Francis Chesley, MD Director of OEREP/AHRQ Kishena Wadhwani, PhD, MPH Director of Division of Scientific Review (DSR)/OEREP Rebecca Trocki, MSHAI Program Analyst, DSR/OEREP
Resources AHRQ study section review committees: www.ahrq.gov/fund/peerrev/peerdesc.htm www.ahrq.gov/funding/process/study-section/peerdesc.html AHRQ research announcements: www.ahrq.gov/funding/research/announcements/index.html AHRQ scoring criteria: www.grants.nih.gov/grants/guide/notice-files/NOT-HS-10- 002.html
