Analysis of Overall Impact Scoring Trends

within AHRQ Peer Review Study Sections• Gabrielle Quiggle, MPH; Rebecca Trocki, MSHAI; • Kishena Wadhwani, PhD, MPH; Francis Chesley, MD• Office of Extramural Research, Education, and Priority

Populations, Agency for Healthcare Research and Quality

Background: Peer Review

The Agency for Healthcare Research and Quality (AHRQ) has a chartered health services research Initial Review Group (IRG) responsible for the peer review of grant applications submitted for funding opportunity. This IRG is comprised of five subcommittees or study sections:

o Healthcare Systems & Value Research (HSVR)o Healthcare Safety & Quality Improvement Research

(HSQR)o Healthcare Information Technology Research (HITR)o Health Care Effectiveness and Outcomes Research

(HEOR)o Health Care Research Training (HCRT)

Background: Peer Review

Research grant applications submitted to AHRQ are reviewed by one of five standing study section committees.

Applications are submitted in response to a Program Announcement (PA) or Request for Application (RFA).

General research grant mechanisms of interest:R01 – Research project grants – (Independent)R03 – Small research project grantsR18 – Research demonstration and dissemination

project grants

AHRQ uses a 9-point overall impact score system to evaluate the scientific/technical merit of research grant applications submitted to AHRQ for funding opportunities.

Score Descriptor Additional Guidance on Strengths/Weaknesses1 Exceptional Exceptionally strong with essentially no weaknesses2 Outstanding Extremely strong with negligible weaknesses3 Excellent Very strong with only some minor weaknesses4 Very Good Strong but with numerous minor weaknesses5 Good Strong but with at least one moderate weakness6 Satisfactory Some strengths but also some moderate weaknesses7 Fair Some strengths but with at least one major weakness8 Marginal A few strengths and a few major weaknesses9 Poor Very few strengths and numerous major weaknesses

High

Medium

Low

AHRQ Peer Review Scoring Descriptor Table

Non-numeric score options:DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND = Not Discussed

Background: Scoring

The final overall impact score reflects the average of the impact scores provided by the study section members as a whole (x10). Percentiles are calculated to rank applications relative to each other.

Background: Scoring

10 90High impact Low impact

Final Overall Impact Score

Preliminary Score

1 9

Exceptional Poor

1. To determine whether trends exist in the scoring of research grant applications submitted for funding by AHRQ

2. To assess potential differences in scoring trends between the five (5) AHRQ study sections

Study Objectives

Final impact scores were obtained from the following applications:

o First-time applicationso Received from October 2009 to June 2014 (15

review cycles)o Submitted to one of five AHRQ study sectionso Not withdrawn

Resubmitted applications and applications not discussed (ND) were excluded from sample

Data collection using NIH eRA Commons and NIH Query View & Report Database (QVR system)

Methods: Data Collection

• Means (SD) and medians (range) were calculated for each quarterly review meeting and fiscal year.

• Score trends were assessed by council meeting for each study section, using all application mechanisms.o Subgroup analysis was conducted on applications

considered under general research mechanisms R01, R03, and R18 in FY 2011-2014

• Percentile standardized scores were used to compare score trends between study sections.

Descriptive statistics and linear regression conducted using MS Excel and SAS 9.3.

Methods: Analysis

• AHRQ received 3,370 applications between Fiscal Year 2010 and 2014.

• 1,752 (52%) applications were discussed and received a final overall impact score.

• Slight trends towards lower (better) median scores were found in four out of five AHRQ study sections:

Results

Study section Trend line R2

HEOR -0.67x + 47 0.203

HSQR -0.98x + 50 0.233

HSVR +0.45x + 36 0.110

HITR -0.23x + 35 0.081

HCRT -0.41x + 37 0.107

Results

2010-10 2011-01 2011-05 2011-10 2012-01 2012-05 2012-10 2013-01 2013-05 2013-10 2014-01 2014-0510

20

30

40

50

60

70

80

90

f(x) = − 0.671328671328671 x + 47.1136363636364R² = 0.202824712659489

HEOR (FY 2011 - 2014)

AHRQ Review Cycle

Impact

Score

Scored = 42 ND = 39% ND = 48%

Scored = 45 ND = 13% ND = 22%

Scored = 27 ND = 26% ND = 49%

Scored = 32 ND = 12% ND = 27%

Scored = 30 ND = 24% ND = 44%

Scored = 21 ND = 15% ND = 42%

Scored = 21 ND = 11% ND = 34%

Scored = 20 ND = 5% ND = 20%

Scored = 22 ND = 19% ND = 46%

Scored = 30 ND = 32% ND = 52%

Scored = 13 ND = 8% ND = 38%

Scored = 25 ND = 15% ND = 38%

Results

2011-10 2012-01 2012-05 2012-10 2013-01 2013-05 2013-10 2014-01 2014-0510

20

30

40

50

60

70

f(x) = − 0.983333333333333 x + 49.5833333333333R² = 0.232531730126921

HSQR (FY 2011 - 2014)

AHRQ Review Cycle

Impact

Score

Scored = 31 ND = 25% ND = 45%

Scored = 35 ND = 34% ND = 49%

Scored = 25 ND = 27% ND = 52%

Scored = 21 ND = 22% ND = 51%

Scored = 15 ND = 7% ND = 32%

Scored = 28 ND = 13% ND = 32%

Scored = 27 ND = 29% ND = 52%

Scored = 22 ND = 19% ND = 46%

Scored = 23 ND = 27% ND = 54%

Results

2010-10 2011-01 2011-05 2011-10 2012-01 2012-05 2012-10 2013-01 2013-05 2013-10 2014-01 2014-0510

15

20

25

30

35

40

45

50

55

60

f(x) = 0.445804195804196 x + 35.6439393939394R² = 0.109632792667431

HSVR (FY 2011 - 2014)

AHRQ Review Cycle

Impact

Score

Scored = 35 ND = 36% ND = 51%

Scored = 30 ND = 27% ND = 47%

Scored = 27 ND = 36% ND = 57%

Scored = 32 ND = 27% ND = 46%

Scored = 27 ND = 16% ND = 37%

Scored = 10 ND = 20% ND = 67%

Scored = 14 ND = 11% ND = 44%

Scored = 9 ND = 9% ND = 50%

Scored = 11 ND = 9% ND = 45%

Scored = 26 ND = 18% ND = 41%

Scored = 14 ND = 9% ND = 39%

Scored = 12 ND = 9% ND = 43%

Results

2010-10 2011-01 2011-05 2011-10 2012-01 2012-05 2012-10 2013-01 2013-05 2013-10 2014-01 2014-0510

20

30

40

50

60

70

80

90

f(x) = − 0.234265734265734 x + 35.1060606060606R² = 0.0809757740110276

HITR (FY 2011 - 2014)

AHRQ Funding Cycle

Impact

Score

Scored = 48 ND = 50% ND = 51%

Scored = 35 ND = 29% ND = 45%

Scored = 44 ND = 28% ND = 39%

Scored = 32 ND = 27% ND = 46%

Scored = 29 ND = 31% ND = 52%

Scored = 28 ND = 17% ND = 38%

Scored = 28 ND = 25% ND = 47%

Scored = 17 ND = 14% ND = 45%

Scored = 21 ND = 21% ND = 50%

Scored = 44 ND = 41% ND = 48%

Scored = 26 ND = 24% ND = 48%

Scored = 32 ND = 24% ND = 43%

Results

2010-10 2011-01 2011-05 2011-10 2012-01 2012-05 2012-10 2013-01 2013-05 2013-10 2014-01 2014-0510

20

30

40

50

60

70

80

90

f(x) = − 0.409090909090909 x + 36.9924242424242R² = 0.107237422771403

HCRT (FY 2011 - 2014)

AHRQ Review Cycle

Impact

Score

Scored = 26 ND = 21% ND = 45%

Scored = 36 ND = 17% ND = 32%

Scored = 22 ND = 29% ND = 57%

Scored = 27 ND = 14% ND = 34%

Scored = 24 ND = 8% ND = 25%

Scored = 14 ND = 8% ND = 36%

Scored = 18 ND = 7% ND = 28%

Scored = 17 ND = 16% ND = 48%

Scored = 11 ND = 13% ND = 54%

Scored = 8 ND = 13% ND = 62%

Scored = 20 ND = 11% ND = 35%

Scored = 26 ND = 21% ND = 45%

• Subgroup analysis included 1,086 applications (57% discussed) considered under general research mechanisms R01, R03, and R18.

• Triaging of applications was highamong R03 (54.8%) and R18 (57.5%) applications across all study sections, compared to R01 applications (11.4%). 

Mean scores:R01 = 34.2 ± 13.3 to 42.8 ± 12.5R03 = 36.4 ± 15.2 to 41.9 ± 12.8R18 = 38.0 ± 13.6 to 43.1 ± 18.0

Results

159 167

37103

177 203

86

154

H Q E R H SV R H SQ R H T D S

SCORED VS. UNSCORED APPLICATIONS

UNSCORED SCORED

ResultsComparison of score distribution between study sections:

Percentile scores did not differ by study section, adjusting for FY, for R01 (F=0.74, p=0.53), R03 (F=0.31, p=0.82), and R18 (F=0.22, p=0.88).

The analysis of impact scores among study sections as a function of time revealed no statistically significant differences.

AHRQ study sections perform consistently over time, reflecting both the assessments of the reviewers and the quality of the applications.

These results show that careful selection of subject-matter experts, and consistency and uniformity in conducting the evaluation of research grant applications, are the best practices for peer review.

Conclusions

FIRST AUTHOR:• Gabrielle Quiggle, MPH

AHRQ/OEREP staff:• Francis Chesley, MD – Director of OEREP/AHRQ• Kishena Wadhwani, PhD, MPH – Director of Division

of Scientific Review (DSR)/OEREP• Rebecca Trocki, MSHAI – Program Analyst,

DSR/OEREP

Acknowledgements

AHRQ study section review committees: www.ahrq.gov/fund/peerrev/peerdesc.htm www.ahrq.gov/funding/process/study-section/peerdesc.html

AHRQ research announcements:www.ahrq.gov/funding/research/announcements/index.html

AHRQ scoring criteria:www.grants.nih.gov/grants/guide/notice-files/NOT-HS-10-002.html

Resources