Connecting the Dots between Research Priorities and Stakeholder Needs:

An Evaluation of Stakeholder Perceptions of theNSRC Grant Program

Curt Grimm, Ph.D.Barbara Wauchope, Ph.D.

Charlie French, Ph.D.Carsey Institute, UNH

Durham, NH

NSRC Leaders MeetingHubbard Brook Foundation

North Woodstock, NHOctober 11, 2011

Purpose of the Evaluation To assess the work of the NSRC from

the perspective of multiple stakeholders

1. NSRC grantees – Principal Investigators on research grants

2. Practitioners and others who partner with University grantees

3. Other practitioners, policy-makers, and citizens who have interest in/use for the research 

Areas of Evaluation1. NSRC grantmaking process

2. Current research needs and priorities re: Northern Forest

3. Relevance and usefulness of the current NSRC research focus

4. Mechanisms grantees use to communicate research results

5. Communication/outreach strategies that might be more effective

 

Methodology:Data collection

Online Surveys:1. NSRC grantee Principal Investigators

(n=80)

2. Users or beneficiaries of NSRC-funded research – identified by the grantees in their survey (n=9)

3. Actual or potential users or beneficiaries of NSRC-funded research – subscribers to the Northern Forest Center Digest subscriber list (n=69)

Methodology:Resulting samples – PI Survey (n=80)

76.3%

5.0%

6.3%

2.5%5.0%

1.3% 1.3% 2.5% University/college

Research institute

State/Federal agency

Forest-serving org

Community non-profit

Business/industry

Non-affiliated

Other

Methodology:Resulting sample – Stakeholder Surveys combined (n=78)

5.2%

15.6%

13.0%

3.9%

28.6%

2.6%

15.6%

15.6%

Scientist/researcher

State/federal agency

Forest-serving org/inst

Forest-based bus/ind

Comm non-profit, econ dev org

Policy-maker

Citizen

Other

Methodology:Data collection

Telephone interviews:

Actual or potential users or beneficiaries of NSRC-funded research

– list generated by NSRC theme directors and Carsey staff (n=24)

Methodology:Resulting sample – Stakeholder interviews (n=24)

Policy-m

aker

Practitioner

NF resid

ent

Research

er

Advocacy

Philanthro

py

Education

0%

20%

40%

60%

80%

100%

Methodology:Data collection & resulting samples

Focus groups/informal interviews:

Northern Forest Higher Education Resource Network members (n=12) University/college presidents, deans,

faculty

Cooperative Extension county-based educators (n=4) Forest and wildlife staff

New Hampshire Local Government Center employees (n=2)

Key Findings Perceptions of NSRC are generally

positive Scientists and non-scientists have

somewhat different perspectives All agree on the need for both basic

and applied research • Scientists somewhat more supportive of

need for basic research

• Non-scientists overwhelmingly say science should address practical problems

Key Findings All four themes are relevant and

important but #1 is of strongest interest: the economy

Current dissemination strategy is not effective

E-mail-based and colleague-based outreach strategies, either your own or through existing networks, are likely to have more impact

NSRC GrantmakingPI experience with NSRC (n=80)

Number of awards: 0 – 6 Mean: 1.89 Median: 2

0%

20%

40%

60%

80%

100%

Completed = 42In-progress = 38

NSRC GrantmakingPIs: How they hear about Request For Proposal

Colle

ague(

s)

Gra

nt al

ert

Prof m

eeting

NSRC w

ebsite

Acad jo

urnal

, new

slet

ter

Oth

er0%

20%

40%

60%

80%

100%

NSRC GrantmakingPIs: Themes broad enough? (n=76)

Yes No Not sure0%

20%

40%

60%

80%

100%

NSRC GrantmakingPIs: Effectiveness of grantmaking process (n=76)

Publ

icizin

g RFP

Awar

d no

tifica

tion

NSR

C gr

ant a

dmin

RFP p

roce

ss

TA d

urin

g pr

opos

al

TA d

urin

g gr

ant

1.0

1.5

2.0

2.5

3.0

3.5

4.0

NSRC GrantmakingPIs: Effectiveness of mission (n=70-76)

1.0

1.5

2.0

2.5

3.0

3.5

4.0

NSRC GrantmakingPI Perceptions of the program Positive about program overall, including its

efficiency and effectiveness Promotes useful research relevant to the Northern

Forest region Promotes collaboration – between scientists,

across disciplines, with non-scientists Increases research on the Northern Forest Supports a broad scope of projects and disciplines Promotes interdisciplinary research

NSRC GrantmakingPI Perceptions of the program “Without this program there would be very little

research that directly deals with issues important to this region.”

“done well to fund a breadth of projects and expertise across disciplines and regions”

“ exceptionally well-positioned to allow researchers to leverage resources from other sources”

“great collaboration and synthesis within themes. Less opportunity to cross themes.”

NSRC GrantmakingPI Concerns

Grant focus Not relevant enough to Northern Forest residents/too

relevant – important scientific questions not funded Not enough social science

Limitations created by small size of grants On research topics imposed by themes On scope that’s possible and resulting impact On fostering grad students

Grantmaking process: Scientific review process not followed in some cases Poor communication to rejected applicants Bias in awards

NSRC GrantmakingPI Concerns

“projects in isolation fail to convey full impact – needs a vehicle for nesting and connecting to NSRC’s work in the big picture”

“funding pool is so small that it’s hard to develop a good collaborative, cross-disciplinary proposal”

“not all themes are equally "open" to competition”

“the themes divide up the pie and a proposal that crosscuts themes is disadvantaged”

“short proposal precludes effective review by panels who are quite mixed in discipline, depth, and breadth of knowledge”

NSRC GrantmakingStakeholder awareness of NSRC (n=76)

Yes No Not sure0%

20%

40%

60%

80%

100%

NSRC GrantmakingStakeholder understanding of NSRC (n=15)

What is the program focus: Research (14) Funding for research (11) Balance of applied/basic science research (3) Applied research (2) Basic science research (1)

Who NSRC serves: Broad range of stakeholders (8) Targeted audience (5)

NSRC GrantmakingStakeholder collaboration on grants (n=75)

Yes No Not sure0%

20%

40%

60%

80%

100%

Research Needs & Priorities: Perceived benefits of NSRC research

0%

20%

40%

60%

80%

100%

PIs (n=77)

Stakeholders (n=75)

Current NSRC Focus: Themes of present research reported by PIs

0%

20%

40%

60%

80%

100%

Research Needs & Priorities:Stakeholder interest in NSRC themes

1: Sustaining forest com-

munities

2: Sustaining ecosystem

health

3: Forest productivity

4: Biodiversity0%

20%

40%

60%

80%

100%

Current NSRC Focus:PI reported themes vs. Stakeholder interest

4

Theme 1

Theme 2

Theme 3

Theme 4

0%

20%

40%

60%

80%

100%

Stakeholders

Them

e 1

Them

e 2

Them

e 3

Them

e 4

0%

20%

40%

60%

80%

100%PIs

Research Needs & Priorities :Stakeholders: Topics of interest to be addressed:

North Fo

rest eco

nomy

North Fo

rest co

mmunities

Land use

change, c

onflcts

Outdoor recre

ation

Land co

nserva

tion

Forest

management

Forest

products in

dustry

Ecosys

tem servi

ces

Forest

science

Other0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Other PI-Perceived Research Needs and Priorities: Understanding local community need and

demand for biological data (and how to get them what they want)

Effects of forest habitat supply on featured wildlife species

Forest hydrology Forest – stream ecosystem connections

Other Stakeholder Perceived Research Needs and Priorities: Effects/impacts of renewable energy

development Climate change impacts on economy,

community, ecosystems…; and vice-versa Tourism’s potential to turn around the

current economic downturn Local food and agricultural systems impact

on economy and ecology Economic and ecological impact of

community forests

Stakeholders Emphasize Healthy Economy and Communities “What’s true in the rainforest is true in the

Northern Forest – we have to deal with the poverty”

“If we don’t have a healthy community, we won’t have a healthy forest. A healthy forest is a working forest”

“really integrative, holistic collaborative engaged work is a far better way to go. I would really like to see that value really written into the RFP and the grant proposal evaluation process”

Relevance and Usefulness of NSRC Foci: PIs: Relevance of NSRC Research by Audience

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Relevance and Usefulness of NSRC Foci: PIs: Do you use results of other NSRC research

Yes No Not sure0%

20%

40%

60%

80%

100%

Relevance and Usefulness of NSRC Foci: Stakeholder perceptions

“absolutely critical…it’s given us the basis of our economic development program”

“helpful in that it confirms other research that has been conducted “

“the one that I collaborated with was really valuable to my department, and my staff”

“great information for scientists and researchers” “able to expand my own work through channels

of NSRC project PIs”

Relevance and Usefulness of NSRC Foci Stakeholder perceptions (continued) “haven’t found them to be very relevant; too

theoretical and abstract … most of the work is ecologically focused and not relevant for communities and specifically for their economic growth”

“I can’t think of anything that has had a direct impact”

“I don’t feel really connected to the research to be honest … but I can see the potential”

“there doesn't seem to be an emphasis on application of the research in the real world”

“some of it, truthfully, just goes over my head”

Current Outreach & Dissemination:

PIs: Most important products of their research

Peer review jo

urnal

Meeting pre

sentation

Seminar, worksh

op

Collaborative partn

er

Technica

l assi

stance

Inform

ational websit

e

Best practi

ces m

anual

Network cr

eation

Policy docu

ment

Newspaper a

rticle

Trade publication

Testimony

Other0%

20%

40%

60%

80%

100%

Current Outreach & Dissemination:

PIs: Which resources are they using

Conf

pre

sent

atio

ns

NSR

C web

site

Peer

revi

ew jo

urna

l

Sem

inar

, wor

ksho

p, m

tg

Info

rmat

iona

l Web

site

New

spap

er A

rticle

Trad

e pu

blicat

ion

Best p

ract

ices

man

ual

Polic

y do

cum

ent

0%

20%

40%

60%

80%

100%

Current Outreach & Dissemination:

Stakeholders: Where they go for information

0%

20%

40%

60%

80%

100%

Current Outreach & Dissemination:

PIs: Where do they go to find NSRC research

Colleagu

e

Academic

journal

NSRC w

ebsite

Prof m

tg/co

nference

Prof s

eminar/work

shop

Newspaper a

rticle

Best pra

ct/mgm

t manual

Trade public

ationOth

er0%

20%

40%

60%

80%

100%

Current Outreach & Dissemination:

PIs: How effective are resources for communication

Peer review jo

urnal

Seminar, worksh

op, mtg

Conference

prese

ntations

Best practi

ces m

anual

Newspaper A

rticle

Policy docu

ment

Inform

ational Websit

e

Trade publication

NSRC websit

e1.0

1.5

2.0

2.5

3.0

3.5

4.0

Current Outreach & Dissemination:

Stakeholders: Average likelihood of using source

1.0

1.5

2.0

2.5

3.0

3.5

4.0

I rely on my colleagues to stay up to date with the latest research findings. That is where I learned about NSRC.

- Focus Group Participant, Forestry Practitioner

Current Outreach & Dissemination:

PIs: Barriers to dissemination of their research

0%

20%

40%

60%

80%

100%

Current Outreach & Dissemination:

Stakeholders: Barriers to obtaining research

Info doesn't g

et to m

e

Not enough

time

Info difficu

lt to acce

ss

Topics not o

f interest

Other

0%20%40%60%80%

100%

Current Outreach & Dissemination:

Stakeholders: Other barriers to obtaining research

Information is not always summarized well into one or a few sources (it's all over the place)

Restricted travel to meetings Cost to obtain peer reviewed articles “Lack of easy access to information about

the grants and the project findings is a challenge, as is lack of time to sort through lots of information”

Current Outreach & Dissemination:

Stakeholders: Effectiveness of NSRC dissemination Some potential audiences are not aware of

NSRC ‘Forestry practitioners’ and forest-serving

institutions typically hear about NSRC research findings through word-of-mouth.

There is no mechanism to drive people to NSRC’s website.

All 14 of those interviewed who knew of NSRC said dissemination was ineffective

Alternative Dissemination Strategies: PI Suggestions Online access to publications and perhaps

even data collected for each project Issue a NSRC press release / press

conference after each project year, with a bit about each project

More face-to-face contact with research users/stakeholders (but that takes time and resources)

develop an outreach document summarizing project results and management implications

Alternative Dissemination Strategies: PI Suggestions (continued)

Work harder with local stakeholder groups such as environmental managers

More informal outlets (workshops for professionals, articles in general interest magazines or newspapers)

More informal public presentations and seek co-sponsorship with aligned local organizations

A more interactive and content-rich website

Alternative Dissemination StrategiesStakeholder suggestions

NSRC should actively reach out to practitioners and other stakeholders.

Use Northern Forest Center’s News Digest to disseminate key findings and links

E-mail announcements with links to websites

Better use of forest-serving organizations’ websites to share research and grant info

Alternative Dissemination StrategiesStakeholder suggestions (continued)

Ensure that info being shared is up-to-date. Keep summary of findings short and simple

and link to a website for more information. Just getting an email saying, “hey, we

updated the project list and here is what is going on across the region” would be helpful… don’t even need to send the project abstracts… just knowing that they have been updated is helpful.”