1

Collaborative Systemwide Monitoring and Evaluation Project (CSMEP)

Presentation to PNAMP Steering Committee

August 21, 2007

2

Presentation Outline

1. CSMEP Overview

2. Examples of CSMEP Work Products2.1 Status and Trend M&E2.2 Habitat M&E2.3 Hydro M&E2.4 Hatchery M&E2.5 Harvest M&E2.6 Integration of M&E

3. Summary & Future Plans

3

CSMEP is an M&E Coordination Forum for Fish and Wildlife Agencies and Tribes

A forum for Federal, State Agencies and Tribes to collaboratively develop a regional M&E program, consistent with:

– 2000 FCRPS BiOp– Fish and Wildlife Program– Subbasin planning– Recovery Planning

Vision: improve the quality, consistency, and focus

of fish population and habitat data

to answer key M&E questions

relevant to major decisions in the Columbia Basin

4

Participants and Approach

CBFWAESSA Technologies Ltd.

State AgenciesIDFGODFWWDFW

Federal AgenciesNOAAUSFWSEPADFO

Tribal AgenciesCRITFCNez Perce TribeColville TribesYakama NationUmatilla Tribes

ConsultantsEco Logical ResearchQuantitative ConsultantsPERWEST

5

2.1 Status and Trends M&E

Purpose: Assess viability of the population and overall management scheme.

4H Impacts Actions Monitoring

Harvest √

Hydro √

Habitat √

Hatchery √

Status and Trends will tell you what the population is doing but not why.

6

CSMEP Pilot for Snake Basin; learn from this and extend to other regions

7

Status and Trends: a Management Example

Priority Question: Are Snake River spring/summer Chinook salmon viable

using IC-TRT guidelines?

Related Decision: Has there been sufficient improvement in the population

status of Snake River spring/summer Chinook salmon to justify delisting and allow removal of ESA restrictions?

8

How likely is the correct decision with different levels of monitoring intensity?

Data Needs:• Abundance• Productivity• Spatial structure• Diversity

We used a model to test the ability of monitoring programs to correctly assess spring/summer Chinook salmon population viability in the Snake River ESU using a simulated abundance dataset.

Technical Recovery Team

viability criteria

Viability Status:• Not Viable• Maintained• Viable• Highly Viable

9

10

11

12

13

Status quo

55% correct

A single run of the simulation

Medium

87% correct

High

94% correct

14

Objectives by Alternatives Matrix:Status & Trends Designs

Status and Trends Subgroup

.94.87.45.55Pr (making the correct viability assessment for Snake ESU)

Statistical Reliability

2,1007101751,170annual cost of design alternatives (x $1,000)

Cost ($)

GoodFairPoorPoor

ability to make viability assessments for each population in the ESU

Inferential ability (Qualitative)

HighMed Low Status Quo

Design examplesPerformance Measures(Abundance and Spatial

Structure)

Design Objectives

.94.87.45.55Pr (making the correct viability assessment for Snake ESU)

Statistical Reliability

2,1007101751,170annual cost of design alternatives (x $1,000)

Cost ($)

GoodFairPoorPoor

ability to make viability assessments for each population in the ESU

Inferential ability (Qualitative)

HighMed Low Status Quo

Design examplesPerformance Measures(Abundance and Spatial

Structure)

Design Objectives

15

Results and Conclusions for Status & Trends M&E

• TRT rule set is conservative, so high uncertainty results in conservative viability assessments.

• Weakness of status quo monitoring design for Snake Pilot:– lack of spatial structure information.– Lack of an abundance estimate in the non-index areas for

populations without weirs. – One MPG lacks a weir.

• The medium design cost less than the status quo, yet performs better (for this question).

• Measurement error may be less than year to year variability, causing small difference between medium and high designs’ reliability

16

2.2 Habitat Effectiveness MonitoringLemhi River example

The task:Apply CSMEP’s adapted DQO process to a habitat effectiveness monitoring example

BC

A

A - Lower Mainstem LemhiB - Upper Mainstem LemhiC - Hayden Creek

RST

RST

The Lemhi Habitat Conservation Plan (HCP)

Lemhi Basin: heavily impacted with agriculture diversions and consequent passage issues

• 10-17 tributary reconnection projects planned under Lemhi HCP

• Phased in over the next 35 years

17

From Questions to Designs

• To what degree have these actions affected fish populations in the Lemhi Basin?

• Developed a general “Question Clarification Process” to make M&E designs rigorous

• Monitored effects: salmon distribution, survival, productivity

• What are the mechanistic connections between recovery

actions, key habitat attributes and fish responses?

18

Information collected with Designs

Design Performance Measures

Status Quo OK for current status and trends data, very limited for habitat effectiveness

Low Fish abundance, juvenile survival, some fish distribution and habitat

Medium Low design + lots of habitat and fish distribution information

High Medium design + juvenile/adult movement, more precise adult returns

19

Performance Measures ConsideredData Performance Measure Low Mid High

Redd Counts Adult Abundance

Juvenile Counts @ traps Juvenile Emigrant Abundance

Parr & Smolt Tag detections Parr-Smolt Survival

Snorkel counts (targeted) Juvenile Distribution

Habitat survey (presence) Effect of Actions on Habitat

Snorkel counts (extensive) Parr Density & Distribution

Habitat survey Covariates for fish population response

Adult detection (& tagging) @ Weirs

Adult Returns & Distribution

PIT detections at treatment/control sites

Fish distribution

Juvenile movement

Carcass surveys Prespawn mortality & Adult distribution

20

Design Top-Down Bottom-Up

Status Quo

$125,000 $125,000

Low $323,000 $354,000

Mid $377,000 $493,400

High $580,000 $643,600

Top-Down = based on per project costs and contracting history

Bottom-up = based on cost per unit time per person multiplied by the sample sizes identified in the plans.

Costs of CSMEP Designs

21

Habitat Effectiveness M&EResults & Conclusions

• Identifying mechanistic effects requires more intense M&E

• Effectiveness monitoring may fulfill requirements of status and trends monitoring

• A one-size-fits-all template for habitat effectiveness monitoring is not workable

• Aspects of the design template for the Lemhi River habitat effectiveness monitoring project may be transferable to other systems.

22

2.3 Hydro Action Effectiveness

• Hydro decisions occur at multiple scales:– Operations at individual projects (spill, bypass, RSW)– Overall operations (within season transport, performance

measure compliance)– Longer term hydro decisions (flow, transportation, system

configuration)– Adequacy of hydrosystem operations for population

recovery

23

CSMEP Hydro M&E Analyses of Survival Rates(SARs, TIRs, Passage survival)

Spatial Scales ExaminedPopulation (Tributary)

Major Population

Group

Snake River

Aggregate

Time Scales Examined

5-10 year Average

X X X

Annual Estimate

X XSeasonal Estimate

X

25

What is our ability to estimate in-river survival from LGR to BON using different M&E designs?

Design 1998 1999 2000 2001 2002 2003 2004 2005 2006

L

SQ, M, H

4 * SQ

Survival exceeds standard Survival below standard

Uncertain, confidence intervals straddle standard

? ? ? ? ? ? ? ?

? ?

? X

?

X X

X

X

X X

X X

X

26

Objectives by Alternatives Matrix:Hydro Example – In-River Survival

7/92/91/92/9For wild fish only

GoodPoorPoorPoor…For wild fish alone

7/97/91/97/9For hatchery + wild fish

Fraction of years in which compliance can be clearly assigned (Y/N)Statisticalreliability

11.09.07.88.5Annual cost of design alternatives (x $1,000,000)

Cost ($)

GoodGoodPoorGoodAbility to assess compliance with 2000 FCRPS BiOp in-river survival stds… For hatchery + wild chinook

Inferential ability (Qualitative)

HighMed Low Status Quo

Design AlternativesPerformance MeasuresDesign Objectives

7/92/91/92/9For wild fish only

GoodPoorPoorPoor…For wild fish alone

7/97/91/97/9For hatchery + wild fish

Fraction of years in which compliance can be clearly assigned (Y/N)Statisticalreliability

11.09.07.88.5Annual cost of design alternatives (x $1,000,000)

Cost ($)

GoodGoodPoorGoodAbility to assess compliance with 2000 FCRPS BiOp in-river survival stds… For hatchery + wild chinook

Inferential ability (Qualitative)

HighMed Low Status Quo

Design AlternativesPerformance MeasuresDesign Objectives

27

Hydro M&E Conclusions

• Optimal design & confidence level in answers depends on: – spatial and temporal scales– decision criteria, and – true value of parameter being estimated relative to target.

• Increasing number of tags/year: – can improve precision of annual estimates; but– doesn’t help multiple-year averages (# years more important)

• Ability to integrate tags from multiple sources, for multiple questions:– depends on how questions are framed, but– great potential for using PIT-tags, and multi-year averages for

many scales & questions

28

Uncertainty regarding the effects of hatchery fish limit management evaluations of :

1. Impacts and benefits of hatcheries.2. De-listing – how do hatchery fish “count” in productivity

estimates?3. Habitat Action Effectiveness – do hatchery fish reduce

Relative Reproductive Success (RRS) and thus mask productivity improvements?

4. Effectiveness of hatcheries for target populations.

2.4 Hatchery Effectiveness

29

Hatchery Evaluations

1. Existing hatchery M&E in the basin is primarily focused investigations at the project scale.

2. Assessing more generic hatchery impacts are likely to require larger spatial scale designs.

3. Designs were developed to investigate the effects of hatchery Straying and Relative Reproductive Success.

30

What is our ability to assess hatchery straying?

Status Quo• Stray rates/ratios currently calculated using CWT recoveries

in non-random locations.

CSMEP Alternative• Representative distribution of tag recovery effort, increasing

sampling intensity & improved reporting.• No need to sample every population, allows statistical

extrapolation

33

Objectives by Alternatives Matrix:Straying

Performance Measures Poor = P, Fair = F, Good = G, Status

QuoLow Med High

Ability to representatively estimate stray ratios and origin of strays in sampled populations.

N/A or P

F G G

Ability to predict stray ratios for unsampled populations.

N/A F G G

Annual/total cost of design alternatives (x $1000)

Statistical Reliability (N)

Maintain coefficient of variation < 0.2

P P G G

Design Objectives Design Alternatives

756/ 6,045

2,503/ 20,025

Inferential ability (Qualitative)

Cost ($)1 432/ 3,456

34

Conclusions

• Problems with Status quo– Current straying estimates and RRS studies are not

representative, are likely biased, and cannot be used in a predictive fashion for un-sampled programs or populations.

• CSEMP Designs:– Do not address all hatchery uncertainties.– Focus on PRIMARY uncertainties that limit management

decisions.– Useful for aggregate and individual benefit/risk

evaluations.

35

2.5 Harvest Effectiveness M&E

Key Management Decision:

• Open or close fisheries based on allowable Impacts on listed stocks?

Priority Questions:

• What are the in-season, post season estimates of run size & escapement for each management group (target & non-target)?

• How do they compare to preseason estimates?

• What is the target and non-target harvest and when is it projected to reach allowable levels?

.

36

Weakness of Current Harvest M&E

• Fisheries are managed to keep listed-stock mortality (“Impacts”) less than permitted rates, but

• Status quo harvest monitoring generally does not provide precision estimates.

• Improved monitoring needed for more accurate and precise information for models (or run reconstructions) used in population viability, status, and trend assessment

37

Assess Precision and Bias of Impact Estimates

Consequences of poor (biased or imprecise) estimates

• Underestimate Impact – over harvest of listed stocks

• Overestimate Impact – lost opportunities for user groups

• Poor impact estimates reduce the utility of harvest mortality estimates in run reconstructions, and in assessments of status, trends, and viability

38

Identify Areas of Uncertainty and Bias

Columbia River Fisheries• Mainstem Sport• LCR Commercial• Zone 6 Tribal• Tributary

Key Fisheries Metrics• Run-Size Estimates • Stock Composition• Harvest Number• Release Rates • Post-Release Mortality

Rates

39

Status Quo

Visual stock ID + Dam Counts + CWT based run reconstruction

Low Status quo + PIT-tag sampling of kept catch under current tagging programs

Medium Representative PIT-tagging of wild populations

CWT-indicator stocks for ESU-level resolution.

Genetic Stock Identification for ESU-level resolution.

High CWT-indicator stocks for MPG-level resolution

GSI for MPG-level stock composition.

Monitoring Design Alternatives

40

Harvest M&E Conclusions

1. Run-Size Estimates – improved at modest cost using available data and methods

2. Stock Composition – could be improved with PIT tags, GSI sampling, or index stock

3. Harvest Number – best in commercial fisheries; good in tribal and sport fisheries

4. Release Rates – good in commercial fisheries; potential biases in sport fisheries

5. Post-Release Mortality Rates – are based on limited studies; difficult to estimate

Harvest Subgroup

41

2.6 Integration of M&EWhat are the features of an ‘integrated’ monitoring program?

• Scale integration: data can be used at the scale of interest for decisions (e.g. ESUs for viability analysis, population level for local management).

• Integrates across separate monitoring programs: information gathered serves multiple functions (e.g. same PIT-tagged fish used for multiple evaluations); coordinated costing.

• Integrates policy and technical domains: precision of data fits time frames and acceptable risks for decisions.

• Integrates across life history: evaluate survival and habitat requirements throughout the life cycle.

• Species integration: collect data for multiple species in an efficient manner.

42

Multiple species

43

Status and Trends

Abundance, Productivity, Distribution

Harvest Hatchery

Straying and Relative Reproductive Success studies

Stock compositionAge structure

Hydrosystem

Survival studies

Habitat

Abundance, productivity,spatial structure, spatial diversity changes from habitat actions

PIT Tags

PIT T

ags PIT Tags

PIT

Tag

sP

IT T

ags

Integration example using PIT-tags

47

The Future of Collaborative Monitoring

• Collaborative monitoring will become increasingly important as recovery plans are implemented

• Coordination among federal, state and tribal agencies remains critical for developing logical cost-effective M&E (especially for fish populations that cross state and tribal boundaries)

• CSMEP provides a forum for federal, state and tribal fish managers to develop cost-effective, and coordinated regional monitoring for status and trends and the 4-“H”s.

51

Questions?