USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station The Science/Policy Interface in Logic based Evaluation of Forest Ecosystem

USDA Forest Service PNW Research StationUSDA Forest Service PNW Research Station

The Science/Policy Interface in Logic‑based Evaluation of Forest Ecosystem Sustainability

The Science/Policy Interface in Logic‑based Evaluation of Forest Ecosystem Sustainability

Keith M. Reynolds, USDA Forest ServiceKeith M. Reynolds, USDA Forest Service

K. Norman Johnson, Oregon State UniversityK. Norman Johnson, Oregon State University

Sean N. Gordon, Oregon State UniversitySean N. Gordon, Oregon State University


AcknowledgmentsAcknowledgments

USDA Forest Service Washington OfficeUSDA Forest Service Washington Office National Forest System, Ecosystem ManagementNational Forest System, Ecosystem Management

Pacific Northwest Research StationPacific Northwest Research Station Social and Economic Values RD&A ProgramSocial and Economic Values RD&A Program


OverviewOverview

IntroductionIntroduction Knowledge bases and logic modelingKnowledge bases and logic modeling AnalysisAnalysis Model design issuesModel design issues Lessons learnedLessons learned RecommendationsRecommendations


Introduction: indicatorsIntroduction: indicators

An indicator is any variable or component of the An indicator is any variable or component of the forest ecosystem … used to infer attributes of the forest ecosystem … used to infer attributes of the sustainability of the resource and its utilization. sustainability of the resource and its utilization. Indicators should convey a ‘single meaningful Indicators should convey a ‘single meaningful message.’ This ‘single message’ is termed message.’ This ‘single message’ is termed information. It represents an aggregate of one or information. It represents an aggregate of one or more data elements with certain established more data elements with certain established relationships (Prabhu relationships (Prabhu et alet al. 2001).. 2001).


Introduction: criterionIntroduction: criterion

A standard that a thing is judged by (Prabhu A standard that a thing is judged by (Prabhu et alet al. 2001). 2001). . Criteria are the intermediate points to which the information Criteria are the intermediate points to which the information

provided by the indicators can be integrated and where an provided by the indicators can be integrated and where an interpretable assessment crystallizes. interpretable assessment crystallizes.

Principles [e.g., sustainability]Principles [e.g., sustainability] form the final point of form the final point of integration. integration.

A criterion should be treated as a reflection of knowledge.A criterion should be treated as a reflection of knowledge. It can be viewed as a large‑scale selective combination … of It can be viewed as a large‑scale selective combination … of

related pieces of information.related pieces of information.


Introduction: measurement endpointIntroduction: measurement endpoint

Some indicators are simple.Some indicators are simple. Their definition suggests an obvious one‑to‑one Their definition suggests an obvious one‑to‑one

correspondence between an indicator and a metric for correspondence between an indicator and a metric for that indicator. that indicator.

Definitions of some indicators are more complex.Definitions of some indicators are more complex. They represent a synthesis of two or more data They represent a synthesis of two or more data

elements, which we refer to as measurement endpoints. elements, which we refer to as measurement endpoints.


Introduction: scales of applicationIntroduction: scales of application

Purpose varies with scale (Purpose varies with scale (Castañeda 2001)Castañeda 2001) National and regionalNational and regional

Policy instruments to evaluate laws, policy, regulationsPolicy instruments to evaluate laws, policy, regulations E.g., Montreal Process, NWFP, ICBEMPE.g., Montreal Process, NWFP, ICBEMP

Management unitManagement unit Evaluation and adjustment of management practicesEvaluation and adjustment of management practices E.g., CIFOR, USDA FS IMI E.g., CIFOR, USDA FS IMI


Introduction: objectivesIntroduction: objectives

1.1. Illustrate the value of a logic-based approach in Illustrate the value of a logic-based approach in designing a formal specification to evaluate the designing a formal specification to evaluate the Montreal criteria and indicators. Montreal criteria and indicators.

2.2. Identify the roles of science and policy in this Identify the roles of science and policy in this effort. effort.

3.3. Highlight lessons learned from this process.Highlight lessons learned from this process.

4.4. Suggest some general recommendations.Suggest some general recommendations.


Knowledge basesKnowledge bases

A form of meta databaseA form of meta database A formal logical representation of how to evaluate A formal logical representation of how to evaluate

informationinformation Networks of interrelated topicsNetworks of interrelated topics Mental mapMental map

AdvantagesAdvantages Interactive, graphic design (modularity)Interactive, graphic design (modularity) Numerous & diverse topics can be analyzed within a Numerous & diverse topics can be analyzed within a

single integrated analysissingle integrated analysis


Knowledge bases: forms of uncertaintyKnowledge bases: forms of uncertainty

ProbabilisticProbabilistic Uncertainty of eventsUncertainty of events

Linguistic (or lexical)Linguistic (or lexical) Lofti Zadeh, 1966Lofti Zadeh, 1966 Uncertainty about the definition of the eventUncertainty about the definition of the event A proposition is the smallest unit of thought to which A proposition is the smallest unit of thought to which

one can assign a measure of truthone can assign a measure of truth Truth value metrics Truth value metrics

Indices that quantify the degree of support for a Indices that quantify the degree of support for a proposition provided by its premisesproposition provided by its premises

Knowledge bases: networks of topicsKnowledge bases: networks of topics

Concern 1

Ecostate A Ecostate B

Ecostate C Ecostate D Data link Data link

Data link Data linkData link

= topicConcern 2

Etc.

Data


Knowledge bases: topicsKnowledge bases: topics

Each typically evaluates a propositionEach typically evaluates a proposition Attributes of topicsAttributes of topics

NameName PropositionProposition Truth value: a measure of support for the propositionTruth value: a measure of support for the proposition DocumentationDocumentation

Explanation, source, citationsExplanation, source, citations

Knowledge bases: evaluationKnowledge bases: evaluation

Concern 1

Ecostate A

Ecostate B Ecostate C

Data link Data linkData link

Get

dat

a re

quir

emen

ts

Eva

luat

e da

ta


Knowledge bases: fuzzy logicKnowledge bases: fuzzy logic


Analysis: Montreal C&IAnalysis: Montreal C&I

The Montreal specifications provide relatively The Montreal specifications provide relatively clear definitions of biophysical, socioeconomic, clear definitions of biophysical, socioeconomic, and framework attributes requiring evaluation and framework attributes requiring evaluation (WGCICSMTBF 1995) ... (WGCICSMTBF 1995) ...

But, design of evaluation procedures that allow But, design of evaluation procedures that allow interpretation of the Montreal C&I is one of the interpretation of the Montreal C&I is one of the major technical issues that remain to be resolved major technical issues that remain to be resolved (Raison et al. 2001). (Raison et al. 2001).


Analysis: conceptual frameworkAnalysis: conceptual framework

1.1. Specified conditions or outcomes to be sustained (the Specified conditions or outcomes to be sustained (the indicators).indicators).

2.2. A measure for each condition or outcome.A measure for each condition or outcome.3.3. Calculation of the level of the indicator over some time Calculation of the level of the indicator over some time

period using the selected measure.period using the selected measure.4.4. A frame of reference for gauging sustainability. A frame of reference for gauging sustainability. 5.5. Rules for deciding when sustainability has been achieved Rules for deciding when sustainability has been achieved

(sustainability check).(sustainability check).6.6. A monitoring program. A monitoring program. 7.7. A formalism that supports requirements 1 to 6.A formalism that supports requirements 1 to 6.


Analysis: logic models as design frameworksAnalysis: logic models as design frameworks

Logic models (knowledge bases) provide a formal Logic models (knowledge bases) provide a formal specification for organizing and interpreting specification for organizing and interpreting informationinformation..

NetWeaver kb developer systemNetWeaver kb developer system Problem represented in terms of propositions about Problem represented in terms of propositions about

topics of interest and their interdependencies.topics of interest and their interdependencies. Topics translated into propositions.Topics translated into propositions. Fuzzy logic to accommodate lexical uncertainty.Fuzzy logic to accommodate lexical uncertainty.


Analysis: logic models as design frameworks (continued)Analysis: logic models as design frameworks (continued)

Need for transparency (Prabhu et al. 2001)Need for transparency (Prabhu et al. 2001) Models embody important policy decisions.Models embody important policy decisions. Models depend on value judgments and critical Models depend on value judgments and critical

assumptions that need clear documentation.assumptions that need clear documentation. Model developmentModel development

Graphic representation is an effective basis for Graphic representation is an effective basis for organizing discussion and for evolution of design.organizing discussion and for evolution of design.

CommunicationCommunication Between scientists and policy makers.Between scientists and policy makers. With interested publics.With interested publics.

Forestsustainability

AND

Biophysical criteria

Socioeconomiccriterion

Frameworkcriterion

Biodiversity

Productioncapacity

Ecosystemhealth

Conservation

Carbon cycle

AND

AND

Production

Recreation

Investment

Culture

Employment

AND

Institutional

Economic

Monitoringcapacity

Legal

R&Dcapacity

1

2

3

4

5

6

7

Design issues: model organizationDesign issues: model organization

Basic organization Basic organization of topics.of topics.

For example, For example, evaluation of evaluation of criteria in the criteria in the current prototype.current prototype.

Forestsustainability

ANDSocioeconomic

criterion

Frameworkcriterion

Biodiversity

Productioncapacity

Ecosystemhealth

Conservation

Carbon cycle

AND

Production

Recreation

Investment

Culture

Employment

AND

Institutional

Economic

Monitoringcapacity

Legal

R&Dcapacity

1

2

3

4

5

6

7

Design issues: model organizationDesign issues: model organization

An alternative An alternative organization with organization with very different very different emphasis on emphasis on criteria.criteria.

Design issues: synthesisDesign issues: synthesis

ADD operator: arguments evaluated as limiting ADD operator: arguments evaluated as limiting factors.factors.

SUM operator: arguments contribute SUM operator: arguments contribute incrementally to evaluation and can compensate.incrementally to evaluation and can compensate.

Design issues: synthesisDesign issues: synthesis

Another example, including the OR operator.Another example, including the OR operator.


Design issues: weightingDesign issues: weighting

Intrinsic weightsIntrinsic weights Each topic in a NetWeaver logic model has an intrinsic Each topic in a NetWeaver logic model has an intrinsic

weight attribute.weight attribute. E.g., set weight attribute on any topic to adjust its E.g., set weight attribute on any topic to adjust its

contribution of evidence to a proposition.contribution of evidence to a proposition. Bad idea: part of specification, but not obvious.Bad idea: part of specification, but not obvious.

Explicit weightsExplicit weights Better, but add another layer of subjectivity.Better, but add another layer of subjectivity. Some valid purposes, however.Some valid purposes, however.

Design issues: reference conditionsDesign issues: reference conditions

Each fuzzy node Each fuzzy node evaluates a evaluates a measurement measurement endpoint against endpoint against reference reference conditions.conditions.

Lack of reference Lack of reference conditions is a basic conditions is a basic problem for most problem for most measurement measurement endpoints.endpoints.

Design issues: reference conditionsDesign issues: reference conditions

Implementation of a fuzzy node to evaluate Implementation of a fuzzy node to evaluate measurement endpoint against reference conditions.measurement endpoint against reference conditions.

Design issues: qualitative measuresDesign issues: qualitative measures

Outcomes evaluated on an ordinal scale.Outcomes evaluated on an ordinal scale.


Design issues: reliability of dataDesign issues: reliability of data

Reliability of data for evaluation of Montreal C&I. Reliability of data for evaluation of Montreal C&I. Stochastic, rather than lexical, uncertaintyStochastic, rather than lexical, uncertainty

Formal representation of stochastic uncertainty is Formal representation of stochastic uncertainty is problematic in the context of a logic model. problematic in the context of a logic model. Not addressed in the current Montreal C&I prototype.Not addressed in the current Montreal C&I prototype.

Possible solutionPossible solution Adjusting topic weights with a normalized metric such Adjusting topic weights with a normalized metric such

as standard error of the mean.as standard error of the mean. Problems: availability, unknown error correlationsProblems: availability, unknown error correlations

Sequential OR Sequential OR (SOR) to (SOR) to specify specify multiple multiple alternative alternative pathways in pathways in order of order of preference.preference.

Design issues: precision of knowledgeDesign issues: precision of knowledge


Lessons learnedLessons learned

1.1. Lexical uncertainty is an important issue in evaluation of Lexical uncertainty is an important issue in evaluation of Montreal criteria and indicatorsMontreal criteria and indicators..

2.2. Many aspects of evaluating sustainability cannot be Many aspects of evaluating sustainability cannot be answered by science aloneanswered by science alone..

3.3. Acquiring data on sustainability is necessary but not Acquiring data on sustainability is necessary but not sufficient for setting policysufficient for setting policy..

4.4. Evaluating the state of sustainability and deciding how to Evaluating the state of sustainability and deciding how to respond are separate but interdependent decision respond are separate but interdependent decision processes.processes.

5.5. Evaluating sustainability is not the same as defining Evaluating sustainability is not the same as defining desired future conditions.desired future conditions.


RecommendationsRecommendations

Assess the policy role in sustainability evaluation, Assess the policy role in sustainability evaluation, and undertake a policy review of model and undertake a policy review of model organization and strategies for integrating organization and strategies for integrating sustainability information.sustainability information.

The clearest, and most critical, role of science is in The clearest, and most critical, role of science is in development of reference conditions.development of reference conditions.

A major effort is needed to identify measurement A major effort is needed to identify measurement endpoints for indicators of the institutional endpoints for indicators of the institutional framework (criterion 7).framework (criterion 7).


AuthorsAuthors

Keith M. ReynoldsKeith M. Reynolds USDA Forest Service, Pacific Northwest Research Stn.USDA Forest Service, Pacific Northwest Research Stn. [email protected]@fs.fed.us

K. Norman JohnsonK. Norman Johnson Oregon State University, College of ForestryOregon State University, College of Forestry [email protected]@orst.edu

Sean N. GordonSean N. Gordon Oregon State University, College of ForestryOregon State University, College of Forestry [email protected]@orst.edu

Documents

USDA Forest Service PNW Research Station USDA Forest Service PNW Research Station The Science/Policy Interface in Logic based Evaluation of Forest Ecosystem