Practical Ways for Evaluating Programs to Collect

Accountability Information

by

Dr. John G. RichardsonAgricultural Programs Accountability Manager

College of Agriculture and Life SciencesDepartment of Agricultural and Extension Education

Box 7607 North Carolina State UniversityRaleigh, North Carolina 27695

USAEmail: john_richardson@ncsu.edu

Phone: (919) 515-6080Fax: (919) 513-3935

Program Priorities Focus on Performance

“You will see even more so in this budget than in the past, a focus on those [domestic programs] that are indeed federal priorities

and delivering results”

 

 Those programs that the administration thinks are performing poorly “are more likely to be reduced in this budget,” (Bolton

statements in an interview, 2-3-05)

Joshua Bolten, White House Budget Director

Reported by Alan Fram, The Associated Press, in an article entitled “Domestic Programs

Squeezed”, The News & Observer, Friday, February 4, 2005. p. 4A.

Impacts

It is Critical That We Understand Impacts

Impact = Learning + Behavior Change + Results

Learning – is the extent to which participants change attitudes, improve knowledge, and/or increase skills by participating in an educational program.

Behavior Change – is the application of what was learned in the educational program. It is the adoption of practice based on knowledge gained and skills developed.

Results – are evidence of participation, attitude change, knowledge gained, skill developed, and/or changes in behavior. What happened as a result of the change in behavior.

Impact may be expressed as a change in economic value or efficiency, environmental quality, societal or individual well-being.

Program Measures

Level 7 – End Results [IMPACT]Level 6 – Practice ChangeLevel 5 – KASA (Knowledge,

Attitude, Skill, Aspiration)Level 4 – ReactionsLevel 3 – People InvolvementLevel 2 – ActivitiesLevel 1 – Inputs

Accountability, Relevance, and Evaluation

Evaluating for Program

Relevance and Accountability

Observation

This means of gathering

accountability information is the

deliberate act of looking at some

object, event, group, individual

actions or behavior to obtain

program assessment information.

Interview

The direct contact of an individual

with another for the purpose of

obtaining information.

Testimonial

This program assessment tool is the

act of an individual or group

expressing, through verbal or

written means, their direct reaction

to, or experiences resulting from,

Extension’s educational program

efforts.

Anecdotal Information

Information that may be difficult to

verify or quantify, but can provide

insights into program outcomes and

successes.

Existing Records

(records/data analysis) –

This program assessment method is

the use of information or records

that already exist or can be obtained

with minimal planning.

Content Analysis

A way of assessing effects of change

through analysis of materials, levels of

nutrients, moisture levels, or other

factors through actual measurement of

components.

Tests

Often, pre-tests and post-tests are

used as program assessment tools.

Questionnaire

A survey instrument is used to

obtain specific reactions or

input from all members of an

audience or a sampling of the

targeted audience.

Unobtrusive Measures

A means of evaluating that can provide

specific information to the evaluator

without any obtrusive presence needed

in order to obtain information.

Determining Program Economic Benefit Values

Numbers Count!“Whenever some quantification is done

—no matter how speculative or limited

—the number tends to get into the public domain and the qualifications tend to get forgotten…The number is the thing.”

(U.S. Congress Report by the Subcommittee on Oversight and Investigation, 1980, 7, 10)

Key Valuing Tips

Must be believable Use common sense Be reasonable and realistic Use credible sources Seek multiple sources of values information (Web, Experts) Ask clientele Use observation skills Anecdotal evidence may be sufficient Use comparative values if needed Historical values or commonly accepted values may be used Extrapolation may be used May use estimates of values lost Shadow pricing may be used (reduced crime rates, education) Think beyond the box Can not readily value everything Beware of unintended messages Some outcomes may take longer than others Jgr 10-04

Determining Extension Program Economic Benefit Value

Value

Added

Savings

Reduced Costs

IncreasedIncome

Increased

Productivity

ExpectedValue

Willingness to Pay

IndirectValues

Non- marketBenefits

How WeAre

Better Off

Multiplier Effect

Alternative Opportunity

Costs of Capital

Valuing

Extension

Programs

URL: http://www.ces.ncsu.edu/AboutCES/Factsheets/EconomicBenefitValues.htm

Reduced costs

This means for estimating program value has been used extensively.

Estimate gains (dollars saved) resulting from participants changing their behaviors/practices as a result of the organization’s educational programs. 

An example: the value of soil sampling education can be the dollars saved when application rates are based on analysis recommendations rather than indiscriminately applying higher rates.

Increased income

Extension programs can: help participants gain new skills that enhance their job prospects, lead participants to adopt innovative practices that increase yields, or encourage other decisions that cause participants to increase their

income.

One can estimate income gained as a result of adopting new or alternative enterprises that enable the more efficient use of available resources

The program's value can be the difference in participants' income levels before and after the program.

Increased Savings

There are a number of Extension programs that focus on savings enhancement

The increased savings realized by program participants can be considered the estimated value of the program

Increased Productivity

This valuation method is commonly used in non- agricultural businesses as well as in agriculture. 

Productivity is the ratio of output per unit of input

Therefore, practices, processes, or innovations that either increases output or reduces the units of inputs would increase productivity.

Examples:

Farming more acres with the same number of employees Larger or more efficient equipment often increases the productivity of a

farm operation Increased crop yield per acre

The difference between the value gained resulting from adopting a new practice, and the costs required to implement the new practice, would be a good estimate of the value of educational program causing such efficiency gains.

Value added

Value added is the increased value from using a product in a new way or modifying the product in a way that generates more profits for the business.

An example: a farmer uses the grain produced on the farm as feed for hislivestock rather than selling the grain for cash. This may significantly improve the returns per bushel of grain produced.

The value of the program that resulted in this alternative use of the grain can be estimated to be the difference in returns produced by feeding the grain to the livestock compared to the price of the same bushel of grain if sold

Expected values

This method of program valuation, requires one to have knowledge of the end product or result that is expected from the educational program.

For example, the value of a program to help people start home-based business may be the projected income generated by such a new business.

The expected income can be estimated based on the known history of similar ventures.

Alternative Opportunity Cost of Capital

Some Extension programs address alternative allocation of resources within one’s business to increase total return.

The increase in total returns can be used as an estimate of the program value.

For example, a farmer has a choice to

1. invest in machinery which would result in a major

cost savings ,or

2. to invest the same funds in a regular savings acct.

The value of the program to assist the farmer in making this decision could be estimated to be the difference between the net income realized from the cost savings of the machinery investment and the interest earned had the money be invested in the savings account.

Willingness to Pay

The willingness of people to pay for some item or service may be used as an estimate of the economic benefit of providing that product or service.

Consumer’s perception of improved quality of products has been successfully parlayed into additional income through improved product packaging or presentation.

People are often willing to state how much they would be willing to pay for certain improved packaging, conveniences, and services.

Such stated willingness-to-pay could serve as the basis for estimating the value of educational programs exploring the costs and benefits of activities such as improved packaging and other market development, or services such as recreational opportunities (trail development, agri-tourism, or similar recreational ventures).

Multiplier effect

This term is often used in economic development circles.

When new money is introduced into a community as a result of economic activity, it is likely that there will be greater demand for other goods and services in the community.

This may result in more new businesses.

The "multiplier" then, is the number of times that the initial dollar of economic activity causes additional dollars to be generated in the community.

The estimated value of the Extension program that stimulated the initial increase in economic activity could be the total value of additional goods and services being generated.

How we are better off?

The insurance industry continually places values on peoples’ lives and serious injuries. 

Changes in habits, lifestyles, practices, infrastructure, and physical and social environment, impacts our well-being. 

Removal of hazards, improvements in physical surroundings, and changed behaviors that results in reduced illnesses or increased life expectancy, all add to the value of life.

Adopting practices that result in safer driving, improved eating habits, improved water quality and safety, increased seat belt use, safer pesticide handling, and numerous other behaviors are examples of changes that impact the protection of human life. 

The value of such changes in behavior can reasonably be quantified using statistical values ascribed to human life, injury, and health, and thus can be used to estimate the economic value of educational programs.

Non-market benefits (cost effectiveness)

Extension programs produce change in human behavior. Easy to identify Difficult to value due to their non-market nature.

Examples of non-market benefits: (not bought or sold) changes (increases) in a person’s education level, changes in one’s aspirations or attitudes, improvement in one’s quality of life, self-esteem, or major leadership skills and

abilities.

Human development and social change have been studied and values projected for these changes. 

These projections can be used to value of such Extension programs. For example, US Census Bureau estimates values of different education levels in

lifetime earnings

EDUCATION PAYS OFF

$0 $1 $2 $3 $4

Less than high school

High school

Some college

Associate's

Bachelor's

Master's

Professional

Doctoral

million million million million

2.5 million

2.1 million

1.6 million

1.5 million

1.2 million

1.0 million

4.4 million

$3.4 million

New data from the Census Bureau shows that people with higher education levels earn more money over a lifetime.

Lifetime earnings estimates(Full-time workers ages 25 to 64)

Source: Census Bureau THE NEWS & OBSERVER

The Associated PressSunday, July 21, 2002

Indirect values Indirect values are difficult to assess.

Indirect benefits are affected by factors beyond the scope the educational program.

Valuation of indirect program results is questionable at best, if attempted by persons not recognized as valuation experts.

Example: the value of refrigeration equipment sold to provide better protection for food in a restaurant

The primary program goal: food safety through proper food preparation and handling practices by food service workers.

Motivation to buy new equipment may be: a result of Food Handlers Extension Certification Program

HOWEVERIn reality, the equipment may have simply been:

A replacement of old equipment, Merely a step toward  expanding the restaurant’s business, Or, simply the result of a visit by a very good refrigeration equipment salesperson.

FACTORS TO ANSWER IN PROVIDING PROGRAM

IMPACT/ACCOUNTABILITY INFORMATION TO IDENTIFIED

AUDIENCES

•What?

•Who?

•When?

•How?

Accountability Key

The Key is to provide the right accountability information to the right people at the right time in

the right format

Accountability

Except for Those Clients Directly Involved……

“If you did not

report it,

you did not

do it”