Chapter 12 Summative Evaluation

By Reginald Smith

November 9, 2016

Summative Evaluation is defined as the design of evaluation studies and

the collection of data to verify the effectiveness of instructional materials

with target learners. According to Dick and Carey (1996), the expert

judgement phase of summative evaluation is used to find out if either current

or candidate instruction can meet an organization's identified instructional

needs. The following activities are part of the expert judgement phase of

summative evaluation when reviewing candidate instruction:

evaluating the congruence between the organization's instructional

needs and candidate instruction

evaluating the completeness and accuracy of candidate instruction

evaluating the instructional strategy contained in the candidate

instruction

evaluating the utility of the instruction

determining current users' satisfaction with the instruction (p. 323)

The Purpose of Summative Evaluation

Make “go-no-go” decisions

Keep current materials?

The expert judgement phase has already been accomplished if the

instruction was tailored to the identified needs of the organization,

systematically design and developer and been through formative

evaluation. However, the instruction must be subjected to expert

judgement if the organization is unfamiliar with the instruction and

its developmental history (Dick & Carey, 1996). Usually expert

judgement is used to select from the available instructional options

in order to choose one or two that are most promising for a field

trial.

Field Trial

The purpose of the field trial phase of summative evaluation is to

determine the effectiveness of instruction with the target group in

the intended setting (Dick & Carey, 1996). There are two parts to

the field trial phase: outcomes analysis and management analysis.

The outcomes analysis reviews the impact of the instruction on the

learner, the job and the organization. Management analysis

assesses "instructor and supervisor attitudes related to learner

performance, implementation feasibility, and costs" (p. 323).

Are learners capable of applying what they learn to their job?

Impact on organizations

Are learners attitudes helping the organizations achieve a positive

difference?

Are the instructor and manager attitudes

satisfactory

?

Are recommended implementation procedures

feasible

?

Are costs relate to time, personnel, equipments and resources

reasonable?

The focus is the outcome of the instruction

Formative vs. Summative Evaluation

Purpose of the Summative Evaluation

To make "go-no-go" decisions

To keep current materials?

OR

Look for something better suited to meet organization's specific instruction

needs?

(v) Current User Analysis

Seek additional information about

the candidate materials

from the organizations that are

experienced

in using them

Field Trial Phase

(Was the instruction effective?)

-Planning

-Preparing

-Implementing

-Summarizing and Analyzing Data.

Management Analysis

Are the instructor and manager attitudes

satisfactory

?

Are recommended implementation procedures

feasible

?

Are costs relate to time, personnel, equipments and resources

reasonable?

Outcome Analysis

REFLECTION

Why is evaluation important? Possible reasons:

I think without evaluation it cannot be said whether the

designers have properly understood the users. Therefore, one

would not know whether the whole design is intolerant of

minor errors.

It'll be difficult to know whether the systems cause

disruption, frustration, unacceptable changes or conflict in

organisations.

Also, one would not know how to improve the systems in

order to fit the users needs better.

Without evaluation, alternative designs could not be

compared

Or whether systems cause a cognitive overload in users

It would also be hard to assess whether computer systems

force users to perform tasks in undesirable ways. Thus, it

would be hard to check conformance to a standard.

Engineering towards a target (often expressed as some form

of metric) would be hard to achieve.

I think it is very important for us to constantly evaluate the

instructional strategy ensure that is staying the course with what

the organization has in mind while being usable and feasible for

the organization. I think the summative evaluation does just

that. Summative evaluation's existence provides monitoring and

help to develop the training for future events.

As future ID, we need to design and conduct summative evaluation

by looking at all four levels of Kirkpatrick. Kirkpatrick stated that

Levels 3 and 4 (Behavior and Results) are important indicators of

the training's value to the organization. Thus, we should not limit

ourselves to only the first two levels (Reaction and Learning).

EXTENDED KNOWLEDGE

Smith and Ragan (1999) suggest to determine goals of evaluation

as the first step in a goal-based summative evaluation. The most

important part of this stage is determining questions that should be

answered as a result of the evaluation. The client organization

and/or funding agencies and other stakeholders should identify the

questions. These questions will guide the remainder of the

summative evaluation. Questions might include:

Does implementation of the instruction solve the problem

identified in the needs assessment?

Do the learners achieve the goals of the instruction?

What are the costs of the instruction? What is the "return on

investment" of the instruction? (p. 355)

Both the client and evaluator should agree on the questions before

moving on to subsequent steps

Contrasting Summative and Formative Evaluations

Summative Evaluation

• Provides teachers and students with information concerning

achievement

• Has a high-point value

• End-goal is to compare student achievement at the end of an

instructional unit by comparing it to a set benchmark

Formative Evaluation :

• Is used to check student progress as an instructional unit

is occurring

• Guides the next steps in instruction by identifying other

opportunities to aid in success

• Must be designed in a way to respond to students’ needs

Type of Evaluation Relative Complexity Types of Activities Descriptive • Simplest form • Least expensive • Conducted by project staff • Analysis of services • How they were operated • How program was administered • Resources consumed • Characteristics of those impacted by project • Describe any outcomes Operational • Slightly more involved • Low expense • Conducted by project staff • All of descriptive evaluation activities •

Goals and objectives • Describe project components (start-up, recruitment, partnerships, etc.) • Explain short-term and intermediate outcomes • Explain project completion or institutionalization Process • Slightly more involved • Moderate expense • Conducted by professional evaluator (may be staff or consultant) • Focused on service delivery and administrative processes • Suggests causal relationships between what was done and outcomes • Generalize your experiences more broadly by providing insights into effectiveness • Look at efficacy of program in terms of outcomes or costs • Investigate operational features against results Outcomes •

More complex • Moderate expense • Conducted by professional evaluator (may be staff or consultant • Use exacting data collection and statistical methods for data analysis • Requires database and analysis software • Focuses on qualitative and quantitative analysis of data Impact Study • Long-term, involved • Most expensive • Requires third- party evaluator • Often contains experimental and control groups • Proves statistical significance • Requires large sample sizes • Long-term analysis of outcomes