CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea A DESIGN OF THE METAEVALUATION MODEL A DESIGN OF THE METAEVALUATION

Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, KoreaKeun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea

CES 20th Annual ConferenceCES 20th Annual Conference

A DESIGN OF THE METAEVALUATION MODELA DESIGN OF THE METAEVALUATION MODEL

16. May. 2000

Keun-bok Kang, Professor

Chungnam National University (CNU), Taejon, KoreaTel & Fax : 82-42-821-5845 ; E-mail : [email protected]

Chan-goo Yi, Principal Researcher

Electronics and Telecommunications Research Institute (ETRI), Taejon, KoreaTel : 82-42-860-6924 ; Fax : 82-42-861-6880 ; E-mail : [email protected]

A DESIGN OF THE METAEVALUATION MODELA DESIGN OF THE METAEVALUATION MODEL

16. May. 2000

Keun-bok Kang, Professor

Chungnam National University (CNU), Taejon, KoreaTel & Fax : 82-42-821-5845 ; E-mail : [email protected]

Chan-goo Yi, Principal Researcher

Electronics and Telecommunications Research Institute (ETRI), Taejon, KoreaTel : 82-42-860-6924 ; Fax : 82-42-861-6880 ; E-mail : [email protected]



1. Introduction1. Introduction

2. Definition of Metaevaluation2. Definition of Metaevaluation

3. Previous Researches on the Metaevaluation3. Previous Researches on the Metaevaluation

4. Building the Metaevaluation Model4. Building the Metaevaluation Model

4-1. Component Factors of Metaevaluation4-1. Component Factors of Metaevaluation

4-2. Design of the Metaevaluation Model4-2. Design of the Metaevaluation Model

1. Introduction1. Introduction

2. Definition of Metaevaluation2. Definition of Metaevaluation

3. Previous Researches on the Metaevaluation3. Previous Researches on the Metaevaluation

4. Building the Metaevaluation Model4. Building the Metaevaluation Model

4-1. Component Factors of Metaevaluation4-1. Component Factors of Metaevaluation

4-2. Design of the Metaevaluation Model4-2. Design of the Metaevaluation Model

CONTENTSCONTENTSCONTENTSCONTENTS



▣ ▣ Purpose of this studyPurpose of this study

- designing of metaevaluation model that can be used in practice

- in order to achieve this :

• review the previous approaches to metaevaluation

(especially the component factors of metaevaluation)

• identify the component factors of metaevaluation

• suggest the metaevaluation model

▣ ▣ Purpose of this studyPurpose of this study

- designing of metaevaluation model that can be used in practice

- in order to achieve this :

• review the previous approaches to metaevaluation

(especially the component factors of metaevaluation)

• identify the component factors of metaevaluation

• suggest the metaevaluation model

1. Introduction1. Introduction1. Introduction1. Introduction

- 1 -



▣ ▣ usually metaevaluation is understood as evaluation of evaluation

▣ ▣ according to how they understand the nature and the purpose of policy evaluation, they define the metaevaluation differently

▣ ▣ policy evaluation :

- (purpose of policy evaluation) produce valid and useful policy relevant

information to stakeholders, to decide in making the policy or changing

the existing policy, to improve the policy implementation, to identify

the responsibility of policy process participants

▣ ▣ usually metaevaluation is understood as evaluation of evaluation

▣ ▣ according to how they understand the nature and the purpose of policy evaluation, they define the metaevaluation differently

▣ ▣ policy evaluation :

- (purpose of policy evaluation) produce valid and useful policy relevant

information to stakeholders, to decide in making the policy or changing

the existing policy, to improve the policy implementation, to identify

the responsibility of policy process participants

2. Definition of Metaevaluation2. Definition of Metaevaluation2. Definition of Metaevaluation2. Definition of Metaevaluation

- 2 -



- 3 -

▣ ▣ Definition of metaevaluation :

- evaluation on the evaluation and evaluation system, during or after

the evaluation, in terms of evaluation paradigm, evaluation resources,

evaluation process, evaluation performance and evaluation utilization

▣ ▣ Purpose of metaevaluation :

- to value the evaluation(system) : summative evaluation

- to improve the evaluation(system) : formative evaluation

- to promote the utilization of evaluation : summative / formative evaluation

▣ ▣ Definition of metaevaluation :

- evaluation on the evaluation and evaluation system, during or after

the evaluation, in terms of evaluation paradigm, evaluation resources,

evaluation process, evaluation performance and evaluation utilization

▣ ▣ Purpose of metaevaluation :

- to value the evaluation(system) : summative evaluation

- to improve the evaluation(system) : formative evaluation

- to promote the utilization of evaluation : summative / formative evaluation



3-1. 3-1. metaevaluation is an evaluation which examines the value

and merits of policy evaluation itself (Stufflebeam, 1981)

- when policy evaluation is understood as examining the value and merits of

policy activities

- metaevaluation is a process of classifying, acquiring and utilizing descriptive

and perceptive information related effectiveness, actuality, ethicality

and technical appropriateness of certain evaluation activities

- the purpose of metaevaluation is to be a guide for evaluation activities, and

openly report upon the merits and demerits of evaluation itself

3-1. 3-1. metaevaluation is an evaluation which examines the value

and merits of policy evaluation itself (Stufflebeam, 1981)

- when policy evaluation is understood as examining the value and merits of

policy activities

- metaevaluation is a process of classifying, acquiring and utilizing descriptive

and perceptive information related effectiveness, actuality, ethicality

and technical appropriateness of certain evaluation activities

- the purpose of metaevaluation is to be a guide for evaluation activities, and

openly report upon the merits and demerits of evaluation itself

3. Previous Researches on the Metaevaluation3. Previous Researches on the Metaevaluation (expecially the components of metaevaluation)(expecially the components of metaevaluation)

3. Previous Researches on the Metaevaluation3. Previous Researches on the Metaevaluation (expecially the components of metaevaluation)(expecially the components of metaevaluation)

- 4 -



3-2. 3-2. metaevaluation is examining the validity of the evaluation

design, data collection and analysis

- when the purpose of evaluation is understood as producing valid

information for policy results

- metaevaluation is focused on the technical or structural adequacy

of evaluation

(for example, Cook & Gruder, 1978 ; Chelimsky, 1987 ;

Smith & Hauer, 1990 ; Greene, 1992)

3-2. 3-2. metaevaluation is examining the validity of the evaluation

design, data collection and analysis

- when the purpose of evaluation is understood as producing valid

information for policy results

- metaevaluation is focused on the technical or structural adequacy

of evaluation

(for example, Cook & Gruder, 1978 ; Chelimsky, 1987 ;

Smith & Hauer, 1990 ; Greene, 1992)

- 5 -



- 6 -

3-3. 3-3. metaevaluation is assessing the usefulness of the evaluation

- when the purpose of evaluation is understood as providing useful

information to stakeholders

(policymakers, decision makers, program managers, clients, etc)

- metaevaluation is focused on assessing the evaluation utilization

(for example, Mackay, 1992)

3-3. 3-3. metaevaluation is assessing the usefulness of the evaluation

- when the purpose of evaluation is understood as providing useful

information to stakeholders

(policymakers, decision makers, program managers, clients, etc)

- metaevaluation is focused on assessing the evaluation utilization

(for example, Mackay, 1992)



- 7 -

3-4. 3-4. metaevaluation is checking the reliability of the evaluative

judgements(evaluation results), and the evaluation process

that ensure the reduction of the bias

- when evaluation is understood as making unbiased evaluative

judgements of merits or worth of the policy results

- for example, Scriven suggests his ‘Metaevaluation Checklist’

(not identified, quoted from Rogers, 1995)

3-4. 3-4. metaevaluation is checking the reliability of the evaluative

judgements(evaluation results), and the evaluation process

that ensure the reduction of the bias

- when evaluation is understood as making unbiased evaluative

judgements of merits or worth of the policy results

- for example, Scriven suggests his ‘Metaevaluation Checklist’

(not identified, quoted from Rogers, 1995)



- 8 -

3-5. 3-5. Larson & Berliner's Standards

▣ ▣ Larson & Berliner(1983) classified the three component factors of metaevaluation, and further divided each factor into detailed evaluation items

- factors of evaluation input

• resources and techniques used in evaluation, environment

- factors of evaluation process

• actual evaluation activities carried out in accordance with evaluation plan

- factors of evaluation outcome

• number of decisions affected by the evaluation

3-5. 3-5. Larson & Berliner's Standards

▣ ▣ Larson & Berliner(1983) classified the three component factors of metaevaluation, and further divided each factor into detailed evaluation items

- factors of evaluation input

• resources and techniques used in evaluation, environment

- factors of evaluation process

• actual evaluation activities carried out in accordance with evaluation plan

- factors of evaluation outcome

• number of decisions affected by the evaluation



3-5. 3-5. Larson & Berliner's Standards (continued)

- this research was the first systematic approach to metaevaluation

- but, they did not concern the evaluation purposes and types

- they suggested 23 evaluation items for metaevaluation, but some items were

overlapped and the classification of items is somewhat inappropriate

- they had no little concern on the rational and due process

3-5. 3-5. Larson & Berliner's Standards (continued)

- this research was the first systematic approach to metaevaluation

- but, they did not concern the evaluation purposes and types

- they suggested 23 evaluation items for metaevaluation, but some items were

overlapped and the classification of items is somewhat inappropriate

- they had no little concern on the rational and due process

- 9 -



3-6. 3-6. The Joint Committee on Standards for Educational

Evaluation (1994)

- developed 30 principles(standards) for metaevaluation.

• they were grouped as such : Utility, Feasibility, Propriety, Accuracy

- these standards are a useful checklist for metaevlauation, but the committee

does not suggest a satisfactory framework for metaevaluation

(little guidance on how to plan evaluation in order to meets these standards)

- it is needed that these standards to be checked for applicability in other

countries and other types of program (Rogers, 1995)

3-6. 3-6. The Joint Committee on Standards for Educational

Evaluation (1994)

- developed 30 principles(standards) for metaevaluation.

• they were grouped as such : Utility, Feasibility, Propriety, Accuracy

- these standards are a useful checklist for metaevlauation, but the committee

does not suggest a satisfactory framework for metaevaluation

(little guidance on how to plan evaluation in order to meets these standards)

- it is needed that these standards to be checked for applicability in other

countries and other types of program (Rogers, 1995)

- 10 -



3-7. 3-7. Patricia Rogers' Metaevaluation Framework (1995)

▣▣ a framework for evaluating evaluation in terms of their intended and actual impact on the implementation of existing programs

▣▣ her metaevaluation framework consists of :

- 5 evaluation criteria for intermediate outcomes of program evaluation

• producing valid information

• providing useful information to decision-makers

• producing unbiased judgements of merit or worth

• involvement & illumination of relevant stakeholders

• empowerment of the intended program clients


▣▣ a framework for evaluating evaluation in terms of their intended and actual impact on the implementation of existing programs

▣▣ her metaevaluation framework consists of :

- 5 evaluation criteria for intermediate outcomes of program evaluation

• producing valid information

• providing useful information to decision-makers

• producing unbiased judgements of merit or worth

• involvement & illumination of relevant stakeholders

• empowerment of the intended program clients

- 11 -




(continued)

- evaluation's impact on processes of implementing existing programs

or replacing existing programs with alternatives

- evaluation's contribution to the development of programs which

meets need

▣▣ this framework is very systematic, but the evaluation object is limited to the existing programs which is implementing


(continued)

- evaluation's impact on processes of implementing existing programs

or replacing existing programs with alternatives

- evaluation's contribution to the development of programs which

meets need

▣▣ this framework is very systematic, but the evaluation object is limited to the existing programs which is implementing

- 12 -



4-1. 4-1. Component Factors of Metaevaluation

- evaluation paradigm

- evaluation resources

- evaluation process

- evaluation performance

- evaluation utilization

4-1. 4-1. Component Factors of Metaevaluation

- evaluation paradigm

- evaluation resources

- evaluation process

- evaluation performance

- evaluation utilization

4. Building the Metaevaluation Model4. Building the Metaevaluation Model4. Building the Metaevaluation Model4. Building the Metaevaluation Model

- 13 -



① Evaluation Paradigm

- evaluation purpose : rationality

- evaluation type : appropriateness

- evaluation object : suitability

• different level and range of evaluation object may lead different

evaluation results

① Evaluation Paradigm

- evaluation purpose : rationality

- evaluation type : appropriateness

- evaluation object : suitability

• different level and range of evaluation object may lead different

evaluation results

- 14 -



② Evaluation Resources

- evaluation manpower

• quality and quantity of evaluators

• user involvement

- evaluation organization

• structural and functional appropriateness of evaluation organization

- evaluation budget and information

• appropriateness of evaluation budget

• quality(adequacy, reliability, etc) and quantity of information

② Evaluation Resources

- evaluation manpower

• quality and quantity of evaluators

• user involvement

- evaluation organization

• structural and functional appropriateness of evaluation organization

- evaluation budget and information

• appropriateness of evaluation budget

• quality(adequacy, reliability, etc) and quantity of information

- 15 -



③ Evaluation Process

- evaluation procedure : objectivity and fairness

- timing of evaluation : fitness to the evaluation type etc.

- evaluation methodology : accuracy and validity

- evaluation criteria : appropriateness

- evaluation indicators : rationality

③ Evaluation Process

- evaluation procedure : objectivity and fairness

- timing of evaluation : fitness to the evaluation type etc.

- evaluation methodology : accuracy and validity

- evaluation criteria : appropriateness

- evaluation indicators : rationality

- 16 -



④ Evaluation Performance

- evaluation outcome : validity

- evaluation information : usefulness

④ Evaluation Performance

- evaluation outcome : validity

- evaluation information : usefulness

- 17 -



⑤ Evaluation Utilization

▣▣ preparation and dissemination of evaluation report

- clarity and impartiality of report

- timeliness and dissemination of report

▣▣ utilization of evaluation results

- instrumental utilization

• contribution to improving the implementing existing program or

to changing the existing program, to developing the new program

- conceptual utilization

• enlightenment of policy

⑤ Evaluation Utilization

▣▣ preparation and dissemination of evaluation report

- clarity and impartiality of report

- timeliness and dissemination of report

▣▣ utilization of evaluation results

- instrumental utilization

• contribution to improving the implementing existing program or

to changing the existing program, to developing the new program

- conceptual utilization

• enlightenment of policy

- 18 -



- 19 -

4-2. Design of the Metaevaluation Model4-2. Design of the Metaevaluation Model4-2. Design of the Metaevaluation Model4-2. Design of the Metaevaluation Model

component factors ofmetaevaluation items of metaevaluation & criteria

purpose rationality

type

object

appropriateness

suitability (level and range)

manpowerquality / quantity

user involvement

organization structural and functionalappropriateness

evaluationparadigm

evaluationresource

budget

information

appropriateness

quantity and quality(adequacy, reliability, etc)



- 20 -

(continued)

items of metaevaluation & criteriacomponent factors ofmetaevaluation

objectivity and fairness

fitness to evaluation type etc.

accuracy and validity

appropriateness

rationality

validity

usefulness

procedure

timing

methodology

criteria

indicator

outcome

information

evaluationprocess

evaluation performance

clarity and impartiality

timeliness and disseminationreport

Improving & changing the existing program

developing the new program

Instrumentalutilization

enlightment of policyconceptualutilization

evaluationutilization

Documents

CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea A DESIGN OF THE METAEVALUATION MODEL A DESIGN OF THE METAEVALUATION