SGDC 5023 EVALUATION OF TEACHING

EXPERTISE-ORIENTED EVALUATION APPROACHESAND

ADVERSARY-ORIENTED EVALUATION APPROACHES

PREPARED BY:GROUP 1

EMPARIE ANAK TOM 807175JAMPONG ANAK SELAT 807186PATRICK ANAK BETOK 807219PETER JAU NGAU 807220

1

EXPERTISE-ORIENTED EVALUATION APPROACHES

• These approaches depend primarily on the direct application of professional expertise to judge the quality of educational endeavours.

• Worthen and Sanders (1987) suggest that this is probably the oldest approach to evaluation and, while all evaluations have an element of subjective professional judgment, only this approach elevates that subjectivism to such rarefied heights.

• In addition to a specific discussion of Eisner’s connoisseurship model, they have organised their discussion of this approach under four categories (Worthen & Sanders, 1987, p. 99).

2

Informal Professional ReviewsSystem

Formal Professional ReviewSystem

Ad Hoc Panel Review

Ad Hoc Individual Reviews

Type Of Expertise-Oriented

Evaluation Approach

Type Of Expertise-oriented Evaluation Approach

1. Formal Profesional Review Systems• Worthen & Sanders (1987) define formal professional review

systems as having :– structures established to conduct periodic reviews of

educational endeavours; – published standards (and possibly instruments) for

conducting reviews; – pre-specified schedules on which reviews will be

conducted; – opinions of several experts combining to reach a judgment

of value; and – an impact on the status of that which is reviewed.

4

• As a major example of this type of review system, they list accreditation procedures for institutions. They acknowledge, however, that such systems are no longer viewed with the same confidence as they once were; external reviews are now occurring in addition to the private accrediting procedures.

2. Informal Review System• Informal systems are likely to have a structure, procedural

guidelines and multiple reviewers, but they lack published standards and/or specified review schedules. As examples, they list Education Department or Government requirements to review funded educational innovations, professorial review procedures, graduate students’ supervisory committees, and peer review of manuscripts for publication.

5

3. Ad Hoc Panel Review• Many systems occur at irregular intervals when circumstances

demand. Usually, such reviews are without structure or standards. As examples, they list funding agency review panels and "blue-ribbon panels", that is, panels where members are appointed because of their experience or expertise.

4. Ad Hoc Individual Review• Another type of ad hoc system occurs where the review was

conducted not by a panel but by an individual.

6

Educational Connoisseurship and Criticism (see Eisner, 1976)

• In discussing Eisndr’sConooisseurphip model, Worthen & Sanders (1987) note that the role of the theatre or literary critic is well known and generally accepted. Eisner proposed that educators should bring their expertise to bear in evaluating the quality of education. The paradigm is not scientific but artistic.

• The approach that Eisner advocated required what he called educational connoisseurship - the art of appreciation or awareness of qualities and relationships in what is being observed - and educational criticism - the art of disclosing the qualities of events or objects that connoisseurship perceives. In other words, the criticism provides the value statement in the evaluation process. 7

How The Expertise-Oriented Evaluation Approach Has Been Used

• Institutional accreditation– Entire institution is accredited, including all of its more

specific entities and activities

• Specialized or program accreditation– Deals with various subunits in an institution, such as

particular academic or professional training programs.

8

Strengths Expertise-Oriented Evaluation Approach

• As list by Kirkwood (1982):1. Fostering excellence in education through development of

criteria and guidelines for assessing institutional effectiveness

2. Encouraging institutional improvement through continual self-study and evaluation

9

3. Assuring the academic community, the general public, the professions and other agencies that an institution or program has clearly defined and appropriate educational objectives, established conditions to facilitate their achievement, and is so organized and supported that it can be expected to continue doing so

4. Providing counsel and assistance to established and developing institutions

5. Protecting institutional from encroachments that might jeopardize their educational effectiveness or academic freedom.

10

Limitation Expertise-Oriented Evaluation Approach

• Problems with accreditation (Scriven,1984):i. Not suggested of “mishmash” of standard ranging from

trivial to important

ii. Fixation on goals that may exclude searching for side effects

iii. Managerial bias that influences the composition of review teams

iv. Processes that prelude input from the istituition’s most severe critics.

11

• Public suspicion that review by one’s peers is inherently conservative, potentially incestuous and subject to possible conflict of interest.

• Confidentially can be another problem because professionals are often loathe to expose their views boldly in the necessary public report.

• Question of interjudge and interpanel reliability because so much depends on the profesionalism and perception of the individual expert

12

ADVERSARY-ORIENTED EVALUATION APPROACHES

• The evaluator should be impartial toward that which he evaluates.

• Evaluation approaches attempt to reduce bias.

• Some type of hearing would be held so that the opposing views could be presented and debated before whoever would serve as “judge” or “jury” to decide on the relatives merits of the opposing cases.

• Planned opposition in the points of view of different evaluators or evaluation terms-a planned effort to generate opposing points of view within the overall evaluation.

13

Developers of

Adversary-Oriented

Evaluation Approaches and

Their Contributions

1. The Judicial Evaluation

Model and Other ‘Pro’

and ‘Con’ Adversary Hearings2. Adversary Hearings

With More Than

Two Opposing Views3. Adversary Debates

and Other Forensic

Structures

1. The Judicial Evaluation Model and Other ‘Pro’ and ‘Con’ Adversary Hearings

• The ‘fight theory’ underlies most models of litigation for resolving differences among opposing parties.

• This theory holds that the facts in a case can best be determined if each side tries as hard as possible, in a keenly partisan spirit, to provide the court with evidence favorable to that side.

• Owens (1973) listed several characteristics of adversary proceedings appropriate for evaluation:i. The rules established for handling the adversary

proceedings are quite flexible;

15

ii. Complex rules of evidence are replaced by free evaluation of evidence based solely upon whether the ;

iii. Both parties can be required before the trial to inform the hearings officer of all relevant facts, means of proof, and names of witnesses.

iv. A copy of the charges is furnished to the hearings officer and defendant before the trial and the defendant has the option of admitting in advance to certain charges and challenging others.

v. Witnesses are allowed to testify more freely and to be cross-examined.

16

vi. Experts are often called upon to testify even before the trial.

vii. Pretrial conferences of the hearings officer with both parties tend to make the trial less a battle of wits and more of a search for relevant facts.

viii. In addition the two parties involved other interested groups may be permitted to participate.

17

• Wolf (1973, 1975) proposed the judicial evaluation model that included a statement of charges, opposing counselors, witnesses, a judge or hearings officer, and a jury panel. The model had four stages:

i. Issue generation

ii. Issue selection

iii. Reparation of arguments

iv. The hearing

18

2. Adversary Hearings With More Than Two Opposing Views

• Smith (1985) has noted;i. They reflect multiple viewpoints.

ii. They frequently use hearing processes, questioning, cross-examination, interaction concerning alternative points.

• St John (undated), in referring to such hearings as the “committee approach” to evaluation:

i. All of those with a stake in the evaluation-decision-markers, evaluators, program personnel, clients, and other interested persons.

19

iii. A public hearing with testimony, questioning, cross-examination and summary statements.

iv. The committee hearing method consists of public, verbal, face-to-face interactions, and therefore generates a high degree of personal involvement.

v. Because interaction between different points of view takes place, a process of communication and education occurs, and the evaluation makes its impact as it is happening.

20

3. Adversary Debates and Other Forensic Structures• Several approaches that qualify as adversary-oriented do not

employ hearing processes. Kourilsky (1973), proposed that pro and con arguments be presented to a decision-marker, who would examine the evidence and questions the presenters, ultimately arriving at the decisions that seemed fair given both positions.

• Nafziger and others (1977) described and adversary evaluation design employing a modified debate model for presenting data collected in a comprehensive evaluation to ensure that both sides of controversial issues were illuminated.

21

How The Adversary-Oriented Evaluation Approach Has Been Used

• Clyne (1982) summarized the uses of the adversary process in educational evaluation : 1. Summative evaluation

2. Formative evaluation

3. Social science debate

4. Policy analysis and debate

5. School governance and local decision making

6. Issue resolution and policy formation

22

Strengths And Limitations Of

The Adversary Oriented Evaluation Approach

23

Speak To The Merits of The Adversarial Concept Itself.

• For example, most observers would agree that building opposing viewpoints into an evaluation tends to illuminate both the positive and negative aspects of an educational program better than most other evaluation approaches

• Adversary approaches also tend to broaden the range of information collected.

• A strength common to all of the adversary approaches is the interest they create on the part of their intended audiences

24

Speak To The Merits of The Adversarial Concept Itself.

• One of this approach’s greatest strenghts is that it can satisy the audience’s informational needs in an interesting, informative manner.

• Adversary-oriented evaluation is also sufficiently broad and pluralistic that it can be combined with other approaches. For example, there is nothing to prevent the use of an expertise-oriented evaluation approch by both teams in an adversary-oriented study, any more than it would violate this approach for the advocate to use a participant-oriented approach while the adversary employed a management-oriented approach.

25

• Another general advantages to adversary-oriented evaluation is that it anticipates (and largely blunts) the nearly inevitable criticisms offered by anyone whose position is not supported by the findings. It is difficult to argue that an evaluation is unfair if it examines and presents both sides of an issue.

• Another advantage is the substantial, rigorous planning required of most adversary evalutions (no one wants to be humiliated by an opponent gloating over an easy victory).

26

• Certain legal metaphors may be particulary useful. For example, the British Judicial system “exploration for discovery” provides an opportunity for opposing barristers to disclose to one another their cases and supporting evidence in the interest of finding any common ground. When two adversaries agree on any data, interpretation, or conclusion, it lends great credence to that aspects of the evalution.

27

Summarize

1. The objects of the evaluation affects many people

2. Controversy about the object of the evaluation has created wide interest

3. Decisions are summative

4. Evaluators are external

5. Clear issues are involed

28

6. Administrators understand the intensity of adversary-orinted evaluations

7. Resources are availabe for additional expenses required by adversarial strategies

• Adversary-oriented approaches to evaluation are not yet sufficiently well developed to serve as a standard or model of the future efforts.

• Evaluators may forthrightly protest (for example, Wolf, 1975, 1975) that rigid adherence to a legal model is not intended, yet many continue clinging to the more trivial courtroom rituals that seem unnecessary or downright inappropriate in educational results

29

• Adversary-oriented evaluation literature that invokes the legal model tends to use terms such as “statement of charges” (Hiscox & Owens, 1975), “defendant” (Levine & Rosenberg, 1979), “not-guilty” (Levine, 1982), “trial by jury”(Wolf, 1975)

• That orientation may be appropriate when there is a formal complaint againts an educational program, as in the occasional investigation of some education program for malfeasance, misuse of funds or gross misteratment of students

• Educational evaluation should aspire to imrove educational programs, not determine their guilt or innocence.

30

• Another general concern with adversary-oriented evalution is whether it provides decision-makers with the full range of needed information. Presentation of strong pro or positions might increase the probability of an extreme decision

• Popham and Carlson (1977) point to “disparity in proponent prowerss” as a deficit of adversary evaluation, claiming it is all too likely that the case will be decided because of a disparity in skill of the competing teams (or individuals), with the audience influenced more by the persuasiveness of the protagonists than by the strength of the evidence that supports their arguments. The potential for a skilled orator without solid supportive data to sway the “jury” by eloquence alone is unfortunately, a real possibility.

31

• The debate model also has irrelevancies that should be strained out before it is applied to education.

• Debate are polemics and persuasion, not truth, which is central to the validty of evaluation studies.

• Probably more sophistry result from debaters perversions of syllogistics logic than any other form of sef-deception. A skilled debater can often build a remakably strong case on very flimsy foundation.

32

• Many commentators have pointed out that adversary-oriented evaluation are time-consuming and expensive, requiring extensive preparation and considerable investment of human and fiscal resources example, Owens, 1973.

• Levine and others (1978) estimated that over 80 percent of their effort in evaluation using an adaptation of a jury trial went into preparing the case and managing the process, and less than 20 percent went into the actual hearing

• The real question is not cost, but cost-effectivenest or cost-benefit.

33

• On these dimensions it seems apparent that benefit must be argued on grounds that adversary evaluation increases representatives of the data, fairness of the instruments, communication between evaluators and decision-makers, and identification of all the pros and cons.

• Final concern of critics of adversary-oriented evaluations is that those who serve as judges are fallible arbiters.

• Popham and Carlson (1977) have worried about the lack of a process for appealing unwise decisions by arbiters, stating that the lack of a “higher court of appeals” in educational evaluation precludes rectifying improper judgments.

34

RUJUKAN

Worthen, B.R. and Sanders, J. R. (1987).Educational evaluation. New York:Longman.

35

Thank

You

36