MacDonald, B -1976-The Portrayal of Persons as Evaluation Data

8/14/2019 MacDonald, B -1976-The Portrayal of Persons as Evaluation Data

1/21


2/21

THE PORTRAYAL OF PERSONS AS EVALUATION DATA

Barry MacDonaldCentre for Applied Research inEducationUniversity of East AngliaNorwich

SAFARIA four year study of curriculummanagement sponsored by theFord Foundation

UNCALA four year evaluation of computer-assisted learning. Commissionedby the National Development Pro-gramme in Computer Assisted Learning.

Paper presented a t the Annual Meeting of the American Educational ResearchAssociation in San Francisco, as part of a symposium enti t led "Issues andMethods in Qualitative Evaluation", April 23 , 1976.


3/21

ACKNOlfTLEDGEr4ENT

New approaches to evaluat ion solve some old problems butcreate fresh ones. This is a "signall ing" paper, seekingto draw attention to what could prove to be a cr i t ica lissue in the application of non-numerical methods toeducational evaluation. The treatment of identif iableperspns in evaluation reports which are defini t ional lyintended to have consequences for them, has alreadybecome acutely problematic for those of us in theUnited Kingdom who have begun, i n eva luati on , toexplore case-study methods and portrayal"'stylereporting. In formulating the problem in th is paper,I am aware of a debt to colleagues on the UNCAL andSAFARI projects , part icular ly to David Jenkins, StephenKemmis, Helen Simons, David Tawney and Rob Walker. Thisclaim to express a shared concern should not, however,be taken to imply an endorsement on the i r part of thep a r t ~ c u l a r constructions or r esponses expressed in th ispaper


4/21

THE PORTRAYAL OF PERSONS AS EVALUATION DATA

"If humanistic science may be said to have any goalsbeyond sheer fascination with the human mystery andenjoyment of i t , these would be to r el ea se the personfrom external control and to make him less predictableto the observer "Abraham Mazlow, The Psychology of Science.

In view of the continuing proliferation of schools of educationaleValuatiqn, i t may be advisable to begin by offering a few propositions as a badge of identity . Evaluation serves decisions abouteducational provision. I t dees so by observing and describingeducational programnes. Evaluators make known, to those who havelegitimate claims upon their services, something of th e circumstances,values, processes and effects of educational programmes. They seek toperform this task, and to present their results, in ways which arecalculated to enhance understanding of the relationships between thecircumstances, values, processes and effects of programmes. Sounddecisions about educational provision always require a tt en tion to theinterdependence of circumstance, action and consequence. Soundevaluation designs r ef le ct th is requirement.Rhetoric of this kind being notoriously non-diVisive, I had better addthat in practice I favour evaluations which work through case studymethods towards integrated portrayals of programmes in action. "Portrayal"is no t a well-defined concept in evaluation, but i t is a provocativeand suggestive one, an intruder in the vocabulary of research, a bridgingconcept between the ar ts and the social sciences. I ts appeal is , Ibelieve, to those evaluators who want to render educational programmesmore knowable to the non-research community, more accessible to thediverse patternings of meanin, significance and worth through whichpeople ordinarily evaluate social l i fe . More immediately, portrayalsuggests that the audiences of evalUation need to know "what goes on"in education, and that an important task for the evaluator is to displaythe educational process in ways which enable people to engage i t withtheir hearts and minds. The "heart" of judgement i s rarely acknowledgedin conventional defini tions o f evaluation purpose, which speak clinicallyof providing decision data for the con tinu ation, rev isio n or terminationof programmes. Harry Walcott was moved earl ier this week to remind usof this when he said to a group of educational ethnographers, "Howwould you feel i f your data were used to continue, revise or terminatea culture?" Rather less dramatically, but in the same vein, I want


5/21

- 2 -

in this paper to explore the social consequences f>r individualpersons of a portrayal approach to educat ional evaluatoin.

Because I want to address a part icular problem within a portrayalstyle of evaluation, I >Iould prefer to avoid taking up a lo t of timeand space arguing the case for this style, as agains t o thers .Stake (1972) introduced the term 'portrayal ' to this Associationin a presentation four years ago, and has since written frequente l ~ r a t i o n s of i t s operational implications, particularly underthe rubric of ' responsive evd.uation. ' Portrayal i s a key conceptof the counter-culture in evaluation which in the las t decade hasmo4Oted an incr '"


6/21

- 3 -

We could do worse than begin by studying the methods of thejournal is t , pa rt icu la r ly the methods of thd "new" journalismthat has flourished ance the 1960s. Tom ~ I o l f e (1973) analysingthe progress of this movement writes:

"by t r ia l and error, by ' ins t inct ' rather than theory.journalists began to d is cove r the devices that gavethe real is t ic novel i ts unique power, variously knowna t i ts ' i ~ n e d i a c y ' , i ts 'concrete real i ty ' , i ts'emotional involvement', i t s 'gripping' or 'absorbingquali ty '" There is a striking resemblance between these aspirations of thenew journalists and the claims made by portrayal-oriented evaluators( fo r ins tance, Parlet t and Hamilton claim a "recognisable reali ty""Stake, "a surrogate experience"). I t is not surprising, thereforeto find that the devices identif ied by Wolfe ~ l s o characterisethe efforts of th e new portrayal school of evaluation.

~ 1 0 1 f e ntunes four key devices ,. scene by scene construction, the useof dialogue, the representation of events as se",n by a third party,and the inclusion of desc ript ive d e ~ ~ i l s that give th e r ead er accessto what Holfe calls the "status l ife" of the subject, "the entire patternof behaviour and possessions through which people express their positionin the world of what they think it i s or what they hope i t will be."Most of th e outstanding examples of th is journalis t ic genre take theform of the interview story cas t i n nar ra tiv e form against a minutelyobserved portxayal of th e soc ia l s et ti ng in which the subject l ives.But th is i s also true of evaluators star t ing to explore a portrayalapproach. 14y colleague, Rob t ~ a l k e r , from the SAFARI project a t East~ g l i a , is presenting to this A.E R.A. meeting a paper called"Stations" which closely approximates this journalist ic form, albei tunintentionally. S l ~ l , R I is an evaluation of the centralised curriculuminnova'tion system that was s",t up in Britain in the early 1960s."Stations" is an attempt to represent what that system means in thel ives of teachers.

" ~ / e stress," writes \ ~ a l k e r , "the importance of por tr ay ing theperceptions, f e ~ l i n g s and r e p o n s ~ s of identifiable individualsin relation to organisational change. Not jus t to give anaccount of what happended, but to COllect an oral historyof what it was lik,) to be involved.".seen as a port rayal, "Sta tions" , uses the same devices as the newjournalism. I t is basically an interview story" cast in narrativeform, with a stY',cng emphasis on scene by Scene construction:


7/21

- 4 -"That evening Ron's qirlfr iend Pat wClnts him to gowith her to a part.y, but he arranges for me to meetJean, th e deputy head'3 wif8 . I meet Jean in thepub where she i s talking to a group of teachers fromthe school."

Incidental d i a l o g u ~ i s featured throughout the report ,"A gir l callle up beh ind and greeted the macher veryloudly ~ l i t h , "Hey Bummer, had any good oneS lately?".To which he replied, "No .. I can ' t get a look-in anywhere since you put the word round about me." Shethen turne d t o the vis i tor and said, "We cal l himBummer I you know.. bE:::cause he' s queer. a 11

The reporting of ' real ist ic" dial:>gue is , according to Wolfe,the lUOSt .effective way of est ab li sh ing character and of involvingth e re...der. In evaluation terms .. dialogue that has that qualityof authenticity that Lou Smith claims for the f ield data of theeducational antlu;opolog is t. (Smith 's 'Tale s from the teletype"s ecti on o f his 'Education, technology and the rural highlands"report is 11 'lood eXaJl'lpl'".)

The third journalist ic devic"), the ' third person 'perspective , ismuch more than a teclmical convention in the context of evaluation.I t is a t th", heart of the eva lua tion purpose. S take has argued that thebest understandings of educational phenomena are l ikely to be heldby those clos.ast to the educat ional process , and i t i s a maj':>r goalof por tr ayal t o r"veal what those understandings are. "Stations".for instance, takes one t",acher, records h is se lf "por traya l, and embedsthat self-portrayal in a context that gives the reader evaluativeaccess to J.t. I t is an attempt to achieve ",hat Eisner says is theaim of I'thick d i . ; s c r i p t i o n ~ ... l ito describe tho m0aning or s ignif icanceof behaViour as i t Q:curs in 11 cultural n e t \ ~ o r k saturated with mGaning."

The attempt depends cri t ical ly on the fourth device mentioned byWolfe th e symbolic detai l of the subject 's l i fe . At one point innStations _1 i Walker describes th8 teacher I S room

"His room is fair ly chaotic. lin enormous hi - f i system(much l ldmired by his pupils who are often to be foundusing i t ) . A collector 's collection of rock records(no jazz) of which ten or e leven LP's seemed in moreor less constant USe. MagaZines piled around theroom, the most used of which was Let it Rock whichcontained several of Ron's act ic les . Books on localindustr ia l history (Ron was jo int author of one), onRussia and a scattaring of sociology (Bernstein'sClass, Codes and Control, Nell Keddie). Most ofthe floor space was taken up by an old mattress, theres t by socks, a t enni s racquet , gym shoes (oncewhite?), a big trunk, assorted le t ters , (oneapplying for the post of 'geography teacher ')


8/21

- 5 -

On the fading wallpaper a Beatle poster and a schoolreport mads out in Ron's name and signed by a pupil("Could do bett"r i f he t r ied harder") "For th e journalist th e purpose of such a description i s toheighten the reader 's sense of involvement, his f eeling o f

L ~ i n g there. .:valuators too talck of providing a vicariousexperience for the reader, but they have another purposeto "hich the su rro gate experience is secondary. I t is toi nc rease the generalisability of th e d ata. I t in a mistaketo assume that evaluators who choose to port ray educa tiona linstances have abandoned the hope of genera li sat ion . On thecontrary. The portrayal evaluator has only shifted the locusof responsibil i ty for generalisation and reduced the size of th esample upon which generalisations will be based. After a l l , iti s an ,xiom of sample-based generalisation that the sample mustbe adequately described in terms of a l l i t s rGlevant charactpristics.And i t is a ' f inding' from our experience of educational evaluation,witness Cronbach's statement quoted ear l ie r , that educationalcases are behaviourally unique. I t is a small step from thesepremises to the conclus ion that , i f we hold to the axiom, wemust f i r s t seek adequate descr ip tions o f ind iv idua l cases, theircharacteristics and in te rac tive e f fec ts . This will not enableus to prescribe action to o th er s. Cronbach writes.

"Though from successive work in many contexts, he mayreach an actuarial generalisation of some pD\


9/21

- 6

head, a knowledge of his own locale that he employs to evaluatethe portrayal in t0rms of what does or does not apply to hissi tuat ion. He is ill fact , gen",ralising from one case t o ano ther ,making' educated jUdgements about th", degree to which knowndifferences in t he r el ev an t v ar ia bl es might lead to or cal l ford if fe rences o f implementation and effects . fiE< is l ikely to payparticular attention to the eKperience and jUdgements. f peoplein the port rayed si tuat ion whose roles and rol'il-sets are simil':!,rto his own. The accurate portrayal of persons i s thereforecrucial to th e re ader I s capacity to use the study b infornhis own actions. Stake te l l s us that portrayal shOUld focusupon personalities, and that th e evaluator should be eKperta t putting into words the "goals. percept ions and valuesthat they hold."

So far it would seem that the portrayal evaluator and the newjournalist have a great deal in common. They share aspecif icity of focus ," an i nt er es t i n persons, as opposed topeople, a concern for conteKtual detai l , an aspiration tocreate vivid images of complex human events. HOtil are drawn,as we have shown, to thQ devices of the rea l i s t school o f f ic ti on . Onemight '\i,J: 'iThy in that case, we have not drawn the comparisonbetween portrayal evaluation and the novel i t se l f . The answeris that a comparison with journal ism compels the confronta tionof issues which the novelist does not face. issues to whichthe journalist and the evaluator respond in ways which distinguishtheir professions quite sharply. The fact that we acquire anintimate knowledge of the c h a r a c t e ~ o f a piece of f ic t ionhas no consequences for them. They are immune. Not so thesubjects of the journalist or the evaluator. They are realpeople, usually known to th e public in th e casa of journalism,always traceable in the case of evaluation. Information abouttileir actions, values and perceptions, made known to o tile rs ,can be used to p raise or censure them, and to manipUlate them.There are always social consequences for tilose who are tilesubjects of journalis t ic or evaluation portrayals. Theconsequences may be welcome or unwelcome, anticipated or unantiCipated, but they are always tilere.


10/21

- 7 -

In evalua tion , which is knowingly consequence-related, suchportrayals may be util ised quite directly in the determinationof consequences for those portrayed, a n d ~ is a t this pointtha t the portrayal of persons as evaluation data becomesacutely probl..:natic. The quotation from lIazlow ",ith whichI introduced this article draws attention to the nature ofthe problem. Elsewhere in the book from which that quotationis drawn. Mazlow says,

., how could it be said that our efforts to knowhtnnan beings are for the sake of prediction andcontrol? we would be horrified by this possibi l i ty "In talking abcut portrayal evaluation up to this point, wehave emphasised i t s uti l i ty for people who are distant from th escene portrayed, people who inhabit other locales, whose onlyconnection to the personalit ies and event s por tr ayed is via th eevaluators report . But there is 'another context of evaluation,one in '11hich th", portrayal is a resource for decision makers whohave pow0r over those portrayed. Cronbach argues in his paperthat evaluators should concentrate on improving "short-runcontrol" i n par ti cu la r sett ings. Does the por trayal of personsincrease the possiblity of the control of persons? A fine ironyindeed i f thos., "valuators who stepped out with Carl Rogers shouldend up in \qalden I I . On this issue we part company, I hope, withthe new journalists . Wolfe dismisses with contempt any concernwith the consequ",nces of personal disclosure.

"People who become overly sensi t ive on this scoreshould never take up the neW style of journalism.They ineVitably turn out second-rate work, biased insuch banal ways that they embarass even the subjectsthey think they are 'protecting ' . A writer needs a tleas t enough ego to believe that what he is doing asa writer is as important as what anyone he is writingabout is doing and that therefore he should notcompromise: his own work.;'That may suffice for journalism. I t certainly goes a long wayto exp la in the merci le ss exposure of vulnerable personalit iestha t marks i t s most celebrated p roduct s, th e substi tution ofaccuracy f:>r truth, the process ing of persons for emotionalconsuu1pti0n, th e d en ia l o f p ri va cy , th e a p p a r e n ~ l y total disregardfor consequences. Such journalism rarely has a purpose beyondthe immediate experience it offers. I t follows the dictate ofthe biographer who, when asked how he modified his portrayalso f living personalities, replied, "I write, as i f they were dead."But journalists and bigraphers can a t leas t clai"1 that they haviO


11/21

. ~ . ~.,.-

~ .

- 8 -

no intention of bringing about consequances for their subjects,whereas eva luator s a re explicit ly in t he bus iness of feeqingdecisions about the si tuat ion, events and people they portray.

Rob Walker and I have written elsewhere about theethicalproblems involved in educational CaSe study, and have art iculateda code of conduct which gives t he sub je ct s of study controlover the form and content of th e portrayal. His "stations"paper was subjected to e x ~ n $ i v c negotiations and modificationsbefore making i t s present public debut, negotiations largelyconcerned with the possible consequences of publication for thosepo11trayed. But ShFr..IU is only one of mal.y poss ib le evaluat ioncontexts in which th por tr ay al o f persons :aay be problematicand contentious and I would l ike to turn now to a different context,one perhaps llIOre ty pic al o f ovaluati.,;n generally.

THE PORTRAYAL OF PERSONS - 1; CASE IN POINTThe British National Development P r o g r ~ e in Computer AssistedLearning (UDPCAL) WaS set up in 1973 for a period of five yearswith a bUdget of two million pounds. I ts primary aim is to secureth e assimilation of computer based le'.,rning ,on a regular insti tutionalbasis a t reasonable cost , and it prOVides f inanci al support t o curriculumdevelopment projects a l l ove r the United Kingdom. The Programmeis funded by the Department of Education and Science (DES) and bys ix other governrdent departments. The Programme Director, RichardHooper, reports to an executive Programme C o ~ n i t t e e on which s i tthe seven sponsoring departments. P ro je ct s a re supported on astepped-funding basis, Programme C o n ~ i t t e e having the option oftermination or extension of funding a t the end of each step.ONCI.L (Understanding Computer Assisted Learning) is lUI independentevaluation study cOIiJLlissioned by the NDPCAL in 1974 for a periodof three years. UNCAL is a tean of four people" David Jenkins,Stephen Kemmis, David Tawney and myself.

The National Programme has a strong commitment to evaluation procedureswithin a t ig htl y k nit management structure. L'valuation is a'contractual requirerr.ent for every project. Through the Direc torateand the mechanism of stepped-funding review, i nt er na l p ro je ctevaluations are link".. to Programme COllJlllitt!0 appraisals. 1\.longside


12/21

- 9 -

this system UNCAL acts as an additional resource, providingindependent accounts of Programme activit ies for a l l threeparties a t prespecified points of policy review, and tryinggenerally to identify and clarify issues and alternativesfacing programme d , ' c i s i o n - ~ .\kers. One of UNCAL' s roles isto provide PrograL1llle COUlmittee with reports on the work ofindividual projects, and it i s in this conteltt that th eissue of personal portrayal has proved to be highly problemat ic .Le t me set th e scene.

Most UNCAL reports to Programme COllllllittee about the workof individual projects have featured, to varying degrees,attempts to portray th e influence of key members of projectteams on the conduct ::.nd course of the work. These portrayals

,....are ne'Jotiated with their subjects and it can reasonably beclaimed that they repr..sent, i f not always endorsed accounts. a tleas t "fai r comment" on the persons concerned. In the area ofpersonalisation, UNCAL i s particularly sensitive to the need forfu l l non-coercive consultation.

'fuere are four uNCAL observers, and their reports d iffer inthe degree to which they offer personalised data of this kind.David TawneY,shares with most of the u niver sity scientistswhose work he has studied, a distaste for this area of evaluation,and considers that his exce l len t re la t ionships with projectpersonnel would be seriously prejUdiced i f he attempted a directassessment of individual contributions. His reports are basicallydepersonalised accounts. On the other hand, David Jenkins'rElports display a taste for and capacity to describe th.,. workof projects in a way which illuminates (or fai ls to ) theinfluence of personal characterist ics (cornpE'tencies, valuesand dispositions) on what is achieved. He considers thesecharacter is t ics to be signif icant determinants of effects, andhe has been able t o ~ e r a t i o n a l i s G this perspective withoutalienating project personnel. The reports of myself and StephenKemmis could be said to variOUSly stand a t intermediate pointson a scale po la rised by Tawney and Jenkins. Thorc is , in otqerwords, no standard UNCAL practice, a situation w h i c h ~ f l e c t sthe novelty of this practice, th e degree to which the obligationto negotiate c v ~ s t r a i n s uniformity, and the uneven distributionof skil ls and confidence in this area among the UNCAL team.


13/21

The following extract fro lD the reportthe content and much of the style of portrayal

.. 10 -.

I t is probably true to say tha t there was from the star tsome unease within P r o g r ~ T m e Committee about the personalisationelement in some UNCAL reports, but tha t this particular conCernwas "c', ,ntaine,,-, ' wi thin mere generalised criticisms of thesereports, and of UNCAL's work as a whole. ,y t a meeting inAutumn 1975, howevur, " s trong r .,servat ion was finally expressedabout U i ~ C A L ' s portrayal of pt;rsons. The issue was pr ccipitabodby an UilCAL report prepared by David Jenkins, in which Committeewas provided with an unusually extensive analysis of individualmembers of the study tama and of their inter-relationships as anad hoc working group.indica te::< s'j12ething ofto which exception was taken.

~ ~ t h 'There is a consensus view of Smith, relativelyunchallenged, tha t points to his openness. hisdedication, his abil i ty to ' think big ' . and atrack record that suggests high, levels of competenceand rel iabi l i ty . I f the National Programme had anAlf Ramsey as eva luator he would doubtless declareSmith's 'work rate ' to be highest of them al l .But some are perplexed by his talkativeness hisover-watchfulness i n S it ua ti ons, a calculating quali tythat does not escape an element 'o f self-regard, andthe fact that he can be a l i t t l e overwhelming ( ifnot manipula tive). But Smith is also valued differentlyby different people and the accounts picked up byUNCAL have varied from near-adulation to indifference.Colleagues trying to bring order to these differenceshave been tempted to see Jim as 'ufward-orientated',more concerned to win approv.:,l of those above himtha t the respect of those below. At one extremehe has been suspected of male chauvinism, but therea5 in&ufficient evidence to make the c h a r g ~ st ick.I t could amount to as l i t t le as a tendency for Jim,finding himself surrounded by fernale aids i toexaggerate his disposition t o del egat e r e s p o n s i b i l i ~ Vrather than 5 u t h o r ~ and to appear ' h o v e r i n ~ ' aroundeverybody else's work si tuat ion ("short termcontract people need support", exp la ins Smith). Whatis ungrudgingly agreed by Jim's admirers and detractorsalike is his ta lent for organisa tion, his meticulousconcern for details and capacity for sheer hard work.His colleagues judge him as 'unrivalled' in committeemanship, although i nc li ned to play the system a l i t t l eunashamedly. He is also patently ambitious ("you canalmost slOlell the mubition"). His success in Committeeis not always fUlly acknowledged, part icularly by thosewho at tr ibute more success to the organ grinder thanth e monkey, ~ n dismiss smith easily as 'Jones' man'Some remember the time when Smith with Jones' approvalwent a round asking people i f their undergraduate courseswere really necessarYe lt


14/21

,.. 11 -

l'


15/21

- 12 -

matter expertise make it di f f icul t to mount contidentevaluations o f the merits o f design,_ and in areilS whiohare l ikely to oall for sensit ive management.

4. Projeot management i s a declared foous o f eva luat ion forthe Programme Direotorate. Suoh evaluation i s conoernedwith the performanoe as well as the st ructure of management.

5. Ono o f UNCAL's functions is to enable Programme Committeeto evaluate both completed work and proposals for fu turework. Although it has been suggested that UNCliL oonfinei t s e l f to an aocount o f a i r . ~ - a o h i e v e m e n t , there i s broadsupport for the view that such aocounts would neitheradvanoe understanding of the problems o f CAL developmentnor represent in a fair Iray the merits and efforts o fNational Programme projeots. I t would oertainly be quiteimpossible to negotiate suoh restrioted aooountswithmore than a few project teams. Aims"aohievement i swidely disputed as a suf f ioient formula for t he eva luat iono f eduoational programmes. UNCAL takes the view tha t i t simposition in eduoational sett ings leads to oautious ratherthan ambitious goal-setting, to the negleot o f unforseenopportunities, and to the manipulation o f data to meeta b lind or it erion . I t i s our assumption that Committeewishes to know not only wbat bas been aocompliShed, but:"Did these people aot in te ll igent ly ._ ef feot ively , and withintegri ty , in the exeoution of the proposal workil"

Issues and AlternativesTraditionally, the et>e>lu


16/21

., 13 ..

3. They are negotiated wi th the "judged", who therefore haveknowledge o f thG data base o f th e evaluat ion.

Thfi'se departures have qJ.lite properly evoked concern and apprehensivaness,wi thin UNCAL as wel l as within [>:!'.-gr-:;:u:,co C.XlloJ.tt: e. lllnong th edangers and p i t f a l l s of th e procedure IRay be counted the fo l lowing:

( i) Interpretat ive accounts o f . ~ p l e ' s actions depend on frameworlcs o f lmalysis and theories of human mo::ivation which arenot always clear to th e observed._ UNCAL observers are morel i ke ly to command these frameworks than those who are portraged,who lllall be thus disadvantaged in negotiat iol l .

(it) Many people f ind it d i f f i cu l t or unpleasant to negotiatesel f- image, and II>lY defer to UNCAL out o f dif f idenoe orembarassment.

(iii) UNCAL may be impel led by n"'!1otiation away from c learstatements towards innuendo.

iv ) UNCAL reporters could be seduced by the " jourmll ism ofexposure" in to sensational accounts which are ho t disc ip l inedbg a s t r i c t cr i ter ion o f relevance to decisions. Seductionmag be a t the leve l o f stg le or content .

(v ) The procedure o f negot iat ion is not a guarantee o f f a i r play.~ ' h e sk i l l s of bargaining are nei ther evenly distr ibuted norequal ly employed.

It must a t on C


17/21

. 14 -

J. The inforlll.:ltion oould be omitted from writ ten reports,but provided orally On request.

2. ~ ' h e evaluation o f performaooe .. individual and collective,oould be undertaken by th e Directorata ,1na th e independentevaluators, jointly, and reach COU/lllittee as a jointrecollllllel1dl1tion.

3. Such information could be presented in a generalisedform, omi tt ing the particularit ies upon which th egeneralisations are based .

lIone o f these alternatives s tr ike UNC1IL as either fe as ib le o rdesirable. UNCAL bas adopted as a firm principle thnt itwill not engage in "secret" evaluations o f projects. 'this mustapply to the evaluation of oompetence and personal influence.The prinoiple exclu,les the adoption of the f i rs t alternativa.1lnother firm principle is that UNCAL will not racommend courseso f action. To abandon either o f these princit>lc$ would beto fundamantally alter the basis o f our relationship to theNational Progrill!lll1e, a relationship to which we are firmlycommitted. This prinoiple exclUllCld the adoption o f the secondalternative, although i t does not exclude the possibi l i ty o faddressing UNCAL reports to the Directorate rather than toProgramme Committee. UNCAL has assumed that ona of thafunctions o f i ts re:or ts i s to enable Committee to evaluatethe recommendations of the Directorate.

The third altarnative course o f action. that information fo rth e Dvaluation ot personnel ef facts and personnal competencabe presentad in a sUllllllarised form, confl ic ts with tha natureof t he rel evan t elata in ralation to UNCAL tasks. In the fi.r s tplaca, persons are embeddad in the contexts of CAL work .. andef facts are impragnated by contaxts. to abstain from accountso f tha wags in IIhich effact-s. contcxts. and persons intar c twould deny Committae a major rasourcc for u n d e r s ~ 1 n J i n qtha programme in acti.on and fo r asscssing its . p ote ntia l. Inthe sacond placa, tlle employment o f a portrayal approachi s particularly appropriate to the di f f icul t and sonsitiva areao f individual and oollective performance. Individuals and theirwork, are susual ly subjec t to mUltiple intarpretations and


18/21

constructions and unless tJds multi.plicity i .s embodiedin accounts, they will .tI.:lt, nor would UNCAL consider, suchaocounts to be fair resrescmtations. Portrayal seemsto us to be the most ef fect ive way, both in terms of truthand justice, to convey the work of projact personnel.

ConclusionUNCAL is not, a t this point in time, convinced that a change ofprocedure or role would reduce the problems of personnelevuuation while maintaining an effective evalUlltioll.lllrvice.We have llO hard and fAst rules in this area, hJwever. Ourpractice is exploratory. guided pragmatically by whatproves to be "reasonable and acceptable".

Rej;lecting upon that statement new. and upon the nature ofdisagreement with COmmittee, a disagreement which s t i l l

persists, I am pUZZled by a paradox. In developing the kindof port rayal report s which we have evolved, we were consciouslyseeking to match the decision-maker's "vocabUlary of action",to borrow Ernest House's phrase. Classical evaluation hasfailed to provide the range of information which the decisionmaker takes account of in sekcting a course of action. Thefocus in personalities and their influence en events was areal is t ic recognition thnt the personal dimension ~ neveE ignored by the decision-maker, i f information about i tis available. I t was an attel:lpt to close one particular gapin the evaluat10n data. Yet that effort was heaVily cri t icised.Could i t be that the portrayal of persons, far from renderingthose persons vulnerable to greater external control, in facterodes that control by introducing into personnel evaluationan element of public unswerability'i I should like to thirik i twere so , but I am not at a l l sure.

~ ~ s t of th e growing l i terature on case study and por tr ayal inevaluation stresses i ts potential for y ie ld ing better understandingsof education. The S/WARI portrayals are certainly undertakenwith this hope in mind and in this spi r i t . But as evaluatorswe need to bear in mind that portrayalS created in this spi r i tmay not always be received in i t . In portrayil19 persons we " i l loften be portraying


19/21

of portrayals beyond the subject audience must be based ontheir act ive consent. In th i s the 90al i s self-knowledgewhich, in 11azlow's words, "decreases control from outsidethe person nt:d increases antrol from within the person",and reduces his predictabi l i ty t o o th ers . But the UNCALi l lus t ra t ion prolTid",s us with ano ther , but rec09'Jlisablyevaluative context of portrayal , one where tb evaluatorhas the task of portraying persons for thc ~ x p l i c i t purposeof enabling deteltlllin"tions of thei r competence and worthto be !llade. In UNCAL. we have put forward two l ines ofjust i f icat ion in support of our practice . The f i r s t i sthat no adequate portrayal of a programme i s possible whichdoes not ;;c:rtray the key personali t ies involved. To suppress theportrayal oS: person... would be to deny t he dec is ion-maker the poss ib i l i ty of undldrstandinq what has happened. Thesecun": i s that in the circumstances of the Nati':.nal Programmet he eva luat ion of p.3rsonnel is an inescapable factor in thedetermination cf courses of action. I t shoUld be based on anegotiated portrayal of those persons in th e re le vant contextof act ion.

But I remain uneasy. I s t i l l remember a docuoentary filmstudy of a school, made by Roger Graef and broadcast onnational television in 1972. The f ilm concent ra ted, r emorsel es sly,bu t ob jec tively , on portraying the experience of one teacheras she t r ied to communicate with a class of "dif f icul t 'adolescents. By the end of i t , I f e l t tha t I knew thattldacher both as a person and as a professional; I shared hercommitment and had a sympathetic i ns ig ht i nt o her professionalproblems. Above a l l , I admired her for agreeing to exposeher exper ience to a wider audience. , that they might developbet te r understandings of schooling. Some months af ter thebroadcast I heard that she has been subjected to a barrageof cri t icist l alleging r-" ,gogical incompetence, had receiveda number of L>oison-pen le t ters , and was on the verge of a breakdown. These consequences were nei ther intended r anticipatedby those who created the F . . , ~ t r a y a l . Should the' :,e held tohave willed th" consequences of thei r acts? P.rhaps not, butourely th u principles and procedures which govern the creationand ut i l i sa t ion of [or trayals cal l for c loser sc ru tiny thanthey have yet received. I leave the f inal word in th is ar t ic leto Dai Vaughan, who e dite d th e Graef film referred to . He wroteth is two years l a t e r '


20/21

- 17

"Among the people who were not consu lted in theshaping of the films were the participants . Thisconforms with time-hallowed practice. which i susually defended on the grounds that those whoappear in a film, would be vit iated by priue,vanity, modesty or embarrassment. Perhaps i twouldperhaps the attempt, throughopen discussion with the crew, to reachagreements between conflicting part ies onwhat constituted a truthful account of a givenevent would bear more resemblance to a psychiatricencounter session than to Ii civi l ised chatbetween colleagues, and the film would end inribbons. But perhaps that i s a better use forsome films than transmission, and perhapsour bugge ts shou ld allow for i t . There i ssomething .t o be said for an a r t which is grounded,a s th erapy in Ii r ea l s it ua tion ; and sincetelevision is a collaborative art , it may aswell be collaborative therapy. The reSUltsmight in fact be impressive."

What pr ice collaborat ive evaluation?


21/21

\ RE}'EaENCES\I Ehn",r . Ji.l1iot W.\"

M a c D o ~ l d . Berryen:l Walker, Rob,\

\\Mazlow, At.>\MAm.\Pal"l"t t . 1lI. 'and

lfmnil ton, n.

Sni th , L.M. andPohlanj. F;A.

Stake. n.

Vaughan, Dai

Wollo, Tom

Tb.;, Per c;"j/t iv,::. Eyo: Towards t i l" Reformation o fEducational EValuation. Inv! ted A1(lress,Division B, Curriculum end ObjoctiV

Documents

MacDonald, B -1976-The Portrayal of Persons as Evaluation Data