Evaluating Program Progress

By ANDIE L. KNUTSON, Ph.D.

EVALUATION is coicerned we-itlh deteirminl-ilug value orl wor-thl. Eacht of uis is coIn-

stantly iniaking eva-luations in ouri everydaylives. W1'e jndgre i(leas oriactions in terms ofr'ighit or' w%vrongr, rood or baid, lonest or dislhonest,anid p)ractical or il)practical so fr-equeneitly andso qulickly thiat we aire ofteni un-aware thtat wN-e,ire (loinig so.

W1'e drawv upl)on the fiull lange of ouri expe-riiences ill making these judgments or evaltua-tions, testingo eaci onie against oiur per'soiial andl)rofessiollal l)attellrs of valtue or conicernl tandagaiiist estima-Xltes of tlheir relevance to the sit-ntationi at hand. This weighlingtc plocess miiay b)eeitlher conscious Or subconiscioIIs objective or in-tuitive. TI'lius, decisioiis are ofteni imiade witlh-o-ut dle(llqate consideratiomi of pertinienit factsavailable; som:ietiiies they are miiade witlhoutconisi(lerilit the l)ossibilitv of ob)tainling suchfacts.

Progrrain evaluation is conicerniied witlh deter-iiuiig the wortlh or valuie of efforts to aclhievea giveni plrpose or objective. The specific puir-poses of evalua,ttion ill tlis situiationi are to pro-ilde objective estinmates of achlievemenit anid toprovile giili(laince for the conduct of programactivities. To alchlieve these purposes, two typesof ev.luai.ltion are required-first', ''evaluation

Dr. Knutson, chief of Beharioral Studies, Divisionof General Health Services. Public Health Service,presented this paper in part at Pennsylvania's ThirdAnnutal Hfeallh Conference field at State College,4ugust 19,54.

of prog'iraill achiexveinent" ali(l seconid, evealu-aItionl of prog'raml p)rogress."

Souniid evaluiation sttudies of progrraml achieVe-iuuenit are essenitial if we are ever to kinow whletlh-er or not w-e are attaining the goals for whlielcwe. stri-ve. T'lie l)p)lic, too, lhas a rihlit to kinowwlhetlher our eflorts aneld mioniey are well spent.

Concreiete aiid valid evidenice of the aclhieve-inent of public lhealtlh prograins is frequentlymnost (lifficlilt to obtaiin. Anyiiv of the objectivesaire broad and(1 hard to reachl. Ofteni we maylyniot reasoalably exl)ect obser-vable, positive re-sulIts uniitil thte programti lhas beeni unider way forye-ars. Witht sonme lhold-tlhe-linie oper;atiolns,failures al1re easier to (lefine thaii suiecesses.Manyv ouitsi(le iniflutenices qulite apartit from pullb-lic health efforts iiay effect the achievementof the lonig-raige objectives in positive or nieg-altive w\layS. On1-roin p I activities lllustle colitinued ill the best possible muaniier evenin the aibsenice of the miieasures of achievementwhich arie onily available after the p)rograim lhasbeeni iii operationi for somiie tinile.

Program Progress

Tlhus, tle l)pocess of ev-alulatingr p)rograinatehiievemiienit is niot aldequate for satisfying.r tImesecon(l of the two objectives of evaluation-providingcr gritildace for p)rogrami activity. Toatchlieve tlle l)lp1l)oSe of pog-(ramll guidlallnce, ob-jective data nee(l to be ol)tained an(l appliedthroughout tIme phases of programn p)lanning,r(levelopmeittt,umid op)eration. Objective evalin-altioll mullst take its deserved role as alait of the(ladily rmoitijie of eachi htealthi worker.

Vol. 70, No. 3. Marech 1955::29,70() -,

305

In the course of evaluatilig programii progress,it is possible to identify difficulties or barriersas they occur and to make necessary changeson the basis of the best objective data available.Sound evaluation of progress made as a pro-gram continues day by day can help to assurethat each new phase of the program is built onthe positive achievements of the precedingphases.The conicept of evaluating program progress

is of value whether we think in terms of thehealth department program as a whole, in termsof a particular program, or in terms of theefforts of the individual health worker with hispersonal and team responsibilities.The purpose of evaluating program progress

is to improve the quality of decisions made atany point during the planning, development,and operation of a program.

Since many programs are in operation con-currently in any health department, differenttypes of decisions may have to be made at thesame time. Overlappings occur, but for thepurpose of this discussion, I have grouped theprogram decisions to be made into the fourbroad types of those concerned with:

1. The nature of the interests, wants, or needsthe program is intended to satisfy.

2. The broad purposes of the program andthe philosophy of approach or policies to befollowed in trying to achieve these purposes.

3. The selection or development of the meth-ods, techniques, or procedures to be used in theprogram.

4. The application of philosophy, policies,and methods in carrying out the operation ofthe program.

Interests and Needs

We all tend to assume that other people wantanid need the same things that we would wantand need if we were in their situations. Weassume we know what they need, without recog-nizing that in making that assumption we aremakin-g a decision. Ani important question ofprogress evaluation is: Have the interests,needs, or wants we are trying to satisfy beenadequately identified?

Exploratory fact finding is one way of as-suring adequate identification of needs or wants.

Questiomnaires, interviews, projective tests,group discussions, analysis of statistical data-all sorts of techniques can be used to gather nec-essary data. Whichever method is used, it isimportant to try to obtain the other person'sidentification of needs or desires from his pointof view-as he describes them. We need to becareful that the very nature of our questions orapproach does not limit his thinking to our ownideas about the possibilities.The methods of obtaining data for use in

evaluating program progress do not need tobe complicated. In fact, often the simplestmethod yields the greatest return.The nurse who encourages a mother to pour

out her troubles and waits patiently for her toidentify the things with which she needs helpis applying one of the best exploratory tech-niques. Sometimes simply helping the motherto formulate the problem clearly will enablethe mother herself to identify some constructiveaction to take. Maximum value of the tech-nique occurs when the nurse makes sure thatshe has given some help on as many as possibleof the specific things identified. Referring themother to some other source of assistance issometimes as effective as providing her with ma-terials or guidance. Knowing how the motherperceives her problem may be the key to devel-oping an effective solution. If the solution ismeaningful to her, there is a much greater like -lihood that she will carry out the action after thenurse leaves.No public health worker has the timiie and op-

portunity to make a thorough study of the otherperson's interests, wants, and needs every timea decision has to be made. But all of us canprofit by identifying as a decision each assump-tion we make about needs and by being as ob-jective as we can in making that decision. Wecan all profit also by trying out the more thor-ough method of exploration with a part of thecase load. For example, it might be possible toapply this principle regularly on the first homevisit or restaurant inspection carried out eachmorning. Although initially more time willbe spent in these instances, the additional effortmay help to identify and solve problems thatmight otherwise drag on indefinitely.The same principle applies in identifying

community needs. Recently a health educator

Public Health Reports306

in a local community seeking to identify theneeds upon which to build a sound health educa-tion program first mnade a survey of publichealth needs. She identified little that was newor unexpected and was unhappy with the find-ings because she was certain many health prob-lems were not being reported.She reviewed her approach and found that

she had used questions about health needs andpublic health problems that meant more to thepublic health worker than to the people she wasinterviewing. She had encouraged responses interms of existing services only and her ownideas about new possibilities. On the basis ofthis evaluative review, she changed her ap-proach and tried a more open interview whichencouraged the respondents to discuss freely thelittle things families run into when they firstmove into the neighborhood.Using this new inethod, she found that many

were concerned about uncontrolled dogs roam-ing the neighborhood. Others mentioned theneed for some type of recreation facilities forpreschool children who were playing in thestreets since there were no sidewalks. Expect-ant mothers did not know where to reach thenearest physicians. IlNothers with young chil-dren desired help on behavior problems. Thus,the open question approach, which was focusedtoward identifying the problems from the moth-ers' point of view, yielded data of great valuein defining the goals for a community actionprogram.

Purposes and Philosophy

The second major type of program decisionsto which progress evaluation can contribute arethose concerned with the development of thepurposes and philosophy of the program. Thesecond question can be stated as follows: Havethe program objectives and philosophy andpolicies been fully agreed upon, formuilated, andwritten down?Misunderstanding about the broad purpose of

a program and the philosophy of approach orpolicies to be followed in trying to achieve thispurpose is a common source of friction. Oftensources of potential conflict can be identified inthe beginning of the program through frankand open discussion by all persons directly con-

cerned. Writing downi agreemnents lhelps to as-sure clear and workable formulations.In one instance, discussion along these linies

after a program had been in operation for tyears revealed serious miiisunderstandingsamong the members of the statff. Some lhad per-ceived the project as a community deinonstra-tion program to be discontinued as sooni as thelocal health department was prepared to takeover. Others saw it as a research project to testmethods of diagnosis and control. They wereunaware of the problem of preparing the com-munity for its role. A few perceived it a1s aprogram to provide services and seemed un-aware of any other purposes. Those who clear-ly identified the purpose as a demonstrationwere in disagreement about what they were try-ing to demonstrate and to whom.

Stating the goals of a program in terms ofspecific health practices is a good way of keep-ing plans concrete and practical. This is morethan an exercise; it is a test of sound planning.Unfortunately, some of our health objectivesare long range and hard to pin down. Whenthey can be specifically identified in this way,however, methods of achieving them can be se-lected with greater confidence.

If the goal is to influence a specific healthpractice, such as food handling, it is possibleto find out how this particular practice is re-lated to the habit patterns and interests of thepeople concerned. We need to know how theprogram ties in with their ways of thinking andbehaving, with their way of life. For it is un-reasonable to expect ready acceptance of healthlpractices that conflict with personal interests ordeep-seated habit patterns.

Clear formulation of objectives requiresidentifying the intermediate goals that need tobe achieved in order to attain the program ob-jectives. Almost any program can be brokendow-n into intermediate steps that can be meas-ured. If the measurement shows that the stephas been achieved, one can feel confident in go-ing forward to the next step. If the data sug-gest that this intermediate step has not beenattained, it will be important to find out thereason for failure and pause long enough to finda more effective method for achieving the inter-nediate goal.In evalubating progress. it is important to dis-

Vol. 70, No. 3, March 1955 307

.illguii betNveeni tile hiOtlt.ekveP'il tNl)'e of dataIreqllire(l fot adminiiistration and data thlit yiel(devidleIIce of p)rogress. For exaIlj)le. tlee liIliell.teOf hlollme Visits or restailtralnt inspections m11adeare* adlninistrative types of dlata, -as are also theunitinber of telel)llolne calls, p)aimplilets or leafletsp)rel)alepd.ld(l (listributed, filhuis usedl. a-Eind imeet-ingslS (lio'll5SiOls hl(le. tey arle accouniits ofactivities (allied outt -s part of the plrograii.The'11 -t.ay or inay niot conitribtute to the aelhieve-mlelit of either iintermiiediate or lonig-rang(,e goals..cecounts of sut'ch activities do nlot yield vali(levidlence of program progress.A public lheailtil ninrse, for exaniple, imay keel

a recor(l of the numiber of lholmie visits, how muchiof hier tim,e w\vas spenit in the lhomiie, whlat imiate-riials were left with tlhe mnother. ani(l whlat in-structionis w\erle oriVell. Such recor'ds are oftenlnecessary' forl administrative re-asons.In evaluating lher owni progrress (luring(r a, liomiie

visit. however, tlhe nurlise needs aniswers to quies-tiouis like tlhese: Did slhe finid ouit from themotlher what lprolblems she was most concerneda)olt ? Did the mnothler have full opportunityto (lefille the problein?Was eaclh question ani-swere(l specifically ailul satisfactorily wheni thequtestionn was ra ised? Did the imotlher hiave anopportunity to tell hllat slhe already knew and(lwhaItt sh1e ald (lonie in tryingc, to solve tile prob-lent. andl what. imiiportaiit steps sihe thioughit

grhit to be taken ? Did the niurse ill soime w-ayobtaini thie mtiothier's interpretation of aniy in-strullctionis or suggrces;tioiis gri'ven, anld in this waymake suire tih.at tile illsttructions w\ere ('o1rectlyinterlprete(ld? Did shle take the timiie to ol)servethe pnatient carrying ouit the actioni recoint-miiended? Ansswers to siuelh quiestionis wvill lhell)the nurse evaluate hier progress in assisting theml,otller to solI1ve hler problems.

Methods and Procedures

thlir( grolil) of decisions, to which lwwrlessevlduation cani colntriulte are those concernied

-ith tile selection or developmenit of themethlodis, teclnll iqtes, anid plrocedlnes. to be uise(dinI tIle p)rogrIllI.tIme1 (Imiest ionl of p)rogress eva.ll-atoion withl relspect to t-htis thiird area is : Is; theietlietlod or l)l)pproach, selected tlie onie illost lilkelyto prove s;c(cessfuill inl aci levil)mm tile(oiJectiv'ysof thle p-nogra 1111

If tlhe pulrposes of tIle p1togL'(,11mi 1ha've bee"mlbtokemi (lowII ilito specific intermediate goals, allongtr stel) hlas beeni taken in ideentifying., tile bestimiethiods ain(l techliniques to use.

the final clhoice of programi ilethiod can betested by listinig in one columnl the objective evi-deuice for a imnetllodl, say interview, film, or l)almm-1)lllet, anid ini a parallel columni the object,ive evi-(leiice agtrainst tile use of tlat mietllo(l. Try thisfor eaclh illetliod uniider conisiderationi. Coonsiderquestionis like tlhese: Is there aniy objective evi-(lence that olle illethod is inore effective tlhantanlotlher in chianging behavior or iii aclhievinoanyi otlier purpose ,youi iay lhave ii mind? Whatwill it. probably cost in time and moniey toaelhieve yolur purpose by eacli method? Wihat;tyvpe of personnel is requiired to apply it? Arethe niecessary resources available? Consideringthe resources available, is it, realistic to try toaply t iemetlhod as a way of achievinig the ulti-miate objective?

For examtiple, the saniitariani may l)e concernedwith the problem of eliminating rat-breedingpl)laces anid fire lhazaids in the community. Oniesolution suggested may be to visit homes anldoffices to grive individual instruction to personsresponsible for these conditions. Before under-takilng tlhisimetliodl, he would probably find itprofitalble to ask: How muclh would it cost illtime and(l mioney to visit all tile hlomes or officesatiid discuss tlhe,problem witlh the ow-ners?Wlhat type of peerson wouldl)e required to carryout these visits effectively? Are there such per-soiinel available? Do they have the means oftrianisportationi and funds for travel to theselplaces? How longc wouild it take to make all thevisits? Considerinig the answers to these ques-tionls, is the metliod lpradctical? Could someotlher uetlhod be developed w-hich would bettersatisfy these criteria?

Oni the same griouniids, onie inight question theatdvisability of continuiing other methods thatre(qumire a lheavy exlellditure of professionaltimiie to ieachl only a few of those members oftile colmnullllmity needinig assistance, such as theulse of professional personnel to give individ-uials orsm'all gropiis instruction oni weigilt comi-tr ol. HowN- mian-y lprofessional people w%vill ittake to comllplete the job ini youir community%vorki*ig this way ?

If thle lroiblemn is to idlenitifv tuberculosis cam-

Puiblic Health Report-30

riers among older men in the marginal incomegroup, one might ask whether the diagnosticfacilities are made available to them at a timeand in a way that does not interfere with theirprime job of earning a living. Are they alllikely to be reached by the method used? Arethey likely to understand the purposes of theprogram and will it make sense to them?

It is often useful in identifying or develop-ing a program method to identify systemat-ically the individuals who are in a position toinfluence the flow of ideas between ourselvesand the person with whom we are seeking tocommunicate. In a local area, the influentialperson may be a minister, political ward leader,county agent, employer, storekeeper, or clubsecretary. These individuals are sometimescalled communication gatekeepers since theymay open the door and encourage the free flowof ideas, or they may close the door and preventthe ideas from passing through. They mayinfluence the acceptance of an idea by lendingit their prestige and support. Or they maydraw influence from it by deprecating it. Theymay willfully distort the idea if they are somotivated. Or they may unknowingly changeits meaning through failure to understand andthus err in its interpretation.For example, a man may have a sore on his

lip that does not heal. He may not know whereto go to get help. After talking it over withmembers of his family or possibly a friend, hemay bring his problem to a pharmacist, a min-ister, or someone else whose medical judgmenthe respects. Whoever it is may direct him toa physician who is able to diagnose the prob-lem and provide adequate treatment. Or theadviser may direct him along some futile routeby suggesting the use of a home remedy orpatent medicine, or by sending him to someother unqualified source of information and inthis way block his efforts to get expert help.Many of the problems we encounter in carry-

ing out public health responsibilities directlyinvolve one or more such communication gate-keepers. We need their lhelp. Analyzinig aprogram in terms of the communication gate-keepers will help to identify them and to definetheir possible roles so that methods can bedeveloped to enlist their support. Pinpointingefforts to these members of the community will

help open new channels through which we caneffectively communicate with the public.

Operation of ProgramThe fourth group of progranm decisions to

which progress evaluation can make a contri-bution are those concerned with putting thephilosophy, policies, and methods of the pro-gram into practice. Questions of progressevaluation concerined witlh the actual opera-tion of the program include: Do people under-stand the purpose of our efforts? Is the pur-pose one they want to achieve? Do the prac-tices we recommend make sense to them? Arethese practices in accord with the ways peopleusually behave? Are our attempts at com-munication successful?A clear understanding of purpose is essential

to correct action, particularly when the indi-vidual himself is the only one who can decidewhen or how to act. Failure to understand whyis one of the most serious sources of programfailure. Yet the cause of failure can often beavoided by the simple practice of obtaining theother person's interpretation of purpose.A nurse in Minnlesota related a cliinic exper-

ience that will illustrate. A motlher at a, well-baby clinic was advised to put her child onorange juice. A few days later the child wasbrought back ill, and an examinationi revealedthat the child was starviing. The mother hadperceived the orange juice not as a supplementbut as a substitute for the former diet. To avoida recurrence of such misunderstanding, thenurse now uses various ways of drawing fromthe patient an interpretationi of how she willgo about carrying out new instructions and wlhyit is important to do so.This same technique of testing to see whether

the purpose is understood can be applied toposters, pamphlets, films, exhibits, and evengroup discussions. It is a very useful tool foridentifying words or concepts that cause diffi-culty. In testing one poster on safe waters itwas found that some people interpreted theposter to mnean that "all waters should be pol-luted." Changes were clearly indicated.

MIembets of the American Public Health As-sociation Committee on Exhibits have had somevery interesting experiences in applying this

Vol. 70, No. 3, March 1955329870-53 6

309

quick and easy approach. Exhibit evaluatorsask each exhibitor to complete in about 25 wordsthis sentence: "The main purpose of this exhibitis . . ." Then they ask a number of membersto complete the sanme sentence while observingthe exhibit. In one instance all the observerswrote, "The main purpose of this exhibit is todemonstrate the various uses of isotopes."' Theexhibitor wrote, "The main purpose of this ex-hibit is to demonstrate various techniques fortesting laundry equipment."A somewhat similar approach can be used to

test whether the practice recommended makessense to the other person and whether he caneasily apply it in the manner intended. A Pub-lic Health Service psychologist is studyingpatient reactions to a low sodium diet booklet.As part of the test, he and his wife went on therecommended diet for several days. They cameout with new respect for the difficulties facedby the patient. They had considerable trouble,for example, in using the index, which had beenset up from the point of view of a nutritionist,to find such common food items as peanut butterand salad dressing.A Public Health Service nurse in developing

materials for the use of diabetics tried to carryout the exact process outlined in the instructionsfor injecting insulin. She found it physicallyimpossible to perform all the tasks alone asrecommended.Many health departments are now routinely

using tests to tell whether their written mate-rials will be understood by the groups for whichthey are prepared. The Dale-Chall and theFlesch tests of readability are easy to use. TheFlesch test requires only a count of the numberof syllables per 100 words and the average num-ber of words per sentence. With these twofacts it is possible to estimate the grade levelof reading ability required. After some train-inig, any clerical person can apply the testsuccessfully.For example, one health department, while

carrying out a mass X-ray campaign for tuber-culosis, received 3 or 4 calls a day from peoplewho asked for an explanation of the notices they

had received about the X-ray findings. Anianalysis of the reading ease of the notice showedthat about eighth grade reading ability was re-quired to understand the message; thus the mes-sage could be understood by only about half ofthe adult population of the community. Sinceit was important that everyone understand thismessage, it was simplified to about the fourthgrade level of reading ability; then it was triedout on a number of people with low educationto make sure that they interpreted it correctly.The health officer has since reported that nocalls for explanation are now received.Major emphasis has been given to the ap-

proach of the social scientist rather than tothat of the statistician, administrator, or per-sonnel officer. Actions taken in selecting andassigning personnel, for example, directly affectmost of the decisions discussed.One frequently hears that "public health is

people," that it is "everybody's business." Ifpublic health is so intimately concerned withpeople, then how well public health is succeed-ing in its programs may depend on how well ourprograms are wrapped around people-theirneeds, their wants, their hopes. The approachesI have been discussing are the types of ap-proaches which are useful in determiningwhether we are wrapping our programs aroundpeople or whether we are attempting to wrappeople around our programs..Through applying the tools of measurement,we can obtain data for the guidance of programimprovement as the program continues. Wemust always be aware, however, that the dataobtained in evaluating program progress arenot valid for determining whether or not thebroad goals of the program are achieved. Forthis purpose, sound studies of program achieve-ment are essential. Both evaluation of pro-gram progress and evaluation of programachievement are essential to satisfy the twomajor purposes of evaluation-guidance for theconduct of program activities and objectiveestimates of achievement. Neither can be neg-lected if we are to make certain that our effortsand money are well spent.

Public Health Reports310