Studies in Educational Evaluation 34 (2008) 194–200

What we learned from three evaluations that involved stakeholders

Jean A. King *, John C. Ehlert

Dept. of Educational Policy & Administration, 330 Wulling Hall, University of Minnesota, Minneapolis, MN 55455-0221, United States

A R T I C L E I N F O

Keywords:

Participatory evaluation

Stakeholder involvement

Evaluation constraints

A B S T R A C T

This study discusses three evaluations completed in a large suburban school district, each of which

involved stakeholders purposefully throughout the inquiry process. The reflective case narrative

addresses three questions: (1) To what extent or in which parts of the evaluations were stakeholders

involved? (2) What actions did the evaluator(s) take to encourage stakeholder involvement? and (3)

What difficulties were encountered in each evaluation, and what actions were taken to resolve them?

Looking across the evaluations, four lessons emerge. First, these experiences suggest that it may be easier

not to involve multiple stakeholders than to involve them. Second, once multiple stakeholders are

actively involved, the evaluator and client must be willing to handle the complexities of logistical

arrangements and the potential conflicts that may arise. Third, it takes skill to establish meaningful

interactive processes and structures that can involve multiple stakeholders over time. Finally, evaluators

must recognize that involving multiple stakeholders may require additional resources.

� 2008 Elsevier Ltd. All rights reserved.

Contents lists available at ScienceDirect

Studies in Educational Evaluation

journal homepage: www.e lsev ier .com/stueduc

Weiss and others who promulgated the stakeholder approachto evaluation envisioned an interactive process involving peoplewith a stake in a program (Bryk, 1983; Weiss, 1983). Just as thoseholding stock in a company get to vote on its management, theapproach suggests that people associated with a program – thosewho administer it, work for it, participate in it, or simply care aboutit – deserve opportunities to shape its evaluative inquiry. Applyingthis approach, those charged with developing an evaluation mustattend to a variety of people’s issues and concerns rather thandesigning and conducting studies separate from the experiences,perceptions, and desires of program stakeholders.

However, given the impossibility of including everyone related toor interested in a program, especially if one considers a nation’scitizenry is in some sense a stakeholder for every social program, twokinds of questions arise. The first set of questions relates to which

participants an evaluator should involve. Do clients always play aprivileged role with full veto power? Do sponsors? Should everyevaluative study create opportunities for those receiving services toshape the issues addressed? Must evaluators attend to broadframing of social issues, or is it acceptable to focus exclusively on thespecific program at hand? The second set of questions relates to how

people should participate. Is it appropriate to include participants inevery stage of an evaluation, even in technical activities like datacollection and analysis? Are certain participants granted larger rolesat certain steps of the process, for example, at the beginning and end,

* Corresponding author. Tel.: +1 612 626 1614; fax: +1 612 624 3377.

E-mail addresses: kingx004@umn.edu (J.A. King), ehle0087@umn.edu

(J.C. Ehlert).

0191-491X/$ – see front matter � 2008 Elsevier Ltd. All rights reserved.

doi:10.1016/j.stueduc.2008.10.003

whereas others may play small parts throughout? To what extentmust an evaluator control the inquiry process to ensure valid dataand meaningful recommendations based on them?

The present study examines three evaluations completed in alarge suburban school district, each of which included purposefulinvolvement of a sizeable number of stakeholders throughout theinquiry process. In each case, the evaluator framing and conductingthe inquiry systematically created interactions, starting withidentifying the issues the study would address and continuingthrough the development of data-based recommendations. Thisreflective case narrative addresses the following questions:

1. T

o what extent or in which parts of the three evaluations werestakeholders involved?

2. W

hat actions did the evaluator(s) take to encourage stakeholderinvolvement in the three studies?

3. W

hat difficulties were encountered in each evaluation, andwhat actions were taken to resolve those difficulties?

In retrospect, the results of interactions in these evaluationsprovide helpful lessons for evaluators interested in stakeholderinclusion. The paper has four sections: a brief grounding in theliterature and the study’s methods, a description of the districtcontext, details of the three evaluations, and discussion.

Conceptual grounding and methods

Since the burgeoning of the accountability movement in the1970s, American school districts have been responsible for

J.A. King, J.C. Ehlert / Studies in Educational Evaluation 34 (2008) 194–200 195

increasing amounts of standardized testing. In the past 30 years,the parallel development of the program evaluation function inmost districts has been largely overwhelmed (King, 2003). Theprocesses of developmental, formative, or summative programevaluation for decision making and program improvement haverarely been fully institutionalized in districts, despite thepurported benefits of doing so (Sanders, 2000). In these same 30years, program evaluation practice has evolved to include a rangeof acceptable alternatives, from evaluator-directed practice toincreasingly collaborative and participatory forms where theevaluator’s role becomes one of training and facilitation (Cousins,2003; Cousins & Whitmore, 1998). Simultaneously, the notion oforganizational learning, the process through which organizationsover time use data to improve themselves, has gained credibility(Cousins, Goh, Clark, & Lee, 2004; Preskill & Torres, 1999).

Two concepts ground the descriptions of the three evaluationsthat follow. The first of these concepts is participatory evaluationbecause each of these studies actively engaged participants frombeginning to end. Cousins and Whitmore’s (1998) oft-citedframework for participatory evaluation focuses on three processdimensions: (1) control of the evaluation process, (2) stakeholderselection for participation, and (3) depth of participation. To createindicators for these three dimensions, Daigneault and Jacob (2007)have recently operationalized the framework’s components,suggesting their value for systematic reflection and research.The second concept, organizational learning, is also important tothese evaluations, since an internal evaluator charged withdeveloping the district’s evaluation capacity (the first author)led them all. If evaluations lead to organizational learning, then aframework for understanding how ‘‘teaching’’ and ‘‘learning’’occurs is relevant. King (2008) has recast Schwab’s commonplacesof learning (1969) – student, teacher, curriculum, and milieu – asthe commonplaces of evaluative learning: potential users,evaluator, evaluation processes and results, and context. Inthinking about organizational learning that results from evalua-tions, then, it makes sense to examine the evaluator’s role asteacher, the users’ role as learners, the evaluative ‘‘curriculum,’’and the setting in which the learning takes place.

Within a single district setting, three criteria identifiedevaluations appropriate for the current study: (1) the evaluationwas participatory in nature, i.e., it purposefully involved multiplestakeholders, (2) it was completed between 1999 and 2004 so thatthe longer-term outcomes had had time to develop, and (3) asufficient number of participating individuals remained to beinterviewed. Three evaluations met these criteria: an implementa-tion study of state graduation standards, a mandatory review of thedistrict’s Special Education Department, and a holistic study ofmiddle school programming. We used two methods to collect data:interviews with 10 participants across the studies, including ineach case the evaluator, the client, and at least one otherparticipant, and extensive document analysis of related meetingnotes, reports, information from the district website, etc. For eachevaluation we prepared a detailed, month-by-month chronology.

District context

This paper presents part of a larger case study of a school districtevaluation system where administrators were keenly aware of theimportance of continuous improvement and sought to buildevaluation capacity purposefully (Ehlert & King, 2007). Given theresource strain and a Board context that made new administrativepositions unlikely, the option of expanding the assessmentdepartment with additional staff was simply not viable. Theevaluators – one full-time internal evaluator (a faculty member onleave from a university position) and, in one of the studies, two

additional external evaluators – sought instead to actively involveparents and district staff (central office staff, building adminis-trators, and teachers alike) in participatory evaluation activitiesrelated to programs in which they were involved, hoping, overtime, to make evaluation an integral part of every person’s job.

The three evaluations took place in Anoka-Hennepin ISD #11,now the largest district in Minnesota. It is a largely suburbandistrict with a student population of over 40,000, known for its lowfunding base, but – despite that – the traditional high quality of itsstudent outcomes. Unlike the state’s two urban districts, the 13communities that comprise this district do not face the challengesof an urban center. Overall, students do well on both state- andBoard-mandated tests in various subjects, and there is a long-standing district commitment to help students reach the max-imum of their potential. The current superintendent is fond ofsaying, ‘‘We will teach each student one at a time,’’ and it has beena Board goal that all children in the district will ‘‘read well andmeaningfully’’ by the end of third grade, regardless of what theyknow when they enter kindergarten. The percentage of free andreduced lunch students, while growing, remains comparativelysmall, and, although the district’s minority population has growndramatically in recent years, it is now only 8%. Eleven percent ofstudents receive special education services. The district is knownfor innovation and a long-time commitment to the use ofstandardized data for accountability and improvement. Yearsbefore the state adopted graduation testing, Anoka-Hennepinroutinely required students to pass such a test, and in the 1990sdistrict personnel were active in the development of performanceassessments and state graduation standards.

Using special state funding, the district hired a professor onleave from a nearby university (the first author) from 1999 to 2001to serve as the Coordinator of Research and Evaluation, thedistrict’s full-time internal evaluator. Her job was to conductprogram evaluations and simultaneously to build the capacity ofthe district’s 40 schools and its central office staff to engage in‘‘data-based decision making’’ apart from responsibility for thedistrict’s testing program (ordering tests, training proctors, troubleshooting, dealing with angry parents, etc.). In the years since then,central office administrators have continued to use developmentalevaluation processes, both through training existing staff andhiring external evaluators, to create a culture of ongoing inquiryaround instructional programming. Given the lean staffing incentral administration and continuing funding issues, this culturalchange remains a challenge.

Three evaluation studies with active stakeholder involvement

Applying the evaluative commonplaces to these reflective caseexamples documents that organizational learning was purposeful.The evaluator/teacher (the first author) and the context (a singledistrict) were constant across the three evaluations. There was alsosome overlap in the ‘‘learners,’’ i.e., the potential users (e.g., certainadministrators, the teachers’ union representative), and simila-rities, too, in the evaluation processes, which were in each caseparticipatory. The three studies will be presented in chronologicalorder because, over the period of 3 years, as will be shown, theinternal evaluator and her central office ‘‘client’’ (who was also herboss) actively incorporated lessons learned about stakeholderinvolvement from one study into the next.

The graduation standards study (1999–2001)

The purpose of this evaluation was twofold. First, an AssociateSuperintendent, the study’s dual sponsor and client – and theindividual to whom the evaluator reported – was responsible for

J.A. King, J.C. Ehlert / Studies in Educational Evaluation 34 (2008) 194–200196

complying with the state’s complex new graduation standards andwanted data from central office and building-level staff about thestatus of the implementation, for example, what issues teachersfaced as they integrated mandatory performance assessments intotheir classroom practice, how parents were reacting, and wherechallenges lay. Second, she saw the potential of a periodic, butongoing collaborative discussion of standards issues among thosewho would be key to successful implementation of the standardsover time. With the help of the newly hired internal evaluator, aparticipatory evaluation process would be the means to that end.Given the contentious nature of these graduation standards,imposed by the state and subject to continuing rumors about theirlikely demise, the timeline began as a single school year, butultimately extended to 2 years.

The structure of the participatory study was as follows:

1. T

he Associate Superintendent and the evaluator collaborated onevaluation design and implementation decisions. This districtleader, who held a doctorate and understood the requirementsof high quality evaluation, intuitively understood the notion ofprocess use (Patton, 1997) and wanted to use interactive,monthly meetings to help staff collectively improve theimplementation of a challenging state mandate. The evaluator,a specialist in participatory evaluation, was the AssociateSuperintendent’s resource to make this happen, both throughthe structure of the meetings and through the collection anddiscussion of standards-related data.

2. T

he 18-person Graduation Standards Implementation (Evaluation)

Committee (GSIC) included representation from principals atevery level (elementary, middle, and high school), the teachers’union, early childhood education, and curriculum specialists(who were also charged with facilitating the standardsimplementation). Given the evaluator’s newness to the district,the Associate Superintendent identified GSIC participants. Ledby her and supported by the evaluator, this group wasresponsible for framing the overarching evaluation questions,agreeing on how to collect data, reviewing the data oncecollected, and developing recommendations based on them.

Although staff was at the same time attending numeroussessions related to Graduation Standards implementation, theAssociate Superintendent held mandatory GSIC meetings atthe district office approximately once a month to ensure theCommittee’s involvement in the evaluation. People attendedbecause they were expected to; participation on the GSIC was apart of their jobs, although the presence and active involvement ofthe Associate Superintendent pointed to the importance of theCommittee’s work. Despite mandatory attendance, the evaluatorand Associate Superintendent paid careful attention, as wascustomary in the district, to making the meetings both pleasantand meaningful. Refreshments were always provided, the locationwas convenient, and interaction was purposefully structured sothat everyone’s input could be given and received. The evaluatorfacilitated discussion.

In the fall the GSIC sponsored a survey and follow-up focusgroups. In the spring, it collected or compiled extensive data usingmultiple methods: focus groups of teachers and students, surveysof administrators and building-level implementation teams, thecompilation of 9th/10th grade implementation data, a review andanalysis of so-called ‘‘Grad Standards [instructional] packages,’’and a summary of building-level implementation funding. Theevaluation’s culmination took place at the end of the school year(mid-June), when the GSIC sponsored three 1-day planningmeetings, one for each level (elementary, middle, and high school).A larger number of participants (roughly 30 at each) attended these

meetings and used the year’s evaluation data to plan for theupcoming year’s implementation, despite the state’s having placeda moratorium on standards requirements. Throughout thefollowing year, people discussed the data as the district continuedto implement the standards. As the Associate Superintendentnoted at that time, ‘‘We’ve always supported standards andaccountability. . . [I am] stating the obvious—we are supportinghigh standards for student achievement [regardless of what thestate requires].’’

The major difficulty encountered during the study did not relateto issues of involvement – people were reportedly pleased to takepart in a systematic effort to collect data and use it to create animplementation appropriate to the district – nor to technical issuesof data collection, analysis, or interpretation, which the evaluatorand the Assessment Office handled. Instead, people at all levels ofthe district were highly frustrated by the uncertainty surroundingthe state’s requirements and the fear that, by moving forward,teachers would be forced to again alter their practice dramaticallyonce the standards went away—which they ultimately did.However, the collaborative evaluation process had two clearbenefits: first, it provided a structure for people to process theircollective frustration over the year, including sharing rumors andangst; and second, it created a model for district-wide programevaluation that would prove useful in later studies.

The special education study (1999–2001)

Initiated 2 months after the Graduation Standards study, theSpecial Education evaluation was part of the district’s 6-yearcurriculum review cycle. Its process, however, was anything butroutine, in part owing to the highly political nature of specialeducation services in the district, but especially because arepresentative of the Minnesota Department of Education – inresponse to parental complaints to the State – attended allmeetings (see King & Fitzpatrick, 2008). At its initial meeting, theAssociate Superintendent suggested to the planning committee –later known as the Data Collection Team – that a participatoryapproach similar to that of the Graduation Standards study wouldappropriately involve multiple stakeholders and increase thedepartment’s transparency. The evaluator enthusiastically con-curred. The group then generated a list of 130 stakeholdercategories requiring representation. This number represents fourlevels (pre-school, elementary, middle, and high school) by 13disability categories by multiple roles (e.g., teacher, parents,community representative). Ultimately, 40–50 people routinelyattended monthly meetings in a process that extended over a yearand a half.

The self-study structure consisted of three teams, each of whichwas responsible for certain decisions about the evaluation:

1. T

he Evaluation Consulting Team (ECT). This team, which met asoften as was needed, included the district’s internal evaluatorand two ‘‘objective’’ outside evaluation consultants hired toguide the design and implementation of data collection andanalysis, as well as to facilitate the Self-Study Team meetingsand processes. The ECT made decisions about how to structuremonthly meetings and about technical issues related to datacollection (e.g., compiling data, checking for accuracy) andreporting (e.g., preparing graphs and tables, developing formalreports). Because the internal evaluator was in the office daily,she had primary responsibility for communicating with theAssociate Superintendent (this evaluation’s sponsor and client)about any concerns or potential roadblocks.

2. T

he Data Collection Team (DCT). Membership included theEvaluation Consulting Team, the Associate Superintendent (who

J.A. King, J.C. Ehlert / Studies in Educational Evaluation 34 (2008) 194–200 197

led the DCT), the Director and Assistant Director of SpecialEducation, three Special Education administrators, one SpecialEducation administrative intern, and the state departmentsupervisor. This team met twice a month and functioned as thecore planning group for the study. Its activities included revisingand approving monthly meeting agendas the ECT developed,handling non-data-related logistics, serving as a sounding boardfor the ECT, and providing feedback on the evaluation processover time.

3. T

he Self-Study Team (SST). This large group included the DataCollection Team plus a broad-based cross-section of stake-holders, including special and regular education teachers andadministrators, a union representative, paraprofessional staff,multicultural student advisors, parents, parent advocates, andservice agency representatives. It met monthly to review datacollection instruments, analyze and interpret data, identify keythemes, and develop the final data-based commendations andrecommendations. Among others, its decisions included whichtopics to include on data collection instruments, final say ondata collection instruments, interpretations of the data, and,eventually, which recommendations to forward to the Board.

The self-study process involved four key components: (1)process planning by the Data Collection Team, (2) data collectionfrom multiple sources using instruments developed by the ECT andDCT and approved by the Self Study Team, (3) data preparation bythe ECT, and (4) Self-Study Team data review, analysis, andinterpretation. Ultimately, the self-study used a variety ofquantitative and qualitative methods to collect data, includingsurveys, stakeholder dialogues, focus groups, e-mail question-naires, community input via an open forum and public hearing,telephone interviews, and observations.

Knowing the difficulty of getting non-school people to attendany meetings, the DCT scheduled the monthly Self-Study Teammeetings from 5:00 to 7:30 in the evening at a community collegestudent center away from the district offices, with both dinner andchildcare provided. Self-Study Team participants were organizedinto consistent work groups or table teams of 6–8 people, whichacted as the primary units for discussion. Building the capacity ofparticipants to contribute their perspectives actively and tosupport their engagement in data-based decision making wasan important part of the evaluative process. The EvaluationConsulting Team took turns leading parts of each session and usedseveral facilitation procedures to foster discussion, includinginitially organizing the table teams, each of which represented amixture of roles (parents, administrators, special and regulareducation teachers, special education administrators, etc.); assign-ing one special educator to act as a resource for each work group;structuring interactive cooperative learning activities to supportinteraction; and establishing feedback loops (e.g., monthlyevaluation forms) into meeting routines. The results of the study,contained in table notebooks that grew from monthly insertions,were a visible presence at meetings, and, even when attendancelagged, discussions were routinely lively.

The size and complexity of this evaluation led to severalchallenges for the ECT.

1. F

irst, conflict was a part of the study from the beginning. Notonly was the state represented at every DCT and Self-StudyTeam meeting, but some Self-Study Team participants saw theirrole as the loyal and highly vocal opposition to the district’sSpecial Education Department. Knowing this, each meetingbegan with the Associate Superintendent answering anyquestions raised on the preceding month’s evaluation forms.The results of the monthly evaluations were also typed up and

available for table review; people saw that negative as well aspositive comments were included and, to the extent possible,addressed. In addition, the table teams’ interactive processesguaranteed that participants had their chance to expresswhatever ideas they wanted to. The important requirementwas that the data presented be analyzed, rather than using anypre-conceived ideas not backed up in the data.

Further, the evaluators were a visible and smiling presence atthe monthly meetings, greeting people, speaking individuallywith those who appeared uncomfortable or who had personalissues to raise, and representing the data. For example,whenever new data were presented, the internal evaluatorasked people to review them and look for obvious errors (e.g., acolumn of numbers that was added incorrectly) before thegroups began work. All activities were structured using thesocial psychological principles of collaboration (Stevahn & King,2005) to ensure that people could actively express theiropinions, discuss the data in small groups, and see ongoingevidence of their input from month to month.

2. A

nother ECT challenge was that it frequently took longer than amonth to collect data and/or to prepare materials for Self-StudyTeam review, but, once set, meeting dates could not easily bechanged. This led to many long days and stressful nights for theECT and ultimately to an extended timeline and additionalconsulting expense for the study. Given its scope and thenumber of individuals involved, it simply could not becompleted in the year originally allotted.

3. T

he longer the study continued, the more attendance at themonthly Self-Study Team meetings became an issue. A sizeablecore group persisted, but many people, especially parents ofcolor, began to drop out or to miss meetings despite theincentives provided. Inconsistent attendance increased theECT’s challenges because participants truly needed to under-stand to collaborate meaningfully. The DCT, which included theAssociate Superintendent, was confident that there was noproblem, i.e., that the 40–50 people who did attend regularlyprovided sufficient representation. To be sure that all voiceswere adequately heard, the ECT insisted on collecting additionaldata from parents of color to confirm that they held similaropinions.

4. S

ome technical issues arose in asking non-evaluators to analyzeand interpret data and make recommendations based on them.The ECT addressed this by taking full responsibility forcompiling and editing as necessary the table teams’ monthlyresults, thus providing the quality control they believedessential for the eventual use of the data.

The lengthy and labor-intensive process of the evaluation led topositive outcomes for the district, including a formal report to theBoard of Education that evaluation representatives presented andthe eventual development of a state-approved system of con-tinuous improvement (Ehlert, King, & Shanker, in preparation). Themodel for such an inclusive and interactive evaluation was not,however, viable in the context of a district continually strapped forresources. The study was simply too expensive, took too long, andwas in the end too complex.

But it did yield one other important outcome: a structure andprocess for district-wide participatory evaluation. Seeing theresults of both the Grad Standards and the Special Educationstudies, the Associate Superintendent recognized the value ofpairing a smaller evaluation planning and implementation team(i.e., the internal evaluator and key central office and building-leveladministrators) with a larger discussion team that could meetregularly to collaborate on the study. The key was to apply themodel using existing resources. As the special education study was

J.A. King, J.C. Ehlert / Studies in Educational Evaluation 34 (2008) 194–200198

winding down, another potentially contentious evaluation projectemerged that the Associate Superintendent believed could benefitfrom interactive discussion to foster support and learning bothabout the subject – middle level education – and about theevaluation process. This became the internal evaluator’s newproject.

The middle school study (2000–2002)

Middle level education in the district had remained somewhatcontroversial since what some saw as the imposition of the middleschool model in the 1990s (Anoka-Hennepin, 1991a, 1991b).People disagreed, for example, about the value of mandatedadvisor–advisee sessions, exploratory classes, and a gradingsystem that eliminated plusses and minuses. Conducting anevaluation that would actively engage multiple stakeholders –teachers, administrators, parents, and even students – became animportant way a decade later to revisit that earlier decision. As wastrue in the two studies previously described, the stakeholders wereto guide the study from beginning to end, framing questions,reviewing data collection instruments, analyzing and interpretingdata, and making district-wide recommendations. Given thecontent and district philosophy, student involvement was anadded component.

The Middle School Study structure consisted of three teams:

1. T

1

Co

he Evaluation Planning Team (EPT). Much smaller than itsspecial education counterpart, this group included only theAssociate Superintendent, an instructional facilitator (a talentedmiddle school curriculum specialist), and the evaluator. Theevaluator had technical responsibility for planning and con-ducting the study with advice and consent from the two otherEPT members.

2. T

he Middle School Study Team (MSST).1 This was the larger groupof about 25 people that met monthly and included the EPT plusseveral instructional facilitators (curriculum leaders), middlelevel teachers, deans, and principals, the Community EducationManager, an Assessment Specialist, and the president of theteachers’ union. Because meetings were held during the day, thegroup decided that in addition to those parent representativesavailable at that time, it would solicit parental input throughstructured discussions of the data and issues during PTO (ParentTeacher Organization) meetings at the middle schools. Thisgroup’s responsibilities paralleled those of the Special EducationSelf-Study Team.

3. T

he Kids’ Study Team (KST). Seeking to model high quality middlelevel practice, the KST included students from each middleschool (grades 6–8), a diverse group representing differentabilities, races, ethnicities, SES, etc., selected by principals inconsultation with the MSST. Given meetings’ timing during theschool day, students did not participate in MSST meetings.

Building on the two earlier studies, the middle schoolevaluation used this team structure and an interactive meetingprocess to facilitate stakeholder involvement. In seeking to create acost-effective version of the special education study, the evaluatorfacilitated meetings and controlled the process, but not the contentof the study. Meetings were held during the school day whenprofessional staff was available. Refreshments were less elaborate,but the use of table teams and systematic interactions for framingquestions, finalizing data collection instruments, and, eventually,reviewing data and making recommendations were the same. At

Following the study’s completion, this group was renamed the Middle School

mmittee and continued to function for an additional 6 years.

MSST meetings people spent time with others in the district whomthey rarely saw, discussing middle school research and broadmiddle school issues across the district, and they reportedlyappreciated the interactions.

The length of the study expanded to allow time to collect andreview the necessary data through surveys, focus groups, and PTOdiscussion meetings. As a positive side effect, this seemed toincrease the buy-in of participants who saw that their work couldbe both used and useful district-wide. In addition, having time tothink about the evaluation appeared to help some stakeholdersassimilate both the evaluation process and its eventual outcomes.The evaluator facilitated the process of thinking like an evaluatorby teaching MSST members how to conduct focus groups, analyzedata, and facilitate small groups (at the PTO meetings). The MiddleSchool study needed to address the controversy over gradingpolicy for the middle schools publicly. The group specificallywanted to address the issue of using plusses and minuses with theletter grades. At the completion of the evaluation, the MSST createda task force consisting of representatives from each of the middleschools to address the grading issue.

While separate from the MSST, the Kids’ Study Team followed asimilar interactive process with support from the evaluator andtwo central office curriculum leaders. Twice in 1 year, a diversegroup of students came together off-site for a full day, facilitatedby the evaluator, to learn about evaluating middle schooleducation and to reflect on their experiences as middle schoolstudents in the district. In between, they helped collect data fromtheir peers. The KST first wrote questions then clarified them withstudents at their home schools prior to developing the instru-ments. They were ultimately involved in all parts of theevaluation, including framing questions, developing and con-ducting student surveys and focus groups, then, at the second all-day meeting, analyzing and interpreting data and developingrecommendations that were formally presented to the MSST andincluded in its deliberations.

Not surprisingly, the middle school study generated challengesfor the EPT.

1. S

election of team members. Given the contentious nature withwhich some teachers viewed the district’s middle gradesphilosophy, the ECT paid close attention to the selection ofMSST members. The Associate Superintendent, the evaluator’sboss and key assessor of the political climate in the district,believed it was important to include people who were visibleopponents of the status quo, but not so many that the groupwould be unable to function. With the exception of oneindividual who routinely caused minor disruptions at meetings,the other members of the team not only came together, but werevisibly engaged in discussions over the course of the study thatgenerated acceptable results for the group.

2. P

reparing materials. As was true during the Special Educationstudy, the turn-around time for collecting and processing data(i.e., data preparation, the copying of materials) put theAssessment Office’s support staff in an impossible time bind.The meeting dates were not alterable so staff simply pitched inand got things done, a major challenge especially during periodsof the district’s standardized testing.

3. D

istrust of student involvement. Certain principals displayedlittle confidence in the students’ abilities to participate in andconduct an evaluation. They especially did not want lowerability students to take part or to have any students conductfocus groups with their peers. With the full support of theAssociate Superintendent, the EPT insisted that the studentsbe allowed to participate equally in the process, and theyacquitted themselves well. Given competing demands on

J.A. King, J.C. Ehlert / Studies in Educational Evaluation 34 (2008) 194–200 199

people’s time, however, the KST did not continue beyond itsinaugural year.

Discussion

The eventual outcomes of these three evaluations documenthow people used the results in future decisions: The results of theGraduation Standards study were directly applied to planning forthe following school year’s implementation; the Special Educationstudy resulted in a state-approved system of continuous improve-ment that continues to the present day; and the Middle SchoolTask Force took on and eventually resolved grading issues in themiddle schools. The outcomes also document the creation ofstructures and processes that helped build the capacity of thisdistrict to conduct and use evaluation. Looking across the studies,four lessons emerge. First, these experiences suggest that it may beeasier not to involve multiple stakeholders than to involve them.The decision to involve many people must be made purposefully, inlight of the resources required for meaningful involvement overtime. This is especially true since an evaluator can be responsive tostakeholders (e.g., by interviewing key individuals and primaryintended users or by conducting targeted focus groups) withoutextensively engaging them in the evaluation process over time.

Second, once the decision is made to involve multiplestakeholders actively, the evaluator and client must be willingto handle the complexities of logistical arrangements and thepotential conflicts that may arise. The Cousins and Whitmore(1998) framework raises three fundamental questions:

1. W

ho has the ultimate control of the evaluation process? Interms of the interactive continuum (King & Stevahn, 2002), thethree evaluations discussed in this paper ranged from anevaluator-directed study (Graduation Standards) to a collabora-tive study (Middle School) to a participant-directed study(Special Education where the DCT and SST were in charge). It isimportant to specify who is in charge of what early on, to reviewthis routinely, and to hold people accountable for theircommitments.

2. W

ho will take part? The key actors – whatever that means in agiven context – must participate if the process and its results areto be meaningful. In these studies, the evaluator in collaborationwith her Associate Superintendent client used a matrix of roleand/or content to select stakeholders. All three studiespurposely included naysayers, which made facilitation difficultin both the Special Education SST and the MSST. The conflict thatdeveloped when certain middle school principals did not wantto include students they perceived as troublemakers required astrong stand that could have derailed the process.

3. W

hat are people being asked to do? One thing is clear: Thelonger the evaluation process takes, the more likely it is thatstakeholders who are participating voluntarily (e.g., parents inthe Special Ed study) will drop out. Identifying specific roles,timelines, and task descriptions will enable people to makeinformed decisions about whether or not they become involvedand whether their involvement will be a positive experience.

Third, it takes skill to establish meaningful interactive processesand structures that can involve multiple stakeholders over time.These evaluations suggest that the logistics can be complex,especially when people meet infrequently over an extendednumber of months. Fortunately, the principles of social psychologyinclude concepts like social interdependence and conflict resolu-tion that can be operationalized in participatory studies. Forexample, the external threat of the state’s impending changes tothe Graduation Standards brought people together with a shared

goal: to make sense of what was happening and to build on thestandards implementation regardless of what the state decided. Inboth the Special Education and the Middle School evaluations, theidea of having public discussion of data created transparency tomove both programs forward.

These evaluations point to two key evaluator roles: (1) theteacher or kindly task master who breaks down the process intobite-size chunks so people will not feel overwhelmed, and (2) thetechnical expert who provides quality control and allows notechnical errors. Evaluators engaging multiple stakeholders dowell to think about what is in it for each individual, knowing thatpeople who are not truly motivated to participate may block theprocess. These studies suggest that the more you are able tostructure involvement into ongoing routines, the better; this bothfacilitates participation and fosters the eventual use of the data.Some may worry that this ultimately blurs the line betweenevaluation use and planning, but by involving multiple people andintegrating activities into ongoing work, an evaluation becomes adevelopmental process akin to action research (King, in press). Thebest case scenario is gathering a group of individuals who bothtruly care about the evaluation and are open to making collectivesense of the data. Evaluators should be forewarned, however, thatwhen people are interested, they may have firm beliefs apart fromthe data that could lead to conflict if they are not open-minded(e.g., some parents in the Special Ed study).

Finally, evaluators must recognize that additional resourcesmay be required to involve multiple stakeholders. Such processesare more expensive for two reasons: (1) you may need to pay toguarantee that materials are prepared on time, that copies areavailable for all participants, that appropriate incentives likerefreshments and childcare are provided, that meeting rooms arerented, and so on; and (2) interaction takes time, and time is often alimited resource in the life of organizations. For those who believethat involving people in evaluations is a way to save money onstudies, beware the Rumplestiltskin effect. The very act ofconducting a sound evaluation on limited resources may leadothers to underestimate exactly how much work went into theprocess.

References

Anoka-Hennepin ISD #11 (1991). Implementation report of the middle school keycharacteristics and philosophy. Coon Rapids, MN: Anoka-Hennepin ISD #11.

Anoka-Hennepin ISD #11 (1991). Middle school transition in the Anoka-Hennepin schooldistrict. Coon Rapids, MN: Anoka-Hennepin ISD #11.

Bryk, A.S., (Vol. Ed.) (1983). Stakeholder-based evaluation. New Directions for programevaluation. 17. San Francisco: Jossey-Bass.

Cousins, J. B. (2003). Use effects of participatory evaluation. In T. Kellaghan & D. L.Stufflebeam (Eds.), International handbook on educational evaluation (pp. 245–266).Boston: Klewer Academic Publishers.

Cousins, J. B., Goh, S., Clark, S., & Lee, L. (2004). Integrating evaluative inquiry into theorganizational culture: A review and synthesis of the knowledge. Canadian Journalof Program Evaluation, 19(2), 99–141.

Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New directionsfor evaluation. San Francisco: Jossey-Bass.

Daigneault, P. M. & Jacob, S. (November, 2007). Rethinking participatory evaluation’sconceptualization: Toward the development of a ‘‘full-blown’’, useful concept.Paper presented at the annual meeting of the American Evaluation Association.Baltimore, MD.

Ehlert, J. C., & King, J. A. (November, 2007). Case studies of evaluation use and influencein a school district. Paper presented at the annual meeting of the American EvaluationAssociation. Baltimore, MD.

Ehlert, J. C., King, J. A., & Shanker, V. (in preparation). Case studies of evaluation use andinfluence in a school district. Minneapolis, MN: University of Minnesota.

King, J. A. (2003). Evaluating educational programs and projects in the USA. In T.Kellaghan & D. L. Stufflebeam (Eds.), International handbook on educational evalua-tion (pp. 721–732). Dordrecht: Kluwer Academic Publishers.

King, J. A. (2008). Bringing evaluative inquiry to life. American Journal of Evaluation,29(2), 151–155.

King, J. A., & Fitzpatrick, J. L. (2008). Dialogue with Jean A. King: Evaluation of the specialeducation program at the Anoka-Hennepin School District #11. In J. L. Fitzpatrick,

J.A. King, J.C. Ehlert / Studies in Educational Evaluation 34 (2008) 194–200200

C. Christie, & M. Mark (Eds.), Evaluation in action: Interviews with expert evaluators.Thousand Oaks, CA: Sage.

King, J. A., & Stevahn, L. (2002). Three frameworks for considering evaluator role. In K. E.Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 1–16).Greenwich, CT: Information Age Publishing.

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.Preskill, H. S., & Torres, R. T. (1999). Evaluative inquiry for learning in organizations.

Thousand Oaks, CA: Sage Publications.

Sanders, J. R. (April, 2000). Strengthening evaluation use in public education. Paperpresented at the annual meeting of the American Educational Research Association.New Orleans, LA.

Schwab, J. (1969). The practical: A language for curriculum. School Review, 78(1), 1–23.Stevahn, L., & King, J. A. (2005). Managing conflict constructively in program evaluation.

Evaluation, 11(4), 415–427.Weiss, C. H. (1983). The stakeholder approach to evaluation: Origins and promise. New

Directions for Program Evaluation, 17, 3–14.