10
Mixed-methods design for an objective-based evaluation of a magnet school assistance project Mingchu Luo, Leon Dappen * Department of Educational Administration and Supervision, College of Education, University of Nebraska at Omaha, Kayser Hall 414, 6001 Dodge Street, 68182 0162 Omaha, NE, USA Received 20 March 2003; received in revised form 3 November 2003; accepted 17 December 2003 Abstract This article describes and discusses the development and implementation of an evaluation design to assess a magnet assistance project, which integrates a mixed-methods approach into the objective-based evaluation. By combining both quantitative and qualitative methods, this design can address some of the weaknesses of objective-based approach in the process of evaluation. Functions of mixed-methods approach, such as initiation, triangulation, complementarity and development can possibly contribute to improving program progress, avoiding information narrowness and uncovering side effects. The examination of this design also reveals that the mixed-methods approach is not only possible, but more effective, and has higher validity. The mixed-methods is a more useful and accountable approach, which can be used in integration with the traditional objective-based approach to conceive and implement evaluation, especially in program evaluations with broader audiences, longer terms, and more complex goals. q 2004 Elsevier Ltd. All rights reserved. Keywords: Evaluation design; Mixed-methods approach; Objective-based evaluation; Magnet assistance project; Formative evaluation; Enabling objective 1. Introduction Evaluation is widely used to examine and judge the worth, merits and shortcomings of various educational programs through collecting and analyzing data. It serves as an important instrument in the improvement process of educational quality. Because of the complexity of the evaluation objects and the evaluators’ different philosophi- cal views of evaluation, the topic of evaluation approach has attained critical status in both research and practice. Worthen, Sander, and Fitzpatrick (1997) described six approaches of evaluation. Stufflemean (2001) identified 21 approaches often employed to evaluation programs by categorizing them into pseudoevaluations, questions- and methods-oriented evaluation approaches, improve/account- ability approaches, and social agenda/advocacy approaches. Objective-based and mixed-methods were included in the category of questions- and methods-oriented evaluation approaches (quasi-evaluation studies). The education evaluation field has increasingly recog- nized the benefits of using the views of system, multiplicity, and integration in both research and practice. The mixed- methods approach, which is broadly used and discussed, technically represents this trend. This article describes and discusses the development and implementation of an evaluation design to assess a magnet assistance project with a strong emphasis on methodological issues. This design integrates mixed-methods, an intensely-discussed topic over the past decade, into the classic and prevalent objective-based evaluation. 2. Theoretical review of the two evaluation approaches 2.1. Objective-based approach The objective-based approach specifies the purpose of educational programs and determines if or to what extent, 0149-7189/$ - see front matter q 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.evalprogplan.2003.12.001 Evaluation and Program Planning 28 (2005) 109–118 www.elsevier.com/locate/evalprogplan * Corresponding author. Tel.: C1 402 554 3485; fax: C1 402 554 2722. E-mail address: [email protected] (L. Dappen).

Mixed-methods design for an objective-based evaluation of a magnet school assistance project

Embed Size (px)

Citation preview

Page 1: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

Mixed-methods design for an objective-based evaluation

of a magnet school assistance project

Mingchu Luo, Leon Dappen*

Department of Educational Administration and Supervision, College of Education, University of Nebraska at Omaha, Kayser Hall 414,

6001 Dodge Street, 68182 0162 Omaha, NE, USA

Received 20 March 2003; received in revised form 3 November 2003; accepted 17 December 2003

Abstract

This article describes and discusses the development and implementation of an evaluation design to assess a magnet assistance project,

which integrates a mixed-methods approach into the objective-based evaluation. By combining both quantitative and qualitative methods,

this design can address some of the weaknesses of objective-based approach in the process of evaluation. Functions of mixed-methods

approach, such as initiation, triangulation, complementarity and development can possibly contribute to improving program progress,

avoiding information narrowness and uncovering side effects. The examination of this design also reveals that the mixed-methods approach

is not only possible, but more effective, and has higher validity. The mixed-methods is a more useful and accountable approach, which can be

used in integration with the traditional objective-based approach to conceive and implement evaluation, especially in program evaluations

with broader audiences, longer terms, and more complex goals.

q 2004 Elsevier Ltd. All rights reserved.

Keywords: Evaluation design; Mixed-methods approach; Objective-based evaluation; Magnet assistance project; Formative evaluation; Enabling objective

1. Introduction

Evaluation is widely used to examine and judge the

worth, merits and shortcomings of various educational

programs through collecting and analyzing data. It serves as

an important instrument in the improvement process of

educational quality. Because of the complexity of the

evaluation objects and the evaluators’ different philosophi-

cal views of evaluation, the topic of evaluation approach has

attained critical status in both research and practice.

Worthen, Sander, and Fitzpatrick (1997) described six

approaches of evaluation. Stufflemean (2001) identified 21

approaches often employed to evaluation programs by

categorizing them into pseudoevaluations, questions- and

methods-oriented evaluation approaches, improve/account-

ability approaches, and social agenda/advocacy approaches.

Objective-based and mixed-methods were included in

0149-7189/$ - see front matter q 2004 Elsevier Ltd. All rights reserved.

doi:10.1016/j.evalprogplan.2003.12.001

* Corresponding author. Tel.: C1 402 554 3485; fax: C1 402 554 2722.

E-mail address: [email protected] (L. Dappen).

the category of questions- and methods-oriented evaluation

approaches (quasi-evaluation studies).

The education evaluation field has increasingly recog-

nized the benefits of using the views of system, multiplicity,

and integration in both research and practice. The mixed-

methods approach, which is broadly used and discussed,

technically represents this trend. This article describes and

discusses the development and implementation of an

evaluation design to assess a magnet assistance project

with a strong emphasis on methodological issues. This

design integrates mixed-methods, an intensely-discussed

topic over the past decade, into the classic and prevalent

objective-based evaluation.

2. Theoretical review of the two evaluation approaches

2.1. Objective-based approach

The objective-based approach specifies the purpose of

educational programs and determines if or to what extent,

Evaluation and Program Planning 28 (2005) 109–118

www.elsevier.com/locate/evalprogplan

Page 2: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118110

these objectives have been attained. Its procedure of having

objective achievement judge the extent of success or failure

is straightforward. Its purposes of justifying improvements,

maintenance, and termination of program activity are

practical. For these reasons, the objective-based approach

has dominated the thinking and development of evaluation

since the 1930s, both in the United States and elsewhere

(Madaus & Stufflebeam, 1989). Evaluation researchers and

practitioners have been contributing their efforts in design-

ing, redesigning and improving this approach.

Ralph Tyler, regarded as the pioneer in the objective-

based approach viewed evaluation as the process of

determining the extent to which the objectives of a program

are actually attained. He emphasized the use of filtering

goals and objectives based on the rationale of being logical,

scientifically acceptable, and readily adoptable by evalua-

tors (Worthen et al., 1997). Metfessel and Michael (1967)

proposed an eight-step model to enrich the Tylerian

approach by adding more pragmatic ideas such as involving

stakeholders as facilitators, carrying out periodic obser-

vations, and developing recommendations for the further

implementation, modification, and revision of broad goals

and specific objectives. Provus (1971) used the Tylerian

tradition in designing the Discrepancy Evaluation Model,

which was based on the rationale that evaluation is a

continuous information-management process designed to

serve as the watchdog of program management and the

handmaiden of administration on the management of

program development through sound decision-making.

He also used new terms such as enabling objective (student

achievement acquired during the program to ensure the

accomplishment of the major program objectives), terminal

objective (immediate outcomes) and ultimate objective

(long-term outcomes) in the stages of process and product.

Hammond (1973) developed a cube model by applying a

three-dimensional framework for analyzing a community-

based youth program. Stufflemean (2001) and Worthen et al.

(1997) also discussed the developmental variations of the

Tylerian evaluation model.

Although the objective-based approach has been exten-

sively used in educational programs, it is commonly

criticized because of its simplicity in operation, emphasis

on defining outcomes, and focused use of behavioral

objectives. Stufflemean (2001) described four major aspects

being criticized: ‘leading to terminal information that is

neither timely nor pertinent to improve a program’s

progress; providing narrow information to judge the merits

and worth of the program; being unable to uncover the

positive and negative side effects; and crediting unworthy

objects’ (p. 18).

2.2. Mixed-methods approach

Evaluation theory and practice today are characteristi-

cally pluralistic, embracing diverse perspectives, methods,

data, and values within and across studies that aim to

generate more insightful and meaningful evaluative claims

(Caracelli & Greene, 1997). The wide use of the mixed-

methods approach stems from the perception that it is

particularly necessary to apply multiple ways of knowing

and acting in evaluation because educational problems are

increasingly complex and intractable. By looking at

educational phenomenon through both quantitative and

qualitative lenses, mixed-methods approach is intended to

ensure dependable feedback on a wide range of questions;

benefit in in-depth understanding of the programs; display a

holistic perspective; and enhance the validity, reliability and

usefulness of the full set of findings (Stufflemean, 2001).

Evaluation designs using mixed-method research (Brewer

& Hunter, 1989; Mark & Shotland, 1987), and specifically

combining qualitative and quantitative methods (Cook &

Reichardt, 1979; Crewell, 1994; Reichardt & Rallis, 1994),

have become a part of methodological guidance for the

practitioner (Caracelli & Greene, 1997). ‘In this troubled

era, with social problems of ever-increasing complexity and

intractability, multiple ways of knowing and acting are

surely needed’ (Greene & Caracelli, 1997, p. 15).

Advantages of using mixed-methods are that they

complement each other in ways that are important to the

evaluation audiences (Stufflemean, 2001). Compared to

single approach designs, mixed methods research can

answer questions in a better way, provide stronger

inferences, and provide opportunity for presenting a greater

diversity of divergent views (Teddlie & Tashakkori, 2003).

Quantitative methods provide relatively standardized,

efficient, amenable information, which can be easily

summarized and analyzed. Qualitative methods add con-

textual and cultural dimensions, which deepens the study by

providing more natural information. Quantitative research is

more directed at theory verification while qualitative

research tends to be more concerned with theory generation

(Punch, 1998).

3. Context and description of program

Since 1970s, magnet schools have been developed in

more and more school systems. Magnet programs are often

designed to promote racial diversity, expand educational

choices, and improve student achievement by providing a

unique or specialized curriculum or approach. Many of

them also enhance parent/community involvement, prepare

students for further education and/or careers in the world of

work, provide field-based and hands-on learning experi-

ences, and offer mentorship, internship, and apprenticeship

opportunities. Dentler (1991) identified that a magnet school

has the following four characteristics: a distinctive curricu-

lum based on a special theme or instructional method; a

unique district role and purpose for voluntary desegrega-

tion; voluntary school choice by the student and the parent,

with variable criteria established for inclusion; and access to

Page 3: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118 111

students beyond an attendance zone or single subdivision of

a district.

The program being discussed in this article is a three-

year Magnet Assistance Project begun in an urban school

district in the Midwest in 2001. The program is being

implemented in three elementary schools, which are

located in the comparatively low socioeconomic areas.

The magnet school assistance program was designed to

meet four objectives, which are: to increase the number

of non-minority students in each of the magnet schools;

to use state approved content standards in instruction and

use reliable assessment for assessing and rating students;

to offer a challenging curriculum; and to ensure that

magnet school participants will show equal or higher

achievement in core subjects as compared to a control

group. Each of the three elementary schools is imple-

menting their marketing/recruiting plans and strategies to

recruit non-minority students from their respective eight

feeder schools. Teachers are involved in the project

through professional development in using standards as

the basis for instruction, assessment, and rating student

performance. Each school communicates with parents

and its community in a variety of ways following each

school’s communication plan. An advisory committee

was established in each school to oversee the implemen-

tation of innovative, exiting, and challenging curriculum.

Standardized norm-referenced and district developed

criterion-based tests are used for assessing student

achievement.

4. Methodology for evaluating the objectives

The evaluation of the Magnet Assistance Project, which

lasts for 3 years, was designed to determine if, or to what

extent, the program’s objectives have been attained. More

importantly, a key purpose of the evaluation was to correct,

revise, and improve the program over the course of its

implementation. A set of enabling objectives is included in

the project proposal for the purpose of accomplishing the

objectives. The mixed-methods approach was integrated

into the process of evaluating grant objectives. For each

objective, data are collected periodically—some three times

annually, some annually, and some twice over the course of

the project. The descriptions of the evaluation activities and

the results of these activities associate with each of the

objectives are summative in nature. There is also a

formative evaluation that is intended to provide periodic

data to assist in making mid-course corrections as the

project is underway. The formative evaluation activities

provide information related to three of the four project

objectives (Objectives I, II, and III). The following sections,

respectively, correspond to an element of the evaluation

plan and discuss how the four objectives are evaluated.

4.1. Objective I

Objective I is to increase the number of non-minority

students in each of the magnet schools. When the magnet

school program is implemented at the three elementary

schools, each school will increase a specific number of non-

minority students in its enrollment at each magnet grade

level. Non-minority students will enroll in the magnet

school by choice without increasing minority isolation in

any of its feeder schools and with minority/non-minority

assignment ratios in each classroom enrolling opting

students consistent with district enrollment for that grade

level.

Evidence of the extent to which this objective is attained

is based on the quantitative enrollment data collected by the

school district and provided to the evaluators. In each of the

three years, analyses using enrollment data are implemented.

Data collected and analyzed include the district enrollment

by minority and non-minority students for the grade levels,

and the comparison of actual and projected minority

enrollment number and percent of minority students in

each magnet school. In addition to this, data were collected

for impact analysis of recruiting for the magnet schools on

the feeder schools; and the distribution data of minority

students in classes are compared to the distribution of

minority students for each grade for the entire school.

Comparison both among the three schools and among the

three years is the primary method to analyze all the data. The

comparison of actual and projected minority enrollment

numbers shows the extent of meeting non-minority enroll-

ment projections in each of the three schools. The impact of

recruiting for the magnet schools on the feeder schools

displays the data of students recruited from each of

respective eight feeder schools and extent of the feeder

school’s impact on the magnet schools. Analysis of the

distribution of students in classes and in schools for each

grade helps us to examine deeply to the classroom level, and

to determine which grade the recruitment targets can be

focused on to increase the non-minority enrollment.

“The information gained from an objective-oriented

evaluation can be used to reformulate the purposes of the

activity, the activity itself, or the assessment procedures and

devices used to determine the achievement of purposes”

(Worthen, et al., 1997, p. 81). By integrating the qualitative

approach, this objective is also examined as part of the

formative evaluation process that looks at each school’s

recruitment plan and determines periodically (three times

annually) if each school is on schedule in terms of its plan.

The purposes of this qualitative evaluation are to determine

whether the activities included in the plan are accomplished;

estimate the value of project activities in reaching the

longer-term project objectives; and determine whether the

enabling objective (to recruit students) of Objective I are

being met for each school and the district. Strategic steps for

adjustment, revisions and improvement are recommended

based on the evaluation results.

Page 4: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118112

A visiting team of three experienced professors who are

former public school administrators and teachers is

conducting the formative evaluation. One team member is

responsible for evaluating one school. They make periodic

visits to the schools involved in the project to examine the

existence of a marketing/recruitment plan, and to determine

if the plan is on schedule regarding implementation of

activities. Each school develops a school portfolio to

provide information to assist in the formative evaluation.

The school portfolio includes: (a) a marketing plan for

recruiting students into the magnet program, (b) minutes of

meetings from appropriate committees, and (c) a communi-

cation plan for communicating with parents and others in

the community.

4.2. Objective II

Objective II is to use state approved content standards in

instruction and to use reliable assessments for assessing and

rating students. Each magnet school uses state approved

content standards and reliable assessment instruments to

evaluate and report student achievement in core subjects.

Magnet schools work with the support of the school district,

the community, and the staff development program to

enhance capacity and longevity. These are judged by

reviewing (1) the results produced with the application of

a set of criteria designed to determine the use and effect of

standards; (2) the records of staff completion of staff

development projects; and (3) a survey of staff, parents and

community members.

Formative evaluation is the focus in evaluating this

objective. Its key purposes are to estimate the value of the

project activities in reaching the long-term project activities

and to determine if enabling objectives are being met for

each school and the district. The enabling objectives are to

use content standards for instructional planning and

reporting, to communicate the standards effectively, and

to conduct successful professional development. The

formative evaluation in this Objective as well as in

Objective I and III was designed to answer the following

questions: (1) are the activities described in the project

being completed and in a timely manner? (2) are the

enabling objectives being met? (3) will continued

implementation of the activities reported make a substantial

contribution to meeting the enabling objectives? (4) what

additional steps could be taken in improving the accom-

plishment of the enabling objectives? The following

methods are integrated to evaluate this objective.

4.2.1. Observation and interview

The visiting team is also empowered to conduct the

evaluation part of this objective. They make periodic visits

(at least three times annually) to each of the three schools

involved in the project. During each visit, a sample of

teachers’ classroom teaching is observed in the specific

content standards. These same teachers are also interviewed

briefly to clarify what they are doing relative to using

content standards in their instruction.

Checklist. In order to aid the visiting team members in

structuring classroom teacher observations’ and interviews,

the district, in cooperation with the evaluators, developed

and validated a checklist (see Appendix). It is a set of

criteria designed to determine the use and effect of standards

in classroom practices, specifically is used to assess the

extent that teachers are using standards as the basis for

instruction, assessment, and rating student performance.

The checklist has three sections: one corresponding to the

use of standards in instruction, a second corresponding

to the use of standards in assessment, and the third related to

the use of standards in rating students performance. This

checklist is the basis for this aspect of the formative

evaluation. The teacher’s progress in accomplishing each of

the indicators will be indicated by the evaluators in three

circumstances: ‘Yes’ representing accomplished, ‘No’

representing not accomplished, and ‘Not Sure’ representing

uncertain.

The visiting team members complete the checklist form

for one-third of the teachers at each visit so that all

participating teachers will be observed over the course of

the year. Thus, about one-third of the teachers are observed

at any one visit. Each year School District Central Office

Curriculum Consultants complete the checklist form twice,

once near the beginning of the year and again near the end of

the year. The first and last set of forms completed by the

visiting team member is compared for consistency to the

forms completed by the District Curriculum Consultants.

The results of this formative evaluation are described in

detail in a series of Interim Reports that are provided to the

school district following each of the three external visits.

The Interim Report includes a report for each of the three

schools and a yearly summary. Accomplishment is

determined by counting the number of ‘Yes’ checks on

the checklist for each teacher within each school and

computing the percentage of ‘Yes’s’ checked in comparison

to the number of possible ‘Yes’ checks. Academic area

specialists as well as regular classroom teachers are also

observed. The checklist is also applied to their classroom

activities. Thus, for each school there are data for each

regular classroom teacher and the specialists. The percent of

‘Yes’ checks on the ‘Using Content Standards Checklist’

and the target percentages for each school across all three

visits during the school year are compared. All these data

are presented separately in the three categories of

compliances using standards in instruction, in assessment,

and in rating student performance for each school.

Another checklist was designed to evaluate all of the

enabling objectives of Objective I, II and III. It is a summary

checklist asking the visiting team members to indicate the

extent that each of the enabling objectives is being met at

the time of visit.

The four open-ended questions asked in the formative

evaluation are also attached to the checklist. The answers to

Page 5: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118 113

these questions are based on the summary of the combined

data from the classroom observations, the teacher interviews

and the portfolio examination.

Student portfolio. One mechanism for facilitating using

standards in assessing students is the development of a

student portfolio that is used at all three schools. All three

schools developed standards-based portfolios for the

purpose of monitoring student progress on content standards

in the magnet focus areas. The portfolio was designed by

teachers with the assistance of a consultant to fulfill the

requirements as specified on the standards checklist. The

portfolio is used to collect student artifacts for various

standards as evidence regarding student achievement.

Portfolio examination is also part of the visiting team’s

work in evaluating the assurance of quality assessment for

all the standards. The visiting team randomly chooses three

to five student portfolios when visiting classrooms, and

reviews presence of student artifacts representing relevant

indicators of the checklist.

4.2.2. Professional development evaluation

Professional development activities are planned to insure

that teachers are prepared to use standards in instruction,

assessment, and rating students in the content areas that are

the focus of the project. It is regarded as a component of

Objective II that the extent that professional development is

occurring as planned, and the extent that the professional

development is successful. There were eight professional

development activities for teachers to effectively learn and

use the content standards in instruction, and reliable

assessments for assessing and rating students at the first

year of the project. Example workshops are the 5E

(engagement, exploration, explanation, extension, and

evaluation) inquiry learning, authentic assessment, coop-

erative learning, high-level questioning, and action

research. The evaluation of these professional development

activities as they occur is another strategy of formative

evaluation.

To evaluate the professional development, an evaluation

form reflecting the overall process and outcomes the

professional development activities was completed at the

end of each professional development activity. An overall

rating scale (1–4) was used for teachers to evaluate each of

the training programs in the following areas: the importance

of professional development activity to the magnet school’s

theme areas, meeting professional needs and expectations,

focusing on helping meet student needs, scheduling of

training, use of technology, and recommending the district

to repeat the training or not. The rating mean was used to

analyze the findings. In addition to this, teachers were also

asked to write down their comments on the training and

suggestions for improving the professional development

activities.

The impact of the professional development in terms of

continuance and focus on long-term change issues are also

examined. This occurs twice during the project, at 18 and 30

months. It is accomplished by forming a committee of staff

and members of advisory groups, selected by the external

program evaluators. The purpose of this committee is to

estimate the success of the professional development

activities related to the project.

4.2.3. Survey and focus group interview

In addition to observing and interviewing teachers, there

is a survey of staff, parents and community partners

associated with each school in the second and third year

of the project. The survey was designed with items

answering the following questions: what changes have

they heard about in the magnet content areas for that school,

what have students talked about regarding their school’s

instruction, what different activities students are involved in

at their schools, what is talked about at parent-teacher

conference, and any concerns or questions they have

regarding the new standards.

All the staff members in the three schools participate in

the survey. In each school approximately one half of the

parents were asked about one of the content area, and the

other half were asked about the other content area. A group

of school-community partners are randomly selected and

surveyed. Two similar surveys are conducted in the second

and the third year. The purpose of the second year survey is

to see the staff members’ and parents’ general perceptions

about using state approved content standards in instruction

and using reliable assessments for assessing and rating

students. The data of interest in the third year survey is the

difference in survey results between the second year and the

third year.

Following each of the two surveys, three parents at each

school who respectively represent three categories of the

high scorers, median scorers and low scorers on the survey

were randomly selected as a focus group for interview. This

interview was designed for the purpose of collecting and

analyzing the detailed and in-depth reflections and opinions

about their respective school’s implementation of the new

standards in instruction, assessment and rating students.

Parents were asked to discuss the specific examples about

the positive and negative changes regarding the implemen-

tation of the standards, such as their children talking about

and reacting to school instruction, student work brought

home, and things they talked about at parent–teacher

conference. They were also asked to give comments and

opinions about the implementation of the new standards.

Example questions include whether the project leads to

student being more excited and challenging at school,

whether they favor this school work, and what are their

concerns.

4.3. Objective III

Objective III is to offer a challenging curriculum. This

curriculum is unique in breadth and depth among schools in

its zone, appropriate to its theme, suitable to the needs of its

Page 6: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118114

students at different achievement levels, and attractive in its

application. As Objective II and Objective III are heavily

connected and intertwined with each other, an integrative

method is applied in part of the work of evaluating these two

objectives.

This objective is evaluated in two ways: the visiting team

examination of school portfolios and direct observation of

teachers. One aspect of the project is that each school has an

advisory committee, the Excels Committee. The committee

meets periodically to assist in assuring that the curriculum is

innovative, exciting, and challenging for the students. The

visiting team examines the effective use of the advisory

committee to judge the innovation and uniqueness of the

curriculum as well as instructional processes and activities.

The team also directly observes the classrooms as operated

and scheduled in Objective II to judge whether teachers use

excellent teaching strategies and innovative materials, to

create a good learning environment and hold the students’

interest and attention. The availability of the advisory

committee’s minutes, which are included in each school’s

portfolio, and the classroom observations provide other

evidence that this objective is being met.

Secondly, there is a survey of staff, parents and

community partners associated with each school in addition

to the evidence collected during the school visits. The

external program evaluators select a sample of parents for

interview as a focus group. The surveys and interviews are

conducted together with those for evaluating Objective II.

The evaluation of Objective III is also integrated with the

additional professional development that focuses on class-

room management, teaching techniques, and the use of the

available materials to make learning more interesting.

4.4. Objective IV

Objective IV is that magnet school participants will show

equal or superior distribution of achievement in core

subjects when compared to a control group of students in

non-magnet schools with similar characteristics of enroll-

ment. Two targets are set. The first is that achievement test

scores for the students in the magnet schools will equal or

exceed the performance of the comparison schools. The

second is that at least 8% of the magnet school students will

be in the top quartile of the CAT scores.

Indicators of success are objective and criterion-judged

performances, including standardized norm-referenced and

criterion-based tests. The external program evaluators

verify the results. The measures include the California

Achievement Tests as a norm-referenced test, and the

following criterion-based tests: the school district’s Third

Grade Reading Test, the Content Standards Criterion

Referenced Tests, and the Extra Value Standards Criterion

Referenced Tests.

The evaluation of achievement in magnet programs is

impossible to be interpreted when there are no comparisons

of magnet students with non-magnet students or with

district-wide scores (Blank, 1989; Gaines, 1987). In this

design, three schools with combined enrollments that are

similar to the combined enrollments of the magnet schools

are selected for comparison studies during the project. The

three schools match the magnet schools in terms of racial

and economic composition, and baseline achievement.

Assessments are completed periodically, consistent with

the regular district assessment schedule. Medians with

quartiles are reported using normal curve equivalents. The

results of all the four measures of tests in each school will be

separately presented, which shows achievement compari-

sons between magnet and comparison schools. Specifically,

the median scores with the top quartiles of the three subject

areas including reading, language arts, and mathematics

from each of the magnet school, the comparison school, and

the district are displayed and compared within the two

categories of minority and non-minority students.

Comparison and analysis of the standardized test score

data in reading, language arts, and mathematics are

undertaken at each of the three project schools to determine

specific objectives for which students had the most

difficulty. Analysis of teaching results is also conducted to

ascertain those teachers who had the greatest successes so

that their instructional techniques can be shared with grade

level colleagues. Additional discussions occur to insure that

instructional time for reading, language arts, and math-

ematics be kept in balanced and avoid its reduction as a

result of instructional time being dedicated to the magnet

focus areas.

5. Discussions

The objective-based approach usually has its distinguish-

ing feature of specifying the purposes of programs and the

extent to which those purposes are achieved. Both practice

and research of this approach have verified its obvious

weaknesses. Use of any single methodology in an objective-

based approach usually leads to narrow, simplistic, or

unrepresentative findings, especially in evaluating the long-

term programs with higher degree of complexity and

diversity in both program components and clients. The

integration of the mixed-methods approach into the

objective-based evaluation, and the combination of for-

mative evaluation with summative evaluation in this design

can address, at least to some degree, the weaknesses of

objective-based approach. It employs mixed methods

primarily to achieve the benefits of initiation (Rossman &

Wilson, 1994), triangulation, complementarity and devel-

opment (Greene, Caracelli, & Graham, 1989).

5.1. Improving the project’s progress

First of all, the formative evaluation conducted by the

visiting team using both quantitative and qualitative

methods overcomes the weakness of the objective-based

Page 7: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118 115

approach in leading to terminal information that is neither

timely nor pertinent to improve a program’s progress

(Stufflemean, 2001). In the evaluation of Objective I of

increasing the number of non-minority students in each of

the magnet schools, the recruitment targets were not met for

the initial project year based on the quantitative data of each

school’s enrollment. The qualitative data from examination

of the school recruitment plans and interviewing teachers

helped to clarify and interpret the findings obtained from the

statistical analysis, which led to the conclusion that the

targets were difficult. Therefore, a careful examination of

the projected targets for the next two years was rec-

ommended. The data showing the distribution of minority

and non-minority students in each grade in each school

suggested where additional recruitment efforts should be

directed. Some staff members participated in a class on

marketing and developed a targeted plan. Several new

activities including open houses for feeder school princi-

pals, summer camp activities, re-evaluation of marketing

materials, and revision of the magnet selection calendar,

were recommended to intensify the focus on increasing the

enrollment of non-minority students for the next two years.

In evaluating Objective II, the qualitative methods such

as class observations, and a parent focus group interview

contributed to benefits in improving the process of

implementing the standards. In addition to the quantitative

percentage results of the checklist items, which captured

general picture of the objectives, the evaluators’ feedback of

the three-time per year class observations to the school

helped to improve the technical issues for teachers in

instruction, assessment and student rating. This enabled the

evaluators to simultaneously answer confirmatory and

exploratory questions (Teddlie & Tashakkori, 2003). Parent

focus group interviews with the purposes of seeking their

ideas, comments, and questions on implementing the

standards led to change in teaching too much content at

the beginning of the year and adopting a more appropriate

teaching approach which was balanced both in depth and

breadth.

To insure that teachers are prepared to use standards in

instruction, assessment, and rating students in the content

areas that are the focus of the project, this design also

focuses one of its lenses on evaluating teacher professional

development activities as they occurred. The impact of the

professional development in terms of continuance and focus

on long-term change issues were examined as well. The

purpose of evaluation of this enabling objective is not only

to judge the success of the professional development

programs, but also provide evidence for adjustment,

revision and improvement of the project implementation.

The formative evaluations of the visiting team and of the

professional development greatly contribute to meaningful

tracking of the project implementation. Its conclusions and

recommendations were effectively applied in improving the

project. The evaluative actions also greatly increased staff

investment, involvement, and result in support for the grant

activities. Indication of this came from comments from

external team members such as ‘we can see teachers over

the year increasing in their understanding of the grant

activities’; ‘we can observe the occurrence of reflective

thought and action as we visit with them’; and ‘teachers are

comfortable and asking for feedback.’

The mixed-methods approach is employed in this

evaluation also for the purpose of initiation, which refers

to the potential of the evaluation to increase the possibility

of eliciting the new ways of looking at the issues. The

process of the three-time per year visits with the school

leaders to review the student portfolio was key to

recognizing positive implementation steps as well as

possible concerns and suggestions as to how they may be

addressed. This facilitated keeping all involved focused on

continuous improvement. With the examination of the

student work in the portfolios, the visiting team rec-

ommended from the view point of staff that more teacher

assistance must be provided to staff at all three project

schools to insure that portfolios were used purposely with

the goal of enhancing standards-focused instruction,

assessment, and reporting. As a result, a professional

development activity was planned for the three project

schools in which staff learned how to use portfolios to

monitor student progress related to content standards and

how to adapt instruction to meet individual students’ needs.

By using this mixed-method design, we not only captured

the various levels of outcomes, but also evaluated the

different facets of the action process of the objectives, which

resulted in improving the implementation of the projects.

The evaluators’ in-depth interactions with the teachers,

curriculum specialists and school administrators in the

mixed methods provide new insights on how the program

has been perceived and valued across sites.

5.2. Avoiding information narrowness

Use of the mixed-methods approach allows for multiple

lenses to meaningfully track for the implementation of the

program and validations of the program outcomes. Three of

the four program objectives are evaluated by using mixed

methods. This design greatly lessens the weakness of the

objective-based approach in providing narrow information

to judge the merits and worth of the program. The functions

of triangulation and complementarity which occurs when

researchers use mixed-methods approach in order to extend

the scope, breadth and range of inquiry, provides richness

and details to the study exploring specific features of each

method, and leads to multiple inferences that conform and

complement each other (Teddlie & Tashakkori, 2003).

In the evaluation of increasing the number of non-

minority students in enrollments, two other important

dimensions were added for analysis in addition to the

comparison of projected and actual numbers of minority and

non-minority students. One was the analysis of the impact

of recruiting on feeder schools, and the other was

Page 8: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118116

the examination of the school portfolio that includes the

marketing plans, communication plans and meeting minutes

for recruiting non-minority students. Therefore, the three-

dimension data: the outcomes, the sources, and the

processes were structured together for analysis, combining

confirmatory and exploratory investigations. This also

added complexity and details for increasing the trustworthi-

ness and validity of the findings.

Another good example is the evaluation of Objective II in

using state approved content standards in instruction and

reliable assessments for assessing and rating students.

Multiple techniques including observation, interview,

checklist, portfolios, surveys, and focus groups were

effectively integrated in this process. It not only improves

the design and analysis by capturing the richness of the

program, but also strengthens comprehensiveness and

efficacy by looking at the diversity of the program. The

visiting team conducted a series of evaluative activities.

They observed a teacher teaching, talked through checklist

items with the teachers, went through samples of their

student portfolios, visited with the principal about plans and

activities, reviewed documents, and completed reports.

Evaluation subjects covered staff members, students,

parents and community members. Consistent data from

different methods and from various stakeholders increases

the validity of the findings. For instance, the checklist itself

did not provide a complete picture of what was happening in

each of the classroom. However, the student portfolio

examination and parent focus group interview sup-

plemented the checklist. One method gives greater depth,

while the other gives greater breadth. Integrating the

methods together, they give results from which one can

make better and more accurate inferences (Teddlie &

Tashakkori, 2003).

5.3. Uncovering side effects

Another weakness of objective-based approach is being

unable to uncover the positive and negative side effects

(Stufflemean (2001)). The objective-based approach tends

to remove the negative connotations attached to the

discovery of unanticipated effects, because the language

of side-effect or secondary effect or even unanticipated

effect tended to be a putdown of what might well be the

crucial achievement, especially in terms of new priorities

(Scriven, 1972). The mixed-methods design makes it

possible and feasible for evaluators to facilitate the

evaluation process in designing evaluations according to

its utility and actual use. The situational use of the mixed

methods to study change in different objectives makes

evaluations developmental and action-oriented. This helps

to consider and address the additional positive effects or

undesired side effects of programs.

In the process of teacher interview and classroom

observation by using the checklist for evaluating Objective

II, the visiting team found that along with filling out

the quantitative checklist relative to specific objectives,

many comments were noted on the form that provided

qualitative data not covered by the checklist. For example,

while all teachers had received the district proposed student

portfolios in the first year of the program, they were

very uneasy and even overwhelmed about the changes.

Through the analysis of comments, the visiting team

recommended in the first year report that teachers needed

more support and understanding from school and district

leaders. Continued staff development needed to coordinate

and support the implementation of portfolios.

Magnet schools are busy places. It is not unusual for

them to already be involved with a number of ‘extra’

initiatives. These three schools were involved with devel-

opment and implementation of statewide and national (No

Child Left Behind) school improvement activities, a math

and science grant, and some special district initiatives

related to their desegregation plans. The frequent visits and

communications with teachers make the visiting team fully

aware of this situation. They suggested that teachers and

school leaders work towards integrating plans and activities

so that programs complemented each other rather than

becoming ‘add-ons’ or competing. Maintenance of a

climate, which is supportive of high academic and

behavioral expectations should be strengthened. A building

level supervisory team was the key in providing the

coordination.

In the initial review of the enabling objective checklist

with staff, staff concern or worry was uncovered that data

regarding staff performance might be shared with the

principal or the human resources office. In order to

decrease this side effect, an agreement were made

between the school staff and the visiting team that data

collected were only for formative evaluation and

improvement of the magnet grant process, and would

not be shared with others. The process of addressing the

concern gained significant trust and support from the

staff that carried over through the project.

The mixed-methods function of development was also

achieved in developing the survey based on the first year

summary reports. The findings from the evaluation by the

visiting team shaped the subsequent survey design from the

original plan. The visiting team used mixed methods to

judge whether continued implementation contributes to

enabling objectives and what additional steps can be taken

to improve accomplishment of the whole program. The

visiting team also covered and summarized the side effects

such as teachers’ worry and uneasiness, the school’s

situation of multi-tasked improvement programs, teachers’

time pressure during their formative evaluation. Based on

all the results, there were discussions regarding how various

survey items would be examined in the second year and

third year. Meetings were held to identify the areas verified

by observation, the areas needing a review of artifacts, and

the areas needing discussion with the staff members.

Page 9: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118 117

6. Lessons learned

The examination of this design also encourages us to

probe the underlying issues assumed by integrating mixed-

methods into objective-based evaluation. Evaluation pro-

fessionals should be equipped with broader technical skills.

Evaluators need expertise in conceptualizing complicated

objectives, designing and implementing different methods,

as well as in analyzing, interpreting and integrating the

findings. These are efforts to be more critical of the

evaluation practice. In this situation, Rossman and Wilson

(1994) recommend that more than one researcher be

engaged in the enterprise, with each researcher bringing

particular methodological expertise to the endeavor. On the

other hand, the design is also featured with integration of

the ‘extra’ formative evaluation including the evaluation of

the professional development, which is intertwined with

different methods. Consequently, the need for more

professional personnel and more efforts in integrating

mixed-methods approach in objective-based evaluation

may result in time consuming and higher cost. Clear

justification, complete planning and flexible adjustment are

also the key elements in achieving successful mixed-

methods evaluation.

7. Concluding comments

Traditionally, methodology of objective-based approach

of magnet school assistance projects has relied heavily on

surveys and student records. In order to gain better

knowledge and understanding of the operation, policy,

outcomes, and community perceptions of magnet schools,

both the data sources and evaluation questions need to be

expanded (Bailey 1987; Laws 1987). The evaluation design

of a magnet assistance project should be multi-directional

and multi-sourced in nature. From the experiences of

designing and implementing the magnet assistance pro-

gram, we propose the mixed-methods design as a possible

solution, which is likely to address the weaknesses of

objective-based approach and to produce better results in

terms of quality and breadth. Functions of mixed-methods

approach, such as initiation, triangulation, complementarity

and development can possibly contribute to improving

program progress, avoiding information narrowness and

uncovering side effects. The mixed-methods design is not

only possible and necessary, but also can achieve better

effectiveness and higher validity. It can be expected as a

creative and valuable approach in integration with the

traditional objective-based approach to conceive and

implement evaluation. In applying this into practice, the

mixed-methods design in an objective-based evaluation can

be more useful and accountable to program evaluations with

broader audiences, longer terms, and more complex goals.

Appendix

Use of Content Standards in Teachers’ Classroom

Practices Checklist

Teacher_

Directions: Please indicate the teacher’s progress in

accomplishing each of the indicators by placing an ‘x’ in the

appropriate column: Yes if the indicator has been accom-

plished, No if it has not been accomplished, or Not Sure if

you are uncertain.

Indicator Yes

Math Economy

No

Ma Ec

Not sure

Ma Ec

Use in instruction

1. The standards are posted in the classroom in language that students will understand.

2. The teacher’s plans for the year include a schedule for including the standards in

teaching and review.

3. Lesson plans reflect the standards

4. Instruction is focused on achievement of standards.

5. The teacher uses student performance on standards to revise instruction

6. Artifacts of proficient and exemplary student work on meeting the standards is shared

with students and parents.

7. Students know what standards they are working on and trying to master.

8. Student work is focused on achievement of standards.

9. The connection between instruction and standards is explicit in the process of delivering

instruction (e.g. lesson plans, other instructional materials).

Use in assessment

10. Evaluations of student performance are explicitly tied to the standards.

11. Diagnostic information is available to identify the needs of students in meeting the

standards.

12. Assessments reflect the standards and are scored using specific criteria.

13. Assessment strategies include performances, projects, and portfolios.

(continued on next page)

Page 10: Mixed-methods design for an objective-based evaluation of a magnet school assistance project

M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118118

Indicator Yes

Math Economy

No

Ma Ec

Not sure

Ma Ec

14. The connection between assessment and standards is explicit (e.g. classroom

assessments, projects dearly reflect the content standards).

Use in Rating Student Performance

15. Student records (e.g. report cards, progress charts) show student status on standards.

16. Grades are influenced by performance on standards.

17. Performance on content standards is reviewed regularly by the teacher to assure skill

maintenance.

18. Student performance on standards is used in making decisions about students (e.g. Title

I, IEP preparation, qualification for special programs).

19. Progress reports (including report cards) communicate student performance on the

standards.

20. Information shared with parents and students include student’s progress on meeting the

standards.

21. Students are informed of the required level of performance for meeting the standards.

22. The overall student evaluation/rating process explicitly reflects the content standards

(e.g. student records, progress reports).

Laws, L. (1987). Assessing outcomes of magnet schools. Washington, DC:

References

Bailey, J. D. (1987). Research needed to assess the performance of magnet

schools. Washington, DC: Office of Educational Research and Improve-

ment. ERIC Document Reproduction Service No. ED 284 952.

Blank, R. K. (1989). Educational effects of magnet high schools. Paper presented

to the conference on choice and control in American education, Madison,

Wisconsin. ERIC Document Reproduction Service No. ED 313 469.

Brewer, J., & Hunter, A. (1989). A multimethod research: A synthesis of

styles. Thousand Oak, CA: Sage.

Caracelli, V. J., & Greene, J. C. (1997). Crafting mix-method evaluation

design New directions for evaluation, Vol. 74. San Francisco, CA:

Jossey-Bass pp. 19–32.

Cook, T. D., & Reichardt, C. S. (Eds.). (1979). Qualitative and quantitative

methods in evaluation research. Thousand oaks, CA: Sage.

Crewell, J. W. (1994). Research design: Qualitative and quantitative

approaches. Thousand oaks, CA: Sage.

Dentler, R. A. (1991). The national evidence on magnet schools. Los

Alamitos, CA: Southwest Regional Laboratory for Educational Research

and Development. ERIC Document Reproduction Service No. ED 335

780.

Gaines, M. L. (1987). Evaluating magnet schools effectively: Challenges and

cautions. Paper presented al the annual meeting of the American

Educational Research Association, Washington, DC. ERIC Document

Reproduction Service No. ED 283 836.

Greene, J. C., & Caracelli, V. J. (1997). Defining and describing the paradigm

issue in mixed-methods evaluation. New Directions for Evaluation, 74,

5–17.

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Towards a conceptual

framework for mixed-methods evaluation designs. Educational Evalu-

ation and Policy Analysis, 11, 255–274.

Hammond, R. L. (1973). Evaluation at the local level. In B. R. Worthen, &

J. R. Sanders, Educational evaluation: Theory and practice. Belmont,

CA: Wadsworth.

Office of Educational Research and Improvement. ERIC Document

Reproduction Service No. ED 284 951.

Madaus, G. E., & Stufflemean, D. L. (Eds.). (1989). Educational evaluation:

Classic works of Ralph W. Tyler. Boston, MA: Kluwer Academic.

Mark, M. M., & Shotland, R. L. (1987). Alternative models for the use of

multiple methods. New Directions for Program Evaluation, 35, 95–

100.

Metfessel, N. S., & Michael, W. B. (1967). A paradigm involving

multiple criterion measures for the evaluation of the effectiveness of

school programs. Educational and Psychological Measurement, 27,

391–943.

Provus, M. M. (1971). Discrepancy evaluation for educational program

improvement and assessment. Berkeley, CA: McCutchan Pub. Corp.

Punch, K. F. (1998). Introduction to social research: Quantitative and

qualitative approaches. Thousand Oaks, CA: Sage.

Reichardt, C. S., & Rallis, S. F. (1994). The relationship between the

qualitative and quantitative research traditions. New Directions for

Evaluation, 61, 5–11.

Rossman, G. B., & Wilson, B. L. (1994). Numbers and words revisited: Being

‘shamelessly eclectic.’. Quality and Quantity, 28, 315–327.

Scriven, M. (1972). Pros and cons about goal-free evaluation. Evaluation

Comment: The Journal of Educational Evaluation (Center for the Study

of Evaluation, UCLA), 3, 1–7.

Stufflemean, D. L. (2001). Evaluation model. New Directions for Evaluation,

89, 7–98.

Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the

use of mixed methods in the social and behavioral sciences. In A.

Tashakkori, & C. Teddlie (Eds.), Handbook of mixed methods in social

and behavioral research. Thousand Oaks, CA: Sage.

Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1997). Program

evaluation: Alternative approaches and practical guidelines (2nd ed.).

White Plain, NY: Longman.