Upload
mingchu-luo
View
216
Download
0
Embed Size (px)
Citation preview
Mixed-methods design for an objective-based evaluation
of a magnet school assistance project
Mingchu Luo, Leon Dappen*
Department of Educational Administration and Supervision, College of Education, University of Nebraska at Omaha, Kayser Hall 414,
6001 Dodge Street, 68182 0162 Omaha, NE, USA
Received 20 March 2003; received in revised form 3 November 2003; accepted 17 December 2003
Abstract
This article describes and discusses the development and implementation of an evaluation design to assess a magnet assistance project,
which integrates a mixed-methods approach into the objective-based evaluation. By combining both quantitative and qualitative methods,
this design can address some of the weaknesses of objective-based approach in the process of evaluation. Functions of mixed-methods
approach, such as initiation, triangulation, complementarity and development can possibly contribute to improving program progress,
avoiding information narrowness and uncovering side effects. The examination of this design also reveals that the mixed-methods approach
is not only possible, but more effective, and has higher validity. The mixed-methods is a more useful and accountable approach, which can be
used in integration with the traditional objective-based approach to conceive and implement evaluation, especially in program evaluations
with broader audiences, longer terms, and more complex goals.
q 2004 Elsevier Ltd. All rights reserved.
Keywords: Evaluation design; Mixed-methods approach; Objective-based evaluation; Magnet assistance project; Formative evaluation; Enabling objective
1. Introduction
Evaluation is widely used to examine and judge the
worth, merits and shortcomings of various educational
programs through collecting and analyzing data. It serves as
an important instrument in the improvement process of
educational quality. Because of the complexity of the
evaluation objects and the evaluators’ different philosophi-
cal views of evaluation, the topic of evaluation approach has
attained critical status in both research and practice.
Worthen, Sander, and Fitzpatrick (1997) described six
approaches of evaluation. Stufflemean (2001) identified 21
approaches often employed to evaluation programs by
categorizing them into pseudoevaluations, questions- and
methods-oriented evaluation approaches, improve/account-
ability approaches, and social agenda/advocacy approaches.
Objective-based and mixed-methods were included in
0149-7189/$ - see front matter q 2004 Elsevier Ltd. All rights reserved.
doi:10.1016/j.evalprogplan.2003.12.001
* Corresponding author. Tel.: C1 402 554 3485; fax: C1 402 554 2722.
E-mail address: [email protected] (L. Dappen).
the category of questions- and methods-oriented evaluation
approaches (quasi-evaluation studies).
The education evaluation field has increasingly recog-
nized the benefits of using the views of system, multiplicity,
and integration in both research and practice. The mixed-
methods approach, which is broadly used and discussed,
technically represents this trend. This article describes and
discusses the development and implementation of an
evaluation design to assess a magnet assistance project
with a strong emphasis on methodological issues. This
design integrates mixed-methods, an intensely-discussed
topic over the past decade, into the classic and prevalent
objective-based evaluation.
2. Theoretical review of the two evaluation approaches
2.1. Objective-based approach
The objective-based approach specifies the purpose of
educational programs and determines if or to what extent,
Evaluation and Program Planning 28 (2005) 109–118
www.elsevier.com/locate/evalprogplan
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118110
these objectives have been attained. Its procedure of having
objective achievement judge the extent of success or failure
is straightforward. Its purposes of justifying improvements,
maintenance, and termination of program activity are
practical. For these reasons, the objective-based approach
has dominated the thinking and development of evaluation
since the 1930s, both in the United States and elsewhere
(Madaus & Stufflebeam, 1989). Evaluation researchers and
practitioners have been contributing their efforts in design-
ing, redesigning and improving this approach.
Ralph Tyler, regarded as the pioneer in the objective-
based approach viewed evaluation as the process of
determining the extent to which the objectives of a program
are actually attained. He emphasized the use of filtering
goals and objectives based on the rationale of being logical,
scientifically acceptable, and readily adoptable by evalua-
tors (Worthen et al., 1997). Metfessel and Michael (1967)
proposed an eight-step model to enrich the Tylerian
approach by adding more pragmatic ideas such as involving
stakeholders as facilitators, carrying out periodic obser-
vations, and developing recommendations for the further
implementation, modification, and revision of broad goals
and specific objectives. Provus (1971) used the Tylerian
tradition in designing the Discrepancy Evaluation Model,
which was based on the rationale that evaluation is a
continuous information-management process designed to
serve as the watchdog of program management and the
handmaiden of administration on the management of
program development through sound decision-making.
He also used new terms such as enabling objective (student
achievement acquired during the program to ensure the
accomplishment of the major program objectives), terminal
objective (immediate outcomes) and ultimate objective
(long-term outcomes) in the stages of process and product.
Hammond (1973) developed a cube model by applying a
three-dimensional framework for analyzing a community-
based youth program. Stufflemean (2001) and Worthen et al.
(1997) also discussed the developmental variations of the
Tylerian evaluation model.
Although the objective-based approach has been exten-
sively used in educational programs, it is commonly
criticized because of its simplicity in operation, emphasis
on defining outcomes, and focused use of behavioral
objectives. Stufflemean (2001) described four major aspects
being criticized: ‘leading to terminal information that is
neither timely nor pertinent to improve a program’s
progress; providing narrow information to judge the merits
and worth of the program; being unable to uncover the
positive and negative side effects; and crediting unworthy
objects’ (p. 18).
2.2. Mixed-methods approach
Evaluation theory and practice today are characteristi-
cally pluralistic, embracing diverse perspectives, methods,
data, and values within and across studies that aim to
generate more insightful and meaningful evaluative claims
(Caracelli & Greene, 1997). The wide use of the mixed-
methods approach stems from the perception that it is
particularly necessary to apply multiple ways of knowing
and acting in evaluation because educational problems are
increasingly complex and intractable. By looking at
educational phenomenon through both quantitative and
qualitative lenses, mixed-methods approach is intended to
ensure dependable feedback on a wide range of questions;
benefit in in-depth understanding of the programs; display a
holistic perspective; and enhance the validity, reliability and
usefulness of the full set of findings (Stufflemean, 2001).
Evaluation designs using mixed-method research (Brewer
& Hunter, 1989; Mark & Shotland, 1987), and specifically
combining qualitative and quantitative methods (Cook &
Reichardt, 1979; Crewell, 1994; Reichardt & Rallis, 1994),
have become a part of methodological guidance for the
practitioner (Caracelli & Greene, 1997). ‘In this troubled
era, with social problems of ever-increasing complexity and
intractability, multiple ways of knowing and acting are
surely needed’ (Greene & Caracelli, 1997, p. 15).
Advantages of using mixed-methods are that they
complement each other in ways that are important to the
evaluation audiences (Stufflemean, 2001). Compared to
single approach designs, mixed methods research can
answer questions in a better way, provide stronger
inferences, and provide opportunity for presenting a greater
diversity of divergent views (Teddlie & Tashakkori, 2003).
Quantitative methods provide relatively standardized,
efficient, amenable information, which can be easily
summarized and analyzed. Qualitative methods add con-
textual and cultural dimensions, which deepens the study by
providing more natural information. Quantitative research is
more directed at theory verification while qualitative
research tends to be more concerned with theory generation
(Punch, 1998).
3. Context and description of program
Since 1970s, magnet schools have been developed in
more and more school systems. Magnet programs are often
designed to promote racial diversity, expand educational
choices, and improve student achievement by providing a
unique or specialized curriculum or approach. Many of
them also enhance parent/community involvement, prepare
students for further education and/or careers in the world of
work, provide field-based and hands-on learning experi-
ences, and offer mentorship, internship, and apprenticeship
opportunities. Dentler (1991) identified that a magnet school
has the following four characteristics: a distinctive curricu-
lum based on a special theme or instructional method; a
unique district role and purpose for voluntary desegrega-
tion; voluntary school choice by the student and the parent,
with variable criteria established for inclusion; and access to
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118 111
students beyond an attendance zone or single subdivision of
a district.
The program being discussed in this article is a three-
year Magnet Assistance Project begun in an urban school
district in the Midwest in 2001. The program is being
implemented in three elementary schools, which are
located in the comparatively low socioeconomic areas.
The magnet school assistance program was designed to
meet four objectives, which are: to increase the number
of non-minority students in each of the magnet schools;
to use state approved content standards in instruction and
use reliable assessment for assessing and rating students;
to offer a challenging curriculum; and to ensure that
magnet school participants will show equal or higher
achievement in core subjects as compared to a control
group. Each of the three elementary schools is imple-
menting their marketing/recruiting plans and strategies to
recruit non-minority students from their respective eight
feeder schools. Teachers are involved in the project
through professional development in using standards as
the basis for instruction, assessment, and rating student
performance. Each school communicates with parents
and its community in a variety of ways following each
school’s communication plan. An advisory committee
was established in each school to oversee the implemen-
tation of innovative, exiting, and challenging curriculum.
Standardized norm-referenced and district developed
criterion-based tests are used for assessing student
achievement.
4. Methodology for evaluating the objectives
The evaluation of the Magnet Assistance Project, which
lasts for 3 years, was designed to determine if, or to what
extent, the program’s objectives have been attained. More
importantly, a key purpose of the evaluation was to correct,
revise, and improve the program over the course of its
implementation. A set of enabling objectives is included in
the project proposal for the purpose of accomplishing the
objectives. The mixed-methods approach was integrated
into the process of evaluating grant objectives. For each
objective, data are collected periodically—some three times
annually, some annually, and some twice over the course of
the project. The descriptions of the evaluation activities and
the results of these activities associate with each of the
objectives are summative in nature. There is also a
formative evaluation that is intended to provide periodic
data to assist in making mid-course corrections as the
project is underway. The formative evaluation activities
provide information related to three of the four project
objectives (Objectives I, II, and III). The following sections,
respectively, correspond to an element of the evaluation
plan and discuss how the four objectives are evaluated.
4.1. Objective I
Objective I is to increase the number of non-minority
students in each of the magnet schools. When the magnet
school program is implemented at the three elementary
schools, each school will increase a specific number of non-
minority students in its enrollment at each magnet grade
level. Non-minority students will enroll in the magnet
school by choice without increasing minority isolation in
any of its feeder schools and with minority/non-minority
assignment ratios in each classroom enrolling opting
students consistent with district enrollment for that grade
level.
Evidence of the extent to which this objective is attained
is based on the quantitative enrollment data collected by the
school district and provided to the evaluators. In each of the
three years, analyses using enrollment data are implemented.
Data collected and analyzed include the district enrollment
by minority and non-minority students for the grade levels,
and the comparison of actual and projected minority
enrollment number and percent of minority students in
each magnet school. In addition to this, data were collected
for impact analysis of recruiting for the magnet schools on
the feeder schools; and the distribution data of minority
students in classes are compared to the distribution of
minority students for each grade for the entire school.
Comparison both among the three schools and among the
three years is the primary method to analyze all the data. The
comparison of actual and projected minority enrollment
numbers shows the extent of meeting non-minority enroll-
ment projections in each of the three schools. The impact of
recruiting for the magnet schools on the feeder schools
displays the data of students recruited from each of
respective eight feeder schools and extent of the feeder
school’s impact on the magnet schools. Analysis of the
distribution of students in classes and in schools for each
grade helps us to examine deeply to the classroom level, and
to determine which grade the recruitment targets can be
focused on to increase the non-minority enrollment.
“The information gained from an objective-oriented
evaluation can be used to reformulate the purposes of the
activity, the activity itself, or the assessment procedures and
devices used to determine the achievement of purposes”
(Worthen, et al., 1997, p. 81). By integrating the qualitative
approach, this objective is also examined as part of the
formative evaluation process that looks at each school’s
recruitment plan and determines periodically (three times
annually) if each school is on schedule in terms of its plan.
The purposes of this qualitative evaluation are to determine
whether the activities included in the plan are accomplished;
estimate the value of project activities in reaching the
longer-term project objectives; and determine whether the
enabling objective (to recruit students) of Objective I are
being met for each school and the district. Strategic steps for
adjustment, revisions and improvement are recommended
based on the evaluation results.
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118112
A visiting team of three experienced professors who are
former public school administrators and teachers is
conducting the formative evaluation. One team member is
responsible for evaluating one school. They make periodic
visits to the schools involved in the project to examine the
existence of a marketing/recruitment plan, and to determine
if the plan is on schedule regarding implementation of
activities. Each school develops a school portfolio to
provide information to assist in the formative evaluation.
The school portfolio includes: (a) a marketing plan for
recruiting students into the magnet program, (b) minutes of
meetings from appropriate committees, and (c) a communi-
cation plan for communicating with parents and others in
the community.
4.2. Objective II
Objective II is to use state approved content standards in
instruction and to use reliable assessments for assessing and
rating students. Each magnet school uses state approved
content standards and reliable assessment instruments to
evaluate and report student achievement in core subjects.
Magnet schools work with the support of the school district,
the community, and the staff development program to
enhance capacity and longevity. These are judged by
reviewing (1) the results produced with the application of
a set of criteria designed to determine the use and effect of
standards; (2) the records of staff completion of staff
development projects; and (3) a survey of staff, parents and
community members.
Formative evaluation is the focus in evaluating this
objective. Its key purposes are to estimate the value of the
project activities in reaching the long-term project activities
and to determine if enabling objectives are being met for
each school and the district. The enabling objectives are to
use content standards for instructional planning and
reporting, to communicate the standards effectively, and
to conduct successful professional development. The
formative evaluation in this Objective as well as in
Objective I and III was designed to answer the following
questions: (1) are the activities described in the project
being completed and in a timely manner? (2) are the
enabling objectives being met? (3) will continued
implementation of the activities reported make a substantial
contribution to meeting the enabling objectives? (4) what
additional steps could be taken in improving the accom-
plishment of the enabling objectives? The following
methods are integrated to evaluate this objective.
4.2.1. Observation and interview
The visiting team is also empowered to conduct the
evaluation part of this objective. They make periodic visits
(at least three times annually) to each of the three schools
involved in the project. During each visit, a sample of
teachers’ classroom teaching is observed in the specific
content standards. These same teachers are also interviewed
briefly to clarify what they are doing relative to using
content standards in their instruction.
Checklist. In order to aid the visiting team members in
structuring classroom teacher observations’ and interviews,
the district, in cooperation with the evaluators, developed
and validated a checklist (see Appendix). It is a set of
criteria designed to determine the use and effect of standards
in classroom practices, specifically is used to assess the
extent that teachers are using standards as the basis for
instruction, assessment, and rating student performance.
The checklist has three sections: one corresponding to the
use of standards in instruction, a second corresponding
to the use of standards in assessment, and the third related to
the use of standards in rating students performance. This
checklist is the basis for this aspect of the formative
evaluation. The teacher’s progress in accomplishing each of
the indicators will be indicated by the evaluators in three
circumstances: ‘Yes’ representing accomplished, ‘No’
representing not accomplished, and ‘Not Sure’ representing
uncertain.
The visiting team members complete the checklist form
for one-third of the teachers at each visit so that all
participating teachers will be observed over the course of
the year. Thus, about one-third of the teachers are observed
at any one visit. Each year School District Central Office
Curriculum Consultants complete the checklist form twice,
once near the beginning of the year and again near the end of
the year. The first and last set of forms completed by the
visiting team member is compared for consistency to the
forms completed by the District Curriculum Consultants.
The results of this formative evaluation are described in
detail in a series of Interim Reports that are provided to the
school district following each of the three external visits.
The Interim Report includes a report for each of the three
schools and a yearly summary. Accomplishment is
determined by counting the number of ‘Yes’ checks on
the checklist for each teacher within each school and
computing the percentage of ‘Yes’s’ checked in comparison
to the number of possible ‘Yes’ checks. Academic area
specialists as well as regular classroom teachers are also
observed. The checklist is also applied to their classroom
activities. Thus, for each school there are data for each
regular classroom teacher and the specialists. The percent of
‘Yes’ checks on the ‘Using Content Standards Checklist’
and the target percentages for each school across all three
visits during the school year are compared. All these data
are presented separately in the three categories of
compliances using standards in instruction, in assessment,
and in rating student performance for each school.
Another checklist was designed to evaluate all of the
enabling objectives of Objective I, II and III. It is a summary
checklist asking the visiting team members to indicate the
extent that each of the enabling objectives is being met at
the time of visit.
The four open-ended questions asked in the formative
evaluation are also attached to the checklist. The answers to
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118 113
these questions are based on the summary of the combined
data from the classroom observations, the teacher interviews
and the portfolio examination.
Student portfolio. One mechanism for facilitating using
standards in assessing students is the development of a
student portfolio that is used at all three schools. All three
schools developed standards-based portfolios for the
purpose of monitoring student progress on content standards
in the magnet focus areas. The portfolio was designed by
teachers with the assistance of a consultant to fulfill the
requirements as specified on the standards checklist. The
portfolio is used to collect student artifacts for various
standards as evidence regarding student achievement.
Portfolio examination is also part of the visiting team’s
work in evaluating the assurance of quality assessment for
all the standards. The visiting team randomly chooses three
to five student portfolios when visiting classrooms, and
reviews presence of student artifacts representing relevant
indicators of the checklist.
4.2.2. Professional development evaluation
Professional development activities are planned to insure
that teachers are prepared to use standards in instruction,
assessment, and rating students in the content areas that are
the focus of the project. It is regarded as a component of
Objective II that the extent that professional development is
occurring as planned, and the extent that the professional
development is successful. There were eight professional
development activities for teachers to effectively learn and
use the content standards in instruction, and reliable
assessments for assessing and rating students at the first
year of the project. Example workshops are the 5E
(engagement, exploration, explanation, extension, and
evaluation) inquiry learning, authentic assessment, coop-
erative learning, high-level questioning, and action
research. The evaluation of these professional development
activities as they occur is another strategy of formative
evaluation.
To evaluate the professional development, an evaluation
form reflecting the overall process and outcomes the
professional development activities was completed at the
end of each professional development activity. An overall
rating scale (1–4) was used for teachers to evaluate each of
the training programs in the following areas: the importance
of professional development activity to the magnet school’s
theme areas, meeting professional needs and expectations,
focusing on helping meet student needs, scheduling of
training, use of technology, and recommending the district
to repeat the training or not. The rating mean was used to
analyze the findings. In addition to this, teachers were also
asked to write down their comments on the training and
suggestions for improving the professional development
activities.
The impact of the professional development in terms of
continuance and focus on long-term change issues are also
examined. This occurs twice during the project, at 18 and 30
months. It is accomplished by forming a committee of staff
and members of advisory groups, selected by the external
program evaluators. The purpose of this committee is to
estimate the success of the professional development
activities related to the project.
4.2.3. Survey and focus group interview
In addition to observing and interviewing teachers, there
is a survey of staff, parents and community partners
associated with each school in the second and third year
of the project. The survey was designed with items
answering the following questions: what changes have
they heard about in the magnet content areas for that school,
what have students talked about regarding their school’s
instruction, what different activities students are involved in
at their schools, what is talked about at parent-teacher
conference, and any concerns or questions they have
regarding the new standards.
All the staff members in the three schools participate in
the survey. In each school approximately one half of the
parents were asked about one of the content area, and the
other half were asked about the other content area. A group
of school-community partners are randomly selected and
surveyed. Two similar surveys are conducted in the second
and the third year. The purpose of the second year survey is
to see the staff members’ and parents’ general perceptions
about using state approved content standards in instruction
and using reliable assessments for assessing and rating
students. The data of interest in the third year survey is the
difference in survey results between the second year and the
third year.
Following each of the two surveys, three parents at each
school who respectively represent three categories of the
high scorers, median scorers and low scorers on the survey
were randomly selected as a focus group for interview. This
interview was designed for the purpose of collecting and
analyzing the detailed and in-depth reflections and opinions
about their respective school’s implementation of the new
standards in instruction, assessment and rating students.
Parents were asked to discuss the specific examples about
the positive and negative changes regarding the implemen-
tation of the standards, such as their children talking about
and reacting to school instruction, student work brought
home, and things they talked about at parent–teacher
conference. They were also asked to give comments and
opinions about the implementation of the new standards.
Example questions include whether the project leads to
student being more excited and challenging at school,
whether they favor this school work, and what are their
concerns.
4.3. Objective III
Objective III is to offer a challenging curriculum. This
curriculum is unique in breadth and depth among schools in
its zone, appropriate to its theme, suitable to the needs of its
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118114
students at different achievement levels, and attractive in its
application. As Objective II and Objective III are heavily
connected and intertwined with each other, an integrative
method is applied in part of the work of evaluating these two
objectives.
This objective is evaluated in two ways: the visiting team
examination of school portfolios and direct observation of
teachers. One aspect of the project is that each school has an
advisory committee, the Excels Committee. The committee
meets periodically to assist in assuring that the curriculum is
innovative, exciting, and challenging for the students. The
visiting team examines the effective use of the advisory
committee to judge the innovation and uniqueness of the
curriculum as well as instructional processes and activities.
The team also directly observes the classrooms as operated
and scheduled in Objective II to judge whether teachers use
excellent teaching strategies and innovative materials, to
create a good learning environment and hold the students’
interest and attention. The availability of the advisory
committee’s minutes, which are included in each school’s
portfolio, and the classroom observations provide other
evidence that this objective is being met.
Secondly, there is a survey of staff, parents and
community partners associated with each school in addition
to the evidence collected during the school visits. The
external program evaluators select a sample of parents for
interview as a focus group. The surveys and interviews are
conducted together with those for evaluating Objective II.
The evaluation of Objective III is also integrated with the
additional professional development that focuses on class-
room management, teaching techniques, and the use of the
available materials to make learning more interesting.
4.4. Objective IV
Objective IV is that magnet school participants will show
equal or superior distribution of achievement in core
subjects when compared to a control group of students in
non-magnet schools with similar characteristics of enroll-
ment. Two targets are set. The first is that achievement test
scores for the students in the magnet schools will equal or
exceed the performance of the comparison schools. The
second is that at least 8% of the magnet school students will
be in the top quartile of the CAT scores.
Indicators of success are objective and criterion-judged
performances, including standardized norm-referenced and
criterion-based tests. The external program evaluators
verify the results. The measures include the California
Achievement Tests as a norm-referenced test, and the
following criterion-based tests: the school district’s Third
Grade Reading Test, the Content Standards Criterion
Referenced Tests, and the Extra Value Standards Criterion
Referenced Tests.
The evaluation of achievement in magnet programs is
impossible to be interpreted when there are no comparisons
of magnet students with non-magnet students or with
district-wide scores (Blank, 1989; Gaines, 1987). In this
design, three schools with combined enrollments that are
similar to the combined enrollments of the magnet schools
are selected for comparison studies during the project. The
three schools match the magnet schools in terms of racial
and economic composition, and baseline achievement.
Assessments are completed periodically, consistent with
the regular district assessment schedule. Medians with
quartiles are reported using normal curve equivalents. The
results of all the four measures of tests in each school will be
separately presented, which shows achievement compari-
sons between magnet and comparison schools. Specifically,
the median scores with the top quartiles of the three subject
areas including reading, language arts, and mathematics
from each of the magnet school, the comparison school, and
the district are displayed and compared within the two
categories of minority and non-minority students.
Comparison and analysis of the standardized test score
data in reading, language arts, and mathematics are
undertaken at each of the three project schools to determine
specific objectives for which students had the most
difficulty. Analysis of teaching results is also conducted to
ascertain those teachers who had the greatest successes so
that their instructional techniques can be shared with grade
level colleagues. Additional discussions occur to insure that
instructional time for reading, language arts, and math-
ematics be kept in balanced and avoid its reduction as a
result of instructional time being dedicated to the magnet
focus areas.
5. Discussions
The objective-based approach usually has its distinguish-
ing feature of specifying the purposes of programs and the
extent to which those purposes are achieved. Both practice
and research of this approach have verified its obvious
weaknesses. Use of any single methodology in an objective-
based approach usually leads to narrow, simplistic, or
unrepresentative findings, especially in evaluating the long-
term programs with higher degree of complexity and
diversity in both program components and clients. The
integration of the mixed-methods approach into the
objective-based evaluation, and the combination of for-
mative evaluation with summative evaluation in this design
can address, at least to some degree, the weaknesses of
objective-based approach. It employs mixed methods
primarily to achieve the benefits of initiation (Rossman &
Wilson, 1994), triangulation, complementarity and devel-
opment (Greene, Caracelli, & Graham, 1989).
5.1. Improving the project’s progress
First of all, the formative evaluation conducted by the
visiting team using both quantitative and qualitative
methods overcomes the weakness of the objective-based
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118 115
approach in leading to terminal information that is neither
timely nor pertinent to improve a program’s progress
(Stufflemean, 2001). In the evaluation of Objective I of
increasing the number of non-minority students in each of
the magnet schools, the recruitment targets were not met for
the initial project year based on the quantitative data of each
school’s enrollment. The qualitative data from examination
of the school recruitment plans and interviewing teachers
helped to clarify and interpret the findings obtained from the
statistical analysis, which led to the conclusion that the
targets were difficult. Therefore, a careful examination of
the projected targets for the next two years was rec-
ommended. The data showing the distribution of minority
and non-minority students in each grade in each school
suggested where additional recruitment efforts should be
directed. Some staff members participated in a class on
marketing and developed a targeted plan. Several new
activities including open houses for feeder school princi-
pals, summer camp activities, re-evaluation of marketing
materials, and revision of the magnet selection calendar,
were recommended to intensify the focus on increasing the
enrollment of non-minority students for the next two years.
In evaluating Objective II, the qualitative methods such
as class observations, and a parent focus group interview
contributed to benefits in improving the process of
implementing the standards. In addition to the quantitative
percentage results of the checklist items, which captured
general picture of the objectives, the evaluators’ feedback of
the three-time per year class observations to the school
helped to improve the technical issues for teachers in
instruction, assessment and student rating. This enabled the
evaluators to simultaneously answer confirmatory and
exploratory questions (Teddlie & Tashakkori, 2003). Parent
focus group interviews with the purposes of seeking their
ideas, comments, and questions on implementing the
standards led to change in teaching too much content at
the beginning of the year and adopting a more appropriate
teaching approach which was balanced both in depth and
breadth.
To insure that teachers are prepared to use standards in
instruction, assessment, and rating students in the content
areas that are the focus of the project, this design also
focuses one of its lenses on evaluating teacher professional
development activities as they occurred. The impact of the
professional development in terms of continuance and focus
on long-term change issues were examined as well. The
purpose of evaluation of this enabling objective is not only
to judge the success of the professional development
programs, but also provide evidence for adjustment,
revision and improvement of the project implementation.
The formative evaluations of the visiting team and of the
professional development greatly contribute to meaningful
tracking of the project implementation. Its conclusions and
recommendations were effectively applied in improving the
project. The evaluative actions also greatly increased staff
investment, involvement, and result in support for the grant
activities. Indication of this came from comments from
external team members such as ‘we can see teachers over
the year increasing in their understanding of the grant
activities’; ‘we can observe the occurrence of reflective
thought and action as we visit with them’; and ‘teachers are
comfortable and asking for feedback.’
The mixed-methods approach is employed in this
evaluation also for the purpose of initiation, which refers
to the potential of the evaluation to increase the possibility
of eliciting the new ways of looking at the issues. The
process of the three-time per year visits with the school
leaders to review the student portfolio was key to
recognizing positive implementation steps as well as
possible concerns and suggestions as to how they may be
addressed. This facilitated keeping all involved focused on
continuous improvement. With the examination of the
student work in the portfolios, the visiting team rec-
ommended from the view point of staff that more teacher
assistance must be provided to staff at all three project
schools to insure that portfolios were used purposely with
the goal of enhancing standards-focused instruction,
assessment, and reporting. As a result, a professional
development activity was planned for the three project
schools in which staff learned how to use portfolios to
monitor student progress related to content standards and
how to adapt instruction to meet individual students’ needs.
By using this mixed-method design, we not only captured
the various levels of outcomes, but also evaluated the
different facets of the action process of the objectives, which
resulted in improving the implementation of the projects.
The evaluators’ in-depth interactions with the teachers,
curriculum specialists and school administrators in the
mixed methods provide new insights on how the program
has been perceived and valued across sites.
5.2. Avoiding information narrowness
Use of the mixed-methods approach allows for multiple
lenses to meaningfully track for the implementation of the
program and validations of the program outcomes. Three of
the four program objectives are evaluated by using mixed
methods. This design greatly lessens the weakness of the
objective-based approach in providing narrow information
to judge the merits and worth of the program. The functions
of triangulation and complementarity which occurs when
researchers use mixed-methods approach in order to extend
the scope, breadth and range of inquiry, provides richness
and details to the study exploring specific features of each
method, and leads to multiple inferences that conform and
complement each other (Teddlie & Tashakkori, 2003).
In the evaluation of increasing the number of non-
minority students in enrollments, two other important
dimensions were added for analysis in addition to the
comparison of projected and actual numbers of minority and
non-minority students. One was the analysis of the impact
of recruiting on feeder schools, and the other was
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118116
the examination of the school portfolio that includes the
marketing plans, communication plans and meeting minutes
for recruiting non-minority students. Therefore, the three-
dimension data: the outcomes, the sources, and the
processes were structured together for analysis, combining
confirmatory and exploratory investigations. This also
added complexity and details for increasing the trustworthi-
ness and validity of the findings.
Another good example is the evaluation of Objective II in
using state approved content standards in instruction and
reliable assessments for assessing and rating students.
Multiple techniques including observation, interview,
checklist, portfolios, surveys, and focus groups were
effectively integrated in this process. It not only improves
the design and analysis by capturing the richness of the
program, but also strengthens comprehensiveness and
efficacy by looking at the diversity of the program. The
visiting team conducted a series of evaluative activities.
They observed a teacher teaching, talked through checklist
items with the teachers, went through samples of their
student portfolios, visited with the principal about plans and
activities, reviewed documents, and completed reports.
Evaluation subjects covered staff members, students,
parents and community members. Consistent data from
different methods and from various stakeholders increases
the validity of the findings. For instance, the checklist itself
did not provide a complete picture of what was happening in
each of the classroom. However, the student portfolio
examination and parent focus group interview sup-
plemented the checklist. One method gives greater depth,
while the other gives greater breadth. Integrating the
methods together, they give results from which one can
make better and more accurate inferences (Teddlie &
Tashakkori, 2003).
5.3. Uncovering side effects
Another weakness of objective-based approach is being
unable to uncover the positive and negative side effects
(Stufflemean (2001)). The objective-based approach tends
to remove the negative connotations attached to the
discovery of unanticipated effects, because the language
of side-effect or secondary effect or even unanticipated
effect tended to be a putdown of what might well be the
crucial achievement, especially in terms of new priorities
(Scriven, 1972). The mixed-methods design makes it
possible and feasible for evaluators to facilitate the
evaluation process in designing evaluations according to
its utility and actual use. The situational use of the mixed
methods to study change in different objectives makes
evaluations developmental and action-oriented. This helps
to consider and address the additional positive effects or
undesired side effects of programs.
In the process of teacher interview and classroom
observation by using the checklist for evaluating Objective
II, the visiting team found that along with filling out
the quantitative checklist relative to specific objectives,
many comments were noted on the form that provided
qualitative data not covered by the checklist. For example,
while all teachers had received the district proposed student
portfolios in the first year of the program, they were
very uneasy and even overwhelmed about the changes.
Through the analysis of comments, the visiting team
recommended in the first year report that teachers needed
more support and understanding from school and district
leaders. Continued staff development needed to coordinate
and support the implementation of portfolios.
Magnet schools are busy places. It is not unusual for
them to already be involved with a number of ‘extra’
initiatives. These three schools were involved with devel-
opment and implementation of statewide and national (No
Child Left Behind) school improvement activities, a math
and science grant, and some special district initiatives
related to their desegregation plans. The frequent visits and
communications with teachers make the visiting team fully
aware of this situation. They suggested that teachers and
school leaders work towards integrating plans and activities
so that programs complemented each other rather than
becoming ‘add-ons’ or competing. Maintenance of a
climate, which is supportive of high academic and
behavioral expectations should be strengthened. A building
level supervisory team was the key in providing the
coordination.
In the initial review of the enabling objective checklist
with staff, staff concern or worry was uncovered that data
regarding staff performance might be shared with the
principal or the human resources office. In order to
decrease this side effect, an agreement were made
between the school staff and the visiting team that data
collected were only for formative evaluation and
improvement of the magnet grant process, and would
not be shared with others. The process of addressing the
concern gained significant trust and support from the
staff that carried over through the project.
The mixed-methods function of development was also
achieved in developing the survey based on the first year
summary reports. The findings from the evaluation by the
visiting team shaped the subsequent survey design from the
original plan. The visiting team used mixed methods to
judge whether continued implementation contributes to
enabling objectives and what additional steps can be taken
to improve accomplishment of the whole program. The
visiting team also covered and summarized the side effects
such as teachers’ worry and uneasiness, the school’s
situation of multi-tasked improvement programs, teachers’
time pressure during their formative evaluation. Based on
all the results, there were discussions regarding how various
survey items would be examined in the second year and
third year. Meetings were held to identify the areas verified
by observation, the areas needing a review of artifacts, and
the areas needing discussion with the staff members.
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118 117
6. Lessons learned
The examination of this design also encourages us to
probe the underlying issues assumed by integrating mixed-
methods into objective-based evaluation. Evaluation pro-
fessionals should be equipped with broader technical skills.
Evaluators need expertise in conceptualizing complicated
objectives, designing and implementing different methods,
as well as in analyzing, interpreting and integrating the
findings. These are efforts to be more critical of the
evaluation practice. In this situation, Rossman and Wilson
(1994) recommend that more than one researcher be
engaged in the enterprise, with each researcher bringing
particular methodological expertise to the endeavor. On the
other hand, the design is also featured with integration of
the ‘extra’ formative evaluation including the evaluation of
the professional development, which is intertwined with
different methods. Consequently, the need for more
professional personnel and more efforts in integrating
mixed-methods approach in objective-based evaluation
may result in time consuming and higher cost. Clear
justification, complete planning and flexible adjustment are
also the key elements in achieving successful mixed-
methods evaluation.
7. Concluding comments
Traditionally, methodology of objective-based approach
of magnet school assistance projects has relied heavily on
surveys and student records. In order to gain better
knowledge and understanding of the operation, policy,
outcomes, and community perceptions of magnet schools,
both the data sources and evaluation questions need to be
expanded (Bailey 1987; Laws 1987). The evaluation design
of a magnet assistance project should be multi-directional
and multi-sourced in nature. From the experiences of
designing and implementing the magnet assistance pro-
gram, we propose the mixed-methods design as a possible
solution, which is likely to address the weaknesses of
objective-based approach and to produce better results in
terms of quality and breadth. Functions of mixed-methods
approach, such as initiation, triangulation, complementarity
and development can possibly contribute to improving
program progress, avoiding information narrowness and
uncovering side effects. The mixed-methods design is not
only possible and necessary, but also can achieve better
effectiveness and higher validity. It can be expected as a
creative and valuable approach in integration with the
traditional objective-based approach to conceive and
implement evaluation. In applying this into practice, the
mixed-methods design in an objective-based evaluation can
be more useful and accountable to program evaluations with
broader audiences, longer terms, and more complex goals.
Appendix
Use of Content Standards in Teachers’ Classroom
Practices Checklist
Teacher_
Directions: Please indicate the teacher’s progress in
accomplishing each of the indicators by placing an ‘x’ in the
appropriate column: Yes if the indicator has been accom-
plished, No if it has not been accomplished, or Not Sure if
you are uncertain.
Indicator Yes
Math Economy
No
Ma Ec
Not sure
Ma Ec
Use in instruction
1. The standards are posted in the classroom in language that students will understand.
2. The teacher’s plans for the year include a schedule for including the standards in
teaching and review.
3. Lesson plans reflect the standards
4. Instruction is focused on achievement of standards.
5. The teacher uses student performance on standards to revise instruction
6. Artifacts of proficient and exemplary student work on meeting the standards is shared
with students and parents.
7. Students know what standards they are working on and trying to master.
8. Student work is focused on achievement of standards.
9. The connection between instruction and standards is explicit in the process of delivering
instruction (e.g. lesson plans, other instructional materials).
Use in assessment
10. Evaluations of student performance are explicitly tied to the standards.
11. Diagnostic information is available to identify the needs of students in meeting the
standards.
12. Assessments reflect the standards and are scored using specific criteria.
13. Assessment strategies include performances, projects, and portfolios.
(continued on next page)
M. Luo, L. Dappen / Evaluation and Program Planning 28 (2005) 109–118118
Indicator Yes
Math Economy
No
Ma Ec
Not sure
Ma Ec
14. The connection between assessment and standards is explicit (e.g. classroom
assessments, projects dearly reflect the content standards).
Use in Rating Student Performance
15. Student records (e.g. report cards, progress charts) show student status on standards.
16. Grades are influenced by performance on standards.
17. Performance on content standards is reviewed regularly by the teacher to assure skill
maintenance.
18. Student performance on standards is used in making decisions about students (e.g. Title
I, IEP preparation, qualification for special programs).
19. Progress reports (including report cards) communicate student performance on the
standards.
20. Information shared with parents and students include student’s progress on meeting the
standards.
21. Students are informed of the required level of performance for meeting the standards.
22. The overall student evaluation/rating process explicitly reflects the content standards
(e.g. student records, progress reports).
Laws, L. (1987). Assessing outcomes of magnet schools. Washington, DC:
ReferencesBailey, J. D. (1987). Research needed to assess the performance of magnet
schools. Washington, DC: Office of Educational Research and Improve-
ment. ERIC Document Reproduction Service No. ED 284 952.
Blank, R. K. (1989). Educational effects of magnet high schools. Paper presented
to the conference on choice and control in American education, Madison,
Wisconsin. ERIC Document Reproduction Service No. ED 313 469.
Brewer, J., & Hunter, A. (1989). A multimethod research: A synthesis of
styles. Thousand Oak, CA: Sage.
Caracelli, V. J., & Greene, J. C. (1997). Crafting mix-method evaluation
design New directions for evaluation, Vol. 74. San Francisco, CA:
Jossey-Bass pp. 19–32.
Cook, T. D., & Reichardt, C. S. (Eds.). (1979). Qualitative and quantitative
methods in evaluation research. Thousand oaks, CA: Sage.
Crewell, J. W. (1994). Research design: Qualitative and quantitative
approaches. Thousand oaks, CA: Sage.
Dentler, R. A. (1991). The national evidence on magnet schools. Los
Alamitos, CA: Southwest Regional Laboratory for Educational Research
and Development. ERIC Document Reproduction Service No. ED 335
780.
Gaines, M. L. (1987). Evaluating magnet schools effectively: Challenges and
cautions. Paper presented al the annual meeting of the American
Educational Research Association, Washington, DC. ERIC Document
Reproduction Service No. ED 283 836.
Greene, J. C., & Caracelli, V. J. (1997). Defining and describing the paradigm
issue in mixed-methods evaluation. New Directions for Evaluation, 74,
5–17.
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Towards a conceptual
framework for mixed-methods evaluation designs. Educational Evalu-
ation and Policy Analysis, 11, 255–274.
Hammond, R. L. (1973). Evaluation at the local level. In B. R. Worthen, &
J. R. Sanders, Educational evaluation: Theory and practice. Belmont,
CA: Wadsworth.
Office of Educational Research and Improvement. ERIC Document
Reproduction Service No. ED 284 951.
Madaus, G. E., & Stufflemean, D. L. (Eds.). (1989). Educational evaluation:
Classic works of Ralph W. Tyler. Boston, MA: Kluwer Academic.
Mark, M. M., & Shotland, R. L. (1987). Alternative models for the use of
multiple methods. New Directions for Program Evaluation, 35, 95–
100.
Metfessel, N. S., & Michael, W. B. (1967). A paradigm involving
multiple criterion measures for the evaluation of the effectiveness of
school programs. Educational and Psychological Measurement, 27,
391–943.
Provus, M. M. (1971). Discrepancy evaluation for educational program
improvement and assessment. Berkeley, CA: McCutchan Pub. Corp.
Punch, K. F. (1998). Introduction to social research: Quantitative and
qualitative approaches. Thousand Oaks, CA: Sage.
Reichardt, C. S., & Rallis, S. F. (1994). The relationship between the
qualitative and quantitative research traditions. New Directions for
Evaluation, 61, 5–11.
Rossman, G. B., & Wilson, B. L. (1994). Numbers and words revisited: Being
‘shamelessly eclectic.’. Quality and Quantity, 28, 315–327.
Scriven, M. (1972). Pros and cons about goal-free evaluation. Evaluation
Comment: The Journal of Educational Evaluation (Center for the Study
of Evaluation, UCLA), 3, 1–7.
Stufflemean, D. L. (2001). Evaluation model. New Directions for Evaluation,
89, 7–98.
Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the
use of mixed methods in the social and behavioral sciences. In A.
Tashakkori, & C. Teddlie (Eds.), Handbook of mixed methods in social
and behavioral research. Thousand Oaks, CA: Sage.
Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1997). Program
evaluation: Alternative approaches and practical guidelines (2nd ed.).
White Plain, NY: Longman.