11

Assessment and Assessment and EvaluationEvaluation

Diane M. BunceDiane M. BunceChemistry DepartmentChemistry Department

The Catholic University of AmericaThe Catholic University of AmericaWashington, DC 2064Washington, DC 2064

Bunce@cua.eduBunce@cua.edu

22

Project EvaluationProject EvaluationNSF 1997NSF 1997

What is it?What is it? Systematic investigation of the worth of a projectSystematic investigation of the worth of a project

• Answers the questions:Answers the questions: What have we accomplished?What have we accomplished? Are the project goals being met?Are the project goals being met? How effectively are the individual parts of the project How effectively are the individual parts of the project

working?working?

• Provides information to communicate to stakeholdersProvides information to communicate to stakeholders NSFNSF Administrators Administrators Faculty Faculty StudentsStudents Others not involved in the projectOthers not involved in the project

• Faculty peersFaculty peers• Other studentsOther students

General AudienceGeneral Audience

33

Role of Evaluation in ProjectRole of Evaluation in Project

Evaluation should be an integral partEvaluation should be an integral part• Address each goal of project.Address each goal of project.• Provide information to help improve project.Provide information to help improve project.• Offer necessary information to different groups Offer necessary information to different groups

of stakeholders.of stakeholders.• Need not be adversarial.Need not be adversarial.

Evaluation should include the criteria of proof Evaluation should include the criteria of proof accepted by each group of stakeholders.accepted by each group of stakeholders.

Funded at a level consistent with the work Funded at a level consistent with the work that needs to be accomplished.that needs to be accomplished.

44

Two Stages of EvaluationTwo Stages of Evaluation

Formative EvaluationFormative Evaluation• Opportunity to measure progress and Opportunity to measure progress and

suggest modifications suggest modifications duringduring the project the project Implementation EvaluationImplementation Evaluation

• Is the project being conducted as planned?Is the project being conducted as planned? Progress EvaluationProgress Evaluation

• Measures impact of project components at Measures impact of project components at various stages.various stages.

Summative EvaluationSummative Evaluation• Assesses quality and impact of the full Assesses quality and impact of the full

project project at the completion of project.at the completion of project.

55

StakeholdersStakeholders Who are the people who have an interest in the results of your Who are the people who have an interest in the results of your

project?project?• Funding AgencyFunding Agency

Judge the success of your individual project.Judge the success of your individual project. Judge the impact your project results have on the broader Judge the impact your project results have on the broader

goals of the program.goals of the program.• Administration of the participating institutionsAdministration of the participating institutions

Are the results compelling enough to make changes in the Are the results compelling enough to make changes in the staffing, scheduling or facilities of the institution?staffing, scheduling or facilities of the institution?

• FacultyFaculty Are the results compelling enough to change the way they Are the results compelling enough to change the way they

teach and the curricula materials they choose?teach and the curricula materials they choose?• StudentsStudents

Does the experience support/challenge their Does the experience support/challenge their academic/career choice?academic/career choice?

• Are more students attracted to science, mathematics, Are more students attracted to science, mathematics, engineering majors?engineering majors?

• Professional PeersProfessional Peers Will your project affect the way others teach?Will your project affect the way others teach?

66

Evaluation QuestionsEvaluation QuestionsQuestions must address the overall goals of Questions must address the overall goals of

the project.the project.

Goal: To provide authentic research Goal: To provide authentic research experience for beginning undergraduatesexperience for beginning undergraduates• Evaluation Question: Evaluation Question:

What constitutes authentic research?What constitutes authentic research? Do students experience authentic Do students experience authentic

research in this project as judged by research in this project as judged by science experts, faculty, science experts, faculty, administrators, and students administrators, and students themselves?themselves?

77

Evaluation QuestionsEvaluation Questions Goal: To increase the number of students who Goal: To increase the number of students who

choose to major in science, mathematics, choose to major in science, mathematics, engineering, and technology.engineering, and technology.• Evaluation Questions: Evaluation Questions:

How many students declare/change their How many students declare/change their majors from non-SMET to SMET?majors from non-SMET to SMET?

Does the % of students who successfully Does the % of students who successfully graduate with SMET majors increase as a graduate with SMET majors increase as a result of this project?result of this project?

Does the % of students who go onto SMET Does the % of students who go onto SMET careers increase as a result of the project?careers increase as a result of the project?

88

Evaluation QuestionsEvaluation Questions

Goal: To change the way introductory Goal: To change the way introductory chemistry is taught to students.chemistry is taught to students.• Evaluation QuestionEvaluation Question

How have faculty changed the How have faculty changed the process or curricula of these process or curricula of these courses as a result of this courses as a result of this project?project?

99

Measurement and Results of Measurement and Results of Evaluation Questions Evaluation Questions

What proof is needed to answer the questions What proof is needed to answer the questions asked?asked?• Quantitative: StatisticsQuantitative: Statistics

Significant Difference between groupsSignificant Difference between groups• Difference unlikely to occur by chanceDifference unlikely to occur by chance

Effect Size Effect Size • Change measured against standard deviationChange measured against standard deviation

• QualitativeQualitative Observation of specific behavior of groupsObservation of specific behavior of groups Perceptions of StakeholdersPerceptions of Stakeholders

• CombinationCombination Can results be Can results be appliedapplied to policy or program that to policy or program that

motivated the study?motivated the study? Will the study have direct implications for Will the study have direct implications for actionaction??

1010

Matching Evaluation Methodology Matching Evaluation Methodology to Evaluation Questionsto Evaluation Questions

What, How or Why something occurs?What, How or Why something occurs?• QualitativeQualitative

Does what occurs make a difference?Does what occurs make a difference?• QuantitativeQuantitative

How important is it?How important is it?• Mixed MethodsMixed Methods

What occurs and why?What occurs and why? What does it mean?What does it mean? How is it valued by stakeholders?How is it valued by stakeholders?

1111

Comparison of Qualitative and Comparison of Qualitative and Quantitative Evaluation DesignsQuantitative Evaluation Designs

QualitativeQualitative AssumptionsAssumptions

• Variables are complex, Variables are complex, interwoven and difficult to interwoven and difficult to measuremeasure

PurposePurpose• Interpretation of reality of Interpretation of reality of

situationsituation ApproachApproach

• Ends with hypothesisEnds with hypothesis• Researcher is the toolResearcher is the tool

QuantitativeQuantitative AssumptionsAssumptions

• Variables can be Variables can be identified and identified and relationships measuredrelationships measured

PurposePurpose• GeneralizabilityGeneralizability• PredictionPrediction

ApproachApproach• Starts with hypothesisStarts with hypothesis• Researcher manipulates Researcher manipulates

formal toolsformal tools

1212

Qualitative Evaluation DesignQualitative Evaluation Design Use when investigating “What”, “How”, or “Why” Use when investigating “What”, “How”, or “Why”

• Results can be used Results can be used To answer the question directly or To answer the question directly or To help develop a quantitative study.To help develop a quantitative study.

Types of Qualitative StudiesTypes of Qualitative Studies• Observer Study: observations of participants Observer Study: observations of participants

augmented by opportunities to talk with augmented by opportunities to talk with stakeholders. stakeholders.

• Case Study: In-depth analysis of a limited Case Study: In-depth analysis of a limited number of participants.number of participants.

1313

Analysis of Qualitative DataAnalysis of Qualitative Data (Miles and Huberman 1994)(Miles and Huberman 1994)

Data ReductionData Reduction• Process of selecting, focusing and transforming Process of selecting, focusing and transforming

datadata• Initial categorizations shaped by pre-Initial categorizations shaped by pre-

established evaluation questions.established evaluation questions. Data DisplayData Display

• Diagram, Chart or Matrix used to organize data Diagram, Chart or Matrix used to organize data for purpose of facilitating construction of for purpose of facilitating construction of conclusions.conclusions.

Conclusion Drawing and VerificationConclusion Drawing and Verification• Consideration of what data mean.Consideration of what data mean.• Revisiting data to verify emerging conclusions.Revisiting data to verify emerging conclusions.

1414

Quantitative Evaluation DesignQuantitative Evaluation Design

Quantitative studies report on aggregate Quantitative studies report on aggregate (average of students in sample)(average of students in sample)• (Qualitative studies report on a limited number (Qualitative studies report on a limited number

of specific cases)of specific cases) Types of Quantitative DesignsTypes of Quantitative Designs

• Pure experimental design: assume Pure experimental design: assume randomization of subjects to treatment and randomization of subjects to treatment and controlcontrol

• Quasi experimental design: randomization not Quasi experimental design: randomization not assumedassumed

1515

Pure Experimental DesignPure Experimental Design(Abraham & Cracolice, 1993-94)(Abraham & Cracolice, 1993-94)

Pretest-Post TestPretest-Post Test• Each student randomly assigned to treatment Each student randomly assigned to treatment

or control groups.or control groups.• Use pretest to measure variables of interest.Use pretest to measure variables of interest.• Implement treatment.Implement treatment.• Use post test (same as or equivalent to Use post test (same as or equivalent to

pretest).pretest). ProblemsProblems

• Possible interaction between pre and post Possible interaction between pre and post tests.tests.

1616

Pure Experimental DesignPure Experimental Design

Post-test only DesignPost-test only Design• Randomly assign students to treatment or control Randomly assign students to treatment or control

groups.groups.• Present treatment.Present treatment.• Administer post test on variables of interest.Administer post test on variables of interest.

Potential problemsPotential problems• Treatment and control groups may not be equivalent Treatment and control groups may not be equivalent

even though randomly assigned.even though randomly assigned.• If there is a differential drop out from treatment vs. If there is a differential drop out from treatment vs.

control groups, a biased sample may result.control groups, a biased sample may result.• Without pretest, cannot analyze students according to Without pretest, cannot analyze students according to

subgroup (low vs. high achieving students).subgroup (low vs. high achieving students).

1717

Quasi-Experimental DesignQuasi-Experimental Design

Students are not randomly assigned to Students are not randomly assigned to groups.groups.

Identification and measurement of Identification and measurement of pertinent covariables are compared pertinent covariables are compared between treatment and control groups between treatment and control groups to assure equality of groups.to assure equality of groups.• Gender, age, mathematical aptitude, Gender, age, mathematical aptitude,

logical reasoning ability, academic logical reasoning ability, academic accomplishment, science vs. nonscience accomplishment, science vs. nonscience major, etc.major, etc.

1818

Quasi-Experimental DesignQuasi-Experimental Design(Jaeger. Pg. 592)(Jaeger. Pg. 592)

Interrupted Time Series Interrupted Time Series • Data recorded for pertinent variable for many Data recorded for pertinent variable for many

consecutive points in time both before and consecutive points in time both before and after treatment is introduced.after treatment is introduced.

• Pattern of graph is studied before and after Pattern of graph is studied before and after treatment.treatment.

If graph shows an abrupt shift in level or direction at If graph shows an abrupt shift in level or direction at point of treatment, then treatment is cause.point of treatment, then treatment is cause.

AdvantageAdvantage• Useful with a small number of subjectsUseful with a small number of subjects

ProblemProblem• Separating out other possible causes of the Separating out other possible causes of the

shift in graph can be difficult.shift in graph can be difficult.

1919

Quasi-Experimental DesignQuasi-Experimental Design

One classroom designOne classroom design• Covariables of interest identified and Covariables of interest identified and

measured.measured. Ex. Logical reasoning ability, gender, Ex. Logical reasoning ability, gender,

academic aptitudeacademic aptitude

• Treatment applied.Treatment applied.• Achievement/Attitude measures Achievement/Attitude measures

administered.administered.• AnalysisAnalysis

Differences in achievement/attitude of Differences in achievement/attitude of subgroups, as identified through use of subgroups, as identified through use of covariables, are compared.covariables, are compared.

2020

Mixed Evaluation MethodsMixed Evaluation Methods(NSF 1997)(NSF 1997)

Approach combining both qualitative and Approach combining both qualitative and quantitative methodsquantitative methods

AdvantagesAdvantages• TriangulationTriangulation

validity of results strengthened by using more than validity of results strengthened by using more than one method to investigate question.one method to investigate question.

• Improved instrumentationImproved instrumentation Qualitative open-ended survey results used to design Qualitative open-ended survey results used to design

Likert scale survey.Likert scale survey.

• Mixed results may lead investigators to modify Mixed results may lead investigators to modify or expand research design.or expand research design.

2121

Mixed Methods:Mixed Methods:SurveysSurveys

Open-ended question surveys Open-ended question surveys (qualitative).(qualitative).• Analysis of responses can lead to Analysis of responses can lead to

development of a validated Likert Scale development of a validated Likert Scale Survey. Survey.

Likert Scale Survey (quantitative)Likert Scale Survey (quantitative)• Subjects respond to a series of Subjects respond to a series of

questions/statements by indicating the questions/statements by indicating the amount of agreement/disagreement on a amount of agreement/disagreement on a scale of 1scale of 15.5.

2222

Mixed Methods:Mixed Methods:InterviewsInterviews

Structured interviewsStructured interviews• Fixed sequence of pre-determined questionsFixed sequence of pre-determined questions• Interviewer does not deviate from script.Interviewer does not deviate from script.

Semi-structured interviewsSemi-structured interviews• Interviewer is free to ask spontaneous questions of Interviewer is free to ask spontaneous questions of

intervieweeinterviewee Think Aloud interviewThink Aloud interview

• Present subjects with a demonstration or Present subjects with a demonstration or problem. problem.

• Interviewee solves and thinks aloud.Interviewee solves and thinks aloud.• Interviewer asks questions such as Interviewer asks questions such as

“ “ What are you thinking?”What are you thinking?”

2323

Mixed Methods:Mixed Methods:InterviewsInterviews

Unstructured InterviewsUnstructured Interviews• Interviewer asks subject questions as Interviewer asks subject questions as

they arise. they arise. Not all subjects are asked the same Not all subjects are asked the same

questions.questions.

2424

SummarySummary Project EvaluationProject Evaluation

• Tied to project goalsTied to project goals• Includes responses of stakeholders in projectIncludes responses of stakeholders in project• Includes both formative and summative evaluationIncludes both formative and summative evaluation

MethodologyMethodology• Match methodology to evaluation questionsMatch methodology to evaluation questions

Mixed Methods uses both qualitative and quantitative Mixed Methods uses both qualitative and quantitative methods.methods.

• Validity of evaluation strengthened by using multiple methods Validity of evaluation strengthened by using multiple methods (triangulation).(triangulation).

ResultsResults• Inform stakeholdersInform stakeholders• Lead to actionLead to action

2525

ReferencesReferences Abraham, M. R. and M. S. Cracolice. (1993-94). Doing Research on Abraham, M. R. and M. S. Cracolice. (1993-94). Doing Research on

College Science Instruction: Realistic Research Designs. Journal of College Science Instruction: Realistic Research Designs. Journal of College Science Teaching. Dec.93/Jan.94, pp.150-153.College Science Teaching. Dec.93/Jan.94, pp.150-153.

Jaeger, R. M. (Ed.) (1988). Complementary Methods for Research Jaeger, R. M. (Ed.) (1988). Complementary Methods for Research in Education. 2in Education. 2ndnd Ed. Washington, DC: American Educational Ed. Washington, DC: American Educational Research Association.Research Association.

Miles, M. B. and Huberman, A. M. (1994). Qualitative Data Miles, M. B. and Huberman, A. M. (1994). Qualitative Data Analysis, 2Analysis, 2ndnd Ed. Newbury Park, CA: Sage. Ed. Newbury Park, CA: Sage.

The 2002 User-Friendly Handbook for Project Evaluation. (2002). The 2002 User-Friendly Handbook for Project Evaluation. (2002). Washington, DC: National Science Foundation (02-057).Washington, DC: National Science Foundation (02-057).

User Friendly Handbook for Mixed Methods Evaluations. (1997). User Friendly Handbook for Mixed Methods Evaluations. (1997). Washington, DC: National Science Foundation (97-153).Washington, DC: National Science Foundation (97-153).