Embed Size (px)
Citation preview
Evaluation of Hawaii’s Supplemental Evaluation of Hawaii’s Supplemental Educational Services ProgramEducational Services Program
DRAFT Evaluation PlanDRAFT Evaluation Plan
Presentation to Service Providers and Complex AreasPresentation to Service Providers and Complex Areas April 2006 April 2006
Office for Evaluation and Needs AssessmentOffice for Evaluation and Needs AssessmentSocial Science Research InstituteSocial Science Research Institute
University of HawaiiUniversity of Hawaii
Evaluation TeamEvaluation Team
• Dr. Judith Inazu, P.I., Associate Director, Social Dr. Judith Inazu, P.I., Associate Director, Social Science Research InstituteScience Research Institute
• Dr. Shuqiang Zhang, Statistician, College of Dr. Shuqiang Zhang, Statistician, College of EducationEducation
• Dr. Daniel Anderson, Consultant, Planning and Dr. Daniel Anderson, Consultant, Planning and Evaluation, Inc.Evaluation, Inc.
• Dr. Aiko Oda, Consultant, Planning and Evaluation, Dr. Aiko Oda, Consultant, Planning and Evaluation, Inc.Inc.
• Ms. Nancy Marker, Educational Specialist, Social Ms. Nancy Marker, Educational Specialist, Social Science Research InstituteScience Research Institute
• Dr. Patty Reiss, Lecturer, College of Dr. Patty Reiss, Lecturer, College of EducationEducation• Ms. Julie Holmes, Graduate Assistant, Social Ms. Julie Holmes, Graduate Assistant, Social
Science Research InstituteScience Research Institute
Goal of SESGoal of SES
To increase the academic To increase the academic achievement of economically achievement of economically disadvantaged students in low-disadvantaged students in low-performing schools by performing schools by providing additional academic providing additional academic instruction outside of the instruction outside of the regular school day. regular school day.
Purpose of EvaluationPurpose of Evaluation
To determine the effectiveness of service To determine the effectiveness of service providers in increasing students’ providers in increasing students’ academic achievement. academic achievement.
Providers must:Providers must:
►► Increase students’ achievement for 2 Increase students’ achievement for 2 consecutive years, andconsecutive years, and
► ► Provide services consistent with Provide services consistent with applicable federal, state, and local health, applicable federal, state, and local health, safety, and civil rights requirements.safety, and civil rights requirements.
Guiding QuestionsGuiding Questions
1.1. EffectivenessEffectivenessDid the provider increase student achievement in Did the provider increase student achievement in reading and/or mathematics?reading and/or mathematics?
2.2. Customer Satisfaction Customer Satisfaction Are parents, schools, and complex areas involved in Are parents, schools, and complex areas involved in SES satisfied with the service provider?SES satisfied with the service provider?
3.3. Service DeliveryService DeliveryDid the provider comply with applicable federal and Did the provider comply with applicable federal and state laws and regulations, and contractual state laws and regulations, and contractual procedures and requirements associated with the procedures and requirements associated with the delivery of SES? delivery of SES?
Interim and Final Evaluation PlansInterim and Final Evaluation Plans
Interim Evaluation PlanInterim Evaluation Plan
Year 1 (Fall 2005 - Summer 2006) Year 1 (Fall 2005 - Summer 2006)
Year 2 (Fall 2006 - Summer 2007)Year 2 (Fall 2006 - Summer 2007)
Year 3 (Fall 2007 - Summer 2008)Year 3 (Fall 2007 - Summer 2008)
Final Evaluation PlanFinal Evaluation Plan
Year 4 (Fall 2008 - Summer 2009)Year 4 (Fall 2008 - Summer 2009)
Measuring Academic AchievementMeasuring Academic Achievement
In Year 1 of the Evaluation, In Year 1 of the Evaluation, academic achievement will be academic achievement will be
the sole criterion for the sole criterion for evaluating service providersevaluating service providers
Measuring Academic AchievementMeasuring Academic Achievement
• Analysis to be conducted Analysis to be conducted separately for each separately for each gradegrade by by subject mattersubject matter, and for , and for each each providerprovider..
• Example: For Example: For 44thth graders graders who who received tutoring in received tutoring in readingreading by by College ConnectionsCollege Connections..
Minimum Data Set for AnalysesMinimum Data Set for AnalysesData on all SES-eligible students. Data on all SES-eligible students.
Arrayed by grade level, subject matter, and service provider.Arrayed by grade level, subject matter, and service provider.
Student IDStudent ID Enrollment in Enrollment in SESSES
Prior HSA Prior HSA ScoreScore
2006 HSA 2006 HSA ScoreScore
001001 YesYes
002002 YesYes
003003 YesYes
004004 YesYes
005005 YesYes
006006 NoNo
007007 NoNo
008008 NoNo
009009 NoNo
010010 NoNo
AnalysesAnalyses
Analyses of Covariance (ANCOVA) with prior Analyses of Covariance (ANCOVA) with prior HSA score as the covariate, SES status HSA score as the covariate, SES status (yes/no) as the binary independent variable, (yes/no) as the binary independent variable, and 2006 HSA score as the dependent and 2006 HSA score as the dependent variable. variable.
The ANCOVA asks the question, “If you hold The ANCOVA asks the question, “If you hold the pre-test scores constant (since all the pre-test scores constant (since all students begin at different levels of students begin at different levels of performance), is there a significant performance), is there a significant difference in post-test scores between difference in post-test scores between students who received tutoring and students students who received tutoring and students who did not?”who did not?”
Analyses for Students Without Pre-test Analyses for Students Without Pre-test ScoresScores
Conduct t-test analyses (difference between Conduct t-test analyses (difference between mean scores)mean scores)
• Compare 2006 HSA test scores between Compare 2006 HSA test scores between students who received tutoring and those students who received tutoring and those who did not.who did not.
• Again, analyses conducted separately by Again, analyses conducted separately by grade level, subject matter, and for each grade level, subject matter, and for each service provider.service provider.
Measuring Customer SatisfactionMeasuring Customer Satisfaction
Pilot tested with selected samples inPilot tested with selected samples in
Years 1-3:Years 1-3:
• ParentsParents
• TeachersTeachers
• Complex Area AdministratorsComplex Area Administrators
• PrincipalsPrincipals
• Students (To-be-determined)Students (To-be-determined)
Measuring Customer SatisfactionMeasuring Customer Satisfaction
Pilot testing to begin in May 2006Pilot testing to begin in May 2006
• Paper and pencil questionnaires with Paper and pencil questionnaires with a select sample of parents, teachers, a select sample of parents, teachers, complex area administrators, and complex area administrators, and principals. principals.
• Translation requirements for parents.Translation requirements for parents.
• Internet survey for teachers, school Internet survey for teachers, school administrators, and complex areas. administrators, and complex areas.
Sample Questions: Parental SatisfactionSample Questions: Parental Satisfaction
1. The school staff was qualified and supportive in helping me get 1. The school staff was qualified and supportive in helping me get tutoring for my child. tutoring for my child.
o YesYeso NoNo
2. I would recommend this tutor to other parents. 2. I would recommend this tutor to other parents. o YesYeso NoNo
3. My child's school work improved because of the tutoring. 3. My child's school work improved because of the tutoring. o YesYeso NoNo
4. My child found tutoring was a positive experience. 4. My child found tutoring was a positive experience. o YesYeso NoNo
5. The tutors were good at reporting to me about my child’s 5. The tutors were good at reporting to me about my child’s progress. progress.
o YesYeso NoNo
6. Overall how would you rate this tutor? 6. Overall how would you rate this tutor? o StrongStrongo AverageAverageo WeakWeak
Measuring Service DeliveryMeasuring Service Delivery
Pilot testing to begin in May 2006Pilot testing to begin in May 2006
• Site visits and observations of Site visits and observations of tutoring sessionstutoring sessions
• Interview(s) with contact person(s) at Interview(s) with contact person(s) at the sitethe site
• Document review during site visitDocument review during site visit
• Self-administered compliance Self-administered compliance checklistchecklist
Instructional ObservationInstructional Observation
• Clear academic expectations are set and articulatedClear academic expectations are set and articulated
• Instruction is on task without interruptionInstruction is on task without interruption
• Criterion material is covered with instructor’s directionCriterion material is covered with instructor’s direction
• Instructional time is adequate for material yet flexibleInstructional time is adequate for material yet flexible
• Learning modalities are active, variable and appropriate Learning modalities are active, variable and appropriate
• Instructional pacing is appropriate for student’s interest Instructional pacing is appropriate for student’s interest
and ability and ability
• Student progress is frequently assessed Student progress is frequently assessed
• Evidence of mutual respect, positive interaction and Evidence of mutual respect, positive interaction and
feedbackfeedback
• Evidence of the instructor’s enthusiasm and interestEvidence of the instructor’s enthusiasm and interest
Instructional DeliveryInstructional Delivery
• Student-teacher ratio
• Computer, lectures, etc.
Compliance Checklist (Examples)Compliance Checklist (Examples)SES Provider AssurancesSES Provider Assurances
• Use research-based strategies designed to improve Use research-based strategies designed to improve
academic achievementacademic achievement
• Offer instruction consistent with state academic content Offer instruction consistent with state academic content
and achievement standardsand achievement standards
• Meet federal, state and local health, safety and civil rights Meet federal, state and local health, safety and civil rights
lawlaw
• Provide services that are secular, neutral, and non-Provide services that are secular, neutral, and non-
ideological ideological
• Provide information on student’s progress, as agreed Provide information on student’s progress, as agreed
upon with the LEAupon with the LEA
• Remain financially viableRemain financially viable
• Abide by Family Educational Rights and Privacy Act of Abide by Family Educational Rights and Privacy Act of
19751975
What We’ll Need From YouWhat We’ll Need From You
Complex AreasComplex Areas
• Written permission to be on campus Written permission to be on campus to conduct site observationsto conduct site observations
• Names and addresses of parentsNames and addresses of parents
• Completion of satisfaction surveyCompletion of satisfaction survey
What We’ll Need From YouWhat We’ll Need From YouService ProvidersService Providers
• Assistance in scheduling site visitsAssistance in scheduling site visits
• Set aside time for an interviewSet aside time for an interview
• Documents available for review (e.g., Documents available for review (e.g., attendance logs, student records, pre/post attendance logs, student records, pre/post test reports)test reports)
• Completion of compliance checklistCompletion of compliance checklist
