Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Evaluating your 21st CCLC Program and Lessons Learned by Veteran
Program DirectorsFY13 Program Training
August 7, 2012
Presenter: Michael Thaler21st CCLC State Program [email protected](404) 232-1197
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Goals of this Presentation
• Understand the importance of program evaluation• Discuss the components of the evaluation reports• Understand the roles and expectations of an external
evaluator• Learn about tools other 21st CCLC programs are using• Receive tips and knowledge that our most veteran
program directors wish they had known their first year
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
“We are doing a great job of running our programs and we are effective, but we aren’t
telling our story and that’s hurting us.”
Dr. Sylvia LylesBeyond School Hours XIII Conference February, 2010
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Perception vs. Reality• Perception
- Daycare centers- Unorganized or ineffective activities- Minimal impact on schools, families and community- Adult employment agency
• Reality- Highly organized out-of-school programs- Utilize research-based activities that specifically
target ‘at-risk’ or ‘high opportunity’ students- Provide critical services to a number of stakeholders
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Our job is to tell the story of our 21st CCLC programs to our various stakeholders so we can change their perception and help educate them about positive impact of our programs. This is accomplished by developing and utilizing an effective evaluation plan and following the
principles of effectiveness.
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Why should you do it?
Principles of Effectiveness (Sec. 4205 (b)) states ‘the program or activity shall undergo a periodic evaluation to assess its progress toward achieving its goal of providing high quality opportunities for academic enrichment’
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Principles of Effectiveness (Sec. 4205 (b))
The program or activity should be based upon:1. An assessment of objective data
regarding the need for the program2. An established set of performance
measures3. Scientifically-based research that
demonstrates the activity will help the students meet State or local standards
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
What can evaluations provide?
• In addition to letting you know how you are doing, effective evaluation plans can help:– Increase student and parent participation– Increase school support– Obtain additional partnerships– Obtain additional funding– Increase the effectiveness of your program
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Essential Steps for an Effective Evaluation Plan
1. Determine the needs of your students and families2. Establish clear goals, objectives, and activities based
on the identified needs3. Develop the timeline for the evaluation plan4. Collect and analyze baseline, formative, and
summative data5. Disseminate the evaluation results to your
stakeholders6. Refine the program and its activities for next year,
based upon the evaluation results
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Evaluation Requirements
• 3 Major Components:- Formative (mid-year) Assessment- Summative (end of year) Assessment- Cayen/Afterschool 21 data entry system
• Summer Evaluation, if your program will run one in 2013
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Formative Assessment• Must undergo at least one and it must be written by the external evaluator• Evaluates from the beginning of the program year to the time of the evaluation• At a minimum, the report must include:
– Student Attendance: Data on average daily attendance and enrollment at each site;
– Program Operation: Synopsis of current level of operation at each site;– Objective Assessment: Data analysis and indication of progress towards
achieving EACH objective (as required, all objectives must include measures that allow for continuous (formative) assessment); and
– Recommendations: Recommendations for programmatic refinement for all objectives where progress is not being achieved and/or where the program is not likely to achieve the stated objective by the end of the program year.
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Formative Assessment
• Due February 1, 2013• Important to continuously assess your
program throughout the year (i.e., grades, attendance)
• Effective programs make revisions throughout, based upon their formative assessments
• Share the results with stakeholders
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Summative Assessment
• Each program must submit a summative report by June 28th, 2012
• Evaluates the entire program year• The results of your summative report will be used to
help determine continuation of funding in subsequent years
• Must be written by your external evaluator and must follow the template on the following slides
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Summative Report Template• Each summative report must include, at minimum, the following major sections: • Overview and History: Provide a general overview of the 21st CCLC program,
including a history of previous operation and how the program has progressed and grown over the past year(s) of operation.
• Student Attendance and Enrollment: Total and Regularly-Participating Student Enrollment (per site); Student Demographics (e.g., gender, ethnicity); Average Daily Attendance (per site)
• Program Operation: The summative report must include information on operation for EACH site within the program (e.g., days, hours, and weeks of operation for each component).
• Quality of Staffing: The composition of site staff is one method for assessing the quality and breadth of 21st CCLC programs. For each site within the program, the summative report must include information about staffing (e.g., Staff Demographics, Student: Staff Ratio, Staff Training, and Use of Certified Teachers).
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Summative Report Template• Objective Assessment: Within the grant application, 21st CCLC programs
proposed both objectives and methods of evaluating progress towards achieving the objectives. The summative report must include detailed information for EACH OBJECTIVE approved by GaDOE (e.g., activities, data collected, timeline, analysis and results). Programs are NOT permitted to revise, reword, or otherwise change their objectives without specific written approval from GaDOE.
• Other Observations (Optional): If appropriate, provide other relevant findings pertaining to this 21st CCLC program. Other findings could include qualitative and/or quantitative data not related to specific objectives; quotes and/or statements from students, parents, and/or teachers; success stories of students within the program; and photographs demonstrating unique program activities and services.
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Summative Report Template
• Progress towards Sustainability: The summative report must provide information about program partnerships (e.g., partnership development, partner maintenance, and contributions to the program).
• Overall Recommendations: Provide an overall assessment of this 21st CCLC program, as well as any program-wide recommendations to enhance program quality.
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Major Outcomes/GaDOE’s Expectations
• GaDOE expects every 21st CCLC program to demonstrate the following:– Increase in grades from fall to spring (no less than 85% of
student grades entered)– Increase in CRCT/EOCT scores (no less than 85% of students
with scores in tested grade levels)
– Increase in school day and program attendance– Improvement in areas such as discipline, homework
completion, parent involvement, parent/student satisfaction, etc.
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Cayen/ Afterschool 21 Data Entry System
• Programs must enter data (grades, attendance, activities, etc.) into this system continuously throughout the life of the grant
• GaDOE uses this to fulfill its requirement of reporting data to US ED
• All new subgrantees must attend webinars and work closely with their assigned AS21 contractor
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
External EvaluatorProvides a critical service to our programs and should be an experienced
professionalContact multiple evaluators and ‘shop around’
- 21st CCLC experience?- Speak with some of their current programs and see if they will provide you with one of the reports
- Ensure you hire one early in the program yearCan only use 3% of your budget for this and must enter in to one year
contractsCannot be the person who wrote the grantThe quality of your evaluator can have a significant impact on the quality
your program!
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Role of an External EvaluatorRemember - They work for you, you don’t work for them
They are there to provide guidance and support, but ultimately the success of the program relies on the program administration
They are somewhat limited by the quality of data you provide to them
Refer to the operations manual for more information requirements/recommendations for the external evaluator
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
What must they provide?
– At a minimum:
- One formative assessment
- Summative assessment after the conclusion of the program year
- A list of recommended revisions based upon the results of the formative and summative assessments
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Suggestions: What to Request
• Conduct site visits at least twice a year– This can vary depending on the size of your grant or
location of your evaluator
• Assist with developing surveys and other measurement instruments
• Assist with the dissemination of evaluation results– i.e., One page flyer and/or brief PowerPoint presentation
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Suggestions: What to Request
• Participate and present at school/advisory/stakeholder meetings
• Conduct focus groups with students, staff, and/or parents
• Provide input in revising objectives and assessments
• Monthly meetings or teleconferences to ensure the evaluation plan is on track
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Helpful Hints
• Set up your contract so your payments align with deliverables, not just timeframes
• Clearly explain to them the evaluation requirements
• If possible, bring them to state/national conferences and state meetings
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Other Helpful Hints
• Disaggregate the results of your summative evaluation based on number of days attended:
• 0-29 days• 30-59 days• 60-89 days• 90+ days• Compare regularly participating students (30+ days) to non-
21st CCLC students• Track students for the life of your grant
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Additional Evaluation Tools
• Needs Assessments or Interest Surveys– Students - determine activities that would interest them– Consider pre and post surveys– Staff - determine the type of training that would benefit
them the most (example)– Staff - determine which types of enrichment activities they
would like to teach– Parents - not only what type of activities, but when and
where to hold them
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Professional Development Activity VI SI NI1. Transition school-day activities to After-School Programactivities2. Integrate related reading and math learning activities for enrichment and non-remediation classes.3. Evaluate and document learning in enrichment and non-remediation classes4. Develop course outlines and weekly lessons in accordance with program objectives5. Identify what teaching techniques promote learning in the After-School Program classroom. 6. Develop additional enrichment activities 7. Promote parental involvement and determine activities to keep them coming back 8. Discover supplemental teaching materials that work 9. Develop/refine classroom management skills for After-School Program10. Maintain a safe, secure classroom environment while promoting learning and creative expression11. Understand the influence of school, family and peers with delinquency12. Gain national, state and DJJ perspectives of After-School Programs13. How is the After-School Evaluated?14. Time to share After-School Program experiences with other after-school teachers15. How to get more community participation for speakers, classroom supplies, etc
DJJ AFTER-SCHOOL PROGRAMPROFESSIONAL DEVELOPMENT SURVEYINTRODUCTION: This professional development survey was developed for DJJ After-School Program teachers. Survey results will help determine future professional development activities. Read each statement carefully and place a check mark in the appropriate space that best represents your answer for Very Interested (VI), Somewhat Interested (SI) or Not Interested.
Comments Suggestions: ____________________________________________________________________________________________________________
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Additional Evaluation Tools
• Student progress reports
• Classroom/teacher observation forms
• Parent satisfaction survey
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Lessons Learned by Veteran 21st CCLC Program Directors
• I asked some of our most successful and experienced 21st CCLC program directors to come up with a list of tips and pieces of knowledge they wish they knew their first year
• Review of Lessons Learned handout– This is GOLD
36
Dr. John D. Barge, State School Superintendent“Making Education Work for All Georgians”www.gadoe.org
Any Questions?
• The biggest thing to remember is you have a support system of GaDOE staff and other subgrantees to help you through this process.