Title

Our Outcomes

• Define evaluation in relationship to professional learning;

• Examine the process of teacher change and its impact on student learning; and

• Acquire strategies, tools, and resources to assist in evaluating professional learning - Building PD Capacity Toolkit (link)

Our Essential Question

• How can evaluating professional learning leverage school, school system, and state improvement effort?

• How will I align professional learning objectives to measurable short, medium and long-term results for educators and students?

• How will I collaborate with others to construct a framework that outlines a detailed plan for evaluation?

• How do I incorporate evaluation into my work and normative practice?

Our ‘Norm’

The Standards for Professional Learning

Professional learning that increases educator effectiveness and results for all students uses a variety of sources and types of student, educator, and system data to plan, assess, and evaluate professional learning.-Standards for Professional Learning, 2011

http://www.ncpublicschools.org/docs/effectiveness-model/ncees/standards/prof-learn-quick-ref.pdf

Link to Student Results

1. Standards-based

professional learning

2. Changes in educator

knowledge, skills, and dispositions

4. Changes in studentresults

3. Changes in educator practice

(Get there by…)Evaluation is:

A systematic, purposeful process of studying, reviewing, and analyzing data gathered from multiple sources in order to make informed decisions about a program. – Killion, 2008

The systematic investigation of merit and worth. – Thomas Guskey, 2000

9

Guskey’s Evaluation Framework

Professional Development

QualityContentContextProcess

Level 1:Reaction to Professional

Development

Level 2:Learning

Level 4:Changed

Instruction

Level 5:Student

Outcomes

Level 3:Organizational Support

(practices, policies, resources)

Adapted from Guskey, 2000, pp. 79–81

4 Key Components to Evaluating PD

Systematic(Established process)

How rigorous is the process?Is it conducted in accordance with standards and guidelines?

Standards(Predetermined criteria)

Does it have merit and/or worth?Does it meet predetermined standards of success?

Audience Who will use the evaluation?For whom is the evaluation being done?

Intended Uses

How will the evaluation be used?What decisions will be made as a result of the evaluation?

8 Smooth Evaluation Steps

Planning

1. Determine ‘what’ to evaluate

2. Formulate Evaluation Questions

3. Construct Evaluation Framework

Conducting

4. Collect Data

5. Organize, Analyze, & Display Data

6. Interpret Data

Reporting

7. Disseminate and Use Findings

8. Evaluate the Evaluation

Your Evaluation Process

Write 4-5 sentences that describe the evaluation process/steps you currently use as a leader responsible for assisting others with evaluating professional learning.

Which best describes your experience?

A B

Externally driven and designed Internally driven and designed

Summative evaluations only Planning, formative, and summative evaluation

Event-based Program-based Looking for answers/solutions form others

Discovering or creating solutions and alternatives with others

Feared Embraced

Filed/shelved Used

Done as an afterthought Planned from the beginning

Documentation Evaluation

Process-focused Results-driven

Presentation of results Reflective dialogue

Shifting PerspectivesFrom To

Externally driven and designed Internally driven and designed

Summative evaluations only Planning, formative, and summative evaluation

Event-based Program-based Looking for answers/solutions form others

Discovering or creating solutions and alternatives with others

Feared Embraced

Filed/shelved Used

Done as an afterthought Planned from the beginning

Documentation Evaluation

Process-focused Results-driven

Presentation of results Reflective dialogue

Group Think – Table talk, share out or Padlet

1. What aspect of evaluating professional learning do you find essential?

2. How do we know that the professional learning is making its way to the classroom?

Means goals vs. End goals

Means End Provide professional learning (Activities or strategies employed to achieve a result)

e.g. Train 100% of the teachers on differentiation.

Increase student achievement results(Results or behavioral changes expected)

e.g. Student achievement increase by 10% on a reading inventory.

Thorough planning facilitates sound

evaluation.

3 Types of Evaluation

1. Planning – before program design to provide information on conditions or needs to address

2. Formative – during implementation to provide information on whether the program is working as designed

3. Summative – after completion to provide information on outcomes or overall impact

Tiers and Benchmarks

1. Multiple settings2. Data sources – affective, quantity,

performance data3. Initial vs. embedded4. 5 year plan for data collection

Think of something to do here

Lead Box Evaluations

Superman xray

Input Output

Actions Results

Glass Box Evaluations

?A simplistic approach to professional learning evaluation that fail to amplify the

underlying theory and operation of the professional learning program.

A comprehensive approach to professional learning evaluation that illuminates how professional learning program components interact to produce results.

Lead Box(Superman xray)

ProfessionalLearningAction

Student Achievement

Results?

Focus on outputs rather than what

occurs in the program or what is presumed to be causing those outcomes and why.

Student Achievement

Results

ProfessionalLearning

CurriculumDevelopment

Nonacademicfactors

?

Lead Box Focus on inputs and fail to shed light on HOW a program’s

activities and resources interact to

produce results.

Student Achievement

Results

ProfessionalLearning

Actions

Glass Box Focus on what occurs and how it occurs

within the program.

Glass Box

Student Achievement

Results

ProfessionalLearning

Coaching/Follow-up

Instructional Resources

Implementation Monitoring

Student Assessment

Focus on illuminating factors contributing to transformation

process.

Lead Box vs. Glass Box

What is the difference?

Transition

Change is Learning

Concerns-Based Adoption Model : Developed by Bill Rutherford, Gene Hall, Shirley Hord, and Susan Loucks-Horsley

4 Components:1. Stage of Concern – 7 stages of responses2. Levels of Use – eight ranges of intervention use3. Innovation Configuration – described actions4. ‘Change’ facilitators – leaders of learning

***(30 min)Change Learning Exchange

• Distribute numbered cards (#1-4) at your table.• Read and the corresponding article on change.• Prepare a two-minute talk about your article. Use

any of the following to prepare.– Why your focus area is important.– Implications of the change process.– Ways to facilitate your area of change. – Challenges you anticipate when helping

others understand this area of change.

Change Spotlight

• Find a partner that read a different article.• Take two minutes each, discuss your article.

Focus on any of the following: – Why your focus area is important.– Implications of the change process.– Ways to facilitate your area of change. – Challenges you anticipate when helping

others understand this area of change. • Listen for the timer to repeat the process.

Tech - Note to Self

• What was one benefit of revisiting the change process?

• What new insights did you gain as a result of your reading and discussion with others?

Planning Tools

• Steps to your own Evaluation• 8 Smooth Steps• Link to DuFour Questions• Long term planning – matrix?

Fix - Setting Standards for Acceptable Performance

• Specifies how good is good enough• Specifies “success” in advance• Provides a benchmark/baseline for

comparison before and after professional learning

Teachers participate in collaborative learning experiences.

Teachers implement new learning in their instruction.

Student performance increases.

Professional development core features:•Content focus

•Active learning

•Coherence•Duration•Collective participation

Increased teacher knowledge and skills; changes in attitude and beliefs

Change in instruction

Change in student learning

Core Conceptual Framework for Studying the Effects of Professional Development on Teachers and Students Desimone, L. (2009)

Professional Learning

Opportunities

Teacher interpretation and

utilization of available

understandings and skills

Teacher outcomes:•Change in practice•No change in practice

Student outcomes:

•Positive changes•No changes•Negative changes

Students’ interpretation and

utilization of available understandings and

skills

Student Learning

Opportunities

Timperley, H. & Alton-Lee, A. (2008). Reframing teacher professional learning: An alternative policy approach to strengthening valued outcomes for diverse learners. Review of Research in Education, 32, pp. 328-369.

Resource: Assumptions About Professional Learning

Do with your team1. Read each assumption statement. 2. Code each with your level of agreement.3. Identify potential implications for the role of

teacher evaluation.4. Share your thoughts and answer the questions

with others at your table.

Ending Thoughts: Evaluation Assumptions

• The staff development program is data-driven, research-based, and well-defined.

• The school, district, or regional agency has the capacity, including fiscal and human capital, to implement both the program and evaluation with fidelity to their designs.

• Key stakeholders in the school, district, or agency intend to use the evaluation results to make decisions about the program.

Debrief

1. Reflect on the learning for the day using the the following three areas to guide your thinking.

– What are the benefits?– What are the barriers?– What are the implications?

2. Consider beginning an evaluation framework (see sample) or theory of change for a current professional learning program.

?? FTT Google Doc and SI common Eval

• How are the ideas presented today CONNECTED to what you already knew?

• What new ideas did you get that EXTENDED or pushed your thinking in new directions?

• What is still CHALLENGING or confusing for you to get your mind around? What questions or wonderings do you now have?