Evaluation in Gifted Programs

June 14-15, 2013University of Iowa

Contact Information

Mary Schmidt9219 Lakewood Pointe DriveNorwalk, IA 50211515.321.6090mrmsjs@gmail.com

Agenda Day 1

Welcome/Introductions Outcomes What is Program Evaluation? Guiding Principles Connecting PE to Standards Program Evaluation as Continuous

Improvement Best Practices in Program Evaluation Framing the Question Data Skills

Ten Things in Common

As a group– Identify ten things you all have in

common (no body parts)– 7 minutes– Discuss what you learned about

classmates

Your Thoughts

What do you know about program evaluation?

What do you want/need to know?

Outcomes

Develop an understanding of the purposes, procedures, and outcomes of program evaluation including formative and summative evaluation

Learn program evaluation models and procedures

Develop questions to drive local program evaluation

Explore data types, purposes, and skills

Expectations

Grade A: Completion of all in-class assignments

Program brochureProgram evaluation plan

Final course reflection Grade B:

Completion of all in-class assignmentsProgram evaluation planFinal course reflectionOr, submissions for an A grade are all completed but

are not of sufficient quality to merit the higher grade.

Managing Complex Change

= Change

Confusion=

=

=

=

=

+ + + +Vision Skills Incentives ResourcesGifted Plan

Adapted from Knoster, T., Villa R., & Thousand, J. (2000). A framework for thinking about systems change. In R. villa & J. Thousand (Eds.), Restructuring for caring and effective education: Piecing the puzzle together (pp. 93-128). Baltimore: Paul H. Brookes Publishing Co.

Anxiety

Resistance

False Starts

Frustration

+ + + +Skills Incentives ResourcesGifted Plan

+ + + +Vision Incentives ResourcesGifted Plan

+ + + +Vision Skills ResourcesGifted Plan

+ + + +Vision Skills IncentivesGifted Plan

+ + + +Vision Skills Incentives Resources

Discuss in a small group…

How effective is your gifted program?

How do you know?

Record your thoughts on TodaysMeet.

http://todaysmeet.com/UI-ProgEval2013

Close Enough…

“There is no such thing as professionalism without a commitment to evaluation of whatever it is that one supervises or produces—and to self-evaluation as well.”

—Michael Scriven, 2000

IN LIGHT OF THIS…

NO MORE

CARDIAC

DATA!!

Program Evaluation Is…

…the systematic collection of information about the activities, characteristics, and outcomes of

programs to make judgments about the program, improve program

effectiveness, and/or inform decisions about future programming.

--Robinson, 2009

What is Program Evaluation?

Chapter 21 “Evaluating Gifted Programs” by James Borland

Discuss the 5 A’s – What assumptions is the author

making?– What would you argue with?– What would you agree with?– What would you aspire to?– What would you ask?

Best Practices

Based on information in Chapters 1 & 2, what are some of the best practices in gifted program evaluation?

Guiding Principles & Attributes

Read p. 196-198 in Designing Services and Programs for High-Ability Learners

Discuss in a small group connections, insights, and questions

Complete the assessment on p. 203-4– Individually– Discuss with a partner– Large group sharing – one strength & one

growth area

National Standards

Visit http://www.nagc.org/index.aspx?id=546

Peruse the revised standards, paying special attention to #2.

Discuss with your partner the guidance these standards provide.

Do the same for Chapter 59 of Iowa Code (handout)

Making Connections

Pro

gra

m E

valu

ati

on a

s C

onti

nuous

Impro

vem

ent

Best Practices in Program Evaluation

Discuss in small groups:– What’s been the program evaluation

practice in your district?– How well do these practices align

with best practice?– What’s working?– What needs to be changed/improved?– How have you used the evaluation

results?

Memory Mingle

1. How has the information you’ve engaged with so far “pushed” your thinking?

2. Share in triads3. Identify connections and/or

conclusions4. Share with large group

Read “Asking the Right Questions: The Central Issue in Evaluating Programs for the Gifted and Talented” (ICON)

Read “General Evaluation Guidelines” found on ICON.

Asking the Right Questions

Work with a partner Identify three big ideas and

discuss how these big ideas play out in your setting.

What questions does the article raise for you?

“…one cannot evaluate what one cannot describe.”(Callahan, p.

41)

Describe your program in terms of its components.

Consider what’s common and different across grade levels/spans/buildings.

Identify audiences for this information

What questions naturally emerge?

Connecting Questions to Data Sources

Consider one to three of your most pressing questions.

Do you have data already available that will help answer the question(s)?

What additional data will you need to collect?

Developing a Program Brochure

Consider:– Purpose(s)– Audience(s)

Create the rough draft of a program brochure using the description you wrote earlier as the basis.

We geek Data Consider the following questions related

to a framework for program evaluation:– What questions do we have about our program?– What data do we need to answer the questions?– What data do we already collect?– Who do we collect data on or from?– How often do we collect it?– How do we use the results?– How and to whom to we provide feedback?– What PD do we provide? How do we identify PD

needs?– Do we have an action plan?

Data Skills

Find a partner Make a list of what you think

qualifies as data skills Identify the data skills you have.

IOW: What do you know how to do regarding data collection, organization, and use?

Data Strengths and Needs

Complete the Data Skills Self-Assessment.

Discuss results with a partner. Set a data improvement goal. What does your district need to

focus on in terms of gifted program evaluation?

Is your home energy efficient?

How would you know?

Is y

our

hom

e e

nerg

y

effi

cient?

Reflecting on the day…

What have you heard/learned today that reinforces your ideas about program evaluation?

What have you heard/learned today that’s changed your thinking about program evaluation?

What’s your most important take away from the day?

Evaluation in Gifted Programs

Day 2June 15, 2014

Agenda Welcome Back Types of Evaluations Evaluation Models Data Types, Purposes, and Uses Developing an Evaluation Plan

Welcome Back!

Form triads Share

– What do you know about your program today that you didn’t know yesterday?

– How did you acquire this knowledge?– How can this information be used to

improve programming?

Program Evaluation Is…

…the systematic collection of information about the activities, characteristics, and outcomes of

programs to make judgments about the program, improve program

effectiveness, and/or inform decisions about future programming.

--Robinson, 2009

Types of Evaluations

Planning– Needs Assessment

Formative– “Miracle in the Middle”

Summative– Impact

Program Performance Measures:

Questions about Program Service Delivery

Quantity(How much did we do?)

Quality(How well did we do

it?)

Input(Effort)

How much service did we deliver?

How well did we deliver service?

Output(Effect)

How much effect/change did

we produce?(#)

What Quality of effect/change did

we produce?(%)Is anyone better off?

Black Box Evaluation

Glass Box Evaluation

Input Output

Input Output

Worth

The extent to which a program or activity is essential to a school’s, district’s, agency’s, or individual’s mission. Worth is an indication of the program’s or activity’s perceived value to constituents or to a single individual.

--Assessing Impact training, 2003

Merit

The value of the program is judged by comparing its performance against established standards of excellence in the profession.

--Assessing Impact training, 2003

How Could a Program Have…

…worth but no merit?

…merit but no worth?

…both worth and merit?

Thus

A program may have great merit yet be of little worth because it is not aligned with the organization’s mission or needs or it may have great worth and little merit…programs can be evaluated both on the basis of their worth and merit.

--Assessing Impact training, 2003

Program Evaluation Models and Tools

Arkansas Evaluation Initiative Borland Evaluation Template Maker’s Responsive Model Self-Audit/Reflection Tool

AEI

Designed to support in-house (formative) evaluation

Don’t need to use all data sources

Borland Template

Types of Data

Outcome– What they got

Key Performance Indicators

Demographic– Who got it

Disaggregation

Process– How they got it

Change Agents (data that guides you toward change)

Which Type of Data?

Percentage of… students identified for gifted services. students scoring a 4 or 5 on the AP exam teachers employing curriculum

compacting the day students receive gifted services students who attain PEP goals identified minority students identified students who are accelerated

(in each acceleration category)

Program Performance Measures:

Questions about Program Service Delivery

Quantity(How much did we do?)

Quality(How well did we do

it?)

Input(Effort)

How much service did we deliver?

How well did we deliver service?

Output(Effect)

How much effect/change did

we produce?(#)

What Quality of effect/change did

we produce?(%)Is anyone better off?

Types of Evaluations

Planning– Needs Assessment

Formative– “Miracle in the Middle”

Summative– Impact

Data Use and Purpose

Download the document Work as a team to complete Large group processing

Data Analysis Questions

What process do we have in place to use data in the areas of curriculum, instruction, and assessment? How does this align with the general curriculum process in the district?

What types of data do we regularly collect? Are there other things we need to collect?

What processes and techniques do we have to organize and analyze our data? What do we need?

What data do we want/need to disaggregate? How are we using data to determine areas of

maintenance, improvement and focus? How effectively are we using data to make

decisions?

Let your direction determine your decisions.Let your direction determine your decisions.

Developing an Evaluation Plan

Consider the place of planning, formative, and summative evaluations in your plan.

What’s the timeline? What will be evaluated and how and by

whom will the evaluation questions be formulated?

Visit the Wiki (link on ICON) for samples and templates.

Required Submissions Develop a program brochure.

– Visit ICON and review the link (under Course Information) that provides steps and tips to develop a brochure.

– Make sure you include Logo District name Clear purpose Distinct audience Layperson-friendly language

Required Submissions Develop a program evaluation plan. Include formative and summative

components Use the template on p. 203-4 to help

you develop the framework Exact format is up to you, but make it

flexible enough that the evaluation question(s), data collection instruments and methods, and communication of results could change from year to year within the same framework.

Required Submissions

Reflection on your learning in this class.– How does your evaluation plan align

with your written plan?– What do you still need to learn/do to

successfully evaluate your program?– How do you see program evaluation as

a tool and a process to improve programming?

Submissions

A copy of your brochure Evaluation plan including at least

one survey Reflection Due June 29