Quality Assessment July 31, 2006Quality Assessment July 31, 2006

Informing Practice

By the end of this session, we will …

• Participate, learn and have fun!• Answer,

– Why is it important to ask?– How do we “inform our practice” through

four stages?– Will assessing make a difference?– Do I have the skills to begin successfully

and feel good about what I am doing?

Why Bother?

• If you always do …

• Our need to know. . . – How are we doing? – How well are our students doing? – How do we know? – Have we made a difference?– Have we met our goals?

• Answer in a systematic way: credibility

Asking:• Provides “informed answers”

- Speak knowledgeably to answer, “How do you know?”

• Demonstrates that we are serving our students:– Interested in knowing if we delivered what we

promised; Listening– Gathering evidence to use to improve

• Contributes to our own learning

Why Bother?

Natural Process

Asking after an event occurs:

• What was good about it?

• What was not so good?

• What will we do different next time? Why?

Guiding Principles

• There is no single “right answer”

• Process of learning together

• Value added process through synergy

Informing Practice

Assessment Assessment

CycleCycle

3. Data Collection

4. Data Analysis,

Reporting, and Action

1. Setting Measurable

Goals

2. Planning to Reach

Goals

Mission Statement

Mission Statement

Strategic Goals

Strategic Goals

1. Setting Measurable Goals

Planning Questions Goals

Students will participate in an effective experience that develops their interpersonal and leadership skills.

Students will be able to rate the two aspects of the experience and demonstrate an example of applying their skills of collaboration and evaluating their leadership actions.

What do we want our students to be able to know and do?

What are observable and measurable outcomes (behaviors to track) that will let us know what our students know and can do?

What tasks will students engage in that demonstrate what we expect of them?

Students will engage in experiences with two aspects where they will apply their skills of collaboration and evaluate their leadership actions.

Tools may be a survey, interview, observation checklist, etc. based on outcomes.

What tool is used to measure the indicator?

2. Planning to Reach Goals

• Advice from “The expert”

• Review goals

• Data collection design

• Data analysis, reporting, and action

• Set future goals

3. Data Collection

Planning for Success:

• Purpose

• Process: – Who? – What? – How? – When?

• Lessons Learned

4. Data Analysis, Reporting, and Action

Results• Analyze data to learn what was said

• Report and communicate to “Close the Loop”

• Action plans for the futureData driven decision makingCompleting the Cycle

3. Data Collection review

Planning for Success:

• Purpose

• Process: – Who? – What? – How? – When?

• Lessons Learned

Purpose

• Clearly write: The purpose for our survey is to. . .

• State:– What do we want to know? – What we will do with the information; how we

will use the assessment results for improvement?

Informs our Data Collection DesignTurns into Letter to Participants

determine

discover

Process – Who?

Who do we ask?

• As researchers, we cannot assume that

we know what everyone is thinking

• The ones who are able to answer the

questions from their perspective

Process – What?

• What do we ask?– Review purpose statement

• Pilot:– Do the questions work? – What information will they give us? – Will the information inform our decision

making?– Be cognizant of time of participant

Process – How?

How do we ask to get information? • Open-ended question format• Close-ended question format

– Likert (feelings / attitude / opinion) scale of 1 to 5; 1 to 7; 1 to 4; others

– Yes / No answers

• Paper / electronic• Focus groups

– Using scripts; recorder/ x-check; skilled interviewer

• Consider need? Consent letter / Institutional Review Board (IRB)?? Anonymous / confidential?

Process – When?

When do we ask?

• Immediately, or risk “time heals” syndrome

• Later, to benefit from reflection

• Check Survey Central– Has it been asked before?– Avoid “survey fatigue”

Lessons Learned

• Critique survey examples

• Response population analyzed

• Letter to Participants

• Pilot

• Scales and ratings

Lessons Learned

Critique

survey

examples

Lessons Learned

Response population analyzed

• What is a good response rate?

• Sample Size Calculator

• Population Profile

Lessons Learned

Letter to

Participants• Content

• Message

Lessons Learned

Write an excellent letter of invitation to participate:

• Identify self, why the survey is happening

• What will be done with the results

• Note changes made in past

• “Data will be reported in aggregate form only”

• Incentives?

• Signatory? Personal connection makes a difference

• Remember: Be empathetic

Lessons Learned

Pilot• Is wording clear?

• Do the questions “work”? – What information will they give us? – Is the information meaningful?

• Be cognizant of time of participant

Lessons Learned

Likert Scale• 5 point scale: mid-point cluster• 7 or higher point scale: many choices• 4 point scale: forced opinion• Define each level

When rating, ask why. . . • Rating was only negative responses. • Rating was replicate the good.

Lessons Learned

Constructing questions:• Short and clear. Avoid misinterpretations.

• Consider pairing statements to ensure reliability and validity

• Avoid “and”. e.g. This limits statements to a single issue.

• Pilot

Will assessing make a difference?

Data has contributed to our need to know. . . – We are… – Our students are… – We know… – We made a difference in these ways…– The goals we met are…

We have the evidence!

Informing Practice

Quality AssessmentQuality Assessment

July 31, 2006July 31, 2006

Written and produced byWritten and produced by

Halyna KornutaHalyna Kornuta