1

Outcome Measurement 101Outcome Measurement 101

145

Step 4Step 4Prepare to collectPrepare to collect

146

Prepare to collect Prepare to collect datadata

ETHICS: ETHICS: Informed Informed ConsentConsent••Subjects Subjects as Volunteersas Volunteers••Risks Risks & Discomforts& Discomforts••No HarmNo Harm••ConfidentialityConfidentiality

147

yy••Right to RefuseRight to Refuse••Free Free to Stop at any to Stop at any timetime

ETHICS:ETHICS:••DeceptionDeception••Denial of treatmentDenial of treatment••Appropriateness of research questionsAppropriateness of research questions••CompensationCompensation

148

pp

2

ConceptsConcepts are not are not directly measurabledirectly measurable

149

So we measure So we measure indicatorsindicatorsto assess the presence or to assess the presence or absence of conceptsabsence of concepts

150

151

•• Indicators: Indicators: proxyproxy measures of outcome concepts measures of outcome concepts •• Proxy indicators are Proxy indicators are measurablemeasurable•• Proxy indicators are Proxy indicators are variablesvariables (vary among clients)(vary among clients)

•• Proxy indicators signal target Proxy indicators signal target outcomeoutcome successsuccess•• Outcome success indicates Outcome success indicates programprogram successsuccess

Program successProgram success program worthprogram worth

152

•• Program success = Program success = program worthprogram worth

3

••ThusThus, indicators measure , indicators measure ““program success/worthprogram success/worth””

••Outcome objectives Outcome objectives hypothesize “hypothesize “SuccessSuccess” targets” targets

••Hypothesize “Hypothesize “SuccessSuccess” targets are usually ” targets are usually expressed as a number, percent, statistic, etc. of expressed as a number, percent, statistic, etc. of persons achieving some prepersons achieving some pre--specified specified targettarget

153

““OutcomeOutcome” objectives” objectives––Describe clients targeted & served Describe clients targeted & served ––Are Are clearclear , , conciseconcise, , measurablemeasurable––Unique to your approachUnique to your approach

WhoWho will be targetedwill be targeted

154

Who Who will be targetedwill be targetedHowHow (in what way) they will change(in what way) they will changeWhatWhat will be donewill be doneWhen When it will occurit will occurWhere Where it will happenit will happen

A sample “A sample “OutcomeOutcome” objective” objective

“Seventy percent (70%) of “Seventy percent (70%) of teen students participating in teen students participating in SynerjexSynerjex onon--site aftersite after--school school

155

mentoring groups will reduce mentoring groups will reduce their truancy rate by 100% by their truancy rate by 100% by

June 1, 2010”June 1, 2010”

Now practice!Now practice!

11 WhWh ill b t t dill b t t d

For each of your 4 proximal outcomes…For each of your 4 proximal outcomes…

156

1.1. Who Who will be targetedwill be targeted2.2. HowHow (in what way) they will change(in what way) they will change3.3. WhatWhat will be donewill be done4.4. When When it will occurit will occur5.5. Where Where it will happenit will happen

4

157

What outcome What outcome indicators to measure?indicators to measure?

158

The measurement instrumentThe measurement instrument

We started building our We started building our variable listvariable list

••Outcomes Outcomes (4)(4)

159

•• EE factorsfactors••Client traitsClient traits••Other stuffOther stuff

Many things on our Many things on our variable may still be variable may still be concepts, so we need to concepts, so we need to ensure we selectedensure we selected

160

ensure we selected ensure we selected measurable indicatorsmeasurable indicators

5

Now, what level of Now, what level of measurement should we measurement should we use?use?

161 162

Examples of Examples of nominalnominal measuresmeasures

1.1. What is your gender? What is your gender? (Please circle one)(Please circle one)........................................Male……FemaleMale……Female

2.2. Who currently lives in your home? Who currently lives in your home? (Please circle all that apply)(Please circle all that apply)

a)a) FatherFatherb)b) M thM th

163

b)b) MotherMotherc)c) SisterSisterd)d) BrotherBrother

No rankingNo ranking

Examples of Examples of ordinalordinal measuresmeasures

1.1. Right now, how Right now, how confident confident are you in client outcome evaluation? are you in client outcome evaluation? (Please circle only one)(Please circle only one)

a)a) Not Not confident confident b)b) KindaKinda confident confident c)c) Pretty Pretty confident confident

Less

164

)) yyd)d) Really Really confident confident e)e) Dead Dead confident confident

RankingRanking

More

6

Examples of Examples of ratioratio measuresmeasures

1.1. How old are you? How old are you? (Please round up)(Please round up)…………….……………...................................

2.2. How tall are you? How tall are you? (Please round up)(Please round up)………….………….…………….….…………….….Response Values

165

True and meaningful zeroTrue and meaningful zero

KindaKinda VeryVery ExtremelyExtremelyUnhappyUnhappy HappyHappy HappyHappy HappyHappy HappyHappy

1. Please rate your happiness1. Please rate your happiness 11 22 33 44 55

Example of a Example of a scaled scaled measuremeasureResponse Values

166

An An IndexIndex (scales (scales combined then summed)combined then summed)NeverNever RarelyRarely DailyDaily FrequentlyFrequently AlwaysAlways

1. How often do you 1. How often do you laugh with friendslaugh with friends?? 00 11 22 33 4+4+

NeverNever RarelyRarely DailyDaily FrequentlyFrequently AlwaysAlways

2. How often each week do you 2. How often each week do you gigglegiggle?? 00 11 22 33 4+4+

167

NeverNever RarelyRarely DailyDaily FrequentlyFrequently AlwaysAlways

3. How often each week do you 3. How often each week do you smilesmile?? 00 11 22 33 4+4+

ΣΣ = = 1212

168

7

Questionnaire Questionnaire developmentdevelopmentTwo types of questionsTwo types of questions

OpenOpen--endedendedClosedClosed--endedended

169

“Please take a minute to tell us about your trip last week.”

KindaKinda VeryVery ExtremelyExtremelyUnhappyUnhappy HappyHappy HappyHappy HappyHappy HappyHappy

1. Please rate your happiness1. Please rate your happiness 11 22 33 44 55

nded

nded

Ope

Ope

170Clo

sed

En

Clo

sed

En n E

ndedn E

nded

171

Questions wording?Questions wording?

172

8

Questionnaire Questionnaire developmentdevelopmentWriting Writing bad bad questionsquestions

Double barreledDouble barreledLeadingLeadingUnavailable informationUnavailable informationJargonJargonSexist racist languageSexist racist languageSexist, racist languageSexist, racist languageInflammatoryInflammatoryNonNon--mutually exclusive questionsmutually exclusive questionsNegatively asked itemsNegatively asked itemsVague Vague questionsquestionsWordinessWordiness

173

Questionnaire Questionnaire developmentdevelopmentWriting Writing good questionnairesgood questionnaires

Mutually exclusive questions Mutually exclusive questions Good question sequencingGood question sequencing

Allows for warmAllows for warm--upupStarts Starts with closedwith closed--endedendedEndsEnds with openwith open--endedendedEnds Ends with openwith open endedendedStarts Starts with easy questionswith easy questionsEnds Ends with heavy questionswith heavy questionsAvoids Avoids respondent fatiguerespondent fatigueClusters topicsClusters topicsUses subheadingsUses subheadingsEmploys concise languageEmploys concise language

174

175

Operational definitionsOperational definitions1.1. Variable nameVariable name2.2. Level of measurementLevel of measurement3.3. Response valuesResponse values44 Data sourceData source (from where will it come?)(from where will it come?)

176

4.4. Data source Data source (from where will it come?)(from where will it come?)

5.5. Who will collect the dataWho will collect the data6.6. When will the data be collected When will the data be collected

(baseline / pretest? Follow(baseline / pretest? Follow--up / posttest?)up / posttest?)

7.7. Actual location of data collectionActual location of data collection8.8. Method of data collectionMethod of data collection

9

177 178

From whom will From whom will data come?data come?

179

Individual Client?Individual Client?Sample?Sample?

180

10

181 182

183 184

11

Sampling Sampling defineddefined

Using Using individual things to represent some wholeindividual things to represent some whole

185

General General sampling sampling processprocessGathers “Gathers “sampling unitssampling units” from “” from “sampling framesampling frame” ” Gathers “Gathers “sampling unitssampling units” ” for for analysisanalysis““Sampling unitsSampling units” may represent ” may represent ““populationpopulation””““PopulationPopulation” represented by “” represented by “sampling framesampling frame””

186

Benefits Benefits of samplingof samplingMoneyMoneyTimeTimeEnergyEnergy

187

Disadvantages Disadvantages of samplingof samplingPotential Potential for… for…

Bias / ErrorBias / ErrorFalse understanding of truthFalse understanding of truth

188

12

Two Two categories of samplescategories of samplesScientific (Probability)Scientific (Probability)Nonscientific (NonNonscientific (Non--probability)probability)

189

Scientific: Scientific: Randomization Randomization (R)(R)Random selection [R (s)]Random selection [R (s)]Random assignment [R (a)]Random assignment [R (a)]

190

191 192

13

NonNon--scientificscientific: : SnowballSnowball

193

NonNon--scientificscientific: : PurposivePurposive

194

NonNon--scientificscientific: : ConvenienceConvenience

195

NonNon--scientificscientific: : QuotaQuota

196

14

197

How to collect the How to collect the actual data?actual data?

198

We must decide how we want We must decide how we want to measure stuff….to measure stuff….

1.1. QualitativelyQualitatively(Observing, watching, describing)(Observing, watching, describing)

2.2. QuantitativelyQuantitatively

199

Q yQ y(measuring, counting, quantifying)(measuring, counting, quantifying)

We measure by….We measure by….1.1. Directly observingDirectly observing2.2. Asking questions (Interviews)Asking questions (Interviews)3.3. Surveying Surveying

(self administered, mail, telephone, interviewer)(self administered, mail, telephone, interviewer)

200

4.4. Consulting recordsConsulting records5.5. TestingTesting6.6. …and other such methods…and other such methods

15

Trained research helpers can-•Build study rapport•Track eye-contact during interview•Track verbal indicators•Record performance

201

Record performance•Record environmental conditions•Skillfully ask questions•Probe responses•Meta analyze the session

We can measure single subjects at a time

202

We can measure a single family at a time

203

We can measure single group at a time

204

16

We can measure single agency at a time

205

We can measure single neighborhood

206

We can measure single neighborhood

207

We can also measure groups

208

17

209

Pretest posttest designPretest posttest design

210

Pretest posttest comparison group designPretest posttest comparison group design

211

SURVEYSSURVEYS( t i t t !)( t i t t !)

We can also administer…

(not survey instruments!)(not survey instruments!)

212

18

Key Key pointspointsNot a “survey instrument”! Not a “survey instrument”! A “survey design”A “survey design”Snapshots of situationSnapshots of situationMostly descriptiveMostly descriptive

213

ExamplesExamplesCommunity needs assessmentsCommunity needs assessmentsClient satisfaction surveysClient satisfaction surveysIntercept surveysIntercept surveysMail Mail surveyssurveysPhone surveysPhone surveysOnOn--line surveysline surveysProduct surveysProduct surveysMarket surveysMarket surveys

214

Develop Develop A Research DesignA Research DesignCommunicationCommunication

215

Develop Develop A Research DesignA Research DesignHuman Reactions to surveysHuman Reactions to surveys

216

19

217

Step 5Step 5Try out yourTry out your

218

Try out your Try out your measurement measurement

systemsystem

The trial run The trial run (aka, field test)(aka, field test)

••Vital to catching glitches, etc.Vital to catching glitches, etc.••Include all aspects of systemInclude all aspects of system••Can be a nonCan be a non--random sample, but…random sample, but…

219

Can be a nonCan be a non random sample, but…random sample, but…••Involve a representative sample, if possibleInvolve a representative sample, if possible••Include all key data collection pointsInclude all key data collection points

FieldField--testtest••Know your system wellKnow your system well••Know Know you key players (Strengths, weaknesses)you key players (Strengths, weaknesses)••Ensure their competenceEnsure their competence••Cross train replacements as indicatedCross train replacements as indicatedM k h b f f ll fi ldM k h b f f ll fi ld t tt t

220

••Make changes before full fieldMake changes before full field--testtest

20

FieldField--testtest••Be vigilantBe vigilant••Meet frequentlyMeet frequently••Anticipate problemsAnticipate problems••Be flexibleBe flexibleD t thiD t thi

221

••Document everythingDocument everything••Do not get discouragedDo not get discouraged

FieldField--testtest••Clear Clear programprogram--wide plan for trial runwide plan for trial run

••All outcome system proceduresAll outcome system procedures••Involve a representative group of participantsInvolve a representative group of participants••Long enough time span to include all key Long enough time span to include all key data collection pointsdata collection points

222

data collection pointsdata collection points

FieldField--testtest••Programs need clear plan for trial runPrograms need clear plan for trial run

••MultiMulti--site site programs, use only one siteprograms, use only one site••MultiMulti--unit structures, use only one unitunit structures, use only one unit••If client groups are assisted, use only If client groups are assisted, use only one client groupone client group

223

one client groupone client group

FieldField--testtest•• Prepare Prepare the data collectorsthe data collectors

••Recruit and train collectorsRecruit and train collectors••Test them prior to trial runTest them prior to trial run••Correct time allotted?Correct time allotted?

224

Correct time allotted?Correct time allotted?••Data collectors working out?Data collectors working out?••Any glitches?Any glitches?

21

FieldField--test test Monitor Monitor the measurement the measurement processprocess

••Perform the actual data collectionPerform the actual data collection••Any Glitches?Any Glitches?••Identify Identify problemsproblems

225

••Work closely with Work closely with ••StaffStaff••AdministrationAdministration

••CAUTIOUSLY make changes now!CAUTIOUSLY make changes now!••NOTE:NOTE: To change or not to change!To change or not to change!

FieldField--test test Monitor the measurement processMonitor the measurement process

••Time Time & Effort (Resource requirements & Effort (Resource requirements -- $)$)••Case response rates (e.g., lost followCase response rates (e.g., lost follow--ups)ups)••Missing dataMissing data

226

••Refusal Refusal ratesrates••Planned observations not completedPlanned observations not completed••Data collection errorsData collection errors••Data needed but unavailableData needed but unavailable

FieldField--testtest

QUESTIONSQUESTIONS::•• What problem areas emerged?What problem areas emerged?

•• Did we get data needed to measure outcomes?Did we get data needed to measure outcomes?

227

gg

•• Did instruments simplify or complicate the process?Did instruments simplify or complicate the process?

•• Did we collected Did we collected any unnecessary data?any unnecessary data?

•• Were all players adequately trained?Were all players adequately trained?

QUESTIONSQUESTIONS::•• Were agency records current?Were agency records current?•• Did data collectors loose interest?Did data collectors loose interest?

FieldField--testtest

228

•• Did the instrument flow?Did the instrument flow?•• Were there too many followWere there too many follow--up calls?up calls?•• Did we encounter overly high refusal rates?Did we encounter overly high refusal rates?

22

Data entry steps:Data entry steps:••Assign Assign someone to enter datasomeone to enter data

••Involves moving information Involves moving information ••From From forms, instruments, recordsforms, instruments, records••To computer To computer (EXCEL)(EXCEL)

229

pp ( )( )••Check for entry error ratesCheck for entry error rates

230

231 232

23

233 234

235

Unique identifiers“10…” = Site 1“20 ” = Site 2

236

“20…” = Site 2“30…” = Site 3

24

Unique identifiers

237

“10…” = Site 1“20…” = Site 2“30…” = Site 3

Step 6Step 6Analyze and reportAnalyze and report

238

Analyze and report Analyze and report your findingsyour findings

Once Once data set is error free, data set is error free, then…then…

TABULATETABULATE

239

ANALYZEANALYZE

Tabulation Tabulation -- UnivariateUnivariate

240

25

Tabulation Tabulation -- UnivariateUnivariate

241

2 x 2 tables2 x 2 tables

Tabulation Tabulation -- BivariateBivariate

242

Tabulation Tabulation -- BivariateBivariate

243

Tabulation Tabulation -- MultMultivariateivariate

244

26

Tabulation Tabulation -- MultMultivariateivariate

Var 1

245

Var 2Var 3Var 4Var 5

Tabulation Tabulation -- MultMultivariateivariate

246

Tabulation Tabulation -- MultMultivariateivariate

247

Analyze - Explain your findings

•Match analysis results to audience•Provide context for results•Use plain language•Use technical writing

248

g• Achievements• Failures

•“Influencing factors”•Age, Gender, Income, Staff differences

•Outline plans for improvement

27

BarchartsBarcharts

249

GraphsGraphs

250

Pie ChartsPie Charts

251

REPORT - Present your data

•Use visuals•Tables•Plots•Charts

252

28

One Year Later:•Present your data

•Using what your planned system, prepare a brief presentation of fake findings!

•Make up three key positive findings

253

•Make up one disappointing finding•Explain finding•Discuss methods to correct program efforts