Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
MeasuringOutcomes
Dr.CatherineWadePrincipalResearchSpecialistParentingResearchCentre
SamHaskardManagerChildrenandFamiliesProgram
AnglicareSA
Overview
Catherine:
• Approachestomeasuringprogramoutcomes
• Howtoeffectivelysourceandselecttherightmeasures
ü Interpretingpsychometrics
ü Measuresdatabases
ü Useofvalidatedtoolsvstailoredmeasures
Sam:
• Apracticalexample
Catherine&Sam:
• Exploringoptions
Definingprogramoutcomes
• Gettingclarityonoutcomes– importanceofprogramlogicinclearlyarticulatingthelinkbetweenprogramactivitiesandanticipatedoutcomes.
Whatoutcomestomeasure?
Whatoutcomestomeasure?
Trytoincludemeasuresforboththekeymediatingandintendedoutcomes,however:
• Considertimeframefortheinterventionandevaluation– whichintendedoutcomescouldwereasonablyexpecttoseeoccur?Someintendedoutcomesmightbebeyondthescopeoftheevaluation.
• Considerfeasibilityofmeasurementi.e.,willitbepossibletoobtainameasureofthetargetedintendedoutcome/s?
Whatoutcomestomeasure?
• Consideralsoincludingmeasuresofkeyimplementationandengagementoutcomes:
Implementationoutcomes considerwhatcriticalquantity,qualityandtimingoutcomesoftheprogramimplementationarenecessarytoachieverequiredengagementandadherencewithprogramsstrategies(e.g.,measuresofimplementationfidelity)
Engagementoutcomes considerwhatlevelsofengagementandadherencewithprogramstrategiesarethoughttobeessentialtoachievemediatingoutcomes (e.g.,measuresofclientattendanceandengagementwithprogram)
Whatoutcomestomeasure?
Approachestomeasuringoutcomes
• Quantitativeversusqualitative– wherepossiblecanbebeneficialtoincludeboth.
• Considerneedsoftheevaluationandthetargetparticipantgroup:
• Rigourofapproach
• Culturalappropriatenessofmethod(e.g.,useofyarningcirclesforevaluationsinvolvingAboriginalfamilies)
• Feasibilityofapproachandacceptabilityofmethodsbyfamiliesandprogramfacilitators
Identifyingandselectingmeasures
• Identifyatleastoneindicatorforeachoutcomeofinterest– engagement,implementation,mediatingandintendedoutcomes
• Astrongindicatorwillprovideameasureofthatoutcomethatisclosesttoa‘realworldresult’– morereliableandvalid
• Aweakindicatorwillbeanapproximationonlyandinvolvegreatererror– lessreliableandvalid
Identifyingandselectingmeasures
Identifyingandselectingmeasures
Typesofmeasures
Observational(video/live/audio):providedirectmeasuresoftheparentand/orthechild'sbehaviour,parent-childinteractions,skillacquisition,andgeneralisationofskillsoutsideoftheprogram.Checklists&toolsavailable,ordevelopyourown.
Self-monitoring:e.g.parentrecordsfrequencyordurationofanevent/behaviour,ortopographyofbehaviour(A’s,B’sandC’s).
Self-report:Checklists,ratingscales,questionnaires.
Interviews:structured/pre-preparedscheduleofquestions&prompts.Acquiescence&socialdesirabilitycanbeanissue.
Directtesting:knowledgeorskillassessmenttools(e.g.,vignettes,quizzeschilddevelopmentalassessments).
Measuresdatabases
CfC outcomesmeasurementmatrixonCFCAwebsite:https://aifs.gov.au/cfca/expert-panel-project/families-and-children-outcomes-measurement-matrix
OntarioCentreofExcellenceforChildandYouthMentalHealth: http://www.excellenceforchildandyouth.ca/resource-hub/measures-list-search-results?p=all
UniversityofMinnesotaEvaluationInstrumentSearch:https://cyfar.org/search_evaluation_instruments
MeasurementInstrumentDatabasefortheSocialSciences:http://www.midss.org/tools/search
TheCaliforniaEvidence-BasedClearinghouseMeasurementToolsforChildWelfare:http://www.cebc4cw.org/assessment-tools/
Usingvalidatedmeasuresversuscustomisedtools:
Prosofestablishedmeasures:
• Reliable
• Valid
• Normed
• Guidelinesforscoringandinterpretation
• Enablesbenchmarking
• Manyfreelyavailabletools
Usingvalidatedmeasuresversuscustomisedtools:Howeverneedtoconsiderfeasibilityofusingestablishedtools:
•Fitforpurpose
•Version
•Restrictions
•Time,cost,length,reportingburden
•Instructions
•Skillsarerequiredoftherespondent
•Populationnormsavailable
•Usedwithclientslikeyours?
Usingvalidatedmeasuresversuscustomisedtools:
Someimportantconsiderationsforthedevelopmentofcustomisedtools:
• Ensuresamequestionsareaskedatpreandposttest– nowordingchanges(thiscanleadtochangesinmeaningandmaythereforenotbemeasuringthesamethingpreandpost)
• Useclear,plainlanguageandavoiddouble-barrelledquestions
• Ensuregoodalignmentbetweentargetedprogramoutcomeandwordingofquestions.
Usingvalidatedmeasuresversuscustomisedtools:
Someimportantconsiderationsforthedevelopmentofcustomisedtools:
• Considerwhetherpossibletoco-designquestionswithclientgroupandincorporateapilottestbeforestartingyourevaluationtoobtainclientfeedback
• Acknowledgelimitationsofusingnon-validatedmeasureswheninterpretingfindings.
Dr.CatherineWadeLevel5,232VictoriaParadeEastMelbourne,Victoria,Australia3002
P:+61386603500F:+61386603599
@WadeCath