EVALUATIONDESIGNUNMEvaluationLabSummerInstitute2018
[email protected] /evallab.unm.edu
Decisionsforyourevaluation
• Whoandhowmany?• Comparedtowhat?• Howwillyoucollectthedata?
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 2
Who?
ParticipantsOtherstakeholders:
• staff• Boardmembers• partners• funders
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 3
Howmany?
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 4
Ifyouwanttomakeclaimsaboutoutcomes,dataneedtoberepresentative• EveryoneinyourprogramOR
• Arandomsampleofthoseinyourprogram(acliniccouldcollectcustomersatisfactiondatafromallvisitorsduringoneweekeveryquarter)
Bewareoflowresponserates• Youwantresponseratesof80%orhigher• Thosenotrespondingareprobablyyourlessenthusiasticparticipants
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 5
DepthUnderstandingdepthofexperienceisharderandmuchmoretimeconsuming
• Howdoparticipantsexperiencetheprogram?• Howdidtheprogramchangetheirlifestory(ifitdid)?
Hereyouarelookingforsimilaritiesanddifferencesandideallyyoukeepcollectinguntilyouhaveallthestories
• Inpracticethereislimitedtimeavailable,andyouneedtotradebreadthfordepth• In-depthexperiencesareincrediblyhelpfulforunderstandingyourprocesses• Justbeawarethattheymaynotberepresentative
Understandthatthosewhovolunteerforafocusgroupwilltypicallybeyourmostenthusiasticparticipants
• Trytofindwaystogetyourlesssatisfiedparticipantsthereaswell
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 6
HowMany?Breadthvs.Depth
DobothCombinerepresentativedataandin-depthdataprovidedbyahandfulofinformants.Buildthatintoyourevaluationdesign.
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 7
ComparedtoWhat?Keyquestionforevaluationdesign
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 8
Oneshot– nocomparison
Source:Russ-EftandPreskill,EvaluationinOrganizations
Posttestofwhatyouknow,yourattitudesorsituationatonepointintime.
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 9
Oneshot– nocomparison, continuedGood for
• establishingbaseline• processevaluations• learningaboutparticipants
Notgoodfor• outcomesevaluation
Foroutcomeevaluationyoualwaysneedacomparison
Unfortunately,youradministrativedataisoftenone-shot
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 10
RetrospectivePretest–individualcomparespreandpost
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 11
Source:Russ-EftandPreskill,EvaluationinOrganizations
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 12
RetrospectivePretest–individualcomparespreandpost,continuedBenefit
• People’sstoriesofhowtheprogramchangedthemcanbeverypowerful
• Andifuniversalorrandomlyselected,thestoriesarecredible(andnotjustanecdotal)
• Youdon’tonlyhavetohaveyourevaluationacttogetherbytheendoftheprogram(notatthebeginning)
Caution• Thereissomeevidencethatpeoplelowertheirretrospectiveassessment ofthe presothattheycanreportanimprovement
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 13
One-GroupPretest-Posttest–comparisonofindividualattwopointsintime
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 14
Source:Russ-EftandPreskill,EvaluationinOrganizations
One-GroupPretest-Posttest–comparisonofindividualattwopointsintimeFANTASTICfortrackingparticipants,especiallypreandpostassessmentsandmaterialconditionsBEWARE: withoutacontrolgroup,youdon’tknowthatprogramcausedchange...
• Yourparticipantscometoyouattheirlowestpoint,maybetheywouldhavedonebetterovertimeevenwithoutyourprogram
• Goodideatocollectadditionaldataabouthowparticipantsratetheprogramasafactorintheirimprovedcondition
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 15
One-GroupPretest-Posttest–comparisonofindividualattwopointsintimeChangeinattitudesor outlook
• ChildHopesurveypreandpost
Changeinknowledge• EnlaceRelaciones Saludables test(butthinkaboutwhetherknowledgetranslatestobehavior)
• Supportedbyqualitativedata– letter,focusgroup
Changeinfamilyfunctioning• PB&JNCFAS,assessedbystaff (butthinkabout “gradeinflation”)• PFS (butthinkaboutlessself-knowledgeinthebeginning)• EnlacePSI
Changeinparticipationingovernmentsupportprograms• ParticipationMedicaid,SNAP,TANF
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 16
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 17
Appendix: Child Hope Scale (modified)
21
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 18
Figure5.PercentofChildrenWhoAgreedwithEachStatementatBeginningandEndofTwo-WeekSummerProgram
Note:15childrencompletedtheBeginningsurveyand14completedtheEndsurvey.Source:Binder,Melissa.2014.EvaluationoftheTaosHealthSystemsPeñascoCommunityHealth�KidsFIRSTInitiative.
93%
79%
64%57%
79%
64%
53%
80%
40%
60%67%
33%
I think I am doing pretty good.
I know what to do when I need
something (hungry, cold, need a hug).
I feel I'm doing just as good as my
classmates.
When I have a problem I can think of things to do to take care of the
problem.
I think the things I do now will help me
later when I am older.
Even when others want to quit, I know that I can find ways
to solve the problem.
Beginning End
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 19
5
Figure 1. Percent Scoring Correctly on Pre-and Post Questions, by Theme
Note: Data collected through pre- and post- tests for 139 Relaciones Saludables students for nine semesters from year to year.
The letters highlight different ways the program has affected the lives of the participants, mainly referring to the enhanced self-knowledge and self-awareness they have experienced. One participant wrote:
A mi en lo personal me ha servido como terapia porque me ha ayudado a conocerme, amarme, valorarme como ser humano y a entender a mi propia familia y a otros a adquirir conocimientos personales para mi vida diaria como el trato a mi pareja y a los que conviven a diario conmigo y que ahora veo la vida diferente. Un cambio total en mi vida. (To me personally, it [Relaciones Saludables] has been like therapy because it’s helped me to know myself, love myself, value myself as a human being and understand my own family and others to acquire personal knowledge for my daily life like how my partner, I, and those that live with me daily, treat each other and now I see life differently. A complete change in my life.) [emphasis added, translated by evaluation team].
65%
50%
39%
49%
29%
84%75% 74% 75%
64%
Self-esteem Communication Conflict Resolution Sexuality IntimateRelationships
Pre Post
“… [Relaciones Saludables] has
been like therapy because it’s
helped me to know myself,
love myself, value myself as a
human being and understand
my own family and others…”
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 20
NMEvaluationLab2016
5
Figure3.FamiliesratedatbaselineorbetterontheNCFASatintakeanddischarge
Note:DatarepresentallclientswithmatchedNCFASassessmentsbetweenJanuary2010andJanuary2016,about900clientsintotal.
NCFASdomainsareratedonaseven-pointscale.Onaverage,familiesinthegeneralprogramimprovebyabouthalfapointontheseven-pointscale.FamiliesintheTime-LimitedReunificationprogramimproveby.6and.7pointsintheareasofCaregiver/ChildAmbivalenceandinReadinessforReunification.
Effectsizesrangefrom.38to.71.Effectsizesshowthechangeinscoreasaproportionofthestandarddeviationofscoresinthefirstassessment.Oncontinuousmeasures,effectsizesforevidence-basedprogramsaretypicallyinthe.3range,sotheseeffectsizesareveryencouraging.Ofcourse,sincewedonothaveacontrolgroup,wecannotsaywithcertaintyhowfamilieswouldfarewithoutPB&Jprogramming.ButwecansaythatfamiliesimprovesignificantlyduringthetimetheyworkwithPB&J.Effectsizesforeachdomainareprovidedintable2intheappendix.
49.3%
51.1%
46.0%
51.9%
52.2%
51.9%
43.2%
51.7%
62.8%
54.5%
34.9%
30.8%
27.6%
33.4%
35.7%
31.7%
23.0%
33.0%
38.6%
36.4%
Environment
ParentCapabilities
FamilyInteractions
FamilySafety
ChildWell-Being
Social/CommunityLife
Self-Sufficiency
FamilyHealth
Caregiver/ChildAmbivalence
ReadinessforReunification
FirstNCFAS LastNCFAS
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 21
NMEvaluationLab2016
7
ProtectiveFactorsSurvey(PFS)
ForthePFS,parentspickafrequencyordegreeofagreementonaseven-pointscale.Examples
offrequencyandagreementquestionsareprovidedinfigure5.
Figure5.SamplequestionsfromtheProtectiveFactorsSurvey
Never
Very Rarely
Rarely
About Half the Time
Frequently
Very Frequently
Always
1. In my family, we talk about problems. 1 2 3 4 5 6 7
Strongly disagree
Mostly Disagree
Slightly Disagree Neutral Slightly
Agree Mostly Agree
Strongly Agree
12. There are many times when I don’t know what to do as a parent.
1 2 3 4 5 6 7
PB&Jfamiliesshowedimprovementineachofthesubscales.Parentsreportedabouta.5point
increaseinFamilyFunctioningandResiliency,a.4pointincreaseinSocialandEmotional
Supportanda.3pointincreaseinConcreteSupport.Alloftheseimprovementsarestatistically
significant.ThechangeinNurturingandAttachmentismuchsmaller;parentsranked
themselvesveryhighinthisareatobeginwith.(Seefigure6belowandtable4inappendix.)
Figure6.Protectivefactorsatintakeanddischarge
*Differencesaresignificantata5%levelusingatwo-tailedt-test.
FivequestionsonthePFSrelatetoparentingknowledgeandstyle.Parents’responsesshow
improvementson3oftheseitems.Thelargestimprovementisinresponsetothestatement:
“TherearemanytimeswhenIdon’tknowwhattodoasaparent.”Thisitemwasreverse-
codedsothatalargerscoremeansthatparentsbecamemoreconfidentabouttheirparenting
knowledge.Theotherimprovementsweregreaterunderstandingthatchildrendonot
misbehavetopurposelyupsettheirparents,andinnotlosingcontrolwhendisciplining.(See
figure7belowandtable5inappendix.)
4.6
5.1 5.0
6.0
5.15.6
5.3
6.1
FamilyFunctioning/
Resiliency*
SocialEmotional
Support*
ConcreteSupport* Nurturingand
Attachment
IntakeScore DischargeScore
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 22
Enlace Comunitario Evaluation Report
10
Figure 3. Parental Stress Index Surveys Show Parents’ Stress Level Percentiles Declined Over Time in the Family Program
Note: Data represent 120 survey responses for 30 unique participants. Data are from mid-2010’s to 2017.
B. Pre- and Post-Program Assessment for Crianza con Cariño. The evaluation team analyzed the 39 matched pre- and post- assessments from the Crianza class, and the findings started a discussion around the questionnaire itself. After review, the Enlace staff realized that this instrument had been created prior to the revision of the curriculum and no longer captures the courses’ content and emphasis. We thus relegated the analysis to the appendix. (See Appendix D.)
C. Post-program evaluation survey for Crianza con Cariño
In the third instrument, the post-program evaluation for the parenting course Crianza con Cariño, the 31 total responses from the last 2 years show participants were overall very grateful and thankful for participating in the course in response to the open-ended questions that asked what the best part of the program was and if they would recommend the program to a friend or relative. 100% of respondents said “Yes” that they would recommend the program to a friend or relative. Additionally, questions 5, 6 and 7 were helpful to this evaluation. In question 5, participants rank on a scale from 1 to 5 (1= “Not at all” to 5= “Very Much”) whether “Before the Crianza program, you used a.) emotional or verbal punishment or b.) physical punishment like spanking.” 40% of respondents reported using emotional or verbal punishment and 43% of respondents reported using physical punishment before the Crianza program (between 3 and 5 on the scale).
39
56
64
70
Final
8-week
4-week
Intake
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 23
Source:Russ-EftandPreskill,EvaluationinOrganizations
Controlgrouponemeasure
Randomlyassignedtoprogram
Randomlyassignedtocontrolgroup
Controlgrouponemeasure
Beopentopossibilityofidentifyingacontrolgroup• Evenifnotrandomassignment,isthereagroupyoucan’tservebecauseofgeographyorsomeothercharacteristicnotrelatedtoneedforprogram?
CouldworkforpeopleonwaitlistYouneedlotsofpeopletomake thiswork
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 24
CrystalBridgesMuseumstudy
• Walton Family Foundation established the museum in Arkansas in 2011
• Free museum tours to schools. • Demand for tours far exceeded capacity and researchers paired
similar classrooms and randomly chose the one to receive the tour. The other classroom, which was offered a tour in the following semester, served as the control.
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 25
Greene, Jay, Brian Kisida and Daniel Bowen. 2014. “The Educational Value of Field Trips.” Education Next (Winter): 79-86.
CrystalBridgesMuseumstudy,continued
• The tour employed student-driven discussions of five paintings. Guides provided artist name and the title of the painting, facilitated the discussion, and gave additional information only in response to student questions. This approach aims to increase student engagement and learning.
• Several weeks after the tour, both the “treated” and control students, more than 10,000 of them, completed a survey.
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 26
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 27
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 28
Source:Russ-EftandPreskill,EvaluationinOrganizations
Controlgrouppre-post
Randomlyassignedtoprogram
Randomlyassignedtocontrolgroup
Controlpre-post
Beopentopossibilityofidentifyingacontrolgroup• Evenifnotrandomassignment,isthereagroupyoucan’tservebecauseofgeographyorsomeothercharacteristicnotrelatedtoneedforprogram?
CouldworkforpeopleonwaitlistYoudon’tneedasmanypeopletomake thiswork- Buthardtofollowupwithcomparisongroup
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 29
CaseStudies
In-depthexplorationstolearnhowprogramimpactedindividualparticipants
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 30
Ideas?Questions?
7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 31