Plan, Evaluate and Improve Your Health Literacy Interventions and Activities
Sabrina Kurtz-Rossi, M.Ed. Health Literacy Consultant
April 10, 2013
ObjectivesAfter attending the session, participants will be able to:Conceptualize a health literacy program evaluation Write measurable health literacy outcome objectivesIdentify indicators to assess both process and outcomes Identify user-friendly evaluation resources for future use
Olney, C. and Barnes, Susan (2006). Planning and Evaluating Health Information Outreach. NN/LM Outreach Evaluation Resource Center. Seattle, WA.
Ten Attributes of Health Literate Health Care Organizations Brach C, Keller D, Hernandez LM, et al. Ten Attributes of Health Literate Health Care Organizations. Washington DC: National Academy of Sciences. 2012
Case StudyProvidence, RIDorcas Place Adult and Family Learning Center Computer LabWestern MaineAuburn Public Library Computer Training Class
Research vs. EvaluationResearch: To draw causal links between observed phenomena and to add to the knowledge base of the field Evaluation: To demonstrate effectiveness, and determine what works and what does not for the purpose of program improvementBoulmetis, J and Dutwin, P (2005). The ABCs of Evaluation. Jossey-Bass. San Francisco, CA
Health Literacy Program / Evaluation PlanningStep 1: Know Your Audience
Step 2: Identify Goals and Objectives
Step 3: Develop Program (and evaluation) Plan
Step 4: Implement and Evaluate
Step 5: Apply and Share Results
Step 1: Know Your AudienceStart with an Assessment Identify needs in context Guide your program Build on already existing capacityIdentify champions within your organization
MethodsInformal discussion, observationFormal Key informant interviews, focused group discussions, survey
Attribute 1: In a Health Literate Organization leadership makes health literacy integral to its mission, structure, and operations.
Case Study Methodology: Leaders at community-based organizations interviewed. Semi-structured interview protocol designed to explore community confidence in using the Internet to find health information and local resources to support access. All interviews conducted in person. Data were coded and analyzed into themes.
Key Findings: Maine has a growing senior population and community-based organizations support public health efforts. Public schools, adult education, and libraries offer computer access and support (n=9).
Latinos get their health information from sources they know and trust, including the radio. A network of minority health programs offer computers access, training and support (n=10).
Step 2: Identify Goals & ObjectivesGoal -- Why you are doing what you are doing and for whom
Increase use of health literacy interventions
Improve community access to reliable online health information
Attribute 2:A Health Literate Organization integrates health literacy into planning, evaluation, patient safety, and quality improvement.
Focused Health Information Outreach ProjectGoal: Increase access to health information among communities experiencing health disparities
National Network of Libraries of Medicine New England Region (NN/LM NER) http://nnlm.gov/ner/
Step 3: Develop Program(and Evaluation) PlanLogic Model: A planning tool connecting activities to outcomes
If you have these resources
and conduct these activitiesyou will get these outcomes
Why a Logic Model?Provides a framework for planningHelps organize your thoughtsClarifies intended outcomesLinks activities to outcomes Acts as a communication tool
Attribute 3:A Health Literate Organization prepares its workforce.
Case StudyGoal: Increase access to health information among communities experiencing health disparitiesIncrease knowledge of NLM resourcesIncrease use of NLM resourcesIncrease community experts to help othersNumber of trainings heldNumber trainedTypes of materials developedNumber of materials distributedDevelop training materialsDistribute materials at community eventsConsultantOutreach coordinatorsNLM materialsNLM fundingConduct trainingsWorking studentLibrarians in communities
ActivityChoose a goal (or use your own)
To increase staff awareness and use of health literacy interventions
To improve community access to reliable online health information
To improve health information literacy skills among patients and their families
Complete the logic model worksheet - identify outcomes, activities, outputs, and resources
Attribute 4:A Health Literate Organization involves the population served in evaluation of health information and services.
Case StudyCommunity AssessmentKey informant interviews
ProgramTailored efforts based on findings Community-based partnerships
Evaluation Outcome assessment: pre-post training evaluations, follow-up assessmentProcess assessment: story-based evaluation, community partner exit interviews
Western Maine (rural) Providence, RI(Latino)
Evaluation PlanProcessTo determine what worked and what didntFor program improvement
OutcomeTo demonstrate effectiveness, value, and influenceFor comparison purposes, to determine which programs are more effective
Attribute 5:A Health Literate Organization meets the needs of the populations and avoids stigmatization.
Process Assessment Methods Activity logsTeacher feedback formsFocused group discussionInterviews
Outcome AssessmentMethodsExisting recordsPre/post testsFollow-up surveysObservation Interview Story-based
Attribute 6:In a Health Literate Organization providers and staff use health literacy strategies in verbal communication and confirm understanding.
Case Study Will report use two weeks later of follow upWill name resources on post-surveysWill document helping others on story-based evaluation forms Outcomes & Indicators
Outcomes (what you want)Indicators (how you will know) Increased knowledge of NLM resourcesIncreased use of NLM resourcesIncreased community experts to help others
Measurable Objectives Four components of a measurable objective1. ConditionFollowing the trainingAfter watching the video
2. Whothe learnerthe patientthe caregiver
Measurable Objectives 3. Performance (knowledge, attitudes, behavior)will name will agree will use
4. Criterion (quality/quantity) 5 out of 6 at least 2 the correct way
Characteristics of a Measurable Objective Is audience focused, not intervention focused (the participant will)
Includes action verbs
Targets one expectation
Includes observable criteria for evaluation
Matches goals to activities/strategies
Attribute 7:In a Health Literate Organization consumers have easy access to health information and services and navigation assistance.
Blooms TaxonomyCognitive explain, recall, list, predict, select, describe, define
Affective agree, avoid, help, join
Psychomotor use, develop, design, repair, taste, use, move
Verbs to AvoidKnowComprehendUnderstandAppreciateStudyGain knowledge ofLearn
Attribute 8:A Health Literate Organization develops plain language materials that are easy to understand and act on.
Case Study Immediately after trainingparticipantswill nameTwo weeks after trainingparticipantswill report usingThree months trainingparticipantsNLM resources on post-surveyNLM resources on follow up interviewwill write a storyDescribing how they helped others find information on story-based evaluation formWriting Measurable Objectives
ActivityChoose an outcome (or use your own)Increase confidence in using plain language/teach-backIncreased awareness of available online health information resourcesIncreased interaction with diabetes educators
Complete the writing measurable objectives worksheet - identify condition, audience, performance, and criterion
Step 4: Implement and Evaluate Expect the Unexpected
Adjust to the environment
Address learners needs
Tweak your notion of success
Attribute 9:A Health Literate Organization addresses health literacy in high-risk situations, including transitions and about medicines.
Case Example Rhode Island: Pre/Post: Confidence in Ability to Find Health Information on the Internet Pre N= 520Post N=503Maine: Pre/Post: Ever Used NLM ResourcesPre N=72Post N=55
Case Example SuccessesTraining participants shared resources with friends and neighborsPartners will continue to promote and distribute resourcesProviders used resources to find accurate information to use in their work
Challenges Pre- and post-evaluations were hard to administer
Step 5: Apply and Share ResultsApply Lessons Learned
Make changes based on what you learned
Replicate with similar or different audiences
Share Your Findings
Attribute 10:At a Health Literate Organization they clearly communicates about health plans and health care costs.
Evaluation ResourcesPlanning and Evaluating Health Information Outreach Projects
Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach
Questions & CommentsSabrina Kurtz-Rossi, M.Ed.Principal | Health Literacy Consultant firstname.lastname@example.org
Kurtz-Rossi & Associates Health*Literacy*Education*Evaluationwww.kurtz-rossi.com
Member, Clear Language Groupwww.clearlanguagegroup.com
Health Literacy Leadership Institute, June 10-14, 2013 http://healthliteracyleadreship.com
**Throughout this session I will be referring to a recent project of mine as a case example. The Focused Heath Information Literacy Outreach Project was funded by NN/LM NER. In order to tailor their program to best meet the communities health information they we began with a community assessment. We worked in two communities: Providence RI and Rural Maine. Through key informant interviews, interviews with key opinion leaders in each community we identified ways to tailor NN/LMs traditional outreach services (teaching and materials). In Maine we identified seniors as a subpopulation of particular need and ended up targeting our efforts towards that population. In RI, based on what we learned form our community assessment, we ended up using a train the trainer model working with CBO namely the Welcome Back and adult education program serving a predominately Spanish speaking student body whose participants were closely connected to the community and seen as leaders in that community.
We used key informant interviews (interviews with key opinion leaders) We tailored our efforts based on findings and build indentified community-based partnersEvaluation Outcome assessment: pre-post training evaluations, follow-up assessmentProcess assessment: story-based evaluation, community partner exit interviews
*First let me be clearDistinguish between research and evalution Not talking about research today (although some methodologies are the same)The purpose of research. Read research who is the audience The purpose of evaluation. Read Evaluation who is the audience Well come back to this point laterThree types of evaluation and well talk about all threeCommunity Assessment ProcessOutcome*I see evaluation as a critical part of program planning and implementation not separate. Evaluation is often an after thought. Who has experience this: your writing a grant proposal, youve got a great program you like to implement and before the deadline of submitting the proposal your project team is like, quick who can we get to be the evaluator? Were not going to look at evaluation as an end project. Were going to look at the role of evaluation at beginning, middle and end of the program planning implementation process. Well go through each of these steps. Starting with knowing your audience. Knowing your audience critically important to everything we do. Always start with that. From there we can identify goals and objectives, develop a program to meet those goals and objectives that includes an evaluation plan for program improvement. You just cant develop program plan with out evaluation plan. Then there is implementation and evaluation of your program, and finally applying and sharing your results. Always need to share what you learned. We tend to be very weak on this. *As mentioned earlier there are three types of evaluation community assessment process evaluation and outcome evaluation. Why do we start with a a community assessment? Why is this so important? It is critical for program planning and development. Your not only identifying community needs, your identifying community support! The most successful and sustainable programs build on community strengths they do no stand alone.
Purpose is to identify needs in order to guide your program, to build on existing capacity and strengths and to identify community-based partners. And here are some of the methods Ive used and find most effective. *Now I want to hear from you. What are some of your program goals. What are you trying to accomplish in your community. This is your big picture statement. This is what you want to accomplished, well discuss program objectives and what youll need to accomplish to achieve your goal next. Here are some examples of big picture statements.
Reduce medical errors by increasing use of health literacy interventions (plain language, and encouraging patient to ask questions) among health care providers Improve community access to reliable online health information Improve health information literacy among patients and families
Objective are what you change to achieve your goal.
Knowledge, attitudes, behaviors, other
*The big picture goal of the Focused Health Information outreach project was to increase access to health information among communities experiencing health disparities. That we know what we want to accomplish the question is how are you going to accomplish it? Next step: Develop your program (and evaluation) plan*Who here is familiar with logic models? Who here loves logic models? Who here hates them? Meant to be a work in progress and to reflect what you do and what happens in the field not as a static document which you try to fit your field into what the logic model says. Living breathing documents. It reflects your program and how your program evolves. Thats why I like them. Can be very useful for many reasons.
The logic model is a planning tool to connect your activities (your program) to you outcomes (the focus of your evaluation). Program implementation and evaluation go hand in hand. *As I said its a planning tool, also very effective community tool. Its a one page diagram of your project. Includes the big picture and all the details. Helps you line your activities to your trying to accomplish, your outcomes. Not linear although, not simply two dimensional piec...