Plan, Evaluate and Improve Your Health Literacy Interventions and Activities

  • Published on

  • View

  • Download

Embed Size (px)


Plan, Evaluate and Improve Your Health Literacy Interventions and Activities. Sabrina Kurtz-Rossi, M.Ed. Health Literacy Consultant April 10, 2013. Objectives. After attending the session, participants will be able to: Conceptualize a health literacy program evaluation - PowerPoint PPT Presentation


  • Plan, Evaluate and Improve Your Health Literacy Interventions and Activities

    Sabrina Kurtz-Rossi, M.Ed. Health Literacy Consultant

    April 10, 2013

  • ObjectivesAfter attending the session, participants will be able to:Conceptualize a health literacy program evaluation Write measurable health literacy outcome objectivesIdentify indicators to assess both process and outcomes Identify user-friendly evaluation resources for future use

    Olney, C. and Barnes, Susan (2006). Planning and Evaluating Health Information Outreach. NN/LM Outreach Evaluation Resource Center. Seattle, WA.

  • Ten Attributes of Health Literate Health Care Organizations Brach C, Keller D, Hernandez LM, et al. Ten Attributes of Health Literate Health Care Organizations. Washington DC: National Academy of Sciences. 2012

  • Case StudyProvidence, RIDorcas Place Adult and Family Learning Center Computer LabWestern MaineAuburn Public Library Computer Training Class

  • Research vs. EvaluationResearch: To draw causal links between observed phenomena and to add to the knowledge base of the field Evaluation: To demonstrate effectiveness, and determine what works and what does not for the purpose of program improvementBoulmetis, J and Dutwin, P (2005). The ABCs of Evaluation. Jossey-Bass. San Francisco, CA

  • Health Literacy Program / Evaluation PlanningStep 1: Know Your Audience

    Step 2: Identify Goals and Objectives

    Step 3: Develop Program (and evaluation) Plan

    Step 4: Implement and Evaluate

    Step 5: Apply and Share Results

  • Step 1: Know Your AudienceStart with an Assessment Identify needs in context Guide your program Build on already existing capacityIdentify champions within your organization

    MethodsInformal discussion, observationFormal Key informant interviews, focused group discussions, survey

  • Attribute 1: In a Health Literate Organization leadership makes health literacy integral to its mission, structure, and operations.

  • Case Study Methodology: Leaders at community-based organizations interviewed. Semi-structured interview protocol designed to explore community confidence in using the Internet to find health information and local resources to support access. All interviews conducted in person. Data were coded and analyzed into themes.

    Key Findings: Maine has a growing senior population and community-based organizations support public health efforts. Public schools, adult education, and libraries offer computer access and support (n=9).

    Latinos get their health information from sources they know and trust, including the radio. A network of minority health programs offer computers access, training and support (n=10).

  • Step 2: Identify Goals & ObjectivesGoal -- Why you are doing what you are doing and for whom


    Increase use of health literacy interventions

    Improve community access to reliable online health information

  • Attribute 2:A Health Literate Organization integrates health literacy into planning, evaluation, patient safety, and quality improvement.

  • Case Study

    Focused Health Information Outreach ProjectGoal: Increase access to health information among communities experiencing health disparities

    National Network of Libraries of Medicine New England Region (NN/LM NER)

  • Step 3: Develop Program(and Evaluation) PlanLogic Model: A planning tool connecting activities to outcomes

    If you have these resources




    and conduct these activitiesyou will get these outcomes

  • Why a Logic Model?Provides a framework for planningHelps organize your thoughtsClarifies intended outcomesLinks activities to outcomes Acts as a communication tool


  • Attribute 3:A Health Literate Organization prepares its workforce.

  • Case StudyGoal: Increase access to health information among communities experiencing health disparitiesIncrease knowledge of NLM resourcesIncrease use of NLM resourcesIncrease community experts to help othersNumber of trainings heldNumber trainedTypes of materials developedNumber of materials distributedDevelop training materialsDistribute materials at community eventsConsultantOutreach coordinatorsNLM materialsNLM fundingConduct trainingsWorking studentLibrarians in communities


  • ActivityChoose a goal (or use your own)

    To increase staff awareness and use of health literacy interventions

    To improve community access to reliable online health information

    To improve health information literacy skills among patients and their families

    Complete the logic model worksheet - identify outcomes, activities, outputs, and resources

  • Attribute 4:A Health Literate Organization involves the population served in evaluation of health information and services.

  • Case StudyCommunity AssessmentKey informant interviews

    ProgramTailored efforts based on findings Community-based partnerships

    Evaluation Outcome assessment: pre-post training evaluations, follow-up assessmentProcess assessment: story-based evaluation, community partner exit interviews

    Western Maine (rural) Providence, RI(Latino)

  • Evaluation PlanProcessTo determine what worked and what didntFor program improvement

    OutcomeTo demonstrate effectiveness, value, and influenceFor comparison purposes, to determine which programs are more effective

  • Attribute 5:A Health Literate Organization meets the needs of the populations and avoids stigmatization.

  • Process Assessment Methods Activity logsTeacher feedback formsFocused group discussionInterviews

  • Outcome AssessmentMethodsExisting recordsPre/post testsFollow-up surveysObservation Interview Story-based

  • Attribute 6:In a Health Literate Organization providers and staff use health literacy strategies in verbal communication and confirm understanding.

  • Case Study Will report use two weeks later of follow upWill name resources on post-surveysWill document helping others on story-based evaluation forms Outcomes & Indicators

    Outcomes (what you want)Indicators (how you will know) Increased knowledge of NLM resourcesIncreased use of NLM resourcesIncreased community experts to help others

  • Measurable Objectives Four components of a measurable objective1. ConditionFollowing the trainingAfter watching the video

    2. Whothe learnerthe patientthe caregiver

  • Measurable Objectives 3. Performance (knowledge, attitudes, behavior)will name will agree will use

    4. Criterion (quality/quantity) 5 out of 6 at least 2 the correct way

  • Characteristics of a Measurable Objective Is audience focused, not intervention focused (the participant will)

    Includes action verbs

    Targets one expectation

    Includes observable criteria for evaluation

    Matches goals to activities/strategies

  • Attribute 7:In a Health Literate Organization consumers have easy access to health information and services and navigation assistance.

  • Blooms TaxonomyCognitive explain, recall, list, predict, select, describe, define

    Affective agree, avoid, help, join

    Psychomotor use, develop, design, repair, taste, use, move

  • Verbs to AvoidKnowComprehendUnderstandAppreciateStudyGain knowledge ofLearn

  • Attribute 8:A Health Literate Organization develops plain language materials that are easy to understand and act on.

  • Case Study Immediately after trainingparticipantswill nameTwo weeks after trainingparticipantswill report usingThree months trainingparticipantsNLM resources on post-surveyNLM resources on follow up interviewwill write a storyDescribing how they helped others find information on story-based evaluation formWriting Measurable Objectives


  • ActivityChoose an outcome (or use your own)Increase confidence in using plain language/teach-backIncreased awareness of available online health information resourcesIncreased interaction with diabetes educators

    Complete the writing measurable objectives worksheet - identify condition, audience, performance, and criterion

  • Step 4: Implement and Evaluate Expect the Unexpected

    Be flexible

    Adjust to the environment

    Address learners needs

    Tweak your notion of success

  • Attribute 9:A Health Literate Organization addresses health literacy in high-risk situations, including transitions and about medicines.

  • Case Example Rhode Island: Pre/Post: Confidence in Ability to Find Health Information on the Internet Pre N= 520Post N=503Maine: Pre/Post: Ever Used NLM ResourcesPre N=72Post N=55

  • Case Example SuccessesTraining participants shared resources with friends and neighborsPartners will continue to promote and distribute resourcesProviders used resources to find accurate information to use in their work

    Challenges Pre- and post-evaluations were hard to administer

  • Step 5: Apply and Share ResultsApply Lessons Learned

    Make changes based on what you learned

    Replicate with similar or different audiences

    Share Your Findings



  • Attribute 10:At a Health Literate Organization they clearly communicates about health plans and health care costs.

  • Evaluation ResourcesPlanning and Evaluating Health Information Outreach Projects

    Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach

  • Questions & CommentsSabrina Kurtz-Rossi, M.Ed.Principal | Health Literacy Consultant

    Kurtz-Rossi & Associates Health*Literacy*Education*

    Member, Clear Language

    Health Literacy Leadership Institute, June 10-14, 2013

    **Throughout this session I will be referring to a recent project of mine as a case example. The Focused Heath Information Literacy Outreach Project was funded by NN/LM NER. In order to tailor their program to best meet the communities health information they we began with a community assessment. We worked in two communities: Providence RI and Rural Maine. Through key informant interviews, interviews with key opinion leaders in each community we identified ways to tailor NN/LMs traditional outreach services (teaching and materials). In Maine we identified seniors as a subpopulation of particular need and ended up targeting our efforts towards that population. In RI, based on what we learned form our community assessment, we ended up using a train the trainer model working with CBO namely the Welcome Back and adult education program serving a predominately Spanish speaking student body whose participants were closely connected to the community and seen as leaders in that community.

    We used key informant interviews (interviews with key opinion leaders) We tailored our efforts based on findings and build indentified community-based partnersEvaluation Outcome assessment: pre-post training evaluations, follow-up assessmentProcess assessment: story-based evaluation, community partner exit interviews

    *First let me be clearDistinguish between research and evalution Not talking about research today (although some methodologies are the same)The purpose of research. Read research who is the audience The purpose of evaluation. Read Evaluation who is the audience Well come back to this point laterThree types of evaluation and well talk about all threeCommunity Assessment ProcessOutcome*I see evaluation as a critical part of program planning and implementation not separate. Evaluation is often an after thought. Who has experience this: your writing a grant proposal, youve got a great program you like to implement and before the deadline of submitting the proposal your project team is like, quick who can we get to be the evaluator? Were not going to look at evaluation as an end project. Were going to look at the role of evaluation at beginning, middle and end of the program planning implementation process. Well go through each of these steps. Starting with knowing your audience. Knowing your audience critically important to everything we do. Always start with that. From there we can identify goals and objectives, develop a program to meet those goals and objectives that includes an evaluation plan for program improvement. You just cant develop program plan with out evaluation plan. Then there is implementation and evaluation of your program, and finally applying and sharing your results. Always need to share what you learned. We tend to be very weak on this. *As mentioned earlier there are three types of evaluation community assessment process evaluation and outcome evaluation. Why do we start with a a community assessment? Why is this so important? It is critical for program planning and development. Your not only identifying community needs, your identifying community support! The most successful and sustainable programs build on community strengths they do no stand alone.

    Purpose is to identify needs in order to guide your program, to build on existing capacity and strengths and to identify community-based partners. And here are some of the methods Ive used and find most effective. *Now I want to hear from you. What are some of your program goals. What are you trying to accomplish in your community. This is your big picture statement. This is what you want to accomplished, well discuss program objectives and what youll need to accomplish to achieve your goal next. Here are some examples of big picture statements.

    Reduce medical errors by increasing use of health literacy interventions (plain language, and encouraging patient to ask questions) among health care providers Improve community access to reliable online health information Improve health information literacy among patients and families

    Objective are what you change to achieve your goal.

    Knowledge, attitudes, behaviors, other

    *The big picture goal of the Focused Health Information outreach project was to increase access to health information among communities experiencing health disparities. That we know what we want to accomplish the question is how are you going to accomplish it? Next step: Develop your program (and evaluation) plan*Who here is familiar with logic models? Who here loves logic models? Who here hates them? Meant to be a work in progress and to reflect what you do and what happens in the field not as a static document which you try to fit your field into what the logic model says. Living breathing documents. It reflects your program and how your program evolves. Thats why I like them. Can be very useful for many reasons.

    The logic model is a planning tool to connect your activities (your program) to you outcomes (the focus of your evaluation). Program implementation and evaluation go hand in hand. *As I said its a planning tool, also very effective community tool. Its a one page diagram of your project. Includes the big picture and all the details. Helps you line your activities to your trying to accomplish, your outcomes. Not linear although, not simply two dimensional piec...


View more >