Assessing Assessing Consumer Health Vocabulary Familiarity: Consumer Health Vocabulary Familiarity:

An Exploratory StudyAn Exploratory Study

Alla Keselman1,2 Tony Tse1, Jon Crowell3

Allen Browne1 Long Ngo3 Qing Zeng3

1 – US National Library of Medicine2 – Aquilent, Inc.

3 – Harvard Medical School

Study BackgroundStudy Background

Consumers have difficulty with health textsConsumers have difficulty with health texts

Study BackgroundStudy Background

Consumers have difficulty with health textsConsumers have difficulty with health texts We would like to provide supportWe would like to provide support

– Authoring guidelines; tools; translatorsAuthoring guidelines; tools; translators

Need a way to evaluate readabilityNeed a way to evaluate readability– Readability formulasReadability formulas

Health domain is unique Health domain is unique – Familiar long words (diabetes); unfamiliar short Familiar long words (diabetes); unfamiliar short

words (apnea)words (apnea)

Term Familiarity Likelihood Term Familiarity Likelihood Regression ModelRegression Model

Computational (regression) modelComputational (regression) model– Each term is assigned 0 – 1 scoreEach term is assigned 0 – 1 score

Algorithm basisAlgorithm basis– Empirical dataEmpirical data– Term frequency counts from health text corporaTerm frequency counts from health text corpora

Term score categoriesTerm score categories– 0.8 – 1.0 score – “likely to be familiar”0.8 – 1.0 score – “likely to be familiar”– 0.5 – 0.8 score – “somewhat likely to be familiar”0.5 – 0.8 score – “somewhat likely to be familiar”– 0.0 – 0.5 score – “not likely to be familiar”0.0 – 0.5 score – “not likely to be familiar”

SourceSource:: Zeng Q, Kim E, Crowell J, Tse T. A text corpora-based Zeng Q, Kim E, Crowell J, Tse T. A text corpora-based estimation of the familiarity of health terminology. Proc ISBMDA estimation of the familiarity of health terminology. Proc ISBMDA 2005: 184-92.2005: 184-92.

ObjectivesObjectives

Validate regression modelValidate regression model– Test with consumersTest with consumers

Effect of demographic factors on familiarityEffect of demographic factors on familiarity– Health literacyHealth literacy

– Education levelEducation level

Relate surface-level and conceptual Relate surface-level and conceptual familiarityfamiliarity– Term vs. conceptTerm vs. concept

HypothesesHypotheses

I.I. Significant effect of predicted familiarity Significant effect of predicted familiarity likelihoodlikelihood

1. Surface-level familiarity1. Surface-level familiarity

2. Conceptual familiarity2. Conceptual familiarity

II.II. Significant effect of demographic factorsSignificant effect of demographic factors

III.III. Surface level familiarity > conceptualSurface level familiarity > conceptual

Survey InstrumentSurvey Instrument

45 items – hypertension, back pain, GERD 45 items – hypertension, back pain, GERD (gastroesophageal reflux)(gastroesophageal reflux)

Random set of terms from MedlinePlusRandom set of terms from MedlinePlus Two types of test items:Two types of test items:

– Surface-level – prominent associationSurface-level – prominent association Surgery => knifeSurgery => knife

– Concept levelConcept level Surgery => removing or repairing a body partSurgery => removing or repairing a body part

45 surface questions; 15 concept questions 45 surface questions; 15 concept questions (GERD)(GERD)

Item FormatItem Format

*Modeled on the Short Assessment of Health Literacy for Spanish-speaking Adults (SAHLSA) Lee S-YD, Bender DE, Ruiz RE, Cho YI. Development of an easy-to-use Spanish health literacy test. Health Serv Res. In press.

ParticipantsParticipants

ProcedureProcedure

Demographic surveyDemographic survey Short Test of Functional Health Literacy Short Test of Functional Health Literacy

in Adults (S-TOFHLA)in Adults (S-TOFHLA) Familiarity testFamiliarity test

ResultsResults

Decrease

ResultsResults

Decrease

ResultsResults

Predictors of Surface-Level FamiliarityPredictors of Surface-Level Familiarity

Regression IRegression I– DV: surface level term familiarityDV: surface level term familiarity– IV: IV: Predicted Familiarity Likelihood Level, Gender, English

proficiency, Highest Education Level, Age, Race, Health Literacy Level

Significant predictors– Predicted Familiarity Likelihood (P<.001)– Health Literacy (P<.001)– English Proficiency (P=.05) Confirms Hypothesis I

Confirms Hypothesis II

Predictors of GERD Concept FamiliarityPredictors of GERD Concept Familiarity

Regression II (GERD)Regression II (GERD)– DV: GERD concept familiarity– IV: Predicted Familiarity Likelihood Level, GERD surface-

level familiarity Gender, English proficiency, Highest Education Level, Age, Race, Health Literacy Level

Predictors of GERD Concept FamiliarityPredictors of GERD Concept Familiarity

Regression II (GERD)Regression II (GERD)– DV: GERD concept familiarityDV: GERD concept familiarity– IV: IV: Predicted Familiarity Likelihood Level, GERD surface-

level familiarity Gender, English proficiency, Highest Education Level, Age, Race, Health Literacy Level

Significant predictors– Predicted Familiarity Likelihood (P=.009)– GERD surface-level familiarity score (P<.001)

Health Literacy (P.06) - trend Confirms Hypothesis I

Predictors of GERD Concept FamiliarityPredictors of GERD Concept Familiarity

Regression II (GERD)Regression II (GERD)– DV: GERD concept familiarityDV: GERD concept familiarity– IV: IV: Predicted Familiarity Likelihood Level, GERD surface-

level familiarity Gender, English proficiency, Highest Education Level, Age, Race, Health Literacy Level

Significant predictors– Predicted Familiarity Likelihood (P=.009)– GERD surface-level familiarity score (P<.001)

Health Literacy (P.06) - trendAddresses Hypothesis III

Predictors of GERD Concept FamiliarityPredictors of GERD Concept Familiarity

Regression II (GERD)Regression II (GERD)– DV: GERD concept familiarityDV: GERD concept familiarity– IV: IV: Predicted Familiarity Likelihood Level, GERD surface-

level familiarity Gender, English proficiency, Highest Education Level, Age, Race, Health Literacy Level

Significant predictors– Predicted Familiarity Likelihood (P=.009)– GERD surface-level familiarity score (P<.001)

Health Literacy (P.06) - trend

Trend for Hypothesis II

Relationship Between Surface Relationship Between Surface Level and Concept Familiarity Level and Concept Familiarity

(GERD)(GERD)

Gap between surface and concept familiarity (P=.001)Gap between surface and concept familiarity (P=.001) Size of gap greater for “likely” than for “unlikely” (P=.006)Size of gap greater for “likely” than for “unlikely” (P=.006) Trend for “somewhat likely” vs. “unlikely” (P=.07)Trend for “somewhat likely” vs. “unlikely” (P=.07)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

1 2

Familiarity type

Sco

re

"Likely to be familiar"

"Somewhat likely"

"Unlikely"

ConclusionsConclusions

Initial validity evidence for CHV familiarity modelInitial validity evidence for CHV familiarity model– Health readability utilityHealth readability utility

Ways to improve the modelWays to improve the model– Allow demographic correctionsAllow demographic corrections– Distinguish between knowledge of terms / conceptsDistinguish between knowledge of terms / concepts

Follow-up workFollow-up work– Increase sample and term poolIncrease sample and term pool– Education level?Education level?– Other predictors?Other predictors?– Work on integrated findings into health readability formulaWork on integrated findings into health readability formula

AcknowledgementsAcknowledgements

Intramural Research Program of the US National Library of Medicine, US National Institutes of Health

NIH grant R01 LM007222-05

Ilyse Rosenberg for medical expertise

Cara Hefner for help with data collection

Thank You!