Rating Health Information Websites

Peter G. Goldschmidt, PresidentHealth Improvement Institute

Bethesda, Maryland

Consumer Reports WebWatch

Trust or Consequence Berkeley, California

June 9, 2005

2

Presentation

1. Introduction – HII-CRW partnership

– Independent ratings

– Project/ratings philosophy

– Types of health websites

2. Methods

3. Results

4. Conclusions

3

1A HII-CRW Partnership

Health Improvement Institute 1990, founded; non-profit 501(c)3 education/research charitable

organization 1991, established Aesculapius Awards for excellence in health

communication 1997, held workshop on quality of health information on the Internet 2003, entered into partnership with CU/CRW

Purpose/objectives regarding health websites To enable consumers to be more effective users To provide consumer with independent ratings

Medical Library Association is actively supporting project

4

1B Independent Ratings

Many past/present activities Criteria

– American Medical Association

– Mitretek Systems Codes of conduct

– eHealth Code of Ethics

– HI-ETHICS Trustmarks/seals

– Health on the Net Foundation (HONcode)

– URAC accreditation

Growing literature on criteria, quality of health websites

5

1C Project/Ratings Philosophy

Transparent regarding– Identity, contact information

– Ownership

– Privacy (or its invasion)

Easy to use regarding– Finding/navigating contents on

website

– Comprehending contents (clarity of writing, reading level, etc)

– Printing contents of interest

– Serving consumers with special needs

Meticulous in distinguishing advertising/selling from contents

Editorially adequate, for example, disclose & be appropriate regarding – Providing authors' credentials

– Selecting/grading & editing contents

– Resulting content

° Accurate, complete, objective, balanced

° Well-organized

° coherent/clear; not muddled

° Referenced

Current Useful to consumers

Health information websites must be

6

1D Types of Health Websites

Health communication– Health information – Decision support tool– Health ratings– Health information resource– Health website search

engine– Health advice on-line

Behavior modification – Behavior self-help – Disease management

On-line product sales/marketing– On-line pharmacy – On-line store – Health product marketing

Health care organization – Health plan

enrollment/transaction – Health care provider– Producer of health resources– Public health program– Other health care organization

To start, focus is "health information websites"

7

2 Health Website Rating Methods

Enter into partnership

HII-CRW

Enter into partnership

HII-CRW

Develop concepts/ approach

Develop concepts/ approach

Select websites to

be rated

Select websites to

be rated

Select additional

websites to be rated

Select additional

websites to be rated

Develop health

website rating instrument

Develop health

website rating instrument

Credential/ select raters

Credential/ select raters

Design ratings website

Design ratings website

Rate health websites

Rate health websites

Display ratings

Display ratings

Rate additional websites

Rate additional websites

Rerate rated websites

Rerate rated websites

Add/update ratings website

displays

Add/update ratings website

displays

Refine methods

Refine methods

Analyze feedback

Analyze feedback

8

2A Ratings Concepts/Strategy

Transparency/accountability

Editorial adequacy

Information reliability — completeness/accuracy of what is stated in any medium for any audience — excluded from present project because– Website may contain information on very many subjects

– To rate validity of health information for given subject requires panel of qualified medical/research experts

– Ultimately, assessments of reliability/validity of health information reflect state of medical science

3 levels for health information websites

9

2B Select Websites

To start Define health information websites Identify 100 most-visited "health" websites (A. C. Nielson) Select top-20 "health information websites"

To continue Rate additional top-100 websites Rate health websites suggested by consumers

Periodically, rerate rated websites

10

2C Develop Ratings Instrument

Developed general instrument applicable to rating all types of websites (CRW principles)

Adapted CRW instrument to meet project purposes

Created "Part I" after evaluation by HII volunteers of adapted CRW instrument (& compiled generic/specific criteria)

Pretested/revised Part I

Analyzed/assessed criteria sets intended to evaluate health websites

Compiled list of (generic & specific) criteria

Asked HII volunteers to evaluate criteria

Created "Part II" to focus on specific criteria

Piloted/refined Part II

Consumer Reports WebWatch Health Improvement Institute

11

2C Contents of Ratings Instrument

I Website transparency/accountability

II Health information editorial policies

III HONcode compliance

IV Raters' feedback

12

2C HWRI, Part I, CRW Principles

Developed/applied by CRW staff Identity

Advertising & sponsorships

Ease of use

Corrections & currency

Privacy

13

2C HWRI, Part II, HII principles

A. Criteria/descriptors – Characteristics of website

contents– Accessibility of contents to

consumers– Editorial policies/procedures– Authors of articles/contents– Articles– Summary scores

Optional – Raters could also complete Part-

I (to facilitate construction of evaluative narratives)

B. Evaluative narratives – Purpose/scope & intended

audience– Characteristics of website– Accessibility– Editorial policies/procedures– Contents– Website's greatest strengths– Website's greatest

weaknesses – Utility of website to

consumers

Developed by HII; applied by volunteers

14

2D Credential/Select Raters

Process Call for raters Request applicants present their

credentials Submit applicants' credentials to raters

credentialing committee Credential health website raters Select credentialed raters for panel

– Type of organization in which employed currently

– Role– Professional background– Region of country

Credentialing criteria Health professional

At least 5 years continuous relevant experience

Currently active in health field

Sufficient qualifications/experience to evaluate health websites

No apparent disqualifying event

After initial credentialing, satisfactory performance as rater

15

2E Rate Websites

CRW staff applied HWRI, Part I

Panel of HII volunteers; each applied HWRI– Raters signed HII "Policy on Conflict of Interest“

– If conflict of interest, website reassigned

HII summarized individual panel members' scores/assessments to produce coordinated ratings

HII/CRW integrated ratings for website display

CRW created ratings website displays

16

2F Create Ratings Website

Scope —website contains – Introductory/explanatory

material– Description of methods– Disclosures/disclaimers– Ratings page for each

rated website

Ratings include – Website's stated purpose– HII-CRW description of

website– Global/attribute scores– Greatest

strengths/weaknesses– Noteworthy items

Purpose/objectives– To display independent health website ratings in consumer-

friendly way

– To solicit feedback

17

3 Results

Rating health websites

Rated websites

Median attribute ratings

Distribution of ratings

Excellent top-20 health websites

Example of ratings webpage

Ratings are accessible on www.healthratings.org

18

3A Rating Health Websites

Process 20 most-visited health websites Rated for

– Transparency, 2 CRW raters

– Editorial policies, HII raters

14 HII raters (3-member panels)– 2 health practitioners

– 5 health information experts

– 2 health education specialists

– 5 media, production & related

Resultant ratings– Excellent– Very good– Good– Fair– Poor

Attributes Identity

Advertising & sponsorship

Ease of use

Corrections & currency

Privacy

Design

Coverage

Accessibility (navigation/reading level)

Contents

Overall rating

19

3B Rated Websites (20 most-visited)

webmd.com nih.gov health.yahoo.com about.com/health mayoclinic.com medicinenet.com emedicine.com drugs.com intelihealth.com pfizer.com

realage.com kidshealth.org rxlist.com qualityhealth.com healthology.com health.ivillage.com medscape.com heartcenteronline.com healthboards.com healthsquare.com

20

3C Median Attribute Ratings

Identity - Excellent Advertising & sponsorship - Excellent Ease of use - Good Corrections & currency - Fair Privacy – Excellent Design – Good Coverage – Very good Accessibility – Very good Contents – Very good

21

3D Distribution of Ratings

0

1

8

5

6 (30%)

(25%)

(40%)

(6%)

(0%)

Excellent

Very Good

Good

Fair

Poor

Top 20 Health Websites

22

3E Excellent Top-20 Health Websites

emedicine.net

kidshealth.org

mayoclinic.com

medscape.com

nih.gov

webmd.com

Ratings are accessible on www.healthratings.org

23

3F Website Ratings Page Example

24

4 Conclusions

Designing, producing & updating excellent health information websites is costly, complex, challenging

Rating websites is equally challenging, but desirable & feasible

6 of 20 most visited health websites, were rated “excellent” overall

Quality of information is limited by state of medical science

25

4A Conclusions: Needed improvements

Generally, rated websites need to improve– Contents - descriptions of editorial policies; also policies

& procedures• Describe how select topics, search/grade information,

develop contents, assure quality of articles/contents• Name authors/reviewers• Provide authors/reviewers’ credentials; must be appropriate

to contents; disclose financial/other interests• State date last reviewed/updated• Refer to sources of facts/citations• Indicate criteria for linking to other websites

– Design & Ease of use

– Accessibility - especially for consumers with special needs

– Currency & corrections

26

4B Feedback

HII/CRW welcome feedback on– Ratings’ utility to consumers

– Suggestions for improving ratings website, process, criteria, etc.

– Health websites to be rated

Volunteer to rate health websites!