March 6, 2007: I. SimResearch on WWW Epi 206 – Medical Informatics Decision Support Systems Ida Sim, MD, PhD February 27, 2007 Division of General Internal

March 6, 2007: I. Sim Research on WWWEpi 206 – Medical Informatics

Decision Support Systems

Ida Sim, MD, PhD

February 27, 2007

Division of General Internal Medicine, and the Center for Clinical and Translational Informatics

UCSF

Copyright Ida Sim, 2007. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.


Outline

• Decision support systems

– background, definition

– clinical versus research decision support• How decision support systems “think”• CDSS Effectiveness• CDSS Adoption


Is Decision Support Effective?

• Moderate benefit found in improving physician behavior (Garg, 2005)

– diagnosis: 4/10 (40%) studies beneficial– reminder systems: 16/21 (76%)– disease management systems: 23/37 (62%)– drug dosing: 19/29 (66%)– few studies improved patient outcomes: 7/52 (13%)

• Counted the number of systems in each category that were “effective” (p>0.05)– but CDSS not all the same (apples and oranges)


CDSS Running Example

• Hypertension treatment Clinical Decision Support System (CDSS)– Clinic has an EHR

– During patient visit, CDSS notes that BP and trend is too high.

– CDSS checks patient’s Cr, diabetes status, cardiac status, current meds and allergies and recommends drug therapy change according to JNC VII guidelines and insurance coverage.

– Presents e-prescription for MD to verify. If verified, order is sent directly to pharmacy and medication list updated.


“Apples” HTN CDSS• Clinical Decision Support Systems (CDSSs)

– software designed to directly aid clinical decision-making• help clinician to prescribe anti-hypertensive

– in which the characteristics of an individual patient are matched to a computerized knowledge base

• match EHR and other data to computable guideline

– and patient-specific assessments or recommendations are presented to the clinician and/or patient for a decision

• recommends drug according to clinical, guideline, and insurance information

• provides clinician with decision choice to prescribe or not prescribe


“Oranges” HTN CDSS• Clinical Decision Support Systems (CDSSs)

– software designed to directly aid clinical decision-making• help clinician to prescribe anti-hypertensive

– in which the characteristics of an individual patient are matched to a computerized knowledge base

• clerk routinely abstracts current BP, A1C, meds, allergies and insurance status from paper chart into a database

• computer runs pt information against computerized guideline

• computer outputs a piece of paper with recommendation

– and patient-specific assessments or recommendations are presented to the clinician and/or patient for a decision

• MD given piece of paper with individualized drug recommendation

• MD writes prescription in usual paper-based way

Taxonomy of CDSSs

OR

INFORMATION DELIVERY•Delivery format•Delivery mode•Action integration•Delivery interactivity/explanation availability

System user/Target decision

maker

DECISION SUPPORT•Reasoning method•Clinical urgency•Recommendation explicitness•Logistical complexity•Response requirement

CONTEXT•Target decision maker•Clinical setting•Clinical task•Unit of optimization•Relation to point of care•Potential external barriers to action

WORKFLOW•Degree of workflow integration

System user/Output

intermediary [ ]

Target decision maker

KNOWLEDGE/DATA SOURCEClinical knowledge source [ ]Patient data source [ ]Data source intermediary [ ]Degree of customizationUpdate mechanism


CDSS Characteristics• Using taxonomy, reviewed and classified 42 RCT-

evaluated CDSSs• Tremendous variation in decision-maker/context, how

recommendation delivered, staff needed to make system run, complexity of recommended actions– 45% targeted to clinician, 55% patient, 5% both– 62% based on national guidelines or literature– 69% “pushed” recommendation to decision maker– 43% collected data directly from the EHR

• 45% required data input intermediary (11% MD)

– 26% required an output intermediary

• Generalizing successes from literature is difficult

(Berlin, Sim, 2006)


CDSS Effectiveness Summary• Current data suggests CDSSs can improve the

process of care and perhaps clinical outcomes– most effective at preventive care reminders– modest at best for drug dosing and active care– generally not helpful for improving diagnosis except with

trainees• Findings limited by

– methodological problems and design type of studies– insufficient appreciation of workflow component of CDSSs– insufficient appreciation of heterogeneity of systems

• Bottom line: only moderate evidence of benefit– limited generalizability of evidence


Ida Sim, MD, PhD

March 6, 2007

Division of General Internal Medicine, and Center for Clinical and Translational Informatics

UCSF

Conducting Research through the Web

Copyright Ida Sim, 2007. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.


Outline

• Internet vs. Web for Research• Example Web Research Project

– design and deploy survey

– recruit participants

– misc challenges

• Summary


Biomedical Informatics Spheres

Administrative Clinical Care Research

ClinicalBilling

Physical Networking

Standard Communications Protocols (e.g., HL-7)

Standard Vocabulary

PracticeManagement

Systems

Medical BusinessData Model

ElectronicMedicalRecord

Clinical CareData Model

??

??


Status of Research IT

• The 3rd wave of health IT development?• Only $2b of $12b RCT industry is electronic• Held back by paper-based processes and

– FDA requirements (for electronic documents (ie PDF) not data)

– lack of mobile hardware

– attitude, inertia (e.g., on Good Clinical Practices)

• Most innovation is in industry, some in academia


2 Types of Research IT Systems

• IT to support traditional studies (ie replace paper)– study design, methodology, etc. all same– gains are in efficiency, accuracy of data– e.g., a clinical trial management system (CTMS)

• IT for more than replacing paper, e.g., – using web to recruit patients– delivering interventions over the web, or via cell

phones– conducting dynamic, responsive web surveys


Internet vs. Web

itsa

medicine

ucsf.edu

nci.nih.gov cochrane.uk myhome.com

Main Trunk Cables

local trunk cablethrough Berkeley

amazon.com

at homedial-in to itsa.ucsf.edu via modem

pacbell.net

aol.com

Internet Service Provider (ISP)via DSLor cable

LAN


Internet vs. Web

• Internet = network of networks– computers and cables all linked to one another

and talking to one another using protocols

– supports lots of different internet protocols• e.g., http, ftp, smtp, https, rdf, doi, etc. etc.

• Web is the internet traffic that uses http– servers send out information in HTML

• Hypertext Markup Language

– web browsers can decode HTML and display it


HTML

<html><body><H1>Level 1 Heading</H1><p>Start of the first paragraph<br><b>bolded</b> or <i>italic</i> words<p><p><a href=http://www.ucsf.edu/>link to UCSF</a></body><html>

http://www.ucsf.edu/


Client/Server Model

• Computers can be servers and/or clients– provide or request “services”

• e.g., web server “serves” web pages to “clients,” who view these pages using a browser – MS Internet Explorer or Netscape Firefox

Clients

WebServer


Clients and Servers

itsa

medicine

ucsf.edu

nci.nih.gov cochrane.uk myhome.com

Main Trunk Cables

amazon.com

at home

pacbell.net

aol.com

LAN

Server

Client


Research IT on Internet/Web

• Research IT using Internet– uses Internet network of networks to send data and

commands back and forth– servers and clients do the storage, query, retrieval,

computation, reporting– may have nothing to do with a web browser

• Research IT using Web– web servers send HTML content over the Internet using

HTTP– web browsers and other “clients” receive that content for

display or computation• What are logistical and methodological issues?


Outline




– misc challenges

• Summary


Eg: Smoking Cessation Trial

• Target audience– English and Spanish-speaking smokers

• Pre- and post demographic, etc. survey• Randomized Interventions

– downloadable brochure vs. brochure + email reminders + diary

• Outcome– quit rate

Slides from Ricardo Muñoz, [email protected] World Health Research Center, www.health.ucsf.edu

mailto:[email protected]


Outline


– design and deploy survey• buy vs. build

• building your own (design considerations, costs, etc)


– logistical challenges

• Summary


Web Surveys are Cheaper

• Web surveys have higher fixed cost but cost per additional respondent is much lower– marginal cost per mail survey respondent $1.93– phone $40 to $100– web $0

• Buy or build?– buy: many companies offer survey design,

deployment, and data management services– build: do-it-yourself


Simple Build

• Requirements– web server

– survey forms • you design and build, or hire web designer

– e.g., using Access Visual Basic, or Access and Front Page (web page design software)

– backend data storage solution• e.g., web responses stored directly in Access

– requires database with ODBC drivers

• Simple survey forms in HTML...


Buying Survey Services

• Many, many companies exist• Survey Monkey www.surveymonkey.com

– free for 10 questions, 100 responses per survey– professional subscription $19.95/mo or $200/yr

• up to 1000 responses per month, $0.05 per additional response

• DatStat’s Illume – web-based survey creation and management– real-time data access and complex query capabilities– exports data to SAS, SPSS, etc. – Internet World Health Research Center is beta user

• $7000/yr first year, $3000/yr thereafter

• $4000 license/user (e.g., you)

Disclosure: I have no ties to SurveyMonkey or DatStat

http://www.surveymonkey.com/


Survey Design

• Usual survey design issues apply, PLUS• Technical design of survey

– platform (e.g., Mac) and browser (e.g., Safari) incompatibilities

– use Flash, Java, etc requiring plug-ins or version compatibility

– readiblity (font too small), need to scroll, confusing navigation, bugs

• What technology does respondent group use?– check some browser statistics sources

• e.g., http://www.w3schools.com/browsers/browsers_stats.asp

– need to test and double-test in various platforms and browsers used, various versions of HTML, Java, Flash, etc.


Outline



– recruit participants• selection, sampling, non-response bias

• sample size calculation

– misc challenges

• Summary


Subject Recruitment

• Recruitment is biggest bottleneck of clinical research

– 30-40% of clinical trial costs

– >80% of trials have recruitment delays

– 1/20 recruited patients actually enroll • Web-based recruitment can be international, cheap,

fast

– e.g., www.stopsmoking.ucsf.edu since Dec 2005• 350,000 hits, 60,000 entered data, 20,000 enrolled

• 2/3 Spanish-speaking, 1/3 English

http://www.stopsmoking.ucsf.edu/

Visits0=>1=>100=>1,000=>10,000

Distribution of Visits to www.stopsmoking.ucsf.edu Jan 12, 2005 to April 5, 2006

(131,517 visits from 121 countries)

http://www.stopsmoking.ucsf.edu/


Threats to Validity

• Selection bias: who is on the web? who isn’t?– digital divide

• Sampling error– non-biased sampling of respondent population

• Non-response bias– enrollees not completing the survey

• Measurement error– poor question wording, variation in how survey

appears on various browsers, non-completion


Digital DivideInternet Access Broadband Access

<$30,000 41% 8%

$30-49,000 71% 16%

>$50,000 89% 39%

No children 59% 16%

Children in home 76% 29%

White 69% 23%

African-American 56% 15%

Hispanic 48% 14%

"Digital Divide" Still Shapes Media Landscape (10/19/04, Knowledge Networks/SRI); http://www.knowledgenetworks.com/info/press/releases/2004/101904_htmtrends.htm

http://www.knowledgenetworks.com/info/press/releases/2004/101904_htmtrends.htm


Digital Health Divide

• Spanish-language sites have lower quality– 45% of English-language sites vs. 22% with minimal

coverage & complete accuracy (JAMA 2001; 285:2612-2621)

• Broadband more available to higher-income white households with children– uneven potential access to Flash, tele-consultation,

etc.


Reducing Sampling Error

• Social sciences and marketing are most advanced in web survey methodology– e.g., Joint Statistical Meetings of the American

Statistical Association

– http://www.knowledgenetworks.com/dmg/index.html

• Recruit a representative sample– e.g., random digit dailing (RDD) sampling for web

• Use a pre-assembled representative cohort

Disclosure: I have no relationships with KnowledgeNetworks


RDD for Web

• Random digit dialing (RDD) analog equally representative as telephone RDD– RDD sampling

– if respondent agrees, provide them with free Internet access (via MSNTV, aka WebTV) or other necessary hardware for duration of participation

– e.g.,http://knowledgenetworks.com/

http://knowledgenetworks.com/




Representative Cohorts

• Maintained by e.g., large survey and marketing firms– www.knowledgenetworks.com

• KnowledgePanel is representative of US• can target specific respondents, “response rates of 65-

75%, abandonment rate <2%”

– www.surveysampling.com• panels in 17 countries totaling 3.8 million respondents

– http://experimentcentral.org/ • NSF-funded representative panel for social science

research

http://www.knowledgenetworks.com/

http://www.surveysampling.com/

http://experimentcentral.org/


Enrollment Rates

• Response rates typically 30-60%• Affected by

– number of (pre) contacts, whether personalized• most influential factors

– incentives (e.g., Amazon certificate)

– population surveyed, nature of topic, official sponsorship, etc.


Other Recruitment Methods

• Higher risk of sampling bias– search engines

• need seach engine optimzation (SEO) techniques

– links from related pages

– email lists, social networking sites, chat rooms, newsgroups

– other Internet communities• friends, webrings

– blend traditional and web• give website on radio, TV, print, brochures


Search Engine Ranking• Search engines have their own (secret) algorithm for

ranking pages– Google uses >100 factors, esp. how many pages link into a

page• Important page attributes influencing ranking include

– keywords in the title tag– keywords in links pointing to the page– keywords appearing in visible text– link popularity– keywords in Heading Tag H1,H2 and H3 Tags in webpage

• Google AdWords – adwords.google.com– put in your keywords, see cost-per-click – pay only if someone click


Measurement Bias

• What you designed may not be what respondent sees

• Client’s browser displays the survey based on – platform, browser, monitor, screen/window siz

– e.g,• small screen/window size makes “Next” button not visible

• text doesn’t fit on small window, or requires scrolling for some respondents and not others

• colors, graphics (e.g., visual analog scales) may appear differently


Non-Completion Bias

• Influenced by– respondent familiarity with web (e.g., click

on link)– technical design of survey– bandwidth– convenience (return to finish?)

• Can use mixed-mode surveys to address– e.g., combined web/phone, web/mail


Note on Sample Size

• Estimating sample size– e.g., Google provides traffic history for various

keywords (adwords.google.com)

• Since incremental cost often negligible, less pressure to minimize sample size– not unusal to get large samples (>10,000)

• But high sample size = high accuracy!– may be precise but inaccurate if sample is non-

representative


Outline




– misc challenges• data integrity

• anonymity/privacy/ethics

• Summary


Data Integrity

• Balancing convenience, privacy, study quality– collecting IP addresses allows easy stop and starts but

compromises privacy– could give respondent an ID to sign back in, but would

reduce completion rates– if anonymous, how to prevent people from participating more

than once? • Reliability

– research suggests that data obtained via internet/computer are somewhat more reliable than what is obtained by phone or in-person

• esp. for socially normative topics (e.g., sex, drug use)


Ethical Considerations

• Anonymity and privacy– IP address (network identifier for client machine) is easily

obtainable• IP hiding and counter-hiding by anonymizer.com

• Informed consent– respondent can’t ask questions to ensure “informed” consent

• could use live chat?

– how to ensure against minors responding without parental consent

• require written parental consent, credit card, etc.

• HIPAA rules– technically does not cover web surveys if not being used for

reimbursable care, but IRBs are increasingly stringent


Interventions on Web

• Huge range of possibilities• Therapeutic/disease management

– online diaries, videos, interactive games, etc• e.g., http://asthma.starlightprograms.org/

• Diagnostic• Shared decision-making/patient education

– see http://decisionaid.ohri.ca/AZinvent.php for large catalog

• “Web 2.0” interventions?– emphasizing online collaboration, sharing, “social computing”

– e.g., Facebook, MySpace, YouTube for health care?

http://decisionaid.ohri.ca/AZinvent.php


Outline




– misc challenges

• Summary


Costs, Services, Support

• Cost categories– recruitment

• access to representative sample

• adwords and ad placement costs

– survey design

– survey deployment and database management

– interventions development

• No central “core” for these services at UCSF– should there be?

– best campus resource right now for this is Ricardo Muñoz ([email protected])

mailto:[email protected]


Summary

• Exciting opportunities for new, broader research

• Lots of technical landmines• Many open questions re: methodology

– minimizing sampling and selection biases, sample size, survey design, etc.

– look outside clinical research for advances

• Need better campus-wide support for these activities?

Documents

March 6, 2007: I. SimResearch on WWW Epi 206 – Medical Informatics Decision Support Systems Ida Sim, MD, PhD February 27, 2007 Division of General Internal