Instructional Design Variations in Internet-Based … · Instructional Design Variations in Internet-Based Learning for Health Professions Education: A Systematic Review ... Compared

Internet-Based Learning: Systematic Review

Instructional Design Variations inInternet-Based Learning for HealthProfessions Education: A Systematic Reviewand Meta-AnalysisDavid A. Cook, MD, MHPE, Anthony J. Levinson, MD, MSc, Sarah Garside, MD, PhD,Denise M. Dupras, MD, PhD, Patricia J. Erwin, MLS, and Victor M. Montori, MD, MSc

Abstract

PurposeA recent systematic review (2008)described the effectiveness of Internet-based learning (IBL) in healthprofessions education. Acomprehensive synthesis of researchinvestigating how to improve IBL isneeded. This systematic review soughtto provide such a synthesis.

MethodThe authors searched MEDLINE, CINAHL,EMBASE, Web of Science, Scopus, ERIC,TimeLit, and the University of TorontoResearch and Development ResourceBase for articles published from 1990through November 2008. They includedall studies quantifying the effect of IBLcompared with another Internet-based orcomputer-assisted instructional

intervention on practicing and studentphysicians, nurses, pharmacists, dentists,and other health professionals. Reviewersworking independently and in duplicateabstracted information, coded studyquality, and grouped studies accordingto inductively identified themes.

ResultsFrom 2,705 articles, the authorsidentified 51 eligible studies, including30 randomized trials. The pooled effectsize (ES) for learning outcomes in 15studies investigating high versus lowinteractivity was 0.27 (95% confidenceinterval, 0.08–0.46; P � .006). Alsoassociated with higher learning werepractice exercises (ES 0.40 [0.08–0.71;P � .01]; 10 studies), feedback (ES 0.68[0.01–1.35; P � .047]; 2 studies), and

repetition of study material (ES 0.19[0.09–0.30; P � .001]; 2 studies). TheES was 0.26 (�0.62 to 1.13; P � .57)for three studies examining onlinediscussion. Inconsistency was large(I2 �89%) in most analyses. Meta-analyses for other themes generallyyielded imprecise results.

ConclusionsInteractivity, practice exercises,repetition, and feedback seem to beassociated with improved learningoutcomes, although inconsistency acrossstudies tempers conclusions. Evidence forother instructional variations remainsinconclusive.

Acad Med. 2010; 85:909–922.

Internet-based medical education hasproliferated rapidly since the advent ofthe World Wide Web in 1991. Potentialadvantages of Internet-based learning

(IBL) over other instructional methodsinclude flexibility in time and locationof learning, economies of scale,facilitation of novel instructionalmethods, and the potential topersonalize instruction to individualneeds.1–3 Hundreds of publishedarticles have described and evaluatedthe use of IBL in health professionseducation.4

Educators need evidence-based guidanceon how to develop effective IBL.3 Over200 studies have reported the results ofcomparing IBL either with nointervention or with traditional(noncomputer) instructional methods inhealth professions education.4 In aprevious report (2008),4 we sought toidentify salient principles regarding whenand how to use IBL. While that meta-analysis supported the effectiveness ofIBL, the evidence did not clearly identifyprinciples to guide futureimplementations. Previous reviews haveencountered similar limitations.5–8

An alternative approach to identifyingevidence-based principles involves directcomparisons of one Internet-basedintervention against another.9 –13

Evidence from such studies, if properlyreviewed and synthesized, could informdecisions on when and how to effectivelyuse IBL. We are not aware of previoussystematic reviews addressing thesequestions.

In the present study, we sought toidentify and quantitatively summarize allstudies involving health professionslearners that compared IBL with anothercomputer-based instructional format. Wefocused our review on health professionslearners because— even if fundamental

Dr. Cook is associate professor of medicine anddirector, Office of Education Research, College ofMedicine, Mayo Clinic, Rochester, Minnesota.

Dr. Levinson is associate professor of psychiatryand director, Division of e-Learning Innovation,McMaster University, Hamilton, Ontario, Canada.

Dr. Garside is associate professor of psychiatry andassociate director, Division of e-Learning Innovation,McMaster University, Hamilton, Ontario, Canada.

Dr. Dupras is assistant professor of medicine, Collegeof Medicine, Mayo Clinic, Rochester, Minnesota.

Ms. Erwin is assistant professor of medicaleducation, College of Medicine, Mayo Clinic,Rochester, Minnesota.

Dr. Montori is professor of medicine and director,Knowledge and Encounter Research Unit, College ofMedicine, Mayo Clinic, Rochester, Minnesota.

Correspondence should be addressed to Dr. Cook,Division of General Internal Medicine, Mayo ClinicCollege of Medicine, Baldwin 4-A, 200 First StreetSW, Rochester, MN 55905; telephone: (507) 266-4156; fax: (507) 284-5370; e-mail:([email protected]).

Supplemental digital content is available for thisarticle. Direct URL citations appear in the printedtext; simply type the URL address into any Webbrowser to access this content. Clickable links tothe material are provided in the HTML text andPDF of this article on the journal’s Web site(www.academicmedicine.org).

Academic Medicine, Vol. 85, No. 5 / May 2010 909

principles of learning apply broadly—thetopics, learning objectives, and learners inhealth professions education vary fromother fields of study.

Method

We planned, conducted, and reportedthis review in adherence to standards ofquality for reporting meta-analyses(QUOROM and MOOSE).14,15

Question

We sought to answer the followingquestion: “What characteristics of IBLinterventions, as compared with othercomputer-based interventions, areassociated with improved outcomes inhealth professions learners?”

Study eligibility

We included studies published in anylanguage that investigated use of theInternet, in comparison with anothercomputer-based intervention, to teachhealth professions learners at any stage intraining or practice, using the Kirkpatrickoutcomes16 of (1) satisfaction, (2)knowledge or attitudes, (3) skills (in a testsetting), and (4) behaviors (in practice)or effects on patients.

Definitions of key variables (e.g.,cognitive interactivity) have been detailedpreviously.4 We defined healthprofessions learners as students,postgraduate trainees (i.e., residents orfellows), or practitioners in a professiondirectly related to human or animalhealth, including physicians, nurses,pharmacists, dentists, veterinarians, andphysical therapists. We defined IBL ascomputer-assisted instruction3 using theInternet or a local intranet as the meansof delivery. This included Internet-basedtutorials, virtual patients, discussionboards, e-mail, and Internet-mediatedvideoconferencing.

We excluded studies if all of thecomputer interventions investigatedresided only on the client computer orCD-ROM or if the use of the Internet waslimited to administrative or secretarialpurposes. We also excluded studies thatdid not report outcomes of interest orwere published only in abstract form.

Study identification

We described our search strategy(Supplemental Digital Content, Box 1,http://links.lww.com/ACADMED/A14)

previously.4 Briefly, one of us (P.J.E.), asenior reference librarian with expertisein systematic reviews, designed a strategyto search MEDLINE, CINAHL, EMBASE,Web of Science, Scopus, ERIC, TimeLit,and the University of Toronto Researchand Development Resource Base. Searchterms included words defining deliveryconcepts (such as Internet, Web, e-learning, and computer-assistedinstruction) and participantcharacteristics (such as “education,professional,” “students, healthoccupations,” and “internship andresidency”). Because the World WideWeb was first described in 1991, oursearch included all articles published afterJanuary 1, 1990. The last search date wasNovember 17, 2008. We identifiedadditional studies both by scanningauthors’ files and previous reviews and byhand searching the reference lists of allincluded articles.

Study selection

Four of us (D.A.C., A.J.L., D.M.D., andS.G.), working independently and induplicate, screened all titles and abstractsto determine whether we should includean article. In the event of disagreement orinsufficient information in the abstract,we reviewed the full text, againindependently and in duplicate. Weresolved conflicts by consensus. Chance-adjusted interrater agreement for studyinclusion, determined using intraclasscorrelation coefficient17 (ICC), was 0.73.

Data extraction

We developed a data abstraction formthrough iterative testing and revision. Weabstracted data independently and induplicate for all variables requiringreviewers’ judgment. We determinedinterrater agreement using ICC, and weresolved conflicts by consensus.

We abstracted the following information:

• number and training level of learners,

• topic,

• study design: presence of pretest (ICC:0.84), number of groups (ICC: 0.97),and method of group assignment (ICC:1.0),

• course length (ICC: 0.70),

• level of cognitive interactivity (low,moderate, high; ICC: 0.71),

• quantity of practice exercises (absent,few, many; ICC: 0.84),

• outcome assessment method(subjective/objective; ICC: 0.79), and

• quantitative outcome results (meanand standard deviation [SD], or otherinformation for calculating effect size[ES]).

When articles contained insufficientoutcomes data, we requested thisinformation from authors.

Desiring to use a common quality metricfor both randomized and observationalstudies, we abstracted information onmethodological quality using anadaptation of the Newcastle–Ottawa scalefor grading cohort studies.4,18 We ratedthe following:

• representativeness of the interventiongroup (ICC: 0.47),

• selection of the control group (ICC:0.93),

• comparability of cohorts: eitherstatistical adjustment for baselinecharacteristics in nonrandomizedstudies (ICC: 0.34) or randomization(ICC: 1.0) and allocation concealmentfor randomized studies (ICC: 0.38),

• blinding of outcome assessment (ICC:0.54), and

• completeness of follow-up (ICC: 0.71).

Three of us (D.A.C, A.J.L., and S.G.)iteratively reviewed all articles toinductively identify themes among theresearch questions or research hypothesesand to achieve consensus on definitions(Table 1). Then, working firstindependently and then by consensus, we(again D.A.C, A.J.L, and S.G.) groupedeach study by research theme.

Data synthesis

We abstracted information separately foroutcomes of satisfaction, knowledge,skills, and behaviors/patient effects. Weconverted means and SDs or odds ratiosto standardized mean differences(Hedges’ g effect sizes).19 –22 Whensufficient data were unavailable, we usedtests of significance (e.g., P values) toback-calculate ESs using standardformulae.23 For crossover studies, weused (1) means or exact statistical testresults adjusted for repeated measures or,if these were not available, (2) meanspooled across each intervention.22,24 Fortwo-group pretest–posttest studies, weused (1) posttest means or exact


Academic Medicine, Vol. 85, No. 5 / May 2010910

Table 1Research Themes Addressed by Studies Included in a Systematic Review ofInternet-based Instruction*

Theme Operational definitionNo. of

studies†

Interactivity: High versus low Compared different levels of cognitive engagement, such as varying the number ofself-assessment questions, adding interactive models, or including thought-stimulatingactivities. Although online discussion would likely enhance interactivity, online discussionalone did not count (i.e., was coded separately).

15

...................................................................................................................................................................................................................................................................................................................Feedback versus no feedback Compared providing feedback versus no feedback. Studies comparing questions and

feedback versus no questions/no feedback (i.e., did not isolate the effect of feedback)were not coded as “yes, feedback” but are nonetheless noted in Table 2.

2 (8‡)

...................................................................................................................................................................................................................................................................................................................Configuration comparison Compared alternate, Internet-based learning systems or interfaces. 8...................................................................................................................................................................................................................................................................................................................Discussion versus no discussion Compared courses that provided for interactions using synchronous or asynchronous

online communication, such as discussion board, e-mail, chat, or Internet conferencing,versus courses with no online communication.

7

...................................................................................................................................................................................................................................................................................................................Patient cases versus no cases Compared the presence/absence of patient cases. No studies completely isolated this

effect (i.e., other features changed as well between the two interventions).0 (4‡)

...................................................................................................................................................................................................................................................................................................................Enhanced to promote participation versusunenhanced

Compared strategies specifically designed and intended to promote greater participationin a course versus courses without such strategies

4

...................................................................................................................................................................................................................................................................................................................Games or simulation versus didactic Compared inductive learning through the use of games and simulations versus learning

through traditional (sequential) presentation of information. Interactive patient scenariosused to introduce otherwise sequential instruction were counted only under “Patientcases.”

3

...................................................................................................................................................................................................................................................................................................................Synthesized information versus existinginformation

Compared information synthesized by the instructor (e.g., a specially prepared e-mail orWeb page) versus access to existing information (e.g., publicly available Web page).

3

...................................................................................................................................................................................................................................................................................................................Adaptive versus nonadaptive navigation Compared instructional designs that substantially altered the course progression in

response to learner characteristics or prior performance versus nonadaptive courses2

...................................................................................................................................................................................................................................................................................................................Audio in discussion Compared written text discussion (i.e., Internet chat) versus auditory discussion

(i.e., videoconference).2

...................................................................................................................................................................................................................................................................................................................Audio in tutorial Compared Internet-based tutorials with versus those without auditory information (audio

versus written text for online discussions was coded separately).2

...................................................................................................................................................................................................................................................................................................................Blended online and face-to-face versus onlineonly

Compared blended instructional formats (online and face-to-face) versus online-onlyformats.

2

...................................................................................................................................................................................................................................................................................................................Repetition of material versus no repetition Compared repeated experience with learning material (e.g., e-mails with identical

information sent multiple times) versus no repetition.2

...................................................................................................................................................................................................................................................................................................................Spacing of material versus no spacing Compared spacing the presentation of learning materials over time versus presenting all

material at once. This is different than repetition: In spacing, less information is presentedat a given time and the total amount of information is kept constant (only the timing ofpresentation changes); in repetition, the same information is presented multiple times.

2

...................................................................................................................................................................................................................................................................................................................Active versus reflective questions Compared self-assessment study questions designed to facilitate active (multiple-choice,

single best answer) versus reflective (open-ended, no best solution) cognitiveengagement.

1

...................................................................................................................................................................................................................................................................................................................Animations versus static images Compared courses enhanced with animations versus courses using static images 1...................................................................................................................................................................................................................................................................................................................Case-first versus didactic-first Compared Internet-based learning tutorials starting with case-based questions and then

didactic information presented as feedback versus didactic information presented firstfollowed by case-based questions.

1

...................................................................................................................................................................................................................................................................................................................Embedded e-mail text versus link to text Compared learning content embedded in e-mail text (“push” information) versus

learning content residing on Web page accessed via link provided in e-mail(“pull” information).

1

...................................................................................................................................................................................................................................................................................................................Synchronous versus asynchronous learning Compared simultaneous interaction between two or more course participants over the

Internet (e.g., chat, live videoconference) versus nonsimultaneous versions of similarmaterial (e.g., discussion board, archived lecture).

1

...................................................................................................................................................................................................................................................................................................................Three-dimensional model Compared presence versus absence of a three-dimensional model. 1...................................................................................................................................................................................................................................................................................................................Timing of discussion live versus delayed Compared discussion occurring during versus after an Internet-mediated lecture. 1...................................................................................................................................................................................................................................................................................................................Video clips versus none Compared courses enhanced with video clips versus no video clips 1

* Themes were inductively developed through iterative review of included studies. The interventions inconfounded studies varied by more than one feature, precluding isolation of this effect.

† Number of studies investigating this as a study theme.‡ Confounded.



statistical test results adjusted for pretestor, if these were not available, (2)differences in change scores standardizedusing pretest variance. If neither P valuesnor any measures of variance werereported, we used the average SD from allother included studies.

Because we anticipated largeinconsistency (heterogeneity), we usedrandom-effects models to pool weightedESs across studies within researchthemes. We used the I2 statistic25 toquantify inconsistency across studies. I2

estimates the percentage of variabilityacross studies that is not due to chance;values greater than 50% indicate largeinconsistency. We conducted meta-analyses to pool study results for allthemes addressed by two or more studies,except where reviewers agreed that theimplementations of that theme were toodissimilar. In addition to the inductivelyidentified themes, we coded the level ofcognitive interactivity and quantity ofpractice exercises, and we performedmeta-analyses pooling the results ofstudies in which these codes variedbetween study arms. Some studiesappeared in more than one meta-analysis. For each meta-analysis, weevaluated separately outcomes ofsatisfaction and learning (i.e., knowledgeor—for the two studies not reportingknowledge—skills or behaviors). Too fewstudies reported skills and behaviors topermit meta-analysis using theseoutcomes alone. To explore therobustness of findings to synthesisassumptions, we conducted subgroupanalyses based on method of groupassignment and study quality.

We used StatsDirect 2.6.6 (Cheshire,United Kingdom) to pool ESs and SAS9.1 (Cary, North Carolina) to calculateICC. We defined statistical significanceby a two-sided alpha of .05.

Results

Trial flow

We identified 2,527 citations using oursearch strategy and an additional 178articles from scanning author files andreviewing reference lists. From these weidentified 369 potentially eligible articles(Figure 1), of which 51 reportedcomparisons between an Internet-basedintervention and other computer-assistedinstruction.26 –76 Eight of thesearticles29,31,33,34,38,39,50,64 also reported

comparisons with no intervention orwith a noncomputer intervention, andthese appeared in our previous systematicreview.4 Two of the studies we includedwere published online ahead of print.60,73

We contacted authors of 17 articles foradditional outcomes information andreceived information from 8. Oneotherwise eligible study32 containedinsufficient data to calculate an ES forany outcome, so we excluded it from ourfinal analysis. Four studies had more thantwo groups and/or used a factorial designto study more than one researchtheme.38,51,59,74 We included these studiesno more than once per meta-analysis.

Study characteristics

Table 2 summarizes key study features.The Internet-based courses addressed awide range of medical topics such aschest pain, geriatric psychiatry,electrocardiogram interpretation, math

skills, periodontology, communicationskills, and psychotherapy. A total of 8,416learners participated, including 3,290medical students, 1,504 postgraduatephysician trainees, 865 physicians inpractice, 303 nurses in training, 485practicing nurses, 153 dental students,151 pharmacists in training, 73practicing pharmacists, and 1,592 otherlearners (other allied health or mixedgroups). Of the 51 studies we included,most (38 [75%]) reported knowledgeoutcomes and 29 (57%) reportedsatisfaction, whereas only 4 (8%) reportedbehaviors or patient effects and 3 (6%)reported skills (See Table 2 andSupplemental Digital Content, Table 1,http://links.lww.com/ACADMED/A14).

Study quality

Table 3 summarizes the methodologicalquality of the included studies. Of the 51studies, 30 (59%) were randomized.Three of 38 (8%) knowledge assessments,

Potentially relevant studies identified and screened for retrieval (n=2,705) • 2,527 from database search • 178 from article reference lists and author files

Studies excluded, with reasons (n=1,613) • Not original research (n=462) • Not instruction predominantly via the Internet (n=950) • No quantitative comparison or qualitative data (n=101) • No health professions learners (n=100)

Studies retrieved for more detailed evaluation (n=1,092)

Potentially appropriate studies to be included in the review (n=369)

Studies excluded, with reasons (n=723) • Not original research (n=50) • Not instruction predominantly via the Internet (n=433) • No quantitative comparison or qualitative data (n=206) • No health professions learners (n=28) • Meeting abstract (n=6)

Studies excluded, with reasons (n=318) • No comparison with another computer-assisted

instruction intervention (n=265) • Qualitative outcomes only (n=35) • No relevant quantitative outcomes (n=8) • Duplicate publication (n=10)

Studies appropriate for inclusion in the review (n=51)

Studies withdrawn from meta-analyses: insufficient data for coding outcomes (n=1)

Studies included in meta-analyses (n=50)

Figure 1 Trial flow for a systematic review of Internet-based instruction.



Tabl

e2

Desc

rip

tio

no

fStu

die

sIn

clu

ded

ina

Syst

em

ati

cR

evie

wo

fIn

tern

et-

Base

dIn

stru

ctio

n(N

�51)*

Firs

tau

tho

r,yearR

ef

Part

icip

an

ts†

Med

ical

top

icStu

dy

inte

rven

tio

nC

om

pari

son

inte

rven

tio

nStu

dy

them

e‡

Ou

tco

mes§

Kap

lan,

1996

26

15;P

-PEv

iden

ce-b

ased

med

icin

eO

nlin

eas

ynch

rono

usco

mm

unic

atio

nsy

stem

1O

nlin

eas

ynch

rono

usco

mm

unic

atio

nsy

stem

2C

oS

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Cha

n,19

9927

23;M

D-P

Ger

iatr

icps

ychi

atry

Inte

rnet

-bas

edm

odul

ew

ithon

line

disc

ussi

onof

case

-ba

sed

prac

tice

exer

cise

sIn

tern

et-b

ased

mod

ule

with

out

disc

ussi

onD

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Papa

,199

928

108;

MS-

PC

hest

pain

diag

nosi

sIn

tern

et-b

ased

case

sw

ithfe

edba

ckIn

tern

et-b

ased

case

sw

ithou

tfe

edba

ckF

K...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.Sc

haad

,199

929

500;

MS-

PH

uman

biol

ogy

Cou

rse

with

Inte

rnet

disc

ussi

ongr

oups

and

virt

ual

clin

ic(o

nlin

epr

actic

eex

erci

ses)

Cou

rse

with

Inte

rnet

disc

ussi

ongr

oups

but

novi

rtua

lclin

icD

,PC

(con

f)S,

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Sche

rly,2

0003

029

;MS-

P,RN

-PV

irolo

gyV

irola

b(t

utor

ial/s

imul

atio

n);l

earn

ing

thro

ugh

expe

rimen

tatio

n(in

duct

ive)

Hyp

erte

xttu

toria

l;in

form

atio

npr

esen

ted

tole

arne

rG

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Bear

man

,200

131

255;

MS-

PC

omm

unic

atio

nsk

ills

Virt

ualp

atie

nt:p

robl

emso

lvin

g(d

iagn

osis

-orie

nted

)fo

rmat

Virt

ualp

atie

nt:n

arra

tive

form

at(a

dapt

ive)

AN

Sk

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Fox,

2001

32

152;

MD

-PC

hang

em

anag

emen

tIn

tern

ettu

toria

lwith

disc

ussi

onfo

rum

Inte

rnet

tuto

rialw

ithou

tdi

scus

sion

foru

mD

—...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.D

uffy

,200

233

154;

O-T

Teac

hing

skill

sD

ista

nce

lear

ning

with

Inte

rnet

-med

iate

ddi

scus

sion

and

face

-to-

face

sess

ions

Dis

tanc

ele

arni

ngw

ithIn

tern

et-m

edia

ted

disc

ussi

onbu

tno

face

-to-

face

sess

ions

BK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Jedl

icka

,200

234

22;O

-TM

enta

lhea

lthpr

ogra

mm

ing

Inte

ract

ive

conf

eren

ce(v

ideo

and

audi

o)C

hat

room

(tex

t)co

nfer

ence

AD

S

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Car

pent

er,2

0033

528

;MS-

C,M

D-G

,RN

-T,R

N-P

,D-T

,O

-P

Mot

ivat

iona

lin

terv

iew

ing

for

toba

cco

cess

atio

n

Inte

rnet

-bas

edtu

toria

lwith

clin

ical

scen

ario

s,qu

estio

ns,f

eedb

ack

Onl

ine

prac

tice

guid

elin

eI,

F(c

onf)

,PC

(con

f),S

IK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Cha

o,20

0336

34;M

S-P

Dia

gnos

isof

mel

anom

aO

nlin

ele

arni

ngsy

stem

Link

toin

form

atio

npa

ges

onth

eIn

tern

etSI

K...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.Fr

ith,2

0033

717

4;RN

-TEC

Gin

terp

reta

tion

Inte

rnet

-bas

edtu

toria

land

onlin

eco

mm

unic

atio

nIn

tern

et-b

ased

tuto

rialo

nly

DS,

K...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.Le

mai

re,2

0033

839

4;RN

-P,O

-P,

Am

big

Phys

ical

reha

bilit

atio

nIn

tern

et-b

ased

mod

ule

CD

-RO

Mm

odul

eC

oS

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Lem

aire

,200

338

(Com

paris

on2)

Inte

rnet

-bas

edm

odul

eC

onfe

renc

e(o

nlin

edi

scus

sion

)via

Inte

rnet

DS

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Maa

g,20

043

996

;RN

-TM

ath

skill

sM

ultim

edia

Inte

rnet

-bas

edm

odul

esw

ithca

se-b

ased

revi

ewqu

estio

nsan

dan

swer

sM

ultim

edia

,Int

erne

t-ba

sed

mod

ules

with

out

ques

tions

and

answ

ers

I,F

(con

f)S,

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Mat

theo

s,20

044

052

;D-T

Perio

dont

olog

yIn

tern

et-b

ased

mod

ules

with

case

-bas

edes

say

ques

tions

with

requ

ired

resp

onse

Inte

rnet

-bas

edm

odul

esw

ithsa

me

case

sbu

top

tiona

llea

rner

resp

onse

I,F

(con

f)S,

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Spic

kard

,200

441

59;M

S-C

Scre

enin

gpr

inci

ples

Inte

rnet

-bas

edle

ctur

ew

ithau

dio

Inte

rnet

-bas

edle

ctur

ew

ithou

tau

dio

AT

S,K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Virv

ou,2

0044

250

;MS-

PG

ener

alm

edic

alkn

owle

dge

Inte

rnet

educ

atio

nvi

am

obile

phon

eD

eskt

opve

rsio

nof

mod

ule

Co

S

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Alli

son,

2005

43

209;

MD

-PC

hlam

ydia

scre

enin

gM

ultic

ompo

nent

,Int

erne

t-ba

sed

mod

ule

with

feed

back

oncl

inic

alpr

actic

eTe

xt,I

nter

net-

base

dm

odul

eI,

F(c

onf)

B

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Beck

er,2

0054

415

3;O

-TIn

terd

isci

plin

ary

team

wor

kW

ebco

urse

with

extr

asu

ppor

tW

ebco

urse

with

out

extr

asu

ppor

tC

o,PP

S,K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Brun

etau

d,20

054

511

;MS-

CIn

flam

mat

ion

Virt

ualc

ampu

se-

lear

ning

man

agem

ent

syst

emW

ebse

rver

(no

form

alm

anag

emen

tsy

stem

)C

o,PP

S

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Muk

ohar

a,20

054

610

7;M

D-P

Gen

eral

inte

rnal

med

icin

eJo

urna

lart

icle

sum

mar

ies

via

e-m

ail

E-m

aill

ink

toon

line

reso

urce

SIS,

K,S

k,B

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Schi

ttek

Jand

a,20

054

728

;D-T

Han

dw

ashi

ngIn

tern

et-b

ased

mod

ule

cons

istin

gof

segm

ente

dvi

deo

clip

sIn

tern

et-b

ased

mod

ule

cons

istin

gof

one

cont

inuo

usvi

deo

clip

SpS,

K,S

k

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Blac

kmor

e,20

064

816

7;O

-TPs

ycho

ther

apy

Inte

rnet

-bas

edm

odul

ew

ithen

hanc

edon

line

colla

bora

tion

Inte

rnet

-bas

edm

odul

ew

ithm

inim

alon

line

colla

bora

tion

D,F

(con

f),

PPS

(Con

tinue

s)



Tabl

e2

(Con

tinue

d)

Firs

tau

tho

r,yearR

ef

Part

icip

an

ts†

Med

ical

top

icStu

dy

inte

rven

tio

nC

om

pari

son

inte

rven

tio

nStu

dy

them

e‡

Ou

tco

mes§

Coo

k,20

0649

121;

MD

-GA

mbu

lato

rym

edic

ine

Inte

rnet

-bas

edm

odul

esw

ithqu

estio

nsan

dfe

edba

ckIn

tern

et-b

ased

mod

ules

with

out

ques

tions

orfe

edba

ckI,

F(c

onf)

S,K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Frie

dl,2

0065

019

5;M

S-C

Car

diov

ascu

lar

surg

ery

Inte

rnet

-bas

edm

odul

ew

ithca

se-b

ased

ques

tions

and

inte

ract

ive

task

sN

onin

tera

ctiv

e,In

tern

et-b

ased

mod

ule

G,I

,PC

(con

f)S,

K,B

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Kem

per,

2006

51

1267

;MD

-P,M

D-O

,RN

-P,P

-P,O

-T,O

-PD

ieta

rysu

pple

men

tsIn

tern

et-b

ased

mod

ule:

info

rmat

ion

spac

edov

ertim

eIn

tern

et-b

ased

mod

ule:

alli

nfor

mat

ion

aton

ceSp

S,K

,B

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Kem

per,

2006

51

(Fac

toria

l2)

Info

rmat

ion

embe

dded

ine-

mai

l(“p

ush”

)E-

mai

llin

kto

info

rmat

ion

onIn

tern

et(“

pull”

)E

S,K

,B

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Ker

foot

,200

652

693;

MS-

C,M

D-G

Syst

ems-

base

dpr

actic

eIn

tera

ctiv

e,In

tern

et-b

ased

mod

ule

Non

inte

ract

ive,

Inte

rnet

-bas

edm

odul

eI

K...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.Li

ttle

,200

653

33;R

N-T

Not

spec

ified

Inte

rnet

cour

sew

ithvo

ice-

over

-Inte

rnet

disc

ussi

onIn

tern

etco

urse

with

out

voic

e-ov

er-In

tern

etD

S...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.N

icho

lson

,200

654

61;M

S-P

Ear

anat

omy

Inte

rnet

-bas

edm

odul

esw

ithth

ree-

dim

ensi

onal

mod

elIn

tern

et-b

ased

mod

ules

with

out

thre

e-di

men

sion

alvi

ews

MK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Rom

anov

,200

655

93;M

S-P

Med

ical

info

rmat

ics

Web

CT-

base

dIn

tern

et-b

ased

mod

ules

with

onlin

epe

erdi

scus

sion

and

self-

test

Iden

tical

,Int

erne

t-ba

sed

mod

ules

with

out

disc

ussi

onor

self-

test

Co

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Thom

pson

,20

065

664

;MS-

CC

ardi

olog

yan

dbi

oter

roris

mIn

tern

et-b

ased

mod

ule

with

case

sum

mar

yIn

tern

et-b

ased

mod

ule

with

out

sum

mar

yI

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Al-R

awi,

2007

57

26;D

-TM

axill

ofac

ialC

Tin

terp

reta

tion

Inte

ract

ive

mod

ule

Stat

icPD

Fof

sim

ilar

cont

ent

IK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Bedn

ar,2

0075

821

;D-T

Ort

hodo

ntic

basi

cco

ncep

tsIn

tern

et-m

edia

ted

vide

ocon

fere

nce

(rea

l-tim

ein

tera

ctio

n)In

tern

et-m

edia

ted

vide

ocon

fere

nce

(del

ayed

inte

ract

ion)

TS,

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Coo

k,20

0759

112;

MS-

C,M

D-G

Com

plem

enta

rym

edic

ine

Inte

rnet

-bas

edm

odul

e;le

arne

rsco

nstr

uct

revi

ewta

ble

Inte

rnet

-bas

edm

odul

e;re

view

tabl

epr

ovid

edI

S,K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Coo

k,20

0759

(Fac

toria

l2)

Inte

rnet

-bas

edm

odul

e;ac

tive

ques

tions

Inte

rnet

-bas

edm

odul

e;re

flect

ive

ques

tions

AR,

F(c

onf)

S,K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Heg

e,20

076

147

5;M

S-C

Inte

rnal

med

icin

eW

eb-b

ased

case

s;m

anda

tory

,lin

ked

toex

amW

eb-b

ased

case

s;vo

lunt

ary,

not

linke

dto

exam

Co,

PPS

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Ker

foot

,200

762

156;

MS-

CU

rolo

gyIn

tern

et-b

ased

mod

ule

with

spac

edre

tent

ion

ques

tions

Inte

rnet

-bas

edm

odul

ew

ithou

tre

tent

ion

ques

tions

IS,

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Ker

foot

,200

763

537;

MD

-GU

rolo

gySp

aced

/rep

eate

dse

lf-as

sess

men

tqu

estio

nsvi

ae-

mai

lSe

lf-as

sess

men

tqu

estio

nsal

lat

once

via

e-m

ail

RK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Mill

er,2

0076

426

;D-T

Ort

hodo

ntic

sV

ideo

conf

eren

cew

ithin

tera

ctiv

evi

deo

disc

ussi

onV

ideo

conf

eren

cew

ithIn

tern

etch

atdi

scus

sion

AD

S...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.M

orid

ani,

2007

65

151;

P-T

Phar

mac

ogen

etic

sA

sync

hron

ous

vide

ole

ctur

esSy

nchr

onou

svi

deoc

onfe

renc

ing

SAK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Ridg

way

,200

766

88;M

S-C

Surg

ery

Inte

rnet

slid

esho

ww

ithvo

ice

Inte

rnet

slid

esho

ww

ithou

tvo

ice

AT

S,K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Rom

anov

,200

767

122;

MS-

PM

edic

alin

form

atic

sBl

ende

dle

arni

ngw

ithvi

deo

clip

sBl

ende

dle

arni

ngw

ithou

tvi

deo

clip

sV

K...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.Be

rger

on,2

0086

889

;Am

big

Radi

atio

nha

zard

med

ical

trai

ning

Serio

usga

me

(inte

ract

ive

gam

ew

ithin

tent

ofte

achi

ngor

enco

urag

ing

appl

icat

ion

ofkn

owle

dge)

Trad

ition

alon

line

(did

actic

)I,

GK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Cam

pbel

l,20

086

911

4;A

mbi

gRe

sear

chm

etho

dsO

nlin

em

odul

esw

ithon

line

disc

ussi

onO

nlin

em

odul

esw

ithfa

ce-t

o-fa

cedi

scus

sion

BK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Coo

k,20

0870

124;

MD

-GA

mbu

lato

rym

edic

ine

Inte

ract

ive

Web

mod

ules

with

addi

tiona

lcom

plex

patie

ntsc

enar

ios

Sam

eW

ebm

odul

esw

ithou

tpa

tient

scen

ario

sI,

F(c

onf)

,PC

(con

f)S,

K

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Coo

k,20

0871

124;

MD

-GA

mbu

lato

rym

edic

ine

Ada

ptiv

eW

ebm

odul

esSt

anda

rdW

ebm

odul

esA

NS,

K (Con

tinue

s)



one of three (33%) skills assessments, andtwo of four (50%) assessments ofbehavior or patient effects used self-report measures. Two studies comparedcourse completion rates as a measure ofsatisfaction51,61; all other studies usedself-reported satisfaction. Fifteen (52%)of the 29 studies assessing satisfaction, 12(32%) of the 38 studies assessingknowledge, 1 (33%) of the 3 studiesassessing skills, and 1 (25%) of the 4studies assessing behaviors and patienteffects lost more than 25% of participantsfrom the time of enrollment, or theyfailed to report follow-up. Quality scores(the number of quality criteria present; 6points maximum) ranged from 0 to 6,with a mean of 3.3 (SD � 1.6).

Research themes

We identified 22 distinct research themes(Table 1). Figure 2 summarizes meta-analysis results for themes addressed by twoor more studies, as well as for studies inwhich the coded level of interactivity orpractice exercises varied between groups.The Supplemental Digital Content(http://links.lww.com/ACADMED/A14,Table 2) has additional analysis detailsincluding subgroup analyses and study-specific ESs. Further details of analysesare available from the authors on request.To illustrate how studies varied while stillfocusing on the same theme, we presentin the Supplemental Digital Content(http://links.lww.com/ACADMED/A14,Box 2) an in-depth examination of thetheories, conceptual frameworks, andinstructional methods for several studiesreflecting one theme.

We did not identify research themes forall coded elements. For example,although we found differences in thecoded quantity of practice exercises, wefound no studies designed to comparesuch differences. Similarly, we foundfewer studies hypothesizing differencesbetween levels of interactivity (n � 15)than we found with differences in ourcoding of this feature (n � 21).

Studies investigating interactivity

Fifteen studies compared different levels ofinteractivity,35,39,40,43,49,50,52,56,57,59,62,68,70,73,74

defined as cognitive (mental) engagementwith the course other than onlinediscussion (which we analyzed separately,see below). Course designers enhancedengagement using a range of tasks, asoutlined below (See also Table 2,Themes).Ta

ble

2(C

ontin

ued)

Firs

tau

tho

r,yearR

ef

Part

icip

an

ts†

Med

ical

top

icStu

dy

inte

rven

tio

nC

om

pari

son

inte

rven

tio

nStu

dy

them

e‡

Ou

tco

mes§

Ker

foot

,200

872

237;

MS-

CU

rolo

gySp

aced

/rep

eate

ded

ucat

ion

Web

trai

ning

mod

ules

RS,

K...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.K

opp,

2008

74

153;

MS-

CSe

cond

ary

hype

rten

sion

Web

mod

ule:

elab

orat

edfe

edba

ckW

ebm

odul

e:br

ief

feed

back

FK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Kop

p,20

0874

(Fac

toria

l2)

Web

mod

ule:

inte

ntio

nalr

easo

ning

erro

rsW

ebm

odul

e:go

odcl

inic

alre

ason

ing

IK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Tunu

gunt

la,

2008

75

156;

MS-

PH

ome

safe

tyM

odul

ew

ithan

imat

ions

Mod

ule

with

stat

icgr

aphi

csA

SK

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

....

Wal

ker,

2008

76

120;

O-P

Inju

rypr

even

tion

Web

-bas

edfo

rmat

CD

-RO

Mfo

rmat

Co

S...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.C

ook,

2009

60**

123;

MD

-GA

mbu

lato

rym

edic

ine

Inte

rnet

-bas

edm

odul

e:ca

sefir

stIn

tern

et-b

ased

mod

ule:

dida

ctic

sfir

stC

aS,

K...

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

......

.K

erfo

ot,2

0097

3*

*11

5;M

S-C

Uro

logy

Web

trai

ning

with

spac

ed/r

epea

ted

educ

atio

nW

ebtr

aini

ngI

K

*A

nex

tend

edve

rsio

nof

this

tabl

e,w

ithad

ditio

nali

nfor

mat

ion

onco

urse

leng

than

dou

tcom

es,i

sav

aila

ble

ason

line-

only

supp

lem

enta

l,di

gita

lmat

eria

l(ht

tp://

links

.lww

.com

/AC

AD

MED

/A14

,Tab

le2)

.†

Ain

dica

tes

ambi

guou

s;D

-T,d

entis

tin

trai

ning

;MD

-G,p

hysi

cian

inpo

stgr

adua

tetr

aini

ng;M

D-P

,pra

ctic

ing

phys

icia

n;M

S-C

,clin

ical

med

ical

stud

ent;

MS-

P,pr

eclin

ical

med

ical

stud

ent;

O-P

,oth

erin

prac

tice;

O-T

,oth

erin

trai

ning

;P-P

,pha

rmac

ist

inpr

actic

e;P-

T,ph

arm

acis

tin

trai

ning

;RN

-P,n

urse

inpr

actic

e;RN

-T,n

urse

intr

aini

ng.

‡St

udy

them

es(s

eeTa

ble

1fo

rde

tails

):A

Din

dica

tes

audi

ove

rsus

text

(cha

t)on

line

disc

ussi

on;A

N,a

dapt

ive

navi

gatio

nve

rsus

nona

dapt

ive

pres

enta

tion;

AR,

activ

eve

rsus

refle

ctiv

equ

estio

ns;A

S,an

imat

ion

vers

usst

atic

imag

es;A

T,au

dio

pres

ent/

abse

ntin

atu

toria

l;B,

blen

ded

onlin

e�

face

-to-

face

vers

uson

line-

only

;Ca,

case

-firs

tve

rsus

dida

ctic

-firs

t;C

o,co

nfig

urat

ion

com

paris

on;C

onf,

conf

ound

edco

mpa

rison

(e.g

.,a

com

paris

onof

ques

tions

and

feed

back

vers

usno

ques

tions

/no

feed

back

);D

,dis

cuss

ion

vers

usno

disc

ussi

on;E

,em

bedd

ede-

mai

ltex

tve

rsus

link

tote

xt;F

,fee

dbac

kve

rsus

nofe

edba

ck;G

,gam

e/si

mul

atio

nve

rsus

sequ

entia

lins

truc

tion;

I,in

tera

ctiv

ityhi

ghve

rsus

low

;M,t

hree

-dim

ensi

onal

mod

el;P

C,p

atie

ntca

ses

vers

usno

case

s;PP

,enh

ance

dto

prom

ote

part

icip

atio

nve

rsus

unen

hanc

ed;R

,rep

etiti

onof

mat

eria

lver

sus

nore

petit

ion;

SA,s

ynch

rono

usve

rsus

asyn

chro

nous

;SI,

synt

hesi

zed

info

rmat

ion

vers

usex

istin

gin

form

atio

n;Sp

,spa

cing

ofm

ater

ialv

ersu

sno

spac

ing;

T,tim

ing

ofdi

scus

sion

live

vers

usde

laye

d;V

,vid

eocl

ips

vers

usno

vide

ocl

ips.

§S

indi

cate

ssa

tisfa

ctio

n;K

,kno

wle

dge;

Sk,s

kills

;B,b

ehav

ior.

**Th

ere

fere

nces

onth

eta

ble

are

sort

edin

orde

rof

year

,the

nau

thor

nam

e.Th

ela

sttw

ow

ere

publ

ishe

don

line

in20

07an

d20

08,r

espe

ctiv

ely,

but

publ

ishe

din

prin

ted

ition

sin

2009

.



Table 3Quality of Studies Included in a Systematic Review of Internet-Based Instruction(N � 51)*

First author,yearRef

Representativeinterventiongroup

Comparison groupselected from samecommunity Comparable cohorts

Blindedoutcomeassessment†

Follow-upadequate†

Kaplan, 199626

...................................................................................................................................................................................................................................................................................................................Chan, 199927 Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Papa, 199928 Yes Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Schaad, 199929 Yes Yes Yes...................................................................................................................................................................................................................................................................................................................Scherly, 200030 Yes...................................................................................................................................................................................................................................................................................................................Bearman, 200131 Yes Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Fox, 200132 Yes Randomized...................................................................................................................................................................................................................................................................................................................Duffy, 200233 Yes Yes...................................................................................................................................................................................................................................................................................................................Jedlicka, 200234 Yes Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Carpenter, 200335 Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Chao, 200336 Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Frith, 200337 Yes Randomized, allocation

concealedYes

...................................................................................................................................................................................................................................................................................................................Lemaire, 200338 Yes Yes Yes...................................................................................................................................................................................................................................................................................................................Maag, 200439 Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Mattheos, 200440 Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Spickard, 200441 Yes Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Virvou, 200442 Yes...................................................................................................................................................................................................................................................................................................................Allison, 200543 Yes Randomized, allocation

concealedYes Yes

...................................................................................................................................................................................................................................................................................................................Becker, 200544 Yes...................................................................................................................................................................................................................................................................................................................Brunetaud, 200545 Yes Yes...................................................................................................................................................................................................................................................................................................................Mukohara, 200546 Yes Randomized, allocation

concealedYes

...................................................................................................................................................................................................................................................................................................................Schittek Janda,200547

Yes Yes Randomized Yes

...................................................................................................................................................................................................................................................................................................................Blackmore, 200648

...................................................................................................................................................................................................................................................................................................................Cook, 200649 Yes Yes Randomized, allocation

concealedYes Yes

...................................................................................................................................................................................................................................................................................................................Friedl, 200650 Yes Yes...................................................................................................................................................................................................................................................................................................................Kemper, 200651 Yes Randomized, allocation

concealedYes

...................................................................................................................................................................................................................................................................................................................Kerfoot, 200652 Yes Yes...................................................................................................................................................................................................................................................................................................................Little, 200653

...................................................................................................................................................................................................................................................................................................................Nicholson, 200654 Yes Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Romanov, 200655 Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Thompson, 200656 Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Al-Rawi, 200757 Yes Randomized Yes...................................................................................................................................................................................................................................................................................................................Bednar, 200758 Yes Controlled for learning

outcomeYes

...................................................................................................................................................................................................................................................................................................................Cook, 200759 Yes Randomized, allocation

concealedYes Yes

...................................................................................................................................................................................................................................................................................................................Hege, 200761 Yes Yes...................................................................................................................................................................................................................................................................................................................Kerfoot, 200762 Yes Yes Randomized, allocation

concealedYes

...................................................................................................................................................................................................................................................................................................................Kerfoot, 200763 Yes Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Miller, 200764 Yes...................................................................................................................................................................................................................................................................................................................Moridani, 200765 Yes Yes Yes...................................................................................................................................................................................................................................................................................................................Ridgway, 200766 Yes Yes

(Continues)



Eight of these studies explored the use ofquestions to enhance interactivity. Twoof three randomized trials that comparedthe use of self-assessment questionsversus no questions reported statisticallysignificantly higher knowledge test scoresfor modules with questions,49,62 while thethird reported no significant difference.39

Another randomized trial reportedsimilar outcomes whether or not learnerswere required to actively respond to thequestion.40 A fifth randomized trialcompared IBL modules with case-basedquestions and tailored feedback againstan online practice guideline withoutquestions and found significantimprovement in knowledge test scoresfor interactive IBL.35 Adding extraquestions before73 or after70 an Internet-based module that already containedseveral interactive questions did notsignificantly alter outcomes in tworandomized trials. Finally, anonrandomized study added case-basedquestions and learning tasks to anInternet-based course consisting of text,video clips, and interactive models andfound a nonsignificant associationbetween lower interactivity (noquestions) and higher knowledge testscores, but similar satisfaction ratings.50

Two randomized trials evaluated theeffect of actively summarizing

information. Creating a summary of apatient scenario improved knowledge testscores compared with not creating asummary.56 However, creating asummary of a tutorial’s didacticinformation did not improve knowledgetest scores in comparison with reviewingan instructor-prepared summary, andpreference was neutral.59

A small randomized trial found a modest(ES 0.57) but nonsignificant benefit ofInternet-based modules with self-evaluation, animations, and video incomparison with static PDFs of the sameinformation.57 Studying Internet-based,worked-example cases with intentionalerrors led to significantly improvedlearning outcomes (knowledge, skills, orbehaviors and patient effects) comparedwith cases without errors in anotherrandomized trial.74

Finally, three studies evaluated Internet-based tutorials with varying levels ofinteractivity, but the differences werepoorly defined. One randomized trialcompared a multicomponent modulewith tailored feedback on clinical practiceperformance against “flat-text Internet-based CME [continuing medicaleducation] modules,”43 while a crossoverstudy compared “interactive Web-basedmodules” with “noninteractive narrated

slide presentations.”52 Both foundsignificant improvements in learningoutcomes for the interactive group. Athird study with ambiguous design founda significant association betweenknowledge scores and use of aninteractive IBL game versusnoninteractive “traditional” computer-assisted instruction.68

For the 15 studies reporting learningoutcomes (knowledge, skills, or behaviorsand patient effects), the pooled ESfavoring interactivity was 0.27 (95%confidence interval [CI], 0.08 to 0.46;P � .006), I2 � 90%. Althoughstatistically significantly different from 0,this is considered a small effect.77 Sevenstudies investigating interactivityreported satisfaction outcomes, with apooled ES of 0.39 (95% CI, �0.12 to0.90; P � .13), I2 � 95%. Subgroupanalyses showed similar findings forhigh- and low-quality studies(Supplemental Digital Content, Table 2,http://links.lww.com/ACADMED/A14).

A second meta-analysis pooled data from21 studies in which the coded level ofinteractivity varied between studyarms.28 –30,33,35–37,39,40,47,49,50,52,54 –58,62,68,74

In this coding, learner tasks such aspractice exercises, information syntheses,essay assignments, and group

Table 3(Continued)

First author,yearRef

Representativeinterventiongroup

Comparison groupselected from samecommunity Comparable cohorts

Blindedoutcomeassessment†

Follow-upadequate†

Romanov, 200767 Yes Yes...................................................................................................................................................................................................................................................................................................................Bergeron, 200868 Yes Yes...................................................................................................................................................................................................................................................................................................................Campbell, 200869 Yes Yes Controlled for other Yes...................................................................................................................................................................................................................................................................................................................Cook, 200870 Yes Randomized, allocation

concealedYes

...................................................................................................................................................................................................................................................................................................................Cook, 200871 Yes Yes Randomized, allocation

concealedYes Yes

...................................................................................................................................................................................................................................................................................................................Kerfoot, 200872 Yes Yes Randomized Yes Yes...................................................................................................................................................................................................................................................................................................................Kopp, 200874 Yes Randomized Yes...................................................................................................................................................................................................................................................................................................................Tunuguntla, 200875 Yes Randomized Yes...................................................................................................................................................................................................................................................................................................................Walker, 200876 Yes Controlled for learning

outcome...................................................................................................................................................................................................................................................................................................................Cook, 200960‡ Yes Yes Randomized, allocation

concealedYes Yes

...................................................................................................................................................................................................................................................................................................................Kerfoot, 200973‡ Yes Randomized Yes Yes

* Quality was assessed using a modification of the Newcastle–Ottawa Scale.4,18 Each study could receive up to sixpoints (maximum two points for comparable cohorts; one point for other criteria). Blank cells indicate thisquality feature was missing (all cells for Kaplan, Blackmore, and Little are intentionally blank).

† Blinding and completeness of follow-up are reported as “yes” if this was true for any reported outcome.‡ The references on the table are sorted in order of year, then author name. The last two were published online in

2007 and 2008, respectively, but published in print editions in 2009.



collaborative projects supported higherinteractivity levels. In this analysis, thepooled ES for learning was 0.53 (95% CI,0.33 to 0.73; P � .001), I2 � 89%. This isconsidered a medium-sized effect.77

Twelve of these studies evaluatedsatisfaction,29,34,37,39,40,47–50,53,58,62 with apooled ES of 0.31 (95% CI, �0.13 to0.74; P � .17), I2 � 92%.

Studies investigating practice exercises

Although not identified as a separateresearch theme, the study protocol specifiedcoding the quantity of practice exercises,and this rating varied between study armsfor 10 studies.27,29,35,39,43,49,50,55,57,62 Meta-analysis revealed a pooled ES of 0.40 (95%CI, 0.08 to 0.71; P � .01), I2 � 92%.Subgroup analyses demonstrated similarfindings for high- and low-quality studies(see Supplemental Digital Content, Table 2,http://links.lww.com/ACADMED/A14).For the five studies reporting satisfaction

outcomes, the pooled ES was 0.59 (95% CI,�0.10 to 1.27; P � .094), I2 � 95%.However, when including only randomizedtrials, this ES was statistically significantlydifferent from 0 (P � .005; seeSupplemental Digital Content, Table 2,http://links.lww.com/ACADMED/A14).

Studies investigating online discussion

Seven studies evaluated the impact ofonline discussion.27,29,32,37,38,48,53 Five ofthese used written asynchronous text-based discussion (e.g., discussion boardsor e-mail). Three randomized trialscompared Internet-based tutorials withand without online discussion.27,32,37

None of these three studies demonstrateda statistically significant effect on learningoutcomes, although students in one studynoted significantly higher satisfactionwith the online discussion format.37 Afourth study added a “virtual clinic” (inwhich students discussed patient cases

online) to an existing IBL activity. Acomparison with historical controlsobserved no association with courseratings but a significant association withhigher test scores.29 Finally, one studyaltered a course to promote greateronline collaboration and found greatersatisfaction among trainees comparedwith historical controls using a previouscourse in which online discussion wasless prominent.48

Two observational studies evaluated theaddition of live audio discussion toInternet-based courses withoutdiscussion. One used historicalcontrols,53 whereas the other allowedparticipants to self-select the presentationformat.38 Both studies found anassociation between live audio discussionand higher satisfaction.

For the three studies of online discussionto report learning outcomes, the pooledES was 0.26 (95% CI, �0.62 to 1.13; P �.57), I2 � 93%. In subgroup analyses, thepooled ES varied by study design (Pinteraction

�.001), with lower pooled ES for the tworandomized trials (�0.14) than for the lonenonrandomized study (1.01; seeSupplemental Digital Content, Table 2,http://links.lww.com/ACADMED/A14, fordetails). For the five studies reportingsatisfaction outcomes, the pooled ES was0.32 (95% CI, 0.14 to 0.51; P � .001) withI2 � 4% and similar effects for qualitysubgroups.

Studies investigating feedback

Two randomized studies28,74 examinedthe effects of feedback. One studycompared feedback with no feedback,28

whereas the other compared detailedfeedback versus providing only thecorrect answer.74 The results of bothstudies favored the more intensivefeedback option, with a pooled learningES of 0.68 (95% CI, 0.01–1.35; P � .047).

A number of other studies used feedbackas part of the instructional design, mostoften in conjunction with self-assessmentquestions35,39,40,49,59,70 but also regardingclinical practice performance43 andexplicit feedback from peers.48

Unfortunately, simultaneous changes inother instructional design features(confounding) precluded isolating theeffect of feedback in these studies.

Figure 2 Random-effects meta-analysis of different Internet-based instructional designs. All studygroupings reflect inductively identified research themes (Table 1), except those marked as“coded” which include studies for which the coded level of that feature (interactivity or practiceexercises) varied between groups; see text for details. Boxes represent the standardized effect size(Hedges’ g), and bars represent the 95% confidence interval (95% CI). The vertical line representsno effect. The pooled estimate is calculated using a random-effects model.

* I2 undefined for analyses with two studies.



Studies investigating repetition andspacing

Several studies addressed the theme ofspacing (spreading a fixed amount ofinstruction over time) and repetition(repeating the same instructionalactivities multiple times). For spacing,one randomized trial comparedspreading 40 short IBL modules across 10weeks using e-mail, versus having all 40modules available in the first week.51 Thesmall benefits (ES 0.12 for bothknowledge test scores and coursecompletion rates) were not statisticallysignificant. Another randomized studycompared a video clip divided into eightsmall segments against the intact clip andfound statistically significantimprovements in knowledge test scores(ES 0.88) for the segmented (spaced)format, but there was no difference insatisfaction or skill ratings.47 We did notpool these results because theinterventions varied substantially.

Considering repetition, two randomizedtrials compared delivering studyquestions and answers via e-mail overseveral weeks and repeating eachquestion once72 or twice,63 againstnonspaced/repeated instruction (either aWeb-based module on the same topic,72

or the same questions/answers deliveredtogether on day 163). The pooled learningES was 0.19 (95% CI, 0.09–0.30; P � .001).

Studies investigating strategies topromote learner participation

Four observational studies evaluated waysto enhance learner participation inInternet-based courses, includingproviding printed course guides andenhanced technical support to courseparticipants,44 using a Web server insteadof a more complicated learningmanagement system,45 modifying thescreen appearance to promoteparticipation in online forums,48 andmaking the course a more integral part ofthe curriculum.61 All of theseinterventions were associated withimproved outcomes; however, because ofthe heterogeneity in approaches, we didnot pool these results.

Studies investigating audio

Two studies explored the use of audio inInternet-based tutorials. One randomizedstudy compared Internet-accessiblePowerPoint presentations using writtentext or an audio voice-over.41 Although

knowledge test scores did not differsignificantly between groups, the audiogroup reported significantly highersatisfaction. However, the audio formatalso took significantly longer to complete.The other study used a single-groupcrossover design in which half theInternet-accessible PowerPointpresentations had supplemental audioinformation that reinforced the text andencouraged further study. Thisintervention was associated withstatistically greater knowledge test scoresand satisfaction.66 Pooled analyses ofthese two studies revealed ESs favoringaudio of 1.26 (95% CI, �0.36 to 2.88;P � .13) for learning and 0.76 (95% CI �0.50 to 1.02; P � .001) for satisfaction.

Another two studies explored the use ofaudio in Internet-based communication.These crossover studies34,64 foundstatistically significantly greater preferencefor Internet-mediated videoconferencesover synchronous, online, text-based chatsessions, with a pooled ES of 1.15 (95% CI,0.15–2.15; P � .02).

Studies investigating instructor-synthesized information

Three randomized trials comparedinstructor-synthesized informationagainst existing information available onthe Internet. One study found asignificant benefit on knowledge testscores from an instructor-synthesizedseries of dermatologic images and brieftext.36 Another study found mixedresults, with no statistically significantdifferences for knowledge, skill, orbehavior scores, but a large, positiveeffect on satisfaction for the instructor-synthesized material.46 The third study,comparing IBL modules against an onlinepractice guideline, has already beendiscussed under Interactivity.35 Thepooled learning ES was 1.09 (95% CI,�0.20 to 2.39; P � .10), I2 � 96%.

Studies investigating games andsimulation

Three observational studies30,50,68

compared learning inductively fromgames or simulations versus learningfrom sequentially presented didacticmaterial, with results favoring didactic,50

favoring games,68 or showing nodifference.30 The pooled ES for learningoutcomes was 0.07 (95% CI, �0.55 to0.68; P � .83), I2 � 95%.

Studies investigating adaptivenavigation

Two randomized trials evaluatedInternet-based interventions that adaptedaccording to learner responses. Onecompared a “narrative” (adaptive) virtualpatient with a “problem-focused” virtualpatient31 and found evidence suggestingimproved communication skills for theadaptive format. The other71 testedlearners’ knowledge before presentinginformation and allowed learners to skipmodule sections if they answeredcorrectly. Although knowledge test scoresdid not significantly differ, the adaptiveformat required significantly less time.Because of the heterogeneity in researchthemes of these studies, we did not poolthese results.

Studies investigating blended learning

Two observational studies comparedInternet-only versus blended Internet/face-to-face courses. In one,33 thecombination of Internet-mediated andface-to-face discussion was associatedwith somewhat lower learning outcomesthan an Internet-only approach, butmultiple variables including culturaldifferences and language barriers betweenthe learner groups could havecontributed to this finding. In anotherstudy, students in an Internet-basedcourse could self-select Internet-based orface-to-face discussion groups.69 Thosechoosing Internet discussion had highercourse grades than those selectingface-to-face.

Other research themes

Eight studies26,38,42,44,45,55,61,76 compareddifferent computer-based learningconfigurations, and a number ofthemes were addressed by one studyeach. These are summarized in theSupplemental Digital Content, Box 3,http://links.lww.com/ACADMED/A14.

Discussion

This systematic review identified amodest number of studies investigatinghow to improve IBL by comparing oneInternet-based intervention with anothercomputer-based intervention. Thesestudies collectively explored a relativelylarge number of research themes.However, in most cases only a smallnumber of studies had investigated a givenresearch theme. Moreover, the operationaldefinitions of the interventions (and the



differences between interventions) variedwidely from study to study even within agiven theme.

Pooled ESs for satisfaction and/orlearning outcomes (knowledge, skills, orbehaviors and patient effects) werepositive but small77 for associations withnearly all of the themes identified.However, the pooled estimates forsatisfaction differed significantly from 0only for associations with interactivity,online discussion, and use of audio forboth tutorials and online discussion,whereas estimates for learning differedsignificantly only for associations withinteractivity, practice exercises, feedback,and repetition. Inconsistency(heterogeneity) between studies was large(�89%) for all but online discussion andsatisfaction. These inconsistencies allowus to draw only weak inferences.

Limitations and strengths

Our review has several limitations. First,only a few studies investigated any givenresearch theme, precluding quantitativesynthesis of results in many instances.Second, even within a given theme, theconceptual definitions (e.g., whatconstitutes “interactivity”?), study andcomparison interventions, outcomes, andresearch methods varied. We emphasizedsimilarities when grouping studies, butwe acknowledge that importantdifferences may explain much of theobserved inconsistency among studies.78

The small number of studies precludedsubgroup analyses to explore thisheterogeneity. We did not use funnelplots to assess for publication biasbecause these are misleading in thepresence of large heterogeneity.79 Third,although some studies reflected highmethodological quality, the averagequality was relatively poor. However,analyses restricted to more rigorousstudies yielded similar findings in mostinstances. Fourth, many articles failed toreport key details of the interventions oroutcome measures. We obtained additionaloutcomes data from some but not allauthors. Finally, space limitations do notpermit a detailed review of the theories andframeworks that support each of thethemes identified. However, in ourSupplemental Digital Content we illustratethis foundation for one theme (Box 2,http://links.lww.com/ACADMED/A14).

Our review also has several strengths. Thequestion of how to improve IBL is timely

and of great importance to medicaleducators. To present a comprehensivesummary of evidence, we kept our scopebroad regarding learners, interventions,outcomes, and study design. Thesystematic literature search encompassedmultiple databases supplemented byhand searches and had few exclusioncriteria. We conducted all aspects of thereview process in duplicate, withacceptable reproducibility.

Comparison with previous reviews

In comparison with our recent reviewsidentifying 130 studies comparing IBLwith no intervention and 76 studiescomparing IBL with non-Internetmethods,4 the 51 studies identified in thisreview represent a relatively small body ofevidence. This is consistent with a reviewsummarizing only the type of study,which reported that only 1% of reviewedstudies on computer-assisted instructioncompared alternate, computer-basedinterventions.80

We are not aware of previous systematicreviews of studies comparing onecomputer-assisted instructionalintervention with another in healthprofessions education, although someprevious reviews included such studiesalong with no-intervention and media-comparative studies.81,82 Outside ofmedical education, authors haveprovided nonsystematic summaries83,84

and broadly focused reviews,6,85,86 butagain we are not aware of comprehensiveand methodologically rigorous synthesesfocusing on how to improve computer-assisted instruction. However, severalreviews5,6,87 and other authors9 –13 haveissued a call for more research of thistype, suggesting that the present reviewfills an unmet need.

Implications

The synthesized evidence suggests thatinteractivity, practice exercises,repetition, and feedback improvelearning outcomes and that interactivity,online discussion, and audio improvesatisfaction in IBL for healthprofessionals. Although educators shouldconsider incorporating these featureswhen designing IBL, the strength of theserecommendations is limited: We foundrelatively few studies; existing studiesaddress a diversity of themes; even withinthemes, the interventions and outcomesvary; study findings are inconsistent; and

methodological quality is relatively low.Clear guidance for practice will requireadditional research. Insights from outsidethe health professions may also be useful.84

To strengthen the evidence base,researchers must first come to agreementon what is being studied. Sharedconceptual and theoretical frameworks,consistent definitions for interventionsand comparison interventions, and theuse of common outcome measures mayhelp. Working from shared frameworks,interventions, and outcomes will permitreplication across learner groups anddifferent educational objectives. Thepresent summary and synthesis ofevidence, along with the research themesidentified, form a foundation for suchwork, and several of the included studiesprovide exemplary models to follow.

As has been documented previously,88

few studies addressed outcomes of skills,behaviors in practice, or effects on patientcare. Such outcomes would be desirablein future research.89,90 However,investigators should ensure thatoutcomes align with interventions,91 andthey might consider demonstratingeffects on applied knowledge and skillsbefore evaluating effects on higher-orderoutcomes.92

Finally, although the evidencesummarized here begins to inform thequestion, “How should we use IBL?” itlargely fails to address the question,“When should we use IBL?” Authors haveargued that these decisions are largelypragmatic, based on relative advantagesof the Internet over other instructionaldelivery systems.2 Evidence to guide thesedecisions will derive from studies,including qualitative analyses, designedto clarify relationships between potentialadvantages and specific topics, courseobjectives, and learner characteristics.13,93

Conclusions

Although existing evidence does notpermit strong recommendations foreducational practice, this review hashighlighted promising areas for futureresearch. Evidence will derive frommultiple sources, including randomizedand observational quantitative studiesand rigorous qualitative research. Clearconceptual frameworks, focused researchquestions, well-defined interventions,study methods appropriate to the



question, and adherence to reportingstandards when disseminating results allwill help advance the science of IBL.

Acknowledgments: The authors thank MelanieLane, Mohamed Elamin, MBBS, and M. HassanMurad, MD, from the Knowledge and EncounterResearch Unit, Mayo Clinic, Rochester,Minnesota, for their assistance with dataextraction and meta-analysis planning andexecution, and Kathryn Trana, from the Divisionof General Internal Medicine, Mayo Clinic, forassistance in article acquisition and processing.

Funding/Support: This work was supported byintramural funds and a Mayo FoundationEducation Innovation award.

Other disclosures: Dr. Levinson is supported inpart through the John R. Evans Chair in HealthSciences Education Research.

Ethical approval: Because no human participantswere involved, this study did not require ethicalapproval.

Previous presentations: Select portions of theseresults were presented in symposia at the 2009meetings of MedBiquitous (April 29, 2009;Baltimore, Maryland), the Association of MedicalEducation in Europe (August 31, 2009; Malaga,Spain), and the Association of American MedicalColleges (November 9, 2009; Boston,Massachusetts).

Disclaimer: Dr. Cook had full access to all of thedata in the study and takes responsibility for theintegrity of the data and the accuracy of the dataanalysis.

References1 Ruiz JG, Mintzer MJ, Leipzig RM. The impact

of E-learning in medical education. AcadMed. 2006;81:207–212.

2 Cook DA. Web-based learning: Pros, cons,and controversies. Clin Med. 2007;7:37–42.

3 Association of American Medical CollegesInstitute for Improving Medical Education.Effective Use of Educational Technology inMedical Education: Colloquium onEducational Technology: Recommendationsand Guidelines for Medical Educators.Available at: https://services.aamc.org/publications/showfile.cfm?file�version80.pdf&prd_id�184&prv_id�224&pdf_id�80.Accessed January 12, 2010.

4 Cook DA, Levinson AJ, Garside S, DuprasDM, Erwin PJ, Montori VM. Internet-basedlearning in the health professions: A meta-analysis. JAMA. 2008;300:1181–1196.

5 Chumley-Jones HS, Dobbie A, Alford CL.Web-based learning: Sound educationalmethod or hype? A review of the evaluationliterature. Acad Med. 2002;77(10 suppl):S86 –S93.

6 Tallent-Runnels MK, Thomas JA, Lan WY,Cooper S. Teaching courses online: A review ofthe research. Rev Educ Res. 2006;76:93–135.

7 Cohen PA, Dacanay LD. Computer-basedinstruction and health professions education:A meta-analysis of outcomes. Eval HealthProf. 1992;15:259 –281.

8 Cook DA. Where are we with Web-basedlearning in medical education? Med Teach.2006;28:594 –598.

9 Clark RE. Reconsidering research on learningfrom media. Rev Educ Res. 1983;53:445–459.

10 Keane DR, Norman GR, Vickers J. Theinadequacy of recent research on computer-assisted instruction. Acad Med. 1991;66:444–448.

11 Friedman CP. The research we should bedoing. Acad Med. 1994;69:455–457.

12 Tegtmeyer K, Ibsen L, Goldstein B.Computer-assisted learning in critical care:From ENIAC to HAL. Crit Care Med. 2001;29(8 suppl):N177–N182.

13 Cook DA. The research we still are not doing:An agenda for the study of computer-basedlearning. Acad Med. 2005;80:541–548.

14 Moher D, Cook DJ, Eastwood S, Olkin I,Rennie D, Stroup DF. Improving the qualityof reports of meta-analyses of randomisedcontrolled trials: The QUOROM statement.Quality of Reporting of Meta-analyses.Lancet. 1999;354:1896 –1900.

15 Stroup DF, Berlin JA, Morton SC, et al.Meta-analysis of observational studies inepidemiology: A proposal for Reporting.Meta-analysis of Observational Studies inEpidemiology (MOOSE) Group. JAMA.2000;283:2008 –2012.

16 Kirkpatrick D. Revisiting Kirkpatrick’s four-level model. Training Dev. 1996;50:54 –59.

17 Shrout PE, Fleiss JL. Intraclass correlations:Uses in assessing rater reliability. PsycholBull. 1979;86:420 –428.

18 Wells GA, Shea B, O’Connell D, et al. TheNewcastle–Ottawa Scale (NOS) for assessingthe quality of nonrandomised studies inmeta-analyses. Available at: http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm. Accessed January 12, 2010.

19 Morris SB, DeShon RP. Combining effect sizeestimates in meta-analysis with repeatedmeasures and independent-groups designs.Psychol Methods. 2002;7:105–125.

20 Dunlap WP, Cortina JM, Vaslow JB, BurkeMJ. Meta-analysis of experiments withmatched groups or repeated measuresdesigns. Psychol Methods. 1996;1:170 –177.

21 Hunter JE, Schmidt FL. Methods of Meta-analysis: Correcting Error and Bias inResearch Findings. Thousand Oaks, Calif:Sage; 2004.

22 Higgins JPT, Green S, eds. CochraneHandbook for Systematic Reviews ofInterventions 5.0.0. Available at: http://www.cochrane.org/resources/handbook/index.htm.Accessed May 29, 2008.

23 Rosenthal R. Parametric measures of effectsize. In: Cooper HM, Hedges LV, eds. TheHandbook of Research Synthesis. New York,NY: Russell Sage Foundation; 1994:231–244.

24 Curtin F, Altman DG, Elbourne D. Meta-analysis combining parallel and cross-overclinical trials. I: Continuous outcomes. StatMed. 2002;21:2131–2144.

25 Higgins JP, Thompson SG, Deeks JJ, AltmanDG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–560.

26 Kaplan IP, Patton LR, Hamilton RA.Adaptation of different computerizedmethods of distance learning to an externalpharmd degree program. Am J Pharm Educ.1996;60:422–425.

27 Chan DH, Leclair K, Kaczorowski J. Problem-based small-group learning via the Internet

among community family physicians: Arandomized controlled trial. MD Comput.1999;16:54 –58.

28 Papa FJ, Aldrich D, Schumacker RE. Theeffects of immediate online feedback upondiagnostic performance. Acad Med. 1999;74(10 suppl):S16 –S18.

29 Schaad DC, Walker EA, Wolf FM, Brock DM,Thielke SM, Oberg L. Evaluating the serialmigration of an existing required course tothe World Wide Web. Acad Med. 1999;74(10suppl):S84 –S86.

30 Scherly D, Roux L, Dillenbourg P. Evaluationof hypertext in an activity learningenvironment. J Comput Assist Learning.2000;16:125–136.

31 Bearman M, Cesnik B, Liddell M. Randomcomparison of ‘virtual patient’ models in thecontext of teaching clinical communicationskills. Med Educ. 2001;35:824 –832.

32 Fox N, O’Rourke A, Roberts C, WalkerJ. Change management in primary care:Design and evaluation of an Internet-delivered course. Med Educ. 2001;35:803–805.

33 Duffy T, Gilbert I, Kennedy D, Wa KP.Comparing distance education andconventional education: Observations from acomparative study of post-registration nurses.Assoc Learning Technol J. 2002;10:70 –82.

34 Jedlicka JS, Brown SW, Bunch AE, Jaffe LE. Acomparison of distance educationinstructional methods in occupationaltherapy. J Allied Health. 2002;31:247–251.

35 Carpenter KM, Watson JM, Raffety B, ChabalC. Teaching brief interventions for smokingcessation via an interactive computer-basedtutorial. J Health Psychol. 2003;8:149 –160.

36 Chao LW, Enokihara MY, Silveira PS, GomesSR, Bohm GM. Telemedicine model fortraining non-medical persons in the earlyrecognition of melanoma. J TelemedTelecare. 2003;9(suppl 1):S4 –S7.

37 Frith KH, Kee CC. The effect ofcommunication on nursing student outcomesin a Web-based course. J Nurs Educ. 2003;42:350 –358.

38 Lemaire ED, Greene G. A comparisonbetween three electronic media and in-personlearning for continuing education in physicalrehabilitation. J Telemed Telecare. 2003;9:17–22.

39 Maag M. The effectiveness of an interactivemultimedia learning tool on nursingstudents’ math knowledge and self-efficacy.Comput Inform Nurs. 2004;22:26 –33.

40 Mattheos N, Nattestad A, Christersson C,Jansson H, Attstrom R. The effects of aninteractive software application on the self-assessment ability of dental students. Eur JDent Educ. 2004;8:97–104.

41 Spickard A 3rd, Smithers J, Cordray D,Gigante J, Wofford JL. A randomised trial ofan online lecture with and without audio.Med Educ. 2004;38:787–790.

42 Virvou M, Alepis E. Mobile versus desktopfacilities for an e-learning system: Users’perspective. In: Conference Proceedings—The Institute of Electrical and ElectronicsEngineers (IEEE) International Conferenceon Systems, Man and Cybernetics. TheHague, The Netherlands: IEEE; 2004;1:48 –52.

43 Allison JJ, Kiefe CI, Wall T, et al.Multicomponent Internet continuing medicaleducation to promote chlamydia screening.Am J Prev Med. 2005;28:285–290.



44 Becker EA, Godwin EM. Methods to improveteaching interdisciplinary teamwork throughcomputer conferencing. J Allied Health. 2005;34:169 –176.

45 Brunetaud JM, Leroy N, Pelayo S, et al.Comparative evaluation of two applications fordelivering a multimedia medical course in theFrench-speaking Virtual Medical University(UMVF). Int J Med Inform. 2005;74:209–212.

46 Mukohara K, Schwartz MD. Electronicdelivery of research summaries for academicgeneralist doctors: A randomised trial of aneducational intervention. Med Educ. 2005;39:402–409.

47 Schittek Janda M, Tani Botticelli A, MattheosN, et al. Computer-mediated instructionalvideo: A randomised controlled trialcomparing a sequential and a segmentedinstructional video in surgical hand wash. EurJ Dent Educ. 2005;9:53–58.

48 Blackmore C, Tantam D, Van Deurzen E. Therole of the eTutor—Evaluating tutor input ina virtual learning community forpsychotherapists and psychologists acrossEurope. Int J Psychother. 2006;10:35–46.

49 Cook DA, Thompson WG, Thomas KG,Thomas MR, Pankratz VS. Impact of self-assessment questions and learning styles inWeb-based learning: A randomized,controlled, crossover trial. Acad Med. 2006;81:231–238.

50 Friedl R, Hoppler H, Ecard K, et al.Comparative evaluation of multimediadriven, interactive, and case-based teaching inheart surgery. Ann Thorac Surg. 2006;82:1790 –1795.

51 Kemper KJ, Gardiner P, Gobble J, Mitra A,Woods C. Randomized controlled trialcomparing four strategies for delivering e-curriculum to health care professionals. BMCMed Educ. 2006;6:2.

52 Kerfoot BP, Conlin PR, McMahon GT.Comparison of delivery modes for onlinemedical education. Med Educ. 2006;40:1137–1138.

53 Little BB, Passmore D, Schullo S. Usingsynchronous software in Web-based nursingcourses. Comput Inform Nurs. 2006;24:317–325.

54 Nicholson DT, Chalk C, Funnell WRJ, DanielSJ. Can virtual reality improve anatomyeducation? A randomised controlled study ofa computer-generated three-dimensionalanatomical ear model. Med Educ. 2006;40:1081–1087.

55 Romanov K, Nevgi A. Learning outcomes inmedical informatics: Comparison of aWebCT course with ordinary Web sitelearning material. Int J Med Inform. 2006;75:156 –162.

56 Thompson GA, Morrison RG, Holyoak KJ,Clark TK. Evaluation of an online analogicalpatient simulation program. In: Proceedingsof the Nineteenth IEEE InternationalSymposium on Computer-Based MedicalSystems. Los Alamitos, Calif: IEEE ComputerSociety Press; 2006:623–628.

57 Al-Rawi WT, Jacobs R, Hassan BA, SanderinkG, Scarfe WC. Evaluation of Web-basedinstruction for anatomical interpretation inmaxillofacial cone beam computedtomography. Dentomaxillofac Radiol. 2007;36:459 –464.

58 Bednar ED, Hannum WM, Firestone A,Silveira AM, Cox TD, Proffit WR.

Application of distance learning to interactiveseminar instruction in orthodontic residencyprograms. Am J Orthod Dentofacial Orthop.2007;132:586 –594.

59 Cook DA, Gelula MH, Dupras DM, SchwartzA. Instructional methods and cognitive andlearning styles in Web-based learning: Reportof two randomised trials. Med Educ. 2007;41:897–905.

60 Cook DA, Thompson WG, Thomas KG,Thomas MR. Lack of interaction betweensensing–intuitive learning styles and problem-first versus information-first instruction: Arandomized crossover trial. Adv Health SciEduc Theory Pract. 2009;14:79–90.

61 Hege I, Ropp V, Adler M, et al. Experienceswith different integration strategies of case-based e-learning. Med Teach. 2007;29:791–797.

62 Kerfoot BP, DeWolf WC, Masser BA, ChurchPA, Federman DD. Spaced educationimproves the retention of clinical knowledgeby medical students: A randomisedcontrolled trial. Med Educ. 2007;41:23–31.

63 Kerfoot BP, Baker HE, Koch MO, ConnellyD, Joseph DB, Ritchey ML. Randomized,controlled trial of spaced education tourology residents in the United States andCanada. J Urol. 2007;177:1481–1487.

64 Miller KT, Hannum WM, Morley T, ProffitWR. Use of recorded interactive seminars inorthodontic distance education. Am J OrthodDentofacial Orthop. 2007;132:408 –414.

65 Moridani M. Asynchronous video streamingvs. synchronous videoconferencing forteaching a pharmacogeneticpharmacotherapy course. Am J Pharm Educ.2007;71:16.

66 Ridgway PF, Sheikh A, Sweeney KJ, et al. Surgicale-learning: Validation of multimedia Web-basedlectures. Med Educ. 2007;41:168–172.

67 Romanov K, Nevgi A. Do medical studentswatch video clips in eLearning and do thesefacilitate learning? Med Teach. 2007;29:484–488.

68 Bergeron BP. Learning & retention inadaptive serious games. Stud Health TechnolInform. 2008;132:26 –30.

69 Campbell M, Gibson W, Hall A, Richards D,Callery P. Online vs. face-to-face discussionin a Web-based research methods course forpostgraduate nursing students: A quasi-experimental study. Int J Nurs Stud. 2008;45:750 –759.

70 Cook DA, Beckman TJ, Thomas KG,Thompson WG. Introducing resident doctorsto complexity in ambulatory medicine. MedEduc. 2008;42:838 –848.

71 Cook DA, Beckman TJ, Thomas KG,Thompson WG. Adapting Web-basedinstruction to residents’ knowledge improveslearning efficiency: A randomized controlledtrial. J Gen Intern Med. 2008;23:985–990.

72 Kerfoot BP. Interactive spaced educationversus Web based modules for teachingurology to medical students: A randomizedcontrolled trial. J Urol. 2008;179:2351–2356.

73 Kerfoot BP, Brotschi E. Online spacededucation to teach urology to medicalstudents: A multi-institutional randomizedtrial. Am J Surg. 2009;197:89 –95.

74 Kopp V, Stark R, Fischer MR. Fosteringdiagnostic knowledge through computer-supported, case-based worked examples:Effects of erroneous examples and feedback.Med Educ. 2008;42:823–829.

75 Tunuguntla R, Rodriguez O, Ruiz JG, et al.Computer-based animations and static graphicsas medical student aids in learning home safetyassessment: A randomized controlled trial. MedTeach. 2008;30:815–817.

76 Walker BL, Harrington SS. Computer-basedinstruction and the Web for delivering injuryprevention training. Educ Gerontol. 2008;34:691–708.

77 Cohen J. Statistical Power Analysis for theBehavioral Sciences. 2nd ed. Hillsdale, NJ:Lawrence Erlbaum; 1988.

78 Colliver JA, Kucera K, Verhulst SJ. Meta-analysis of quasi-experimental research: Aresystematic narrative reviews indicated? MedEduc. 2008;42:858 –865.

79 Lau J, Ioannidis JP, Terrin N, Schmid CH,Olkin I. The case of the misleading funnelplot. BMJ. 2006;333:597–600.

80 Adler MD, Johnson KB. Quantifying theliterature of computer-aided instruction inmedical education. Acad Med. 2000;75:1025–1028.

81 Greenhalgh T. Computer assisted learning inundergraduate medical education. BMJ. 2001;322:40 –44.

82 Wutoh R, Boren SA, Balas EA. eLearning: Areview of Internet-based continuing medicaleducation. J Contin Educ Health Prof. 2004;24:20 –30.

83 Dillon A, Gabbard RB. Hypermedia as aneducational technology: A review of thequantitative research literature on learnercomprehension, control, and style. Rev EducRes. 1998;68:322–349.

84 Clark RC, Mayer RE. E-Learning and theScience of Instruction: Proven Guidelines forConsumers and Designers of MultimediaLearning. 2nd ed. San Francisco, Calif:Pfeiffer; 2007.

85 Kulik C-LC, Kulik JA, Shwalb BJ. Theeffectiveness of computer-based adulteducation: A meta-analysis. J Educ ComputRes. 1986;2:235–252.

86 Fletcher-Flinn CM, Gravatt B. The efficacy ofcomputer-assisted instruction (CAI): A meta-analysis. J Educ Comput Res. 1995;12:219–242.

87 Lewis MJ, Davies R, Jenkins D, Tait MI. Areview of evaluative studies of computer-based learning in nursing education. NurseEduc Today. 2001;21:26 –37.

88 Curran VR, Fleet L. A review of evaluationoutcomes of Web-based continuing medicaleducation. Med Educ. 2005;39:561–567.

89 Prystowsky JB, Bordage G. An outcomesresearch perspective on medical education: Thepredominance of trainee assessment andsatisfaction. Med Educ. 2001;35:331–336.

90 Chen FM, Bauchner H, Burstin H. Acall for outcomes research in medicaleducation. Acad Med. 2004;79:955–960.

91 Cook DA, Bowen JL, Gerrity MS, et al.Proposed standards for medical educationsubmissions to the Journal of GeneralInternal Medicine. J Gen Intern Med. 2008;23:908 –913.

92 Shea JA. Mind the gap: Some reasons whymedical education research is different fromhealth services research. Med Educ. 2001;35:319 –320.

93 Cook DA. The failure of e-learning research toinform educational practice, and what we cando about it. Med Teach. 2009;31:158–162.



Documents

Instructional Design Variations in Internet-Based … · Instructional Design Variations in Internet-Based Learning for Health Professions Education: A Systematic Review ... Compared