Systematic Systematic Reviews: Reviews:

9 Lessons Learned9 Lessons LearnedBeth A. Tarini, MDBeth A. Tarini, MD

Robert Wood Johnson Clinical ScholarRobert Wood Johnson Clinical ScholarUniversity of Washington University of Washington

July 14, 2005July 14, 2005

Needed a project for residency research Needed a project for residency research

electiveelective

No other specific research interests at the No other specific research interests at the

timetime NOTE:NOTE: Shortly after I began this project, I Shortly after I began this project, I

developed an interest in the role of diagnostic developed an interest in the role of diagnostic

testing in generating harm in pediatric patientstesting in generating harm in pediatric patients

Why did I do this?Why did I do this?

Lesson # 1: Be interested (very, if possible)

in your topic

Question generated from patient encounter Question generated from patient encounter during residency:during residency:

Attending told me “what we do”, BUT Attending told me “what we do”, BUT ALSO…ALSO… Challenged the evidence behind the practiceChallenged the evidence behind the practice And suggested that I turn the question into a And suggested that I turn the question into a

systematic reviewsystematic review

How did I chose the topic?How did I chose the topic?

Is it OK to give peanut butter to a 12 Is it OK to give peanut butter to a 12 mo. old?mo. old?

How did I chose my How did I chose my mentors?mentors?

Attending was too busy mentoring fellowsAttending was too busy mentoring fellows RWJ fellow was willing to mentor meRWJ fellow was willing to mentor me Good situation for meGood situation for me

I had a lot of questionsI had a lot of questions The fellow was accessible & very competent The fellow was accessible & very competent

(had done a systematic review before)(had done a systematic review before)

Accessibility of mentor may be more important than job title

Lesson # 2:

Finding a questionFinding a question First, formulated the questionFirst, formulated the question

Does early solid feeding in infants Does early solid feeding in infants increase the risk of allergy?increase the risk of allergy?

NOTE:NOTE: I modified it after searching the I modified it after searching the literatureliterature Wanted an interesting questionWanted an interesting question Wanted to appeal to a broad audienceWanted to appeal to a broad audience

Find an interesting question that will lead to a publication

Lesson # 3:

Finding a questionFinding a question Then, refined the question…Then, refined the question…

EarlyEarly – before 4 months of age – before 4 months of age SolidsSolids – any food stuff other than breastmilk, – any food stuff other than breastmilk,

cow’s milk or formulacow’s milk or formula Allergy Allergy – any clinically diagnosed allergic – any clinically diagnosed allergic

disease (disease (initiallyinitially)) In the endIn the end – asthma, eczema, food allergy, – asthma, eczema, food allergy,

pollen allergy, allergic rhinitis, and animal pollen allergy, allergic rhinitis, and animal dander allergy*dander allergy*

Lesson # 4: Be Specific

The search…The search… Searched databases (MEDLINE, Searched databases (MEDLINE,

EMBASE, COCHRANE) & bibliographies EMBASE, COCHRANE) & bibliographies of relevant studies and review articlesof relevant studies and review articles

Working with the librariansWorking with the librarians 11stst SEARCH SEARCH

Searched Medline with MeSH headings ~ Searched Medline with MeSH headings ~ 1800 titles1800 titles

Found multiple “missed” articles during Found multiple “missed” articles during bibliography searchbibliography search

The search…The search…

22ndnd SEARCH SEARCH Searched MEDLINE with keywords Searched MEDLINE with keywords

“infant” and “food” and “allergy” “infant” and “food” and “allergy”

~ 2500 citations~ 2500 citations

Lesson # 5: Cast a wide net

Reviewing the studiesReviewing the studies Inclusion/Exclusion criteria modified during Inclusion/Exclusion criteria modified during

reviewreview Reviewed Reviewed 39 articles*39 articles* In general, studies of poor qualityIn general, studies of poor quality

No randomized controlled trialsNo randomized controlled trials Majority were cohort studiesMajority were cohort studies Many studies inadequately controlledMany studies inadequately controlled

The literature is like a box of chocolates…you never know what you’re gonna get

Lesson #6:

What did I find?What did I find?

The evidence is of mediocre quality & The evidence is of mediocre quality &

conflictingconflicting Some evidence to support an Some evidence to support an

increased risk of eczema with early increased risk of eczema with early

solid feedingsolid feeding Worse for those with a family history of eczemaWorse for those with a family history of eczema

BudgetBudget Didn’t have one – Didn’t have one – ““do as I say, not as I do”do as I say, not as I do”

Photocopying and ordering articles was the Photocopying and ordering articles was the

biggest expensebiggest expense The RWJ fellow gave me photocopy cardsThe RWJ fellow gave me photocopy cards

Children’s orders articles for staff for freeChildren’s orders articles for staff for free

Overall estimate - $1000-1500?Overall estimate - $1000-1500?

Lesson #7: You always need some $

My “expected” timelineMy “expected” timeline

October 2002: Begin researchOctober 2002: Begin research November 2002: Finish researchNovember 2002: Finish research December 2002: Submit for December 2002: Submit for

publicationpublication

Clearly, I was new to this…Clearly, I was new to this…

The real timelineThe real timeline

Oct

ober

’02:

Began

rese

arch

Decem

ber ’

02:

Applie

d to

RW

J

….res

iden

cy con

t’d

Decem

ber ’

03:

Subm

itted

abs

trac

t

for p

eds m

eetin

g

July

– D

ec ’0

4:

Fini

shed

writ

ing

man

uscr

ipt

Janu

ary

’05:

Subm

itted

man

uscr

ipt

Mar

ch ’0

5:

Rejec

ted

April

’05:

Revised

&

Resub

mitt

ed

May

‘05:

Man

uscr

ipt

acce

pted

for

publ

icat

ion

2002 2003 2004

2005

The real timelineThe real timeline

It always takes longer than you think it will

Lesson #8:

What has this study done for What has this study done for me?me?

Reinforced my skepticism that standard Reinforced my skepticism that standard practice is often not based on solid practice is often not based on solid evidenceevidence

Taught me the value Taught me the value & necessity& necessity of being of being specific specific

Improved my ability to critique studiesImproved my ability to critique studies Improved my patience and staminaImproved my patience and stamina

Lesson #9: You always learn something