Journal of Clinical Apheresis 2:105-111 (1984)

KEYNOTE ADDRESS: THE NEED FOR RANDOMIZED CONTROL TRIALS

Thomas C. Chalmers, M.D.

Mount S i n a i School of Medicine, New York, New York

A l i b r a r y is a good source of f a b u l o u s l y i n t e r e s t i n g and under-analyzed m a t e r i a l on randomized c o n t r o l t r i a l s . I ' m convinced t h a t a medical s t u d e n t can do u s e f u l l i b r a r y r e s e a r c h and ana lyze t h e d a t a by paying a t t e n t i o n t o t h e p r i n c i p l e s of s t a t i s t i c s and app ly t h e s c i e n t i f i c d a t a t o t h e p r a c t i c e of medicine. on a p h e r e s i s which were ga the red p r i m a r i l y by a pre-medical s t u d e n t doing such l i b r a r y r e s e a r c h . We worked t o g e t h e r over t h e l a s t s i x months, s t i m u l a t e d by t h e i n v i t a t i o n t o speak t o t h i s group, bu t had begun t h i s review b e f o r e , because a p h e r e s i s p r e s e n t s such a b e a u t i f u l example of how p h y s i c i a n s have become concerned about t h e c o s t s involved i n d e l i v e r i n g h e a l t h c a r e . It seems a s i f t h e h i g h - q u a l i t y of medical p r a c t i c e is tremendously t h r e a t e n e d by t h e fo l lowing two means used by a d m i n i s t r a t o r s f o r c u t t i n g h e a l t h c a r e c o s t s : One is capping t o t a l expenses and t h e o t h e r is compe t i t i on . Both of t hese methods p r e s e n t a d i sadvan tage t o some peop le , e s p e c i a l l y t h e aged and t h e poor. The re fo re , we have t o f i n d some o t h e r way t o reduce t h e i n c r e a s e d c o s t of h e a l t h d e l i v e r y . Another a t t r a c t i v e way i s t o apply t h e s c i e n t i f i c method t o t h e p r a c t i c e of medicine and f i n d whether we're was t ing money by p rov id ing the rapy f o r people who may no t b e n e f i t from it and determine i f we can save money by us ing those t h e r a p i e s i n d i s e a s e s i n which they do work.

Apheresis i s t h e i d e a l example of t h i s . I t ' s an expensive the rapy t h a t i s being overused and probably i s a l s o underused. The only way t o a r r i v e a t r e l i a b l e answers a s t o when t o use and when n o t t o use a p h e r e s i s is t o g a t h e r d a t a from t h o s e c a s e s i n which it seems t o be i n d i c a t e d and then determine when and how t o use i t , and how it can be modif ied t o have an i n c r e a s e d e f f e c t . But, we need s c i e n t i f i c d a t a t o have those r easons evolve.

We covered a Medline review of s i x yea r s from 1977-1983. We found t h e t r e a t m e n t of 60 d i s o r d e r s i n 151 papers pub l i shed i n t h a t s i x year pe r iod . I d o n ' t know whether exchange t r a n s f u s i o n f a l l s i n t h i s group o r n o t , b u t I do want t o s t a t e t h a t I d i d p a r t i c i p a t e i n t h e f i r s t exchange t r a n s f u s i o n (whenever you s a y you a r e t h e f i r s t , you a r e always wrong) f o r fulminant h e p a t i t i s around 1956, and saved a p a t i e n t who was obv ious ly going t o d i e [ l ] . Because of t h a t one c a s e , I concluded I could n o t do a randomized c o n t r o l t r i a l (RCT) on t h a t d i s o r d e r because it obv ious ly worked. The on ly RCT w a s done by Redeker, of Los Angeles [ 2 ] . This was e n t i r e l y n e g a t i v e and t h e p a t i e n t s who d i d n ' t g e t t h e exchange l i v e d longer b u t no t s i g n i f i c a n t l y so than those who were exchanged. But i t w a s no t a good t r i a l because too many p a t i e n t s were randomized t o exchange who d i d not r e c e i v e i t . There s t i l l h a s n ' t been a randomized t r i a l of t h a t d i s e a s e , bu t I l e a r n e d from t h a t one case about t h e dangers of be ing e n t h u s i a s t i c w i th one expe r i ence .

r e p o r t s and t h e u n c o n t r o l l e d s t u d i e s . The d a t a e s t a b l i s h one of Munch's laws. (Munch was a b i o s t a t i s t i c i a n and chairman of t h e department a t t h e Harvard School of P u b l i c H e a l t h ) . Th i s law s t a t e d t h a t t h e b e s t way t o g e t p o s i t i v e r e s u l t s i n a c l i n i c a l t r i a l is t o l eave o u t t h e c o n t r o l s . There are few exep t ions t o t h i s Law. For example, when you have t o g ive p e n i c i l l i n f o r l o b a r pneumonia, i n s u l i n f o r d i a b e t i c k e t o a c i d o s i s , v i t amin B p e r n i c i o u s anemia and p o s s i b l y p l a smapheres i s f o r Goodpas tu re ' s Syndrome, you don t need c o n t r o l s because you have a s i t u a t i o n i n which t h e f a t a l i t y r a t e i s c l o s e t o 100% and

By reviewing papers I w i l l p r e s e n t d a t a

Most of t h e papers on a p h e r e s i s were e n t h u s i a s t i c about a p h e r e s i s , e s p e c i a l l y t h e case

12 fo:

0 1984 Alan R. Liss, Inc.

106 Chalmers

s u r v i v a l w i th the rapy i s demonstrated t o be 90%. You a l s o do no t need c o n t r o l s f o r demons t r a t ing l a r g e d i f f e r e n c e s . Medicine was

advancing long b e f o r e t h e advent of randomized c o n t r o l t r i a l s , and t h e g r e a t advances i n chemotherapy, a n t i b i o t i c t h e r a p y and s u r g i c a l t r e a t m e n t of some d i s e a s e a r e b e a u t i f u l examples of t h a t . But t h e r e a r e a l s o examples such a s gas t r ec tomy f o r p e p t i c u l c e r and r a d i c a l mastectomy f o r cance r of t h e b r e a s t i n which, a l t h o u g h t h e t r e a t m e n t seemed obvious and a l though one d i d n ' t seem t o need c o n t r o l s , i t l a t e r t u r n e d o u t t h a t one d i d . Those i n which one does need c o n t r o l s , t h e r e f o r e , a r e those i n which t h e d i f f e r e n c e s a r e r e l a t i v e l y small and a r e sma l l enough s o t h a t b i a s i n t h e s e l e c t i o n of p a t i e n t s and i n t h e i n t e r p r e t a t i o n of t h e outcome i s a v e r y s t r o n g d e t e r m i n a t e of t h e c o n c l u s i o n s .

There a r e s t i l l s i t u a t i o n s i n which t h e d i f f e r e n c e can be l a r g e r and we have many examples of t h a t i n modern medical t h e r a p y . The q u e s t i o n today is how o f t e n is t h i s t h e c a s e i n a p h e r e s i s . The d i s e a s e s a r e v a r i a b l e , t h e s e l e c t i o n f a c t o r v e r y impor t an t and t h e e f f i cacy -a l though p h y s i o l o g i c a l l y mean ingfu l - i s t o o s m a l l t o be r e a l l y obvious i n u n c o n t r o l l e d t r i a l s .

A c l a s s i c example of t h i s i n a n o t h e r f i e l d i s a c u t e myoca rd ia l i n f a r c t i o n , of which t h e r e a r e about 400,000 c a s e s pe r yea r i n t h e United S t a t e s . The h o s p i t a l d e a t h r a t e i s about 15% and we s t i l l have n o t h i n g t h a t is going t o reduce t h a t t o z e r o o r one p e r c e n t . We do have some t h i n g s however, t h a t can r educe it from 15% t o 14% and obv ious ly you need c o n t r o l s t o demonstrate t h a t w i t h some a s s u r a n c e . Fu r the rmore , a 10% r e d u c t i o n i n d e a t h ra te when a p p l i e d t o t h e h a l f m i l l i o n people who have myocardial i n f a r c t i o n each y e a r i n the United S t a t e s w i l l r e p r e s e n t 6,000 l i v e s saved. T h e r e f o r e , it i s most impor t an t t o t r y t o demonstrate t h e s e sma l l improvements.

who a r e unable t o work t o r e t u r n t o work, we would pay f o r t h e randomized c o n t r o l t r i a l s many t i m e s ove r and i t would be worthwhile . The re fo re when we are d e a l i n g w i t h s m a l l d i f f e r e n c e s , we need t h e d a t a .

I want t o emphasize t h e f a c t t h a t t h e mere do ing of an RCT does n o t mean t h a t t h e answer i s going t o be obvious and it does n o t mean t h a t i f you 've randomized, you 've done a good job . I n f a c t , t h e r e a r e r e a s o n a b l e numbers of poor ly done randomized t r i a l s .

f o r looking a t RCTs and d e c i d i n g whether or n o t t h e c o n c l u s i o n s of t h e a u t h o r s w i t h r e g a r d t o t h e outcome i n t h a t s t u d y a r e war ran ted by t h e d a t a 131. Whether o r n o t t h e y are warranted by t h e d a t a depends on how w e l l t h e s t u d y i s c a r r i e d o u t and a l s o how a p p l i c a b l e the s t u d i e s may be t o common p r a c t i c e .

m a t e r i a l , and i n t h i s we have f i v e i t e m s f o r which we g i v e t h r e e p o i n t s each i n our s c o r i n g system. We reviewed d a t a on 373 m i s c e l l a n e o u s t r i a l s , i n abou t 10 t o 15 d i s o r d e r s . We s a w t h a t t h e s e l e c t i o n d e s c r i p t i o n was a d e q u a t e , by t h a t , we meant t h a t i t was enough f o r t h e phys ic i an who is p r a c t i c i n g medicine t o know whether o r n o t t h e c o n c l u s i o n s of t h i s t r i a l app ly t o any p a t i e n t he o r she may see i n 46%. important . Although you can se t up c r i t e r i a i n a s t u d y f o r a c c e p t a n c e and r e j e c t i o n of t h e p a t i e n t , i n o r d e r t o app ly t h a t s t u d y , you need t o know whether t h e a u t h o r s a c t u a l l y d i d fo l low t h e c r i t e r i a o u t l i n e d i n t h e p r o t o c o l . The o n l y way t o know t h a t is t o look a t t h e p a t i e n t s who d i d n ' t g e t i n t o t h e s t u d y and s e e how t h e y d i f f e r e d , because then you w i l l g e t some impression about how t h i s s t u d y does n o t app ly . Only 9% k e p t a r e j e c t l og i n t h e g e n e r a l s t u d i e s . Also, p a t i e n t s can be e n t e r e d a t random, b u t i f t hey a r e n o t withdrawn a t random and no t i nc luded i n t h e a n a l y s i s , ( i f t h e i n v e s t i g a t o r s d o n ' t f o l l o w t h e p r i n c i p l e of a n a l y z i n g a l l p a t i e n t s e n t e r e d i n t o t h e s t u d y ) they end up w i t h a v a r i a b l e number of withdrawals . Then you can d e s t r o y t h e e f f i c a c y of t h e r andomiza t ion because of s e l e c t i o n r a t h e r t han t h e the rapy . We found t h a t on ly 55% of t h e s t u d i e s l i s t e d t h e wi thd rawa l s and the t h e r a p e u t i c regimen i s a d e q u a t e l y d e f i n e d i n 78%.

bu t i t a l s o , most i m p o r t a n t l y , i n c l u d e s t h e r andomiza t ion p rocess . W e have d a t a demons t r a t ing t h a t even i n RCTs when t h e d i s t r i b u t i o n of pre-randomizat ion r i s k f a c t o r s d i f f e r s , we can demons t r a t e b i a s i f t h e d o c t o r knows when he se lec ts a p a t i e n t t o go i n t o a t r i a l whether o r n o t t h a t p a t i e n t is going t o r e c e i v e t h e new o r t h e expe r imen ta l t h e r a p y . I n t h i s c a s e , t h e so -ca l l ed good p a t i e n t s d o n ' t g e t i n t o t h e t r i a l i f t h e y a r e due t o have t h e s t a n d a r d t h e r a p y , o r t h e bad p a t i e n t s d o n ' t g e t i n t o t h e t r i a l because they a r e due t o have t h e new t h e r a p y , o r v i c e v e r s a . The r andomiza t ion p r o c e s s must be b l i n d e d , e i t h e r by

For a p h e r e s i s , i f we o b t a i n 10% improvement i n rheumatoid a r t h r i t i s and cause people

Because we can o b t a i n v a l u a b l e i n f o r m a t i o n from R C T s , we've s e t up a s c o r i n g system

Our s c o r i n g system is d i v i d e d i n t o t h r e e s e c t i o n s . One is t h e e s s e n t i a l d e s c r i p t i v e

Now t h e f l i p s i d e of t h a t is e q u a l l y

B l ind ing i s n o t j u s t t h e u s u a l doub le -b l ind h a n d l i n g of t h e p h y s i c i a n and t h e p a t i e n t ,

Need for Randomized Control Trials 107

app ly ing a b l i n d envelope t echn ique , by a t e l ephone c a l l t o a randomizat ion c e n t e r o r by some o t h e r method, e.g. a pharmacy p u t t i n g up t h e drug o r subs t ance ahead of t i m e so t h a t you w i l l no t s u s p e c t i t when you s e e the p a t i e n t . i n s t u d i e s wi th a major end p o i n t , l i k e d e a t h . I n t h i s s c o r i n g , however, we d i d not g i v e c r e d i t when they d i d n ' t d e s c r i b e the randomizat ion method a c c u r a t e l y . t hen by w r i t i n g t o a u t h o r s , t h a t i n about one - th i rd of t h e s t u d i e s , t hose who d i d n ' t d e s c r i b e i t a c c u r a t e l y a c t u a l l y d i d it p r o p e r l y . So, t h e s e pe rcen tages a r e s l i g h t l y low.

done. The r eason we t h i n k t h a t i t is c r i t i c a l i n a good t r i a l i s t h a t i t i n f l u e n c e s both the s c i e n t i f i c and e t h i c a l a s p e c t s of t h e s t u d i e s . From t h e s c i e n t i f i c s t a n d p o i n t i f t h e doc to r knows the t r e n d , b i a s can be in t roduced which w i l l d e s t r o y t h e t r i a l . From t h e e t h i c a l s t a n d p o i n t , one conve r t s an e t h i c a l RCT i n t o an u n e t h i c a l s i t a t i o n by knowing t h e t r e n d . V i s u a l i z e an RCT i n which d e a t h is t h e end p o i n t and i n which p a t i e n t s a r e admi t t ed i n p a i r s . I f t h e s t u d y cannot be double-bl ind, you w i l l know a l l t h e t i m e which t r ea tmen t is doing b e t t e r . need one more p a i r of p a t i e n t s t o b r i n g it under 0.05, can you look t h o s e two p a t i e n t s i n the eye and s a y , "I want you t o v o l u n t e e r t o be randomized. I d o n ' t know which t r ea tmen t i s b e t t e r and we'd l i k e you t o e n t e r t h e study' '? s i g n i f i c a n t e t h i c a l or s c i e n t i f i c d i f f e r e n c e between a P-value of 0.06 and one of 0.04, YOU

f a c e a dilemma f o r which t h e r e is no answer o t h e r t han t o keep t h e ongoing r e s u l t s from t h e i n v e s t i g a t o r s so t h a t t hey a r e n o t b i a sed a s they go a long . p a t i e n t s g e t t i n g i n t o t h e t r i a l d o e s n ' t change and t h e r e j e c t r a t e does no t go up h i g h e r a s t he i n v e s t i g a t o r s a r e n o t faced wi th a t e r r i b l e e t h i c a l dilemma of needing t o e n t e r a few more p a t i e n t s i n o r d e r t o prove t h a t something is more l e t h a l than a n o t h e r . The problem is b e s t a p p r e c i a t e d when you r ecogn ize t h a t t h e odds f avor one t r ea tmen t o r t h e o t h e r a s soon a s t h e r e s u l t s a r e i n on t h e f i r s t p a i r .

The t h i r d method of s c o r i n g has t o do w i t h how w e l l t h i n g s were t e s t e d , and how the i n v e s t i g a t o r s f i g u r e d o u t beforehand what d i f f e r e n c e s they were i n t e r e s t e d i n and c a l c u l a t e d how many p a t i e n t s t hey had t o s t u d y . Deplorably, only 6% of t h e s t u d i e s s o f a r reviewed d i d t h a t .

may be b reaks i n procedure. The d o c t o r s may ho ld t h e envelope up t o a b r i g h t l i g h t t o see which the rapy i s up nex t i n o r d e r t o be s u r e t h e i r p a t i e n t g e t s t he the rapy they would l i k e . We have a documented i n s t a n c e of t h a t i n t h e medical l i t e r a t u r e . So you t e s t t o s e e i f t h e r e has been any gypping, b u t a l s o t o see whether a chance m a l d i s t r i b u t i o n of p a t i e n t s has occurred (which does occur 5% of t h e t ime) . 47% of t h e t r i a l s d i d t es t t h e randomizat ion by adequate methods. T e s t i n g of b l i n d i n g i s something seldom done. One p e r c e n t , ou t of t h e 309 papers ( 3 p a p e r s ) , d i d a sk t h e d o c t o r s and p a t i e n t s which t h e r a p y they thought t hey r e c e i v e d .

Only 36% had t h a t i n fo rma t ion . I n a p h e r e s i s t r i a l s , measurement of immunologic phenomena would be measuring a b i o l o g i c a l e q u i v a l e n t . I f a l l of t h e s e a r e combined i n t h e s c o r i n g system, we come up w i t h a r e l a t i v e q u a l i t y s c o r e . We f i n d t h a t t h e mean q u a l i t y i n most randomized t r i a l s is .45.

pape r s , and it is no more work-or v e r y l i t t l e more work-than it i s t o do a t r i a l t h a t ' s only 15% p e r f e c t .

o v e r a l l , b u t v e r y impor t an t , conc lus ion from a l l t h e s e d a t a . Never b e f o r e has t h e r e been such a need f o r l a r g e numbers of p a t i e n t s and so few s t u d i e s done. I a p p r e c i a t e t h e f a c t t h a t i t ' s d i f f i c u l t t o do an adequate s t u d y i f sham a p h e r e s i s is n e c e s s a r y bu t it is c l e a r l y an u n d e r - u t i l i z e d method.

h e a r t . I f I had randomized t h a t f i r s t p a t i e n t t h a t we t r e a t e d w i t h exchange blood t r a n s f u s i o n f o r fulminant h e p a t i t i s [ l ] and t h e p a t i e n t had been randomized i n t o t h e c o n t r o l group, it is e n t i r e l y p o s s i b l e t h e p a t i e n t would have g o t t e n w e l l and w e would have gone ahead w i t h an RCT. Some s k e p t i c s had t o do it i n s t e a d . I t h i n k t h a t is t h e s i t u a t i o n q u i t e f r e q u e n t l y . Someone has a b r i g h t i d e a , does a p i l o t s t u d y , and t h e p i l o t s t u d y is conceived of a s one i n which t h e r e i s no r andomiza t ion , b u t a consecu t ive series of p a t i e n t s a r e s t u d i e d and t h e r e s u l t w i t h r e g a r d t o t h e r a p e u t i c outcome is looked a t and

This is c r i t i c a l l y impor t an t , e s p e c i a l l y

We have found s i n c e

We a l s o measured t h e b l i n d i n g of t h e p h y s i c i a n as t o t h e r e s u l t s . This is ve ry seldom

When you g e t t o the s t a g e where t h e P-value is 0.06 and YOU

When you know t h a t t h e r e r e a l l y i s no

This ensu res t h a t t h e type of

T e s t i n g of randomizat ion is important because even w i t h b l i n d e d randomizat ion, t h e r e

I n o t h e r words, t hey measured enough d i f f e r e n c e s .

B i o l o g i c a l e q u i v a l e n t i s measuring something o t h e r t han t h e t h e r a p e u t i c end p o i n t .

It i s e n t i r e l y p o s s i b l e t o do a p e r f e c t t r i a l . There a r e two o r t h r e e i n our 309

With r e g a r d t o RCT ' s of a p h e r e s i s , 14 were a l l I could f i n d . I can only make one

Why shou ld t h i s be so? I t h i n k t h e r e are s e v e r a l r e a s o n s , and one i s v e r y d e a r t o my

108 Chalmers

t h e r e a r e t h r e e p o s s i b i l i t i e s . One i s t h a t t h e the rapy i s d r a m a t i c a l l y e f f e c t i v e so t h a t it is no longer e t h i c a l l y p o s s i b l e f o r t h e i n v e n t o r of t h e t r e a t m e n t t o do a RCT and depr ive people of t h e the rapy . That happens v e r y f r e q u e n t l y among i n s t a n c e s i n which l a t e r RCT have been shown t o be n e g a t i v e . The f l i p s i d e i s t h a t t h e t h e r a p y appea r s awful so t h a t it is dropped, and it wou ldn ' t be f a i r t o randomize p a t i e n t s i n t o t h i s awful appea r ing the rapy . This e s p e c i a l l y a p p l i e s t o t h e cance r f i e l d . One i s d e a l i n g wi th a l e t h a l d i s e a s e and the the rapy may have been e f f e c t i v e , b u t less e f f e c t i v e than s t a n d a r d the rapy . The t h i r d p o s s i b i l i t y of an u n c o n t r o l l e d p i l o t s t u d y is t h a t t h e t h e r a p y looks t o be equ ivoca l and t h e r e f o r e t h e a u t h o r drops i t and goes on looking f o r something t h a t might be more d r a m a t i c a l l y e f f e c t i v e . That is o f t e n a p i t y because a c c u r a t e in fo rma t ion about t h e e f f i c a c y of t h i s t h e r a p y would s t i l l be impor t an t i n t h e development of a new one. We have a l l t h r e e p o s s i b l e outcomes of p i l o t t r i a l s . A l l a r e f o r c e s f o r n o t u n d e r t a k i n g a randomized t r i a l and I know of no way t o avo id t h a t l e t h a l e f f e c t o t h e r t han t o randomize the f i r s t p a t i e n t .

Whenever a q u e s t i o n is be ing asked a s t o whether o r n o t a t h e r a p y i s e f f e c t i v e o r no t o r whether i t ' s b e t t e r o r worse than s t a n d a r d t h e r a p y o r even whether it does a n y t h i n g , o r how it works, o r what t h e dose i s , one shou ld s t a r t r i g h t o f f w i t h a RCT. There a r e hundreds and hundreds of new t h e r a p i e s be ing t r i e d eve ry y e a r , p robab ly thousands. P r a c t i c a l l y no one bu t me randomizes the f i r s t p a t i e n t [ 4 ] , bu t I h a v e n ' t g i v e n up. I t h i n k i t ' s t h e only way t o go, once you t ake a s your end p o i n t whether t h e p a t i e n t was improved o r n o t . Twice as many p a t i e n t s have t o be s t u d i e d i n i t i a l l y , bu t f o u r t i m e s a s much in fo rma t ion is o b t a i n a b l e .

There is a v e r y impor t an t s u b s e t t o t h i s problem and t h a t i s p u b l i c a t i o n b i a s . One t ends t o p u b l i s h t h e s u c c e s s e s and no t t h e f a i l u r e s and t h i s is e s p e c i a l l y t r u e when you d o n ' t have c o n t r o l s . D r . Benhamou [ 5 ] reviewed the s i t u a t i o n w i t h r e g a r d t o exchange t r a n s f u s i o n f o r fulminant h e p a t i t i s and found t h a t t h e d a t a a r e v e r y s i m i l a r t o what I ' v e shown he re wi th a p h e r e s i s . With s i n g l e c a s e r e p o r t s , t h e e f f i c a c y w a s about 90%. But whenever the number of c a s e s r e p o r t e d g o t above 10 o r 20, s t i l l d e a l i n g wi th non-randomized c o n t r o l l e d t r i a l s , t h e e f f i c a c y went down. I t h i n k t h e assumption is a p r e t t y good one t h a t i n sma l l s i t u a t i o n s where j u s t a few a r e t r i e d , only t h e s u c c e s s e s were r e p o r t e d . I f t h e i n v e s t i g a t o r needs t o p u b l i s h t h e pape r s t o g e t promoted, t h e y may w r i t e it up and send it i n , bu t t he j o u r n a l r e j e c t s them because of i t s n e g a t i v e c o n c l u s i o n s .

We've done some sma l l m a i l su rveys of RCTs and found ( a l t h o u g h t h e d a t a a r e v e r y u n r e l i a b l e ) t h a t more n e g a t i v e s t u d i e s a r e no t r e p o r t e d . Another r e a s o n why s t u d i e s a r e no t done more o f t e n and why people d o n ' t randomize ve ry e a r l y i s t h e c o s t . Everyone assumes t h a t you can t r e a t a c o n s e c u t i v e s e r i e s of p a t i e n t s and be pa id f o r it by t h e t h i r d p a r t i e s because you a r e no t doing r e s e a r c h and t h e p a t i e n t ' s c a r e i s be ing pa id f o r . But once you randomize, and once you a s k a q u e s t i o n of whether t h a t t h e r a p y is good o r n o t , you a r e doing r e s e a r c h and t h e t h i r d p a r t i e s might no t pay f o r i t . You can r e p o r t u n c o n t r o l l e d d a t a i n a s loppy manner wi thou t l o t s of d e t a i l e d o b s e r v a t i o n s , b u t i f you randomize you have t o be ve ry c a r e f u l t o r e c o r d a l l t h e d a t a e x a c t l y and expens ive ly . You have t o do a l o t of e x t r a t e s t s and you d o n ' t have t h e money t o do t h a t , t h e r e f o r e you d o n ' t randomize because you d o n ' t have the money t o t a k e p rope r c a r e of t h e p a t i e n t . I t h i n k one can have a very a c c e p t a b l e coun te r argument, w i t h good r eco rdkeep ing a s a p a r t of good med ica l c a r e . Randomization r e q u i r e s good r e c o r d k e e p i n g , e r g o , r andomiza t ion i s b e t t e r medical c a r e .

audience a l i t t l e b i t . W e had a meet ing l as t J u l y t o which we asked a l l of t h e t h i r d p a r t y paye r s : t h e f e d e r a l government, s t a t e government, p r i v a t e i n s u r a n c e companies, Blue Cross/Blue S h i e l d , and v a r i o u s o t h e r people . W e u s e d , a s examples f o r t h e d i s c u s s i o n , m u l t i p l e s c l e r o s i s and rheumatoid a r t h r i t i s and s p e c i f i c a l l y , p l a smapheres i s , and found t h a t t h e r e was a v a r i a b l e r a t e of payment by t h i r d p a r t i e s throughout t h e coun t ry . We proposed t h a t t h i r d p a r t i e s p i c k up t h e c o s t of R C T s , and they r e f u s e d . They d i d n o t want t o do any th ing t o i n c r e a s e t h e i r c o s t and d i d n ' t want t o add r e s e a r c h t o what we were paying f o r because t h a t would i n c r e a s e t h e c o s t t o t h e s u b s c r i b e r . They wanted us t o a p p r e c i a t e t h a t t h i s was a c o m p e t i t i v e b u s i n e s s and i f Blue Cross r a i s e d i t s r a t e s i n o r d e r t o pay f o r c l i n i c a l t r i a l s , t hen P r u d e n t i a l would g e t a l l €he Blue Cross s u b s c r i b e r s and Blue Cross would go ou t of bus ines o r v i c e v e r s a . I n t h i s h i g h l y c o m p e t i t i v e b u s i n e s s of hand l ing money f o r h e a l t h c a r e , t hey had t o keep t h e i r r a t e s down o r t hey would l o s e b u s i n e s s and they must t h i n k on ly of t h i s y e a r , n o t 5 y e a r s down t h e road . The people who a r e making t h e s e d e c i s i o n s a r e t h e i n d u s t r i a l people who manage t h e l a r g e amounts of funds

I would l i k e t o propose a s o l u t i o n t o t h e problem which I ' m s u r e w i l l shake up t h i s

Need for Randomized Control Trials 109

f o r t h e b i g in su rance companies. S t e e l o r t h e automobile b u s i n e s s t o come t o t h e meet ing. We should have, because a l l t h e t h i r d p a r t i e s h i d behind t h e f a c t t h a t t hey d i d n ' t make t h e d e c i s i o n bu t t h a t t h e people who pa id t h e premiums made t h e d e c i s i o n s . with a s e n s i b l e recommendation and t h e s e people have gone back t o t h e i r companies t o d i s c u s s it. It has been pub l i shed i n t h e New England J o u r n a l of Medicine [ 6 ] . I t h i n k i t w i l l g e t people t h i n k i n g about t h i s v e r y impor t an t problem.

We propose t h a t t h e t h i r d p a r t y payers should s t o p paying f o r expensive t h e r a p i e s t h a t have no t proven t o be e f f e c t i v e . t h e r a p i e s , or beg in t o pay f o r new a p p l i c a t i o n s of t he rapy i f t hey are p a r t of an RCT. We have k ind led a s p a r k of i n t e r e s t and the t h i r d p a r t i e s r ecogn ize t h a t i f t hey could save money, maybe they could h e l p suppor t r e s e a r c h . f o r c l i n i c a l r e s e a r c h was n o t an anathema t o t h e people managing p a t i e n t c a r e money provided it was n o t an add-on c o s t . They went back wi th t h e idea t h a t t h i s would be i n v e s t i g a t e d f u r t h e r . Apheresis is c e r t a i n l y a good example of an expensive the rapy t h a t might be looked a t i n t h i s manner. NOW t h e r e are l o t s of human r i g h t s problems which would a r i s e . kllat would you do abou t a p a t i e n t who might b e n e f i t from a p h e r e s i s bu t l i v e s i n an a r e a where t h e r e is no t r i a l going on? Well o b v i o u s l y , t h e r e would have t o be some kind of geographic excep t ion and t h e r e would have t o be c o n s i d e r a b l e d e t a i l s worked ou t through t h e Peer Review o r Human Research Review Committee mechanism. But I t h i n k it can be done and I b e l i e v e t h a t i f we had begun t e n y e a r s ago r e q u i r i n g t h a t nobody be plasmapheresed u n l e s s it was done a s p a r t of a RCT, no t only would t h e answer be obvious i n t h e h i g h l y e f f e c t i v e t h e r a p i e s , bu t we would a l s o have had a tremendous amount of i n fo rma t ion about a l l o t h e r d i s e a s e s which might have been b e n e f i t e d .

One could unde r t ake such a s i t u a t i o n i n an e n t i r e l y e t h i c a l manner such a s t h e randomized c o n t r o l t r i a l of myasthenia g r a v i s t h a t was c a r r i e d ou t a t Mount S i n a i , NY. A f t e r a few p a t i e n t s w i th v e r y s e v e r e myasthenia had been t r e a t e d , i t was found t h a t one c o u l d n ' t c o n t i n u e the t r i a l because t h e c o n t r o l s were doing so badly wh i l e t h e plasmapheresed p a t i e n t s seem t o be so much b e t t e r . An e t h i c a l d e c i s i o n was then made t o con t inue t h e t r i a l and t o a l s o s ta r t t r e a t i n g t h e c o n t r o l s . Some people might argue t h a t t h a t was n o t q u i t e warranted a t t h e t i m e and I ' m n o t s u r e it was, bu t i t is p e r f e c t l y unde r s t andab le . It is a p o s s i b i l i t y , provided t h e d e c i s i o n is made by a group of d i s i n t e r e s t e d people who look a t t h e d a t a on t h e p a t i e n t s and have no per sona l b i a s f o r t h e s t u d y one way or t h e o t h e r . It can be done. We've g o t t o do R C T s , e s p e c i a l l y when t h e d i f f e r e n c e s we're looking a t a r e sma l l . Those s m a l l d i f f e r e n c e s a r e impor t an t t o anyone wi th a d i s e a s e . To my mind we on ly have two h u r d l e s t o overcome. One is t o educa te and persuade people t o do good randomized t r i a l s , and t h e second is t o f u r t h e r educa te and persuade people t o pay f o r t h e s e t r i a l s .

We were v e r y n a i v e , we h a d n ' t asked anybody from U.S.

However a t t h i s meet ing, we were a b l e t o come up

They should on ly c o n t i n u e t o pay f o r proven o l d

So, t h e concept of u s i n g p a t i e n t c a r e money

REFERENCES

1.

2.

3.

4. 5.

6.

Berger RL, Live r sage RM, Chalmers TC, Graham J H , McGoldrick DM and Stohlman F : Exchange t r a n s f u s i o n i n t h e t r ea tmen t of f u l m i n a t i n g h e p a t i t i s . N Engl J Med 274:497- 499, 1966. Redeker AG and Yamahiro HS: C o n t r o l l e d t r i a l of exchange-transfusion t h e r a p y i n fu lminan t h e p a t i t i s . Lancet 1:3-6, 1973. Chalmers T C , Smith H, Blackburn B y Silverman B y Schroeder B y Reitman D and Ambroz A: A method f o r a s s e s s i n g t h e q u a l i t y of a randomized c o n t r o l t r i a l . Contr C l i n T r i a l s 2:31-49, 1981. Chalmers TC: Randomization of t h e f i r s t p a t i e n t . Med C l i n N Am 59:1035-38, 1975. Rueff B and Benhamou J P : Acute h e p a t i c n e c r o s i s and fu lminan t h e p a t i c f a i l u r e . Gut 14:805-15, 1973. Chalmers TC, van den Noort S , Lockshin MD and Waksman BH: Summary of a workshop on the r o l e of t h i r d - p a r t y payers i n c l i n i c a l t r i a l s of new a g e n t s ( S p e c i a l Repor t ) . N Engl J Med 309:1334-36, 1983.

110 Chalmers

DISCUSSION

Quest i o n

I and my c o l l e a g u e s a t t h e N I H have g r e a t d i f f i c u l t y w i t h t h e i d e a of randomizing t h e f i r s t p a t i e n t . I would l i k e you t o a d d r e s s two i s s u e s t h a t come up r e c u r r e n t l y . One i s t h e i s s u e i n t h e d i s e a s e p o p u l a t i o n w i t h which you a r e d e a l i n g where t h e r e is an i n a p p a r e n t h e t e r o g e n e i t y of t h e d i s e a s e w i t h i n t h e d i s e a s e a t t h e t i m e t h e s t u d y is conceived. That i s , p a t i e n t s may appear s i m i l a r bu t have d i f f e r e n t r e s p o n s e p o t e n t i a l t o t h e t h e r a p y t o be performed. Second, when we d i s c u s s a p h e r e s i s , t h e r e a r e m u l t i p l e forms of i t . There is a s h o r t d u r a t i o n and a long d u r a t i o n , w i t h and wi thou t v a r i o u s forms of ad juvan t t h e r a p y , such t h a t t h e v a r i e t y of m o d a l i t i e s t h a t can be a p p l i e d a r e a c t u a l l y q u i t e l a r g e so t h a t t he number of c o n t r o l l e d t r i a l s t h a t one is f a c i n g i s q u i t e enormous. I f one is look ing f o r r e l a t i v e l y sma l l d i f f e r e n c e s , t h e number of p a t i e n t s one has t o randomize t o g e t s t a t i s t i c a l l y s i g n i f i c a n t becomes overwhelming. So, many of us f e e l t h a t a s m a l l p i l o t t r i a l t o s o r t o u t some of t h e s e q u e s t i o n s of d o s e , d u r a t i o n and i n t e n s i t y have t o be exp lo red f i r s t .

Answer

Do me a f avor . Would you go back t o your c o l l e g u e s a t N I H and a s k them t h e f o l l o w i n g q u e s t i o n . I n t h e c a s e of unknown h e t e r o g e n i t y of t h e d i s e a s e , how do you t e l l which group of p a t i e n t s might respond and which group might n o t i n t h e case where t h e r e sponse may n o t be overwhelming u n l e s s you do an RCT from t h e beg inn ing? Secondly, e s p e c i a l l y w i t h r e g a r d t o dosage, d u r a t i o n o r f r equency , obv ious ly you want t o choose one t h a t is going t o be t h e b e s t . You are d e a l i n g a g a i n w i t h t h e same end p o i n t a s you a r e when you f i n a l l y do t h e randomized t r i a l which i s t h e t h e r a p e u t i c e f f i c a c y compared t o s i d e e f f e c t s . How t h e n , would you a s k them, do t h e y make t h o s e d e c i s i o n s where t h e r e w i l l be much s m a l l e r d i f f e r e n c e s than t h e o v e r a l l d i f f e r e n c e wi thou t r andomiza t ion?

c r i t i c a l . I t was i l l u s t r a t e d by t h e r e sponse of a surgeon t o me when I advocated t h i s i n terms of a new o p e r a t i o n f o r p o r t a c a v a l s h u n t . Th i s i s a complex o p e r a t i o n i n which you i s o l a t e t h e s p l e e n and do a l e f t s i d e d anas tomos i s between t h e p o r t a l v e s s e l s and t h e main c i r c u i t . I ' m t r y i n g t o pu t t h i s i n non- su rg ica l , non- t echn ica l terms, which is more complex, more d i f f i c u l t t o perform, bu t cou ld have marked advantage over the u s u a l p o r t a c a v a l shunt s u r g e r y i n t h a t it presumably would n o t r e s u l t i n such a h i g h d e g r e e of h e p a t i c encephalopathy. I n o t h e r words, p a t i e n t s would n o t have t h e i r b r a i n damaged by t h e f a c t t h a t t h e i r blood was s h o r t - c i r c u i t i n g t h e l i v e r . I t r i e d t o persuade t h e su rgeons d e v i s i n g t h e s e o p e r a t i o n s t o randomize t h e f i r s t p a t i e n t . They s a i d we neve r cou ld do t h a t a s we have t o work o u t t h e d e t a i l s of t h e o p e r a t i o n and were l i a b l e t o have a h i g h d e a t h r a t e , which they d i d i n t h e beg inn ing . That would be a t e r r i b l e a c t because t h e y would be demons t r a t ing t h a t t h e l e a r n i n g cu rve e x i s t s and would be comparing a l e a r n i n g s i t u a t i o n with an e s t a b l i s h e d o p e r a t i o n i n which t h e r e is o n l y 5-10% m o r t a l i t y . So t h e y had t o develop t h e o p e r a t i o n w i t h a c o n s e c u t i v e series of p a t i e n t s b e f o r e t h e y s t a r t e d randomizing.

Now t h i n k a moment about t h e f u n c t i o n s of informed consen t . How, i f you a r e no t doing an RCT, do you t e l l a p a t i e n t i n 1, 2 , o r 3 pages of w r i t t e n m a t e r i a l t h a t you a r e d e v i s i n g a new o p e r a t i o n and t h a t you want t o t r y t h i s s e r i o u s new o p e r a t i o n o u t on them and you would l i k e t o have t h e i r pe rmis s ion t o do so? 20 p a t i e n t s because you r e a l l y h a v e n ' t worked o u t t h e t h e r a p y y e t , so you are go ing t o l o s e a l o t more of t h e p a t i e n t s i n t h e beg inn ing . You w i l l need t o have e x p e r i e n c e w i t h t h e o p e r a t i o n b e f o r e you can s t a r t randomizing and you would l i k e t h i s p a t i e n t t o v o l u n t e e r t o be one of t hose i n i t i a l 10-20 w h i l e you are s t i l l f i n d i n g o u t the op t ima l t h e r a p y . You would l i k e t h e p a t i e n t t o v o l u n t e e r t o be a s u b j e c t i n t h a t i n i t i a l p i l o t t r i a l so t h a t you can then go on t o h e l p o t h e r people . I t h i n k t h e e t h i c a l answer t o t h a t is obvious. I f t h e informed consent document i s h o n e s t , no one would v o l u n t e e r fo r t h e p i l o t s t u d y . p a t i e n t s would p r e f e r t o be randomly a s s i g n e d t o e i t h e r t h e new t h e r a p y which might t u r n

There is an e t h i c a l r e a s o n f o r randomizing t h e f i r s t p a t i e n t which I t h i n k is

You a r e n o t doing an RCT w i t h t h e f i r s t 10-

Most

Need for Randomized Control Trials 111

out t o be b e t t e r because you might, by luck, have h i t upon the success fu l method, o r be randomly assigned t o s tandard therapy, i n which case you might be the lucky con t ro l . I th ink , in gene ra l , p a t i e n t s and Human Research Review Committees w i l l recognize t h a t i t is twice as e t h i c a l t o do a s tudy i n which you o f f e r a p a t i e n t a 50150 chance of g e t t i n g the s tandard therapy than it is t o assume t h a t t he new therapy is going t o be worse a t f i r s t , then b e t t e r l a t e r . So from the e t h i c a l s t andpo in t , you owe i t t o the p a t i e n t t o randomize.

Comment

D r . Chalmers, i f I understand what you j u s t s a i d c o r r e c t l y , you a r e saying t h a t the e t h i c s of it would d i c t a t e t h a t the p a t i e n t not be involved i n knowledge of t he f a c t , t h a t the learn ing curve does, e x i s t because it would make the choices more d i f f i c u l t . I d o n ' t th ink t h a t changes the e t h i c s of i t . I t seems t o m e t h a t i t j u s t changes the f a c t t h a t we have t o recognize t h a t t he re w i l l not be anything new i n t h a t l earn ing per iod . It makes it e a s i e r t o present i t , but I don ' t th ink i t changes the e t h i c s of i t .

Answer

I th ink it changes the e t h i c s t o the ex ten t t h a t t he p a t i e n t is informed t h a t t he re is a learn ing curve and an oppor tuni ty t o be randomized because the re it may not be a learn ing curve. I n o the r words, t he f i r s t p a t i e n t s may do f i n e . We don ' t know t h a t t he re is a learn ing curve.

Comment

But we do know almost always t h a t t he re i s a learn ing curve.

Answer

I don ' t th ink so. I th ink the re a r e ins tances where, hopefu l ly , t he technique has been worked out in experimental animals wel l enough, s o a learn ing curve should not e x i s t .

Comment

D r . Chalmers, I ' d l i k e t o expand on a remark you made about your c r i t e r i a f o r a good or bad t r i a l . You mentioned t h a t some of the information was the re . In o the r words, the people involved had done it but they hadn ' t repor ted it . I would j u s t l i k e t o make a comment t h a t cont inues t o amaze me t h a t j ou rna l s , p a r t i c u l a r l y very e s t ab l i shed jou rna l s , don ' t have the s o r t of check l i s t t h a t you so b e a u t i f u l l y l a i d out f o r us . It seems t o me tha t the vas t ma jo r i ty of us a r e depending on the jou rna l s t o make t h a t judgement fo r us and it seems we c a n ' t r e l y on them.

Answer

We have a l o t of educat ing t o do. A j ou rna l w i l l expect a biochemist t o descr ibe i n g rea t d e t a i l t he chemical procedure t h a t he went through but tends t o want the author t o leave out d e t a i l s of t he design of the experimental t r i a l .