JOURNAL OF CHILD AND ADOLESCENT PSYCHOPHARMACOLOGYVolume 6, Number 4, 1996Mary Ann Liebert, Inc.Pp. 215-228
Double-Blind Versus Open Evaluations ofStimulant Drug Response in Children withAttention-Deficit Hyperactivity Disorder
JOYCE SPRAFKIN, Ph.D. and KENNETH D. GADOW, Ph.D.
Although placebo controls and double-blind conditions are considered to be essential for theunbiased scientific assessment of drug effects, there is very little research on these proce-dures in the pdiatrie psychopharmacology literature. To examine the impact of controlledassessment procedures on the magnitude of observed drug effects, two groups of childrenwith attention-deficit hyperactivity disorder (ADHD) were evaluated for response tomethyIplienidate under two different assessment procedures. One group (n = 33) was partof a placebo-controlled, double-blind crossover research protocol, with randomized dosesequences, compliance checks, numerous dependent measures, written informed consent,and a considerable amount of staff involvement. The other group (n = 43) received phar-macotherapy at a community-based child psychiatry outpatient service where they were fol-lowed in a routine clinical manner, with "no treatment" as the only control condition, stan-dard fixed-dose titration, parental responsibility for data collection, use of form letters, andminimal staff involvement. Each individual in both groups received divided doses of 0.3 and0.5/0.6 mg/kg daily for a minimum of 1 week at each dose. Comparisons of teacher ratingsobtained for the two assessment procedures revealed highly similar findings. The results ofthis study are discussed with regard to both their methodological and clinical implications.
For research PSYCHOPHARMACOLOGiSTS, placebos and double-blind conditions are sine qua non for theconduct of scientific investigations and the satisfactory documentation of the safety and efficacy of ther-apeutic agents. In fact, it would be pure heresy to even suggest that there may be situations where suchcontrols are not necessary. Many clinical psychopharmacologists, most notably those who have adopted ap-plied behavioral analysis methodologies (behavioral pharmacologists), have even argued that placebo con-trol and double-blind procedures should be incorporated into routine clinical evaluations to ensure reliableand valid assessment of drug response (Ullmann and Sleator 1986).
Some of the earliest examples of placebo controls in drug research were studies of the effects of ethanoland caffeine in normal volunteers (e.g., Gilliland and Nelson 1934, Hollingworth 1923, Rivers 1908). Itwas not until the 1960s, however, that they actually became commonplace. The use of a placebo prepara-
Department of Psychiatry and Behavioral Science, State University of New York, Stony Brook, NY.
SPRAFKIN AND GADOW
tion in a clinical pharmacological study is a direct application of John Stuart Mill's method of difference,which maintains that:
If an instance in which the phenomenon under investigation occurs, and an instance in which it does notoccur, have every circumstance in common save one, that occurring in the former; the circumstances inwhich alone the two instances differ, is the effect, or the cause, or an indispensable part of the cause, ofthe phenomenon (Mill JS, A System ofLogic (Vol. 1), p. 458, quoted by Cohen and Nagel, 1934, p. 256).
Although it is impossible to create identical circumstances in a drug study, investigators are in general agree-ment that every effort should be made to achieve this goal, especially with regard to variables that are knownto influence drug response. Variables that could conceivably be relevant, but which are nevertheless be-yond manipulation, are conveniently ignored. This is best illustrated by comparing procedures for evaluat-ing pharmacological versus behavioral interventions (Gadow et al. 1986b, Gadow 1985).
One of the earliest drug studies to use double-blind procedures was an experiment to determine if theanesthetic properties of bottled ether were in fact more powerful than ether purchased in large drums (Goldand Gold 1935). Another early double-blind study examined the effects of amphetamine sulfate on the cog-nitive performance of juvenile delinquents (Molitch and Eccles 1937). During the 1950s, several psychol-ogy experiments were conducted illustrating how instructions provided by, or the behavior of the experi-menter (whether intentional or not), could influence the outcome of an experiment (Gadow et al. 1986b).In time, psychopharmacologists began to use double-blind procedures, and by the late 1960s, a growingnumber of scientists believed that their use should become a mandatory requirement for publication.
In spite of their now obvious benefits for obtaining reliable and valid information in a drug evaluation,placebo controls and double-blind procedures are not without limitations, controversy, and confusion. Inprinciple, placebo preparations should be identical to the experimental drug formulation in every way (color,size, shape, taste) except for the "active" ingredient. This sounds relatively straightforward but can be dif-ficult to achieve in actual practical application. Even more problematic for researchers is the conceptual-ization of the placebo (e.g., Gadow et al. 1986a, Jospe 1978, Grunbaum 1981, White et al. 1985), particu-larly the notion of active and inactive ingredients. Numerous studies have shown that double-blind conditionsare rarely maintained because the care provider and patient can often differentiate treatment from placeboconditions. In fact, if this were not the case, one might seriously question the efficacy of the treatment.Double-blind procedures are not only difficult to maintain, but there is relatively little consensus as to whatconstitutes a "blind" state. For example, in some research protocols, only the manufacturer knows the iden-tity of the treatment formulation, whereas in others, some individuals actually involved in patient manage-ment (but not all) know or have access to the treatment code. Blindness is therefore a continuum that rangesfrom the highly rigorous standards that characterize most federally funded drug studies to the poorly con-trolled efforts of the dilettante. The reluctance of many scientific journals to publish single-blind and opentrials no doubt contributes to authors' obfuscations and ambiguities in procedural descriptions and, at times,exaggerated claims of methodological rigor (see Aman and Singh 1986).
These commonly recognized limitations notwithstanding, it is generally believed that the failure to useplacebos and double-blind conditions results in erroneous conclusions about drug effects, most typically,exaggerated claims of efficacy. Although we do not question either the purpose or importance of controlledtrials, it is noteworthy that in the child psychopharmacology literature, there is almost no research on therelationship between methodological control and the perceived magnitude of drug effects, and the little re-search that has been done has yielded mixed results. Sulzbacher (1973) published a widely cited review ar-ticle comparing the results of 208 studies that used controlled conditions (placebos, blind evaluations) and548 uncontrolled studies. He concluded that the likelihood of obtaining statistically significant results wascomparable for both controlled and uncontrolled studies. However, he did note that when "professionalopinion" or behavior ratings scales were the dependent variable (as compared with standardized psycho-logical tests or direct measurements of behavior), favorable drug effects were much more likely to be re-ported. Kavale (1982) examined the findings from 135 stimulant drug studies of children with MBD/hy-peractivity and concluded that the effect sizes for controlled and uncontrolled studies were comparable. Inother words, the degree of methodological rigor was not an important variable with regard to assessing stim-
STIMULANTS AND ADHD
ulant drug effects. Nevertheless, there really is very little systematic research on this topic in the pdiatriepsychopharmacology literature.
We have conducted a "natural" experiment comparing the results of two different procedures for evalu-ating response to drug treatment that were used in the same child psychiatry outpatient service. One pro-cedure ("controlled experiment") involved placebos, double-blind conditions, and a highly rigorous, labor-intensive assessment battery that was part of a research protocol. The second procedure ("medication clinic")did not use placebos and relied on parents to supervise the collection of drug response data. The primarygoal for comparing the findings from the controlled experiment and the medication clinic was to determineif the latter was generating information about stimulant drug effects that was consistent with a more sci-entifically rigorous procedure.
SubjectsThe controlled experiment sample consisted of 33 boys between 5 and 13 years old (mean SD, 9.03
2.02) who were referred for psychiatric evaluation (often at the urging of the school personnel) for behav-ior problems or academic difficulties and who participated in one of several research protocols that exam-ined the safety and efficacy of methylphenidate for the treatment of ADD/ADHD (Gadow et al. 1990,Gadow et al. 1995, Sverd et al. 1989). The distribution of race was as follows: Caucasian = 88%, AfricanAmerican = 9%, and other = 3%. All controlled research subjects were diagnosed as meeting DSM-III(American Psychiatric Association 1980) criteria for attention deficit disorder (ADD) with or without hy-peractivity or DSM-III-R (American Psychiatric Association 1987) criteria for attention-deficit hyperac-tivity disorder (ADHD) by a board certified child psychiatrist with 20 years research and clinical experi-ence with hyperactive children. These diagnoses were made on the basis of unstructured clinical interviewswith the parent (typically the mother), a mental status examination of the child, and an extensive battery ofteacher- and parent-completed behavior rating scales for assessing ADD/ADHD symptoms as part of a com-prehensive clinical evaluation.
The medication clinic sample was comprised of 42 boys and 1 girl between 4 and 12 years old (8.20 1.95) who were evaluated for behavior or learning problems in a child psychiatric outpatient clinic, werediagnosed with ADHD, and then referred to the medication clinic for a trial of methylphenidate. Eighty-one percent of this sample were Caucasian, 7% were African American, and 12% were of other racial/eth-nic backgrounds.
The ADHD measures employed in both the controlled experiment and medication clinic diagnosticevaluations included the IOWA Conners Teacher's Rating Scale (Loney and Milich 1982), the Primaryand Secondary Symptom Checklist (PSSC) (Loney 1984) completed by the mother, and the parent andteacher versions of the Child Symptom Inventory-3R (CSI-3R) (Gadow and Sprafkin 1994). TheCSI-3R is a rating scale based on DSM-III-R (American Psychiatric Association 1987) symptoms for13 psychiatric disorders and is used as a symptom screening device. Items are scored on a 2-point scale(never or sometimes = 0; often or very often =1). Cutoff scores are based on the number of symp-toms indicated by DSM-III-R as being necessary for a diagnosis but are used only to indicate the pres-ence and the relative frequency of these behaviors in the school and home setting. The CSI-3R hasbeen shown to be a useful clinical tool for guiding a comprehensive clinical interview (Grayson andCarlson 1991). Early in the study, some of the children were evaluated using a DSM-III version ofthis checklist (Gadow and Sprafkin 1994), which was designed, structured, and scored in a similarmanner.
All children from the medication clinic and all but 1 boy from the controlled experiment sample scoredabove cut-off on at least one of the two teacher measures of hyperactivity/ADHD: the Inattention-Overactivity subscale of the IOWA Conners Teacher's Rating Scale (clinical cut-off score >7) and theADHD index of the CSI-3R (score >7). The boy who was not rated as hyperactive by his teacher wasabove cut-off on both parent measures of hyperactivity/ADHD. With the exception of 1 boy in the con-trolled experiment sample and 2 children in the medication clinic sample, all of the children scored above
SPRAFKIN AND GADOW
cut-off on at least one of the two parent measures of hyperactivity/ADHD: the Hyperactivity subscale fromthe PSSC (score >2) and the ADHD index of the CSI-3R (score >7). The means and SDs for each of thesehyperactivity/ADHD measures for each sample is shown in Table 1. T tests revealed no significant (p 4), the Oppositional Defiant Disorder index of the CSI-3R (score >4), or the ConductDisorder index of the CSI-3R (score >2). The means and SDs for each of these aggression measures areshown in Table 1. T tests revealed no significant differences between the controlled experiment and med-ication clinic samples. However, one measure yielded a marginally significant difference; the IOWA-Aggression score was slightly higher for the controlled experiment than medication clinic sample, p < 0.08.
Similarly, 77% of the controlled experiment sample and 56% of the medication clinic sample were abovecut-off on at least one of the parent measures of aggression: the Aggression subscale of the PSSC (score>2), the Oppositional Defiant Disorder index of the CSI-3R (score >4), or the Conduct Disorder index ofthe CSI-3R (score >2). T tests on the means (shown in Table 1) of the two samples did not reveal any sig-nificant differences on these measures.
In the controlled experiment sample, 18 of the 33 children had comorbid Tourette's disorder or chronicmotor tic disorder (see Gadow et al. 1992, Gadow et al. 1995, Sverd et al. 1989). There was no systematictic status evaluation for the medication clinic sample, although it is well documented that approximatelyone-third of clinic-referred ADHD children exhibit one or more motor or vocal tics (Conners 1970, Muniret al. 1987).
Table 1. Sample CharacteristicsMeans (and SDs)
Controlled MedicationVariable experiment clinic p