Are All Significant P Values Created Equal?The Analogy Between Diagnostic Tests and Clinical ResearchWarren S. Browner, MD, MPH, Thomas B. Newman, MD, MPH

Just as diagnostic tests are most helpful in light of the clinical presentation,statistical tests are most useful in the context of scientific knowledge. Knowingthe specificity and sensitivity of a diagnostic test is necessary, but insufficient: theclinician must also estimate the prior probability of the disease. In the same way,knowing the P value and power, or the confidence interval, for the results of aresearch study is necessary but insufficient: the reader must estimate the priorprobability that the research hypothesis is true. Just as a positive diagnostic testdoes not mean that a patient has the disease, especially if the clinical picturesuggests otherwise, a significant P value does not mean that a researchhypothesis is correct, especially if it is inconsistent with current knowledge.Powerful studies are like sensitive tests in that they can be especially useful whenthe results are negative. Very low P values are like very specific tests; both resultin few false-positive results due to chance. This Bayesian approach can clarifymuch of the confusion surrounding the use and interpretation of statistical tests.

(JAMA 1987;257:2459-2463)

IN THE four original contributionsin this issue of The Journal, the au¬thors report the results of statisticaltests of 76 hypotheses.1"4 Of these, 32had significant P values (P<.05). But dothese P values imply that the 32 hypoth¬eses are true? Or that 95% of them aretrue? Are all significant P values cre¬ated equal?The answer to these questions is

"No!" What then is a P value? It is thelikelihood of observing the study resultsunder the assumption that the null hy¬pothesis of no difference is true. Proba¬bly because this definition is elusive andintimidating, understanding P values(and other statistical concepts likepower, confidence intervals, and multi¬ple hypothesis testing) is often left toexperts in the field. It is easier just tocheck whether a P value is .05 or less,call the result "statistically significant,"regard the tested hypothesis as proba¬bly true, and move on to the next para¬graph.Readers of medical literature need

not give up quite so quickly, however. AsDiamond and Forrester5 pointed out,many statistical concepts have remark¬ably similar analogues in an area famil-

iar to clinicians—the interpretation ofdiagnostic tests. In the diagnosis ofCushing's syndrome, for example, mostclinicians recognize that an elevated se¬rum cortisol level is more useful than anelevated blood glucose level, and that anelevated cortisol level is more likely tobe due to Cushing's syndrome in amoon-faced patient with a buffalo humpand abdominal striae than in an over¬weight patient with hypertension.6,7Why? Because the interpretation of atest result depends on the characteris¬tics of both the test and the patientbeing tested.8"13The same type of reasoning—called

Bayesian analysis after Thomas Bayes,the mathematician who developed itmore than 200 years ago14—can also beused to clarify the meaning of theP value and other statistical terms.Although this application ofBayes' ideashas been discussed in epidemiologic andstatistical literature,1518 it has receivedless attention in the journals read byclinicians. In this article, we begin withthe basic aspects of the analogy be¬tween research studies and diagnostictests, such as the similarity between thepower of a study and the sensitivity of atest, and then examine more challeng¬ing issues, such as how a study withmultiple hypotheses resembles a serumchemistry panel.THE ANALOGYAn overview of the analogy between

research studies and diagnostic tests isshown in Table 1. In this analogy, a

clinician obtains diagnostic data to testfor the presence of a disease, such asbreast cancer, and an investigator col¬lects study data to determine the truthof a research hypothesis, such as thatthe efficacies of two drugs differ in thetreatment of peptic ulcer disease. (Theresearch hypothesis is often called thealternative hypothesis in standard ter¬minology.) The absence of a disease (nobreast cancer) is like the null hypoth¬esis of no difference in the efficacy of thetwo drugs.The term "positive" is used in its usual

sense: to refer to diagnostic tests thatare consistent with the presence of thedisease and to studies that have statis¬tically significant results. Similarly,"negative" refers to diagnostic testsconsistent with the absence of diseaseand research results that fail to reachstatistical significance. Thus there arefour possible results whenever a patientundergoes a diagnostic test. Considerthe use of fine-needle aspiration in theevaluation of a breast mass, for example(Table 2). If the patient has breast can¬cer, there are two possibilities: the testresult can either be correctly positive orincorrectly negative. On the otherhand, if the patient actually does nothave cancer, then the result will eitherbe correctly negative or incorrectlypositive. Similarly, there are four possi¬ble results whenever an investigatorstudies a research hypothesis (Table 3).If the efficacies of the two drugs reallydo differ, there are two possibilities: thestudy can be correctly positive if it findsa difference or incorrectly negative if itTable 1.—The Analogy Between Diagnostic Testsand Research Studies

Diagnostic Test Research StudyAbsence of disease Truth of null hypothesisPresence of disease Truth of research

(alternative)hypothesis

Positive result Positive result (reject(outside normal limits) null hypothesis)

Negative result Negative result (fall to(within normal limits) reject null hypothesis)

Sensitivity PowerFalse-positive rate P value

(1-

specificity)Prior probability Prior probability ofof disease research hypothesis

Predictive value of a positive Predictive value of(or negative) test result a positive (or negative)

study

From the Departments of Medicine (Dr Browner),Pediatrics (Dr Newman), and Epidemiology and Inter-national Health (Drs Browner and Newman), School ofMedicine, University of California at San Francisco, andthe Clinical Epidemiology Program, Institute for HealthPolicy Studies, San Francisco (Drs Browner andNewman).Reprint requests to the Division of General Internal

Medicine 111A1, Veterans Administration Medical Cen-ter, San Francisco, CA 94121 (Dr Browner).

Table 2.—The Four Possible Results of a Diagnostic TestIf Breast Mass Is Actually:

Malignant Benign

And Result ofFine-Needle Aspirate is:

Positive This is a true-positivetest: result Is correct

Negative This is a false-negativetest: result is incorrect

This is a false-positivetest: result Is incorrect

This is a true-negativetest: result is correct

Table 3.—The Four Possible Results of a Research StudyIf Research Hypothesis

is Actually:True

(Efficacy of Drug Aand Drug B Differ In

Treatment of Ulcer Disease)

False(Drug A Has SameEfficacy as Drug B

in Treatment of Ulcer Disease)

And Resultof Study Is:

Positive This is a true-positivestudy: result is correct

Negative This is a false-negativestudy: result is incorrect

This is a false-positivestudy: result is incorrect

This is a true-negativestudy: result Is correct

misses the difference. If the two drugsactually have the same efficacy, then thestudy can either be correctly negative ifit finds no difference or incorrectlypositive if it does find one.The relationships between the four

possible outcomes of a diagnostic testare usually expressed as the sensitivityand specificity of the test, which aredetermined by assuming that the pres¬ence or absence of the disease is known.Sensitivity is the likelihood that a testresult will be positive in a patient withthe disease. Specificity is the likelihoodthat a test result will be negative in apatient without the disease. If the re¬sult from a fine-needle aspiration, ispositive in 80 of 100 women with breastcancer, and negative in 95 of 100 womenwithout cancer, the test would have asensitivity of 80% and a specificity of95%. There is another term that isuseful in the analogy: the false-positiverate (1

-

specificity), which is the likeli¬hood that a test result will be (falsely)positive in someone without the dis¬ease. In this example, the false-positiverate is 5%: of 100 women without breastcancer, five will have falsely positive testresults.Similarly, the relationships between

the four possible outcomes of a researchstudy are usually expressed as thepower and P value of the study, whichare determined by assuming that thetruth or falsity of the null hypothesis isknown. Power is the likelihood of astudy being positive if the researchhypothesis is true (and the null hypoth¬esis is false); it is analogous to thesensitivity of a diagnostic test. TheP value is the likelihood of a study beingpositive when the null hypothesis istrue; it is analogous to the false-positiverate (1

-

specificity) of a diagnostic test.A study comparing two drugs in thetreatment of ulcers that has an 80%chance of being correctly positive if

there really is a difference in theirefficacies would have a power of 0.80. Astudy with a 5% chance of being incor¬rectly positive if there is no differencebetween the drugs would have a"P valueof .05. (Conventionally, when theP value is less than a certain predeter¬mined "level of statistical significance,"usually .01 or .05, the results are said tobe "statistically significant.")Knowing the sensitivity and speci¬

ficity of a test is not sufficient, however,to interpret its results: that interpreta¬tion also depends on the characteristicsof the patient being tested. If the pa¬tient is a 30-year-old woman with sev¬eral soft breast masses, a positive resultfrom a fine-needle aspiration (even witha false-positive rate of only 5%) wouldnot suffice to make a diagnosis of cancer.Similarly, if the patient is a 60-year-oldwoman with a firm solitary breast mass,a negative aspirate result (with a sensi¬tivity of 80%) would not rule out malig¬nancy.19 Clinicians use these sorts ofpatient characteristics to estimate theprior probability of the disease—thelikelihood that the patient has the dis¬ease, made prior to knowing the testresults. The prior probability of a dis¬ease is based on the history and physicalfindings, previous experience withsimilar patients, and knowledge ofalter¬native diagnostic explanations. It can bevery high (breast cancer in the 60-year-old woman with a single firm mass),very low (breast cancer in the youngerwoman), or somewhere in between. Al¬though they may not realize it, cliniciansexpress prior probabilities when usingphrases such as "a low index of suspi¬cion" or "a strong clinical impression."In the same way, knowing the power

and the P value of a study is not suffi¬cient to determine the truth of the re¬search hypothesis. That determinationalso depends on the characteristics ofthe hypothesis being studied. Suppose

one drug is diphenhydramine hydro-chloride (Benadryl) and the other ischlorpheniramine maléate (Chlor-Tri-meton): a positive study (at P = .05)would not ensure that one of the drugs iseffective in the treatment of ulcers.Similarly, if one drug was ranitidinehydrochloride (Zantac) and the other aplacebo, a negative study (even withpower of 0.80) would not establish theineffectiveness of ranitidine. The char¬acteristics of a research hypothesisdetermine its prior probability—an es¬timate of the likelihood that the hypoth¬esis is true, made prior to knowing thestudy results. The prior probability of ahypothesis is based on biologic plau¬sibility, previous experience withsimilar hypotheses, and knowledge ofalternative scientific explanations.Analogous to the situation with diag¬nostic tests, the prior probability of aresearch hypothesis can be very high(that an H2-blocker, such as ranitidine,is more effective than placebo in thetreatment of ulcers), very low (that theefficacies of two Hrblockers, such as

diphenhydramine and chlorphenira¬mine, differ in the treatment of ulcerdisease), or somewhere in between. Au¬thors of research reports indicate priorprobabilities with terms like "unantici¬pated" or "expected" when they discusstheir results.The advantage of Bayesian analysis in

interpreting diagnostic tests is that itcan determine what the clinician reallywants to know—the likelihood that thepatient has the disease, given a certaintest result. Bayesian analysis combinesthe characteristics of the patient (ex¬pressed as the prior probability of dis¬ease), the characteristics of the test(expressed as sensitivity and speci¬ficity), and the test result (positive ornegative) to determine the predictivevalue of a test result. The predictivevalue of a positive diagnostic test is theprobability that given a positive result,the patient actually has the disease.(The predictive value ofa negative test isthe probability that given a negativeresult, the patient does not have thedisease.)As an example, recall the 60-year-old

woman with a firm breast mass. Theprior probability that the mass is malig¬nant is moderate, say 50%. A positiveresult from a fine-needle aspirate (witha specificity of 95% and a sensitivity of80% for cancer) results in a very highpredictive value for malignancy, about94% (Figure). Next, consider the 30-year-old woman with multiple softmasses. The prior probability of canceris low, say 1%. Even given a positiveaspirate result, the likelihood that shehas breast cancer is still small (about14%).

09_ Specificity = 0.999^^^"'^ ^^ /^/

£ / / 0.95 / /| 05- / / //0.90

.001 .005 0.1 .05 .1 .5 1.0

Very Low Low Moderate HighPrior Probability

Relationship between prior probability and predictive value of positive result of diagnostic test with sensitivityof 0.80, at several specificities. Figure can also be used to estimate predictive value of positive research studywith power of 0.80 by substituting (1

-

P value) for specificity, and prior probability of hypothesis for priorprobability of disease (see "Limitations" section).

A Bayesian approach can also be usedto determine what the reader of a re¬search study really wants to know—thelikelihood that the research hypothesisis true, given the study results. It com¬bines the characteristics of the hypoth¬esis (expressed as prior probability),the characteristics of the study (ex¬pressed as power and the P value), andthe study results (positive or negative)to determine the predictive value of astudy. The predictive value ofa positivestudy is the probability that given a

positive result, the research hypothesisis actually true. (The predictive value ofa negative study is the probability thatgiven a negative result, the researchhypothesis is false.)The predictive value of a research

study, however, is usually harder toestimate than the predictive value of adiagnostic test (see "Limitations" sec¬tion). Nonetheless, the basic analogyremains valid: the prior probability ofthe hypothesis must be combined withthe power and the P value of the studyto determine the likelihood that theresearch hypothesis is true. In the nextsection, we discuss how this analogy canbe used to understand several statisticalconcepts.IMPLICATIONSSpecificity and the P ValueHow low must a P value be for it to be

accepted as evidence of the truth of a

research hypothesis? This question isanalogous to asking: how high must thespecificity of a test be to accept a

positive test result as evidence of adisease? Requiring that a P value beless than .05 before it is "significant" isas arbitrary as requiring that a diag¬nostic test have a specificity of at least95%. A more important criterion, butone that is not as easy to quantitate, iswhether the results of the study com¬bined with the prior probability of theresearch hypothesis are sufficient tosuggest that the hypothesis is true.Consider the hypothesis, tested in theLipid Research Clinics Primary Preven¬tion Trial,20 that cholestyramine resindecreases the incidence of coronaryheart disease in hypercholesterolemicmen. This research hypothesis had atleast a low to moderate prior proba¬bility, based on previous evidence. Evenwith a "nonsignificant" P value of .094(the two-sided equivalent of the contro¬versial one-sided P = .047 reported bythe investigators), the hypothesis islikely to be true.It is also a mistake to believe a re¬

search hypothesis just because a Pvalue is statistically significant. Con¬sider a study that found that drink¬ing two or more cups of coffee a daywas associated with pancreatic cancer(P<.05).21 This hypothesis had a verylow prior probability: the authors calledthe association "unexpected." Thus,

finding a significant P value did notestablish the truth of the hypothesis;subsequent studies, including one bythe same authors, failed to confirm theassociation.2227Of course, many diagnostic test re¬

sults are not simply reported as "posi¬tive"; they also indicate how abnormalthe result is. The more abnormal thatresult, the less likely that it is just achance finding in a normal person. If theupper limit of normal for a serum thy-roxine level at a specificity of 95% is 12.0u-g/dL (154 nmol/L), then a thyroxinelevel of 18.0 p.g/L (232 nmol/L) is almostcertainly abnormal. The question be¬comes whether it represents hyper-thyroidism, another disease, or a labo¬ratory error. By analogy, if the cutoff forcalling a study positive is a P value lessthan .05, then a P value of .0001 meanschance is an extremely unlikely expla¬nation for the findings. The questionbecomes whether the results indicatethe truth of the research hypothesis orare a result of confounding or bias (see"Laboratory Error and Bias" and "Al¬ternative Diagnoses and ConfoundingExplanations" sections). Because theP value is analogous to the false-positiverate (1

-

specificity), a study with a verylow P value is like a test with very highspecificity: both give few false-positiveresults due to chance, but may requirecareful consideration of other possibleexplanations.Sensitivity and PowerWhen the result of a diagnostic test

that has a high sensitivity is negative,such as a urinalysis in the diagnosis ofpyelonephritis, it is especially useful forruling out a disease. Similarly, when a

powerful research study is negative, itstrongly suggests that the research hy¬pothesis is false. However, if the sensi¬tivity of a test is low, such as a sputumsmear in a patient with possible tuber¬culosis, then a negative result does notrule out the disease.9 In the same way, anegative study with inadequate powercannot disprove a research hypoth¬esis.28,29

Laboratory Error and BiasWhen unexpected or incredible re¬

sults on a diagnostic test are found, suchas a serum potassium level of 9.0 mEq/L(mmol/L) in an apparently well person,the first possibility to consider is labora¬tory error: Was the test adequatelyperformed? Did the sample hemolyze?Was the specimen mislabeled? Simi¬larly, readers of a research study, suchas a trial of biofeedback in the treatmentof hypertension, must always considerthe possibility of bias, especially if thestudy yields surprising results: Was thestudy adequately designed and ex-

ecuted? Did the investigators assignsubjects randomly? Was blood pressuremeasured blindly?30 Improperly per¬formed tests and biased studies do notyield reliable information, no matterhow specific or significant their results.Alternative Diagnoses andConfounding ExplanationsEven if a diagnostic test is adequately

performed, there may be several expla¬nations for the result. An elevated se¬rum amylase level, for example, has ahigh specificity to distinguish patientswho have pancreatitis from those withnonspecific abdominal pain. However,there are extrapancreatic diseases(such as bowel infarction) that elevatethe amylase level and that must beconsidered in the differential diagnosis.In the same way, although a lowP valuemay indicate an association between anexposure and a disease (like the associa¬tion between carryingmatches and lungcancer), a confounder (cigarette smok¬ing) may actually be responsible. Read¬ers of research studies should alwayskeep in mind potential confounding ex¬

planations for significant P values.

Better Tests and Bigger StudiesIncreasing the sample size in a re¬

search study is similar to using a betterdiagnostic test. Better diagnostic testscan have more sensitivity or specificityor both; large studies can have greaterpower or lower levels of statistical sig¬nificance or both. Often the choice of adiagnostic test is a matter of prac¬ticality: biopsies are not feasible in ev¬

ery patient for every disease. Similarly,power or the significance level may bedetermined by practical considerations,since studies of 20000 or more subjectscannot be done for every research ques¬tion. Of course, bigger studies may findsmaller differences, just like bettertests may detect less advanced cases of adisease. A small but statistically signifi¬cant difference in a research study islike a subtle but definite abnormality ona diagnostic test; its importance is amatter of judgment.Intentionally Ordered Tests andProspective HypothesesA positive result on a single inten¬

tionally ordered test is more likely toindicate disease than the same resultthat turns up on a set of routine admis¬sion laboratory tests. Similarly, theP value for a research hypothesis statedin advance of a study is usually more

meaningful than the same P value for ahypothesis generated by the data. Thereason is that clinicians usually ordertests and investigators state hypoth¬eses in advance when the prior proba¬bility is moderate or high. Thus the

predictive values of positive results aregenerally greater for intentionally or¬dered tests and prospectively statedhypotheses.Not all unexpected results, however,

have low prior probabilities. Occasion¬ally, clinicians or investigators are justnot smart or lucky enough to considerthe diagnosis or hypothesis in advance.For example, a house officer caring fora patient with fatigue and vague ab¬dominal symptoms might ignore a se¬rum calcium level of 10.5 mg/dL(2.62 mmol/L) until the attending physi¬cian mentions the possibility of hyper-parathyroidism in rounds the nextmorning. Similarly, researchers mightdisregard the association betweensmoking and cervical cancer until aplausible biologic explanation is sug¬gested.3134 Estimating the prior proba¬bility of a hypothesis on the basis ofwhether it was considered prospec¬tively is a useful, but not infallible,method. The truth, elusive though itsometimes may be, does not depend onwhen a hypothesis is first formulated.Multiple Tests andMultiple HypothesesMost of us are intuitively skeptical

when one of50 substances on a checklistis associated with a disease at P<.05because of the likelihood of finding suchan association by chance alone. A stan¬dard technique for dealing with thisproblem of testing multiple hypothesesis to use a more stringent level of statis¬tical significance, thus requiring a lowerP value.36,36 This approach is simple andpractical, but it leads to some unsatisfy¬ing situations. It seems unfair, for ex¬ample, to reduce the required signifi¬cance level for a reasonable hypothesisjust because other, perhaps ridiculous,hypotheses were also tested. What ifthe disease was mesothelioma and oneof the exposures was asbestos: should amore stringent level ofstatistical signifi¬cance be required because 49 other sub¬stances were also included? Should thelevel of significance be reduced whentesting the main hypothesis of a studywhenever additional hypotheses areconsidered? Need statistical adjust¬ments for multiple hypothesis testingbe made only when reporting all of thehypotheses in a single publication?This vexing problem of multiple hy¬

pothesis testing resembles the inter¬pretation of a serum chemistry panel.When a clinician evaluates a patientwith a swollen knee, a serum uric acidlevel of 10.0 mg/dL (0.6 mmol/L) has thesame meaning no matter how manyother tests were also performed on thespecimen by the autoanalyzer. How¬ever, an unanticipated abnormal valueon another test in the panel is likely to

be a false-positive: that is because thediseases itmight represent usually havelow prior probabilities, not because sev¬eral tests were performed on the same

sample of serum. Similarly, testing mul¬tiple hypotheses in a single study causesproblems because the prior proba¬bilities of such hypotheses tend to below: when investigators are not sure ofwhat they are looking for, they testmany possibilities. The solution is torecognize that it is not the number ofhypotheses tested, but the prior proba¬bility of each of them, that determineswhether a result is meaningful.37Confirmatory Tests and PooledStudiesWhen a single diagnostic test is insuf¬

ficient to make a diagnosis, additionaltests are often ordered, some results ofwhich may be positive and some nega¬tive. The clinician revises the proba¬bility of the disease by combining theseresults, often weighting them by thetests' characteristics. In a patient with aswollen leg, for example, a normal re¬sult from a Doppler study would lowerthe probability of deep venous throm¬bosis, but an abnormal result of a fi-brinogen scan might raise it sufficientlyto make the diagnosis. In the same way,it may be necessary to combine theresults of several research studies,weighting them by the characteristicsof each study. This process, known as

pooling, allows studies with both signif¬icant and nonsignificant P values tochange incrementally the likelihoodthat a research hypothesis is true. How¬ever, just as only those tests that arerelevant to the diagnosis in questionshould be combined, only those re¬search studies that address the sameresearch hypothesis should be pooled.Confidence IntervalsThere is no ready diagnostic test

analogy for confidence intervals fromresearch studies (the concept of testprecision comes closest). But becauseconfidence intervals are commonly mis¬interpreted as expressions of predictivevalue, they merit a short discussion.The term "confidence interval" is unfor¬tunate, because it leads many people tobelieve that they can be confident thatthe interval contains the true value be¬ing estimated. Actually, confidence in¬tervals are determined entirely by thestudy data: the prior probability thatthe true value lies within that interval isnot at all considered in the calculations.A 95% confidence interval is simply therange of values that would not differfrom the estimate provided by the studyat a statistical significance level of.05.38,39Confidence intervals are useful be-

cause they define the upper and lowerlimits consistent with a study's data.But they do not estimate the likelihoodthat the results of the research arecorrect. A confidence interval providesno more information about the likeli¬hood of chance as an explanation for afinding than does a P value.40 As anexample, suppose a well-designed studyfinds that joggers are twice as likely as

nonjoggers to develop coronary heartdisease, with a 95% confidence intervalfor the relative risk of 1.01 to 3.96. (Thisis equivalent to rejecting the null hy¬pothesis of no association between jog¬ging and heart disease at P = .05.)Despite a 95% confidence interval thatexcludes 1.0, there is obviously not a95% likelihood that joggers are at anincreased risk of coronary heart dis¬ease. There are many other studies thathave found that exercise is associatedwith a reduced risk of heart disease.Given the low prior probability of thehypothesis that jogging increases therisk of coronary heart disease, chance(or perhaps bias) would be a more likelyexplanation for the results.LIMITATIONSWhile it provides several useful in¬

sights, the analogy between diagnostictests and clinical research is not perfect.It is easier to determine the prior prob¬ability of a disease, based on the preva¬lence of the disease in similar patients,than the prior probability of a hypoth¬esis, based on the prevalence of thetruth of similar hypotheses. Similarityin patients can be defined by character¬istics known to be associated with adisease, such as age, sex, and symp¬toms.11 But what defines similar hy¬potheses? Thus the prior probability ofmost research hypotheses tends to be a

subjective estimate (although, in prac¬tice, estimates of the prior probability ofa disease are generally subjective aswell).Second, as long as there is a gold

standard for its diagnosis, a disease iseither present or absent: there are onlythese two possibilities. If a group ofpatients known to have the disease isassembled, a single value for the sensi¬tivity of a test can be determined em¬pirically. But there is no single value forthe power of a research study: it de¬pends on the sample size, as well as themagnitude of the actual difference be¬tween the groups being compared. Astudy comparing IQ in internists andsurgeons, for example, might have apower of only 50% to detect a differencebetween them if surgeons actuallyscored five points higher than inter¬nists, but a power of 98% if surgeonsactually scored ten points higher. Sincethe actual difference is unknown, a

unique value for power cannot be calcu¬lated.

CONCLUSIONSClinicians do not simply decide that a

patient has a disease when a diagnostictest result is positive or rule out thedisease when the test result is negative.They also consider the sensitivity andspecificity of the test and the character¬istics of the patient being tested. In thesame way, readers should not believe ordisbelieve the research hypothesis of astudy on the basis of whether the re¬sults were statistically significant. Theyshould also take into account the study'spower and P value and the characteris¬tics of the hypothesis being tested.Thus, all significant P values are not

created equal. Just as the accuracy of adiagnosis depends on how well the clini¬cian has estimated the prior probabilityand considered alternative diagnosesand laboratory errors, the interpreta¬tion of a research study depends on howwell the reader has estimated the priorprobability and considered confoundersand biases. Knowing the power andP value (or the confidence interval) for astudy's results, like knowing the sensi¬tivity and specificity of a diagnostic test,is necessary but not sufficient. ThisBayesian approach requires the activeparticipation of the reader and empha¬sizes the importance of scientific con¬text in the interpretation of research.This project was supported by a grant from the

Andrew W. Mellon Foundation.

References

1. Schade DS, Mitchell WJ, Griego G: Addition ofsulfonylurea to insulin treatment in poorly con-trolled type II diabetes: A double-blind, ran-domized clinical trial. JAMA 1987;257:2441-2445.2. Cramer DW, Goldman MB, Schiff I, et al: Therelationship of tubal infertility to barrier methodand oral contraceptive use. JAMA 1987;257:2446-2450.3. Bennett KJ, Sackett DL, Haynes RB, et al: Acontrolled trial of teaching critical appraisal of theclinical literature to medical students. JAMA1987;257:2451-2454.4. Chaiken BP, Williams NM, Preblud SR, et al:The effect of a school entry law on mumps activityin a school district. JAMA 1987;257:2455-2458.5. Diamond GA, Forrester JS: Clinical trials andstatistical verdicts: Probable grounds for appeal.Ann Intern Med 1983;98:385-394.6. Nugent CA, Warner HR, Dunn JT, et al: Proba-bility theory in the diagnosis of Cushing's syn-drome. J Clin Endocrinol Metab 1964;24:621-627.7. Crapo L: Cushing's syndrome: A review of diag-nostic tests. Metabolism 1979;28:955-977.8. Vecchio TJ: Predictive value of a single diag-nostic test in unselected populations. N EnglJMed1966;274:1171-1173.9. Boyd JC, Marr JJ: Decreasing reliability of acid\x=req-\fast smear techniques for detection of tuberculosis.Ann Intern Med 1975;82:489-492.10. Jones RB: Bayes' theorem, the exercise ECG,and coronary artery disease. JAMA 1979;242:1067-1068.U. Diamond GA, Forrester JS: Analysis of proba-bility as an aid in the clinical diagnosis of coronary-artery disease. N Engl J Med 1979;300:1350-1358.

12. Griner PF, Mayewski RJ, Mushlin AI, et al:Selection and interpretation of diagnostic tests andprocedures: Principles and applications. Ann In-tern Med 1981;94:553-600.13. Havey RJ, Krumlovsky F, delGreco F, et al:Screening for renovascular hypertension: Is renaldigital-subtraction angiography the preferred non-invasive test? JAMA 1985;254:388-393.14. Bayes T: An essay towards solving a problem inthe doctrine of chances. Philos Trans R Soc Lond1763;53:370-418.15. Phillips LD: Bayesian Statistics for SocialScientists. New York, Crowell, 1974.16. Donner A: A Bayesian approach to the in-terpretation of subgroup results in clinical trials.J Chronic Dis 1982;35:429-435.17. Pater JL, Willan AR: Clinical trials as diag-nostic tests. Controlled Clin Trials 1984;5:107-113.18. Thomas DC, Siemiatycki J, Dewar R, et al: Theproblem of multiple inferences in studies designedto generate hypotheses. Am J Epidemiol 1985;122:1080-1095.19. Mushlin AI: Diagnostic tests in breast cancer:Clinical strategies based on diagnostic proba-bilities. Ann Intern Med 1985;103:79-85.20. Lipid Research Clinics Program: The LipidResearch Clinics Coronary Primary PreventionTrial results: I. Reduction in incidence of coronaryheart disease. JAMA 1984;251:351-364.21. MacMahon B, Yen S, Trichopolous D, et al:Coffee and cancer of the pancreas. N Engl J Med1981;304:630-633.22. Jick H, Dinan BJ: Coffee and pancreatic cancer.Lancet 1981;2:92.23. Goldstein HR: No association found betweencoffee and cancer of the pancreas. N Engl J Med1982;306:997.24. Wynder EL, Hall NEL, Polansky M: Epi-demiology of coffee and pancreatic cancer. CancerRes 1983;43:3900-3906.25. Kinlen LJ, McPherson K: Pancreas cancer andcoffee and tea consumption: A case-control study.Br J Cancer 1984;49:93-96.26. Gold EB, Gordis L, Diener MD, et al: Diet andother risk factors for cancer of the pancreas.Cancer 1985;55:460-467.27. Hsieh C, MacMahon B, Yen S, et al: Coffee andpancreatic cancer (chapter 2). N Engl J Med1986;315:587-589.28. Frieman JA, Chalmers TC, Smith H Jr, et al:The importance of beta, type II errors and samplesize in the randomized control trial: Survey of 71'negative' trials. N Engl J Med 1978;299:690-694.29. Young MJ, Bresnitz EA, Strom BL: Samplesize nomograms for interpreting negative clinicalstudies. Ann Intern Med 1983;99:248-251.30. Sackett DL: Bias in analytic research. JChronic Dis 1979;32:51-63.31. Winkelstein W Jr: Smoking and cancer of theuterine cervix: Hypothesis. Am J Epidemiol1977;106:257-259.32. Wright NH, Vessey MP, Kenward B, et al:Neoplasia and dysplasia of the cervix uteri andcontraception: A possible protective effect of thediaphragm. Br J Cancer 1978;38:273-279.33. Harris RWC, Brinton LA, Cowdell RH, et al:Characteristics of women with dysplasia or car-cinoma in situ of the cervix uteri. Br J Cancer1980;42:359-369.34. Lyon JL, Gardner JW, West DW, et al: Smok-ing and carcinoma in situ of the uterine cervix. AmJ Public Health 1983;73:558-562.35. Godfrey K: Comparing the means of severalgroups. NEngl J Med 1985;313:1450-1456.36. Cupples LA, Heeren T, Schatzkin A, et al:Multiple testing of hypotheses in comparing twogroups. Ann Intern Med 1984;100:122-129.37. Cole P: The evolving case-control study. JChronic Dis 1979;32:15-27.38. Fleiss JL: Statistical Methods for Rates andProportions, ed 2. New York, John Wiley & SonsInc, 1981, p 14.39. Rothman K: A show of confidence. N Engl JMed 1978;299:1362-1363.40. Browner WS, Newman TB: Confidence inter-vals. Ann Intern Med 1986;105:973-974.