Hypothesis Testing - ?· Non‐Commercial Share Alike License, which permits unrestricted use, distribution,…

  • View
    213

  • Download
    0

Embed Size (px)

Transcript

  • Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page1

    HypothesisTesting

    LearningObjectives

    Aftercompletingthismodule,thestudentwillbeableto

    carryoutastatisticaltestofsignificance calculatetheacceptanceandrejectionregion calculateandinterpretthepvalueofastatistical

    test

    calculateandinterprettype1andtype2errors calculatethepowerofatest

    KnowledgeandSkills

    Concepts:nullhypothesis,alternative,teststatistic,rejectionregion,acceptanceregion,pvalue,significancelevel,type1error,type2error,falsepositive,falsenegative,powerofatest

    Resamplingmethod Fishersexacttest

    Prerequisites

    binomialdistribution hypergeometricdistribution Normaldistribution Sampleaverage Samplestandarddeviation macrosinExcel

  • Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page2

    PrologueTheproblemofdecisionmakingisubiquitous.Almostdaily,youcanreadinthenewsaboutstudiesthatleadtorecommendationsbasedonstatisticalevidence.TheU.S.DepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality(http://www.ahrq.gov/)provideshealthcarerecommendations,forinstance,throughitsU.S.PreventiveServicesTaskForce(http://www.ahrq.gov/clinic/uspstfix.htm),anindependentpanelofexpertsinprimarycareandprevention,whichreviewsresearchresultsanddevelopsrecommendations.Theserecommendationsarebasedonanalysesoftensorhundredsofclinicalstudies,andrecommendationsmaychangeasnewevidenceaccumulatesovertime.

    Frequently,clinicalstudiesarephrasedintermsofhypothesistesting.Forinstance,ifanewtreatmentforadiseaseisdeveloped,wemightwishtoknowwhetheritperformsbetterthanthecurrenttreatment.Wesetupaclinicaltrialwherepatientsarerandomlyassignedtooneortheothertreatment.Wethencomparethenumberofsuccessfultreatmentsineachgroup.Letsassumethatthetwogroupshavethesamenumberofpatients.Inordertoconcludethatthenewtreatmentisbetterthanthecurrenttreatment,wewouldneedtodemonstratethatthenumberofsuccessfultreatmentsinthenewtreatmentgroupislargerthanthenumberofsuccessfultreatmentsinthecurrenttreatmentgroup.Thequestionishowmuchlargerthenumberofsuccessfultreatmentsinthenewtreatmentgroupwouldneedtobetoconvinceotherinvestigatorsthatthenewtreatmentisindeedbetter.Thesekindsofquestionscanbeansweredwithintheframeworkofhypothesistesting.

    InclassActivity1

    Assumethecurrenttreatmentforadiseaseissuccessfulin30%ofallcases.Anewtreatmentisbeingdevelopedandapreliminaryclinicaltrialshowedthat5outof10patientsweresuccessfullytreated.Canyouconcludethatthenewtreatmentismoresuccessful?

    Ifthenewtreatmentwasnotbetterthanthecurrenttreatment,wewouldhypothesizethatthenewtreatmenthasprobability0.3ofbeingsuccessful.Alternatively,ifthenewtreatmentisbetterthanthecurrenttreatment,wewouldhypothesizethatthenewtreatmenthasprobabilitygreaterthan0.3ofbeingsuccessful.

    Ifthenewtreatmenthasthesamelikelihoodofsuccessthanthecurrenttreatment,namelyprobability0.3,thenthenumberofpatientsinthesmallclinicaltrialwhoaretreatedsuccessfullyunderthenewtreatmentisbinomiallydistributedwith10trialsandsuccessprobability0.3.ThefollowingtablewascreatedinEXCELusingtheBINOMDISTfunctionandshowsthisprobabilitydistribution:

  • Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page3

    x P(X=x)

    0 0.02821 0.12112 0.23353 0.26684 0.20015 0.10296 0.03687 0.00908 0.00149 0.0001

    10 0.0000

    Weseethattheprobabilityoffiveormoresuccesseswhenthesuccessprobabilityis0.3is

    0.1029 0.0368 0.0090 0.0014 0.0000 0.1502+ + + + =

    Thus,itisnotunlikelytosee5(ormore)outof10patientsrecoverwhenthesuccessprobabilityofrecoveryis0.3.Weconcludethatthereisnotenoughevidencetoconcludethatthenewtreatmentisbetter.

    Discussinyourgroupthefollowingquestions:

    1. Whydidweadduptheprobabilitiesintheaboveexample?2. Wouldyoubeabletoconcludedefinitivelyfromthisstudythatthenewtreatmentisntanybetter?3. Whatwouldbeyournextstepindeterminingwhetherthenewtreatmentisbetter?

  • Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page4

    InclassActivity2

    Supposeyouhaveacoininyourpocket.Youwanttodecidewhetherthecoinisfairorbiased.Youhypothesizethatthecoinisfair.Totestthishypothesis,youtossthecoin30times.Thenumberofheadsisbinomiallydistributedwiththenumberoftrialsbeing30andtheprobabilityofheads(success)being0.5.Belowisthehistogramoftheprobabilitydistribution.

    Supposetheexperimentresultedin18headsand12tails.Discussthefollowingquestionsinyourgroup:

    1. Whatcanyousayaboutthecoin?Isitafaircoinorabiasedcoin?2. Whatwouldyourconclusionbeiftheexperimentresultedin24headsand6tails?3. Whatcriteriadidyouusetomakethedecisionineachofthetwocases?4. Canyoubesurethatyourdecisioniscorrect?

  • Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page5

    SomeTheoryInbothInclassActivities,youhadtomakeadecisionbetweentwoalternatives.Inthefirstcase,youneededtodecidewhetherthenewtreatmentwasbetterthanthecurrenttreatment.Inthesecondcase,youneededtodecidewhetherthecoinwasfairorbiased.Inbothcases,youreliedonaprobabilitymodel,andyoubasedyourdecisiononhowlikelytheoutcomeoftheexperimentwascomparedtotheexpectationofthemodel.Inbothcases,therewasalsothepossibilitythatyouarrivedatthewrongdecision.

    Inthefollowing,wewilldiscussthebasicelementsofhypothesistesting.Wewillusetheexampleofthefaircoinversusthebiasedcoinbecauseofitssimplicity.Onehypothesisisthatthecoinisfair,thatis,thattheprobabilityofheadsis0.5.Thealternativehypothesisisthatthecoinisbiased,thatis,theprobabilityofheadsisdifferentfrom0.5.Webaseourdecisionofwhetherornotthecoinisfaironcomparingtheresultofourexperimenttowhatweexpectbasedonaprobabilisticmodel.Namely,ifthefractionofheadsintheexperimentiscloseto1/2,theexperimentprovidesevidenceforthecoinbeingfair;ifthefractionofheadsiseitherloworhigh,theexperimentprovidesevidenceforthecoinbeingbiased.

    Thehypothesisthecoinisfairiscalledthenullhypothesisandisdenotedby 0H .Thealternativethe

    coinisbiasedisdenotedby 1H .(Wewillsaymoreaboutwhichofthetwohypothesesisthenull

    hypothesisandwhichisthealternativelater.)Wesummarizethisas

    =

    0

    1

    : 0.5

    : 0.5

    H p

    H p

    Wedesignedanexperimentinwhichwetossedthecointhirtytimes.Thedatacollectedintheexperimentprovidedevidencefororagainstthenullhypothesis.Thedatainourexperimentwerethesequenceofheadsandtailsinthethirtytrials.Thedatasuggestthatwecancalculateasinglenumber,namelythenumberofheads,whichwecancompareagainstwhatwewouldexpectunderthenullhypothesis.Thissinglenumberiscalledtheteststatistic.Aprobabilisticmodelfortossingafaircoinallowsustocalculatetheprobabilitydistributionoftheteststatistic.Namely,underthenullhypothesis,thenumberofheadsisbinomiallydistributedwith30trialsandsuccessprobability = 0.5p .Inthe

    experiment,weobserved18heads.Howlikelyisitthatweobserve18ormoreheads?IfXdenotesthe

    numberofheads,weareaskingfor ( 18)P X ,whichcanbecalculatedbyaddinguptheprobabilitiesof

    theevents{ } { } { }= = =18 , 19 ,... 30X X X .Refertothespreadsheet(tabFairCoin)toverifythat

    =( 18) 0.1808P X

  • Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page6

    Sincethealternativeistwosided