MethodsWorkshopfortheMinistryofHealthandLong-TermCare

AssessingResearchProtocols:MixedMethodsResearchBy:YelenaPetrosyan,MD

Mixedmethods:Definition

Mixedmethodsconsistsofthecollectionoranalysisofbothquantitativeandqualitativedatainasinglestudyinwhichthedataarecollectedconcurrentlyorsequentially,aregivenapriority,andinvolvetheintegrationofdataatoneormorestagesintheprocessofresearch.Themajormixedmethodsdesigns

1. Theconvergentparalleldesign

Theconvergentparalleldesign(convergent/triangulationdesign)occurswhentheresearchersuseconcurrenttimingtoimplementthequantitativeandqualitativestudiesduringthesamephaseoftheresearchprocess.Thetwomethodsinthisdesignhaveanequalprioritysothatbothplayanequallyimportantroleinaddressingtheresearchproblem.Thisdesignkeepsthestudiesindependentduringthedatacollectionandanalysisandthenmixesormergestheresultsduringtheoverallinterpretation.Thepurposefortheconvergentdesign

• Totriangulatethemethodsbydirectlycomparingandcontrastingquantitativeresultswithqualitativefindingsforcorroborationandvalidationpurposes;

• Toillustratequantitativeresultswithqualitativefindingstodevelopamorecompleteunderstandingofaphenomenon.

Advantagesoftheconvergentdesign

• Itisanefficientdesign,inwhichbothtypesofdataarecollectedduringonephaseoftheresearchatroughlythesametime.

• Eachtypeofdatacanbecollectedandanalyzedseparatelyandindependently,usingthetechniquestraditionallyassociatedwitheachdatatype.

Challengesinusingtheconvergentdesign

• Mucheffortandexpertiseisrequired,particularlybecauseoftheconcurrentdatacollectionandthefactthatequalweightisusuallygiventoeachdatatype.Thiscanbeaddressedbyformingaresearchteamthatincludesmemberswhohavequantitativeandqualitativeexpertise,orbytrainingsingleresearchersinbothquantitativeandqualitativeresearch.

MethodsWorkshopfortheMinistryofHealthandLong-TermCare

• Researchersneedtoconsidertheconsequencesofhavingdifferentsamplesanddifferentsamplesizeswhenmergingthetwodatasets.Differentsamplesizesmayoccurbecausethequantitativeandqualitativedataareusuallycollectedfordifferentpurposes.

• Itcanbechallengingtomergetwosetsofverydifferentdatainameaningfulway;• Itcanbechallengingifthequantitativeandqualitativeresultsdonotagree.

Contradictionsmayprovidenewinsightsintothetopic,butthesedifferencescanbedifficulttoresolveandmayrequirethecollectionofadditionaldata.

2. Theexplanatorysequentialdesign

Theexplanatorysequentialdesignusessequentialtiming.Thisdesignstartswiththecollectionandanalysisofquantitativedata,whichhasthepriorityforaddressingthestudy’squestions.Thisfirstphaseisfollowedbythesubsequentcollectionandanalysisofqualitativedata.Thesecond,qualitativephaseofthestudyisdesignedsothatitfollowsfromtheresultsofthefirst,quantitativephase.Finally,theresearcherinterpretstowhatextentandinwhatwaysthequalitativeresultshelptoexplaintheinitialquantitativeresults.Thepurposefortheexplanatorydesign

• Touseaqualitativestrandtoexplaininitialquantitativeresults;• Tousequantitativeresultsaboutparticipantcharacteristicstoguidepurposeful

samplingforaqualitativephase.Advantagesoftheexplanatorydesign

• Thisdesignappealstoquantitativeresearchers,becauseitoftenbeginswithastrongquantitativeorientation.

• Itstwo-phasestructuremakesitstraightforwardtoimplement,becausetheresearcherconductsthetwomethodsinseparatephasesandcollectsonlyonetypeofdataatatime.Therefore,singleresearcherscanconductthisdesign.

• Thisdesignlendsitselftoemergentapproacheswherethesecondphasecanbedesignedbasedonwhatislearnedfromtheinitialquantitativephase.

Challengesinusingtheexplanatorydesign

• Thisdesignrequiresalengthyamountoftimeforimplementingthetwophases;• Itcanbedifficulttosecureinstitutionalreviewboard(IRB)approvalforthisdesign,

becausetheresearchercannotspecifyhowparticipantswillbeselectedforthesecondphaseuntiltheinitialfindingsareobtained.

• Theresearchermustdecidewhichquantitativeresultsneedtobefurtherexplained.• Theresearchermustdecidewhotosampleinthesecondphaseandwhatcriteriato

useforparticipantselection.

MethodsWorkshopfortheMinistryofHealthandLong-TermCare

3. Theexploratorysequentialdesign

Theexploratorysequentialdesignalsousessequentialtiming.Incontrasttotheexplanatorydesign,theexploratorydesignbeginswithandprioritizesthecollectionandanalysisofqualitativedatainthefirstphase.Buildingfromthequalitativeresults,theresearcherconductsasecond,quantitativephasetotestorgeneralizetheinitialfindings.Finally,theresearcherinterpretshowthequantitativeresultsbuildontheinitialqualitativeresults.Purposefortheexploratorydesign

• Togeneralizequalitativefindingsbasedonafewindividualsfromthefirstphasetoalargersamplegatheredduringthesecondphase;

• Theresultsofthefirst,qualitativemethodcanhelpdeveloporinformthesecond,quantitativemethod;

• Todevelopandtestaninstrumentbecauseoneisnotavailable;• Toidentifyimportantvariablesforthequantitativestudywhenthevariablesare

unknown;• Toexploreaphenomenonindepthandmeasuretheprevalenceofitsdimensions.

Advantagesoftheexploratorydesign

• Separatephasesmaketheexploratorydesignstraightforwardtodescribe,implement,andreport.

• Theresearchercanproduceanewinstrumentasoneofthepotentialproductsoftheresearchprocess.

Challengesinusingtheexploratorydesign

• Thetwo-phaseapproachrequiresconsiderabletimetoimplement,potentiallyincludingtimetodevelopanewinstrument.

• ItisdifficulttospecifytheproceduresofthequantitativephasewhenapplyingforinitialIRBapprovalforthestudy;

• Researchersshouldconsiderusingasmallpurposefulsampleinthefirstphaseandalargesampleofdifferentparticipantsinthesecondphasetoavoidquestionsofbiasinthequantitativestudy.

• Proceduresshouldbeundertakentoensurethatthescoresdevelopedontheinstrumentarevalidandreliable.

4. Theembeddeddesign Theembeddeddesignoccurswhentheresearchercollectsandanalyzesbothquantitativeandqualitativedatawithinatraditionalquantitativeorqualitativedesign.Inanembeddeddesign,theresearchermayaddaqualitativestudywithinaquantitative

MethodsWorkshopfortheMinistryofHealthandLong-TermCare

design,suchasanexperiment,oraddaquantitativestudywithinaqualitativedesign,suchasacasestudy.Intheembeddeddesign,thesupplementalstudyisaddedtoenhancetheoveralldesigninsomeway.

Thepurposefortheembeddeddesign• Toincludequalitativedatatoanswerasecondaryresearchquestionwithinthe

predominantlyquantitativestudy;• Toimproverecruitmentprocedures;• Toexaminetheprocessofanintervention;• Toexplainreactionstoparticipationinanexperiment.

Advantagesoftheembeddeddesign• Thisdesigncanbeusedwhentheresearcherdoesnothavesufficienttimeor

resourcestocommittoextensivequantitativeandqualitativedatacollectionbecauseonedatatypeisgivenlessprioritythantheother.

• Bytheadditionofsupplementaldata,theresearcherisabletoimprovethelargerdesign.

• Researchersusinganembeddeddesigncankeepthetwosetsofresultsseparateintheirreportsorevenreporttheminseparatepapers.

Challengesinusingtheembeddeddesign

• Theresearcherneedstohaveexpertiseinthequantitativeorqualitativedesignusedinadditiontoexpertiseinmixedmethodsresearch.

• Theresearchermustspecifythepurposeofcollectingqualitative(orquantitative)dataaspartofalargerquantitative(orqualitative)study.

• Itcanbedifficulttointegratetheresultswhenthetwomethodsareusedtoanswerdifferentresearchquestions.

Samplingissues

• Somechallengesspecifictoconcurrentdesigns(i.e.,mergingquantitativeandqualitativeresearch)includehavingadequatesamplesizesforanalyses,usingcomparablesamples,andemployingaconsistentunitofanalysisacrossthedatabases.

• Forsequentialdesigns(i.e.,onephaseofqualitativeresearchbuildsonthequantitativephaseorviceversa),theissuesrelatetodecidingwhatresultsfromthefirstphasetouseinthefollow-upphase,choosingsamplesandestimatingreasonablesamplesizesforbothphases,andinterpretingresultsfrombothphases.

Analyticandinterpretiveissues

MethodsWorkshopfortheMinistryofHealthandLong-TermCare

Issuesariseduringdataanalysisandinterpretationwhenusingspecificdesigns.Whentheinvestigatormergesthedataduringaconcurrentdesign,thefindingsmayconflictorbecontradictory.Astrategyofresolvingdifferencesneedstobeconsidered,suchasgatheringmoredataorrevisitingthedatabases.Fordesignsinvolvingasequentialdesignwithonephasefollowingtheother,thekeyissuessurroundthe“pointofinterface”inwhichtheinvestigatorneedstodecidewhatresultsfromthefirstphasewillbethefocusofattentionforthefollow-updatacollection.Makinganinterpretationbasedonintegratedresultsmaybechallengingbecauseoftheunequalemphasisplacedoneachdatasetbytheinvestigatororteam,theaccuracyorvalidityofeachdataset,andwhetherphilosophiesrelatedtoquantitativeorqualitativeresearchcanorshouldbecombined.IntegrationofQualitativeandQuantitativedataMergingdata• Byreportingresultstogetherinadiscussionsectionofastudy,suchasreportingfirst

thequantitativestatisticalresultsfollowedbyqualitativequotesorthemesthatsupportorrefutethequantitativeresults.

• Theconversionortransformationofonedatatypeintotheothersothatbothcanbeanalyzedtogether:

o Quantitativedataarenumericallycodedandincludedwithquantitativedatainstatisticalanalysis.

o Quantitativedataaretransformedintonarrativeandincludedwithqualitativedatainthematicanalysis.

Connectingdata

o Thisintegrationinvolvesanalyzingonedataset(e.g.,aquantitativesurvey),andthenusingtheinformationtoinformthesubsequentdatacollection(e.g.,interviewquestions,identificationofparticipantstointerview).Inthisway,theintegrationoccursbyconnectingtheanalysisoftheresultsfromtheinitialstudyorphasewiththedatacollectionfromthesecondstudyorphase.

Embeddingdata

o Theresearchermayembedasupplementalqualitativedatawithinalargerquantitative(e.g.,experimental)designorembedaquantitativedatawithinalargerqualitative(e.g.,narrative)design.Theembeddednatureoccursatthedesignlevel,inthattheembeddedmethodisconductedspecificallytofitthecontextofthelargerquantitativeorqualitativedesignframework.

MethodsWorkshopfortheMinistryofHealthandLong-TermCare

AchecklistforreviewingNIHmixedmethodsapplication

Significance• Doestheapplicationmakeaconvincingcasethattheproblemisrelevant(e.g.,if

aimsareachieved,theworkwillimproveknowledgeorpractice)?• Cantheproblembebeststudiedthroughthemultipleperspectivesofmixed

methodsresearch?

Investigator(s)

• Dotheinvestigator(s)havetherequiredskillstoconductallproposedmethods(e.g.,investigator(s)havepriorpublicationsand/orgrantsrelatedtoproposedqualitative,quantitative,andmixedmethods;co-investigatorswithappropriateexpertiseareidentifiedtoleadeachmethodasneeded)?

• Isthereevidencethattheprojectleadershipiscommittedtomixedmethodsresearch(e.g.,eachcomponentofthestudyisaddressedsufficientlyandconsistentlythroughouttheapplication;therearereferencestocurrentrelevantliteratureonmixedmethods;investigatorshaveexperiencesinprofessionaldevelopmentinmixedmethods)?

• Hastheapproachtocollaborationbeendescribed(e.g.,frequencyofmeetingsbetweenleadersofdifferentcomponents,managementofdifferencesbetweenco-investigators)?

Innovation

• Doestheuseofmixedmethodsprovideaplatformforinnovativeinvestigationoftheresearchproblem(s)(e.g.,providesinsightsintomechanismsoforganizationalchangenotpossiblewithasinglemethod)?

• Isthecombinationofmethodsusedinnovative,orthewayinwhichtheyareintegratedinnovative?

Approach

• Isthereadescriptionofthephilosophyortheoryinformingtheresearchandthewaysthisphilosophyortheoryshapestheinvestigation?

• Havetheapplicantsofferedaconvincingexplanationofwhymixedmethodsresearchisneededtoaddressthestudyaimsandthevalueaddedbyusingthisapproach(e.g.,explainedhowalternativedesignswouldbeinappropriateorinadequate)?

• Isthereacleardescriptionofthefullstudydesign,includingwhereintegrationoccurs(e.g.,usingacomprehensivefigureormatrix)?

• Istheintegrationofthemethodswelldescribed,includingthetiming,techniques,andresponsibilitiesforintegration?

• Isthedesignappropriateforthestudyaims?

MethodsWorkshopfortheMinistryofHealthandLong-TermCare

• Arethemethodsconsistentwithestablishedstandardsofrigorforquantitativeandqualitativedatacollectionandanalysis(e.g.,sampling,samplesizeandanalysisplansarespecifiedforeachmethod,withappropriatecitations)?

• Willappropriatecomputersoftwarebeusedforeachanalyticcomponent,andifnot,isaconvincingrationaleprovided?

• Isthestudyfeasiblewithinitsproposedtimeframeandresources(e.g.,atimetableisprovidedthatallocatestimefordataintegration)?

Environment• Isthereevidencethattheinstitutionsupportsmixedmethodsresearch(e.g.,forums

formultidisciplinarycollaborations,facultywithfundingformixedmethodsresearch)?

MethodsWorkshopfortheMinistryofHealthandLong-TermCare

ReferencesCreswell,J.W.,&PlanoClark,V.L.(2007).Designingandconductingmixedmethods

research(Chapter4Choosingamixedmethodsdesign).ThousandOaks:SagePublication.Cohen,D.,&Crabtree,B.(2008).RobertWoodJohnson,qualitativeresearchguidelines

project.Retrievedfromhttp://www.qualres.org/.Creswell,J.W.,&PlanoClark,V.L.(2011).Designingandconductingmixedmethods

research(2nded.).ThousandOaks,CA:Sage.Frechtling,J.(January,2002).The2002userfriendlyhandbookforprojectevaluation.WashingtonD.C.:NationalScienceFoundation.Retrievedfromhttp://www.nsf.gov/pubs/2002/nsf02057/start.htm.LevinJ.S.,Glass,T.A.,Kushi,L.H.,Schuck,J.R.,Steele,L.,&Jonas,W.B.(1997).Quantitativemethodsinresearchcomplementaryandalternativemedicine,MedicalCare,

35,1079-1094.O’Cathain,A.(2010).Assessingthequalityofmixedmethodsresearch:Towardacomprehensiveframework.InA.Tashakkori&C.Teddlie(Eds.),SAGEhandbookofmixedmethodsinsocial&behavioralresearch(2nded.)(pp.531-555)ThousandOaks,CA:Sage.O’Cathain,A.,Murphy,E.,&Nicholl,J.(2008).Thequalityofmixedmethodsstudiesin

healthservicesresearch.JournalofHealthServicesResearchPolicy,13(2),92-98.Tashakkori,A.,&Creswell,J.W.(2007).Theneweraofmixedmethods.[Editorial].Journal

ofMixedMethodsResearch,1(1),3-7.Caracelli,V.J.,&Greene,J.C.(1993).Dataanalysisstrategiesformixed-methodevaluationdesigns.EducationalEvaluationandPolicyAnalysis,15,195-207.Greene,J.C.(2007).Mixedmethodsinsocialinquiry.SanFrancisco:JohnWiley&Sons.Morse,J.,&Niehaus,L.(2009).Mixedmethoddesign:Principlesandprocedures.WalnutCreek,CA:LeftCoastPress.O’Cathain,A.O.,Murphy,E.,&Nicholl,J.(2010).Threetechniquesforintegratingdatainmixedmethodsstudies.BMJ,341,c4587.Song,M.,Sandelowski,M.,&Happ,M.B.(2010).Currentpracticesandemergingtrendsinconductingmixedmethodsinterventionstudiesinthehealthsciences.In:A.Tashakkori&

C.Teddlie(Eds.),Handbookofmixedmethodsinsocial&behavioralresearch(2nded.,pp.

725–747).ThousandOaks,CA:Sage.NationalInstitutesofHealth.Bestpracticesformixedmethodsresearchinthehealth

sciences,2010.