Qualitative Research in Accounting & ManagementApplying mixed methods research in evaluating clinical trialsLyn Murphy William Maguire

Article information:To cite this document:Lyn Murphy William Maguire, (2011),"Applying mixed methods research in evaluating clinical trials",Qualitative Research in Accounting & Management, Vol. 8 Iss 1 pp. 72 - 90Permanent link to this document:http://dx.doi.org/10.1108/11766091111124711

Downloaded on: 07 November 2014, At: 08:09 (PT)References: this document contains references to 73 other documents.To copy this document: permissions@emeraldinsight.comThe fulltext of this document has been downloaded 800 times since 2011*

Users who downloaded this article also downloaded:Jennifer Grafton, Anna M Lillis, Mary A. Malina, Hanne S.O. Nørreklit, Frank H. Selto, (2011),"Lessonslearned: advantages and disadvantages of mixed method research", Qualitative Research in Accounting& Management, Vol. 8 Iss 1 pp. 59-71Jennifer Grafton, Anne M. Lillis, Jennifer Grafton, Anne M. Lillis, Habib Mahama, (2011),"Mixed methodsresearch in accounting", Qualitative Research in Accounting & Management, Vol. 8 Iss 1 pp. 5-21Jennifer Grafton, Anna M Lillis, Eeva#Mari Ihantola, Lili#Anne Kihn, (2011),"Threats to validity and reliabilityin mixed methods accounting research", Qualitative Research in Accounting & Management, Vol. 8 Iss1 pp. 39-58

Access to this document was granted through an Emerald subscription provided by 549136 []

For AuthorsIf you would like to write for this, or any other Emerald publication, then please use our Emerald forAuthors service information about how to choose which publication to write for and submission guidelinesare available for all. Please visit www.emeraldinsight.com/authors for more information.

About Emerald www.emeraldinsight.comEmerald is a global publisher linking research and practice to the benefit of society. The companymanages a portfolio of more than 290 journals and over 2,350 books and book series volumes, as well asproviding an extensive range of online products and additional customer resources and services.

Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committeeon Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archivepreservation.

*Related content and download information correct at time of download.

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

Applying mixed methodsresearch in evaluating clinical

trialsLyn Murphy

School of Accounting and Management, Manukau Institute of Technology,Manukau, New Zealand, and

William MaguireSchool of Accounting and Corporate Governance,

University of Tasmania, Hobart, Australia

Abstract

Purpose – The purpose of this paper is to report on the decision process that the authors follow inapplying mixed methods research to evaluate the benefits and costs of conducting sponsored clinicaltrials in a publicly funded New Zealand hospital.

Design/methodology/approach – A simultaneous parallel mixed method design was adopted. Thisdesign builds on a health outcomes study that involves a retrospective cohort study of changes inparticipants’ health status and mortality rates. Although a team of medical researchers conducted thatstudy (i.e. the current authors were not involved), it is one of the three strands of the current research as itforms the platform for the other two strands, namely the multiple stakeholder perception strand and theeconomic outcomes strand. In the multiple stakeholder perceptions strand, qualitative methods wereused to explore the benefits and costs perceived by stakeholders. In the economic outcomes strand,quantitative methods were used to estimate the benefits and costs of clinical trials.

Findings – The economic outcomes strand and the multiple stakeholder perceptions strand arecomplementary. Each strand delivers dimensions to the analysis that are not apparent from the other.

Originality/value – The value of the paper lies in improved understanding of the process of mixedmethod research through communicating choices and decisions made in response to the challenges faced.

Keywords Research methods, Cost benefit analysis, Stakeholder analysis, Project evaluation, Hospitals,New Zealand

Paper type Research paper

1. IntroductionThe setting of this paper is the design and execution of a study that evaluates clinicaltrials in a publicly funded New Zealand hospital[1]. We aim to show how we use mixedmethod research by describing the techniques we use and the decisions we make in theresearch process. We apply mixed method research in a large case study that evaluatesthe benefits and costs of conducting sponsored clinical trials in a publicly fundedNew Zealand hospital. It involves three strands:

(1) a health outcomes strand involving a retrospective cohort study of participantsin sponsored clinical trials;

(2) a multiple stakeholder perceptions strand assessing stakeholder perceptions ofclinical trials being performed at the Centre for Clinical Research and EffectivePractice (CCRep); and

(3) an economic outcomes strand evaluating the benefits and costs of clinical trials.

The current issue and full text archive of this journal is available at

www.emeraldinsight.com/1176-6093.htm

QRAM8,1

72

Qualitative Research in Accounting &ManagementVol. 8 No. 1, 2011pp. 72-90q Emerald Group Publishing Limited1176-6093DOI 10.1108/11766091111124711

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

Figure 1 shows how the economic outcomes multiple stakeholder perceptions strandsbuilds on the health outcomes strand.

Although researchers have traditionally chosen between qualitative and quantitativemethods, they now consider mixed method research a viable option. Doyle et al. (2009)suggest that this is a response to the limitations that researchers experience when usingmethods individually as well as the greater acceptance of mixed method research by theacademic community. Tashakkori and Creswell (2007, p. 4) define mixed methodresearch as:

[. . .] research in which an investigator collects and analyses data, integrates the findings, anddraws inferences using both qualitative and quantitative approaches or methods in a singlestudy or program of inquiry.

The paper proceeds as follows. Section 2 provides an overview of the study, includingthe role of the research team in deciding to use mixed method research. Section 3identifies the stakeholders and examines their role in the study. We then present theresearch design, including sampling techniques and data collection and analysis.The paper concludes by drawing out the combined potential of qualitative andquantitative methods as applied across the health outcomes, economic outcomes andmultiple stakeholder perceptions strands to produce a strong evaluation of clinical trials.

2. Overview of clinical trials studyThe genesis of the current study lies in a health outcomes study by the medical researchteam at the CCRep. Relative to other research units that conduct clinical trials inAustralasia, CCRep has conducted a significant number of clinical trials (CountiesManukau District Health Board (CMDHB), 2009) and CCRep clinicians believe that theyhave developed the expertise to add value through these trials. The health outcomesstudy uses quantitative methods to evaluate health outcomes from two randomisedclinical trials with a cohort control group. While the study would serve to raise CCRep’sprofile in this area, the medical team felt that it should be extended to an economicevaluation. Accordingly, they sought a research partner to evaluate the economicoutcomes for the two clinical trials and cohort control groups. This is consistent withBabour and Babour’s (2003) recommendation of collaborative exercises and parallel datacollection as an efficient and effective research process when resources are scarce.CCRep approached the school of nursing at a local institute of technology for assistancein recruiting a team and researchers from the school of business expressed interest.

Figure 1.Three strands: health

outcomes, economicoutcomes, multiple

stakeholder perceptions

Health outcomes

Economic outcomes

Multiple stakeholder perceptions

• Quantitative

• Quantitative

• Qualitative• Nursing school

• Business faculty

• CCrep

Evaluatingclinical trials

73

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

During a brainstorming session by researchers from the nursing (qualitative focus) andbusiness schools (mainly quantitative focus), they concluded that a quantitativeevaluation alone would not tell the whole story, because only a qualitative study wouldreveal a sense of the value that stakeholders place on clinical trials. They also recognisedthe difference in the relative sensitivity of data collected from quantitativeand qualitative studies. Consistent with Moffatt et al. (2006), a qualitative study mayidentify important impacts on stakeholders, whereas a large health board may not findpractical or economic interest in a quantitative study indicating that the economicoutcome of the trial is not significantly positive in economic terms.

During the early preparations for the project, the CCRep research team suggested thatit would be advantageous to include a PhD student in the team. A member of the teamsubsequently committed to a PhD and two other members acted as the principal andco-supervisors. The researchers at CCRep managed the health outcomes project andmaintained a deep interest in all other aspects.

Clinical trials are the most reliable way to assess the efficacy and safety of healthinterventions ( Jull et al., 2005). They provide information for national treatmentguidelines on patient management and are required for the approval and registration ofnew medicines. The evaluation of new pharmaceutical products through clinical trialsplays an important role in modern evidence-based medical practice. Clinical trials followlaboratory testing in cell and animal studies and occur in four distinct phases, all ofwhich involve humans. We examine two phase-III studies which assess the effectivenessof the new drug as compared with treatments currently being used. These trials involvemany participants, normally occur over several years and incur higher costs than earlierphase trials (Getz, 2008). Researchers select participants for these trials randomly foreither a drug trial group or a placebo control group.

Recent literature reveals little information on the benefits and costs related to clinicaltrials in New Zealand, whether they relate to health outcomes, stakeholder perceptions oreconomic outcomes. Watson (2006) authored one of the few New Zealand studies and thefirst of its kind based on New Zealand data. This study was, however, commissioned byPfizer Pharmaceuticals (i.e. not an independent study) and based on secondary data.He suggests that sponsored clinical trials could have an important role in New Zealand’shealth system by providing the potential to contribute to reducing the costs ofNew Zealand health boards, retaining and developing a pool of internationally recognisedNew Zealand researchers and sustaining New Zealand’s clinical research infrastructurewhile at the same time allowing New Zealanders timely access to new drugs.

International studies have produced a variety of results on the benefits and costs ofclinical trials. The studies noted below, all based on quantitative data, calculate thesavings from drug cost avoidance for hospitals running clinical trials. LaFleur et al.(2004) review the study protocols and dispensing data for clinical trials in one unit overtwo years and calculate the revenue generated and drug cost avoided. They find drugtrial participation achieves substantial drug cost avoidance. In another comparable USstudy, McDonagh et al. (2000) examine the costs and savings resulting from drug costavoidance for two organisations during the fiscal year 1996-1997. They use pharmacydispensing records to tabulate the number of drugs provided free for each study. Drugcost avoidance results in significant savings equivalent to 8 per cent of the institutions’annual drug budget. Acquired immune deficiency syndrome and oncology trials deliverthe highest drug cost avoidance savings. Braunholtz et al. (2001) focus on the immediate

QRAM8,1

74

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

benefit resulting from clinical trials. They summarise trial data to determine whether atrial is beneficial or detrimental to patients participating in it. Their findings suggestthat trials on average and if conducted well tend to benefit the participants and do not, inmost instances, cause harm. These studies suggest that the conduct of clinical trialsoffers potential benefits to trial participants and cost savings to the New Zealand healthsector.

Edwards et al. (1998) undertake a review of 61 studies of the perspectives of patients,the general public and healthcare professionals on the ethics of clinical trials. Of these,53 use quantitative, and seven use qualitative methods alone, while one study uses both.They identify stakeholder concern surrounding the ethics of clinical trials, are highlycritical of the quality of the research methods used and call for further research into“what well informed members of the public really think about trials” (p. 1211).

CMDHB anticipates that by 2012 it will have 3,851 or 11.1 per cent of all of itsoutpatients enrolled in sponsored clinical trials (CMDHB, 2009). In 2015, the percentageof outpatients participating in sponsored clinical trials is expected to increase to15 per cent (Ryan, 2009). Publicly funded institutions have a responsibility to considerstakeholders in their actions. CMDHB values stakeholder involvement and has as one ofits objectives “to exhibit a sense of social responsibility by having regard to the interestsof the people to whom it provides, or for whom it arranges the provision of services”(www.cmdhb.org.nz/About_CMDHB/Overview/objectives-functions.htm).

The foregoing sections show that the nature of the study, the way in which it developsand the evolution of the research team all play a part in shaping a mixed method study.The next section identifies and classifies the stakeholders in clinical trials.

3. StakeholdersAccording to Freeman (1984, p. 46), a stakeholder is “any group or individual who canaffect or is affected by the achievement of the organisation’s objectives.” However, not allstakeholders affect or are affected by an organisation equally. The literature reveals anumber of proposals for identifying, classifying or mapping stakeholders according toattributes such as power, influence and interest (Mitchell et al., 1997). To identifystakeholders, a value judgment is required about the legitimacy of each stakeholder andthe informants who will represent them (Mark and Shatland, 1985). The stakeholdersin the clinical trials are anchored in the health outcomes strand, which is the platformfor the study as a whole.

We identify the trial participants as the primary stakeholders, given that it is theirhealth that is at stake. In one sense, they have little power to influence clinical trials – ifthey wish to benefit, they must volunteer; in another sense, if conditions are sufficientlyadverse, aversion to risk may cause them to withhold their cooperation. The interests offamily and caregivers are so closely entwined with those of the trial participants that itseems appropriate to include them as primary stakeholders. CMDHB researchers andthe pharmaceutical industry run a close second to participants and caregivers; theysimilarly have a deep interest in the health outcomes, albeit for quite different reasons.Researchers may benefit from satisfaction from research activities, publicationopportunities and career advancement. Members of the pharmaceutical industry rely onclinical trials for the development of their products because drug testing on humans isrequired by law, and this is the avenue to marketable products and financial rewards.At the same time, their financial resources afford them the power, or at least the power

Evaluatingclinical trials

75

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

to influence research institutions and others in relation to clinical trials and theirconduct. The Counties-Manukau Community represents New Zealand society in thisstudy – ultimately, clinical trials are of concern to all members of society – an effectivedrug can bring relief and perhaps happiness to a great number of people. For example,lithium carbonate and antidepressants have improved the quality of life for many,whereas thalidomide, inadequately tested in the 1960s, has brought misery to thousandswith consequences that continue to this day. If the concerns of the community aresufficiently strong, the engagement of government bodies and politicians follows,bringing with it considerable potential power or power to influence. A salient example ofgovernment’s potential influence takes the form of the recently appointed New ZealandParliament Health Select Committee Enquiry into improving New Zealand’senvironment to support innovation through clinical trials.

In addition to the above approach to classifying stakeholders, it may be useful togroup stakeholders at the macro-level (society, government bodies and politicians),meso-level (CMDHB, pharmaceutical industry) and micro-level (trial participants,caregivers and family and researchers). Qualitative data are especially good at exploringthe micro-level and revealing feelings and emotional responses that are not discoveredthrough quantitative research. Quantitative data help us to understand the meso-levelbenefits and costs. For example, they allow an assessment of the relative value of clinicaltrials against other treatment methods and the savings that CMDHB achieves whensponsors subsidise the cost of treating trial participants.

4. Developing the initial research designCreswell (2003) and O’Cathain et al. (2007) have called for increased documentation of theway researchers design and combine research methods in mixed method studies.Documentation is a means to both improve understanding about the process of mixedmethod research and to justify the research conclusions reached (Bryman, 2007b). Morse(2001, p. 435) believes that presenting research as a sterile process can “appear effortlessand trite, concealing the conceptual struggles and skills essential in producing suchresearch”. Bechhofer (1974, p. 73) observes that:

[. . .] the research process is not a clear cut sequence of procedures following a neat pattern,but a messy interaction between the conceptual and the empirical world, deduction andinduction occurring at the same time.

As we develop the research design, we seek to articulate worldview and methodology.Positivist versus constructivist debates abound in the literature (Bryman, 2007a) andresearchers have historically opted for either a positivist worldview, associated withquantitative methods or a constructivist worldview associated with qualitative methods(Doyle et al., 2009). While purists argue that positivism and constructivism areepistemological extremes and cannot be combined (Smith and Heshusius, 1986; Lincolnand Guba, 2000) others maintain that these epistemological points of view are part of acontinuum and that claimed distinctions between them are less obvious under scrutiny(Bryman, 2001; Brannen, 2005). As an alternative to these approaches, we adopt apragmatic worldview, where the empirical and practical consequences are consideredwhen evaluating ideas (Johnson and Onwuegbuzie, 2004).

Our rationale for using a combination of data sources is that mixed methods exposedifferent views, perceptions and experiences of clinical trials. As Bryman et al. (2008, p. 264)

QRAM8,1

76

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

suggest, if we view the evaluation of clinical trials as a large jigsaw puzzle, each method ofdata collection becomes “an important piece of a jigsaw”. Regarding the research design,Teddlie and Tashakkori (2009, p. 163) observe:

It is important to recognize that it is impossible to enumerate all possible mixed methoddesigns. Therefore, you should look for the most appropriate or single best available researchdesign, rather than the “perfect fit”. You may have to combine existing designs, or createnew designs for your study.

Johnstone’s (2004) design attracts our attention as a suitable model for the current studybecause the research questions she asks are broadly similar to ours. She explores theadoption of new surgical technology in a health-care setting. Her question is, “what are topmanagers’ assumptions about, and expectations of, their hospitals’ investments in newsurgical artefacts?” This question calls for a concurrent design allowing corroboration offindings. Johnstone investigates work-related benefits of new technology and collects datafrom internal stakeholders and hospital records. She limits her analysis to thechoice-making process and the organisational consequences of technological changes thatcan be directly related to the adoption of new technology. The design incorporatestriangulation at two levels; when:

(1) several qualitative methods are used to gather data that are analysedconcurrently; and

(2) quantitative data are analysed with qualitative data.

Johnstone’s (2001) main emphasis is to draw out similarities with her data. She usesgrounded theory – constant comparison method which involves an “analytic processand the continuous interplay between previously collected and analysed data and newinformation” (p. 21). Triangulation, Johnstone suggests, assists this constantcomparison process. Triangulation in research derives from positioning geometry,in which selected stable and objectively verifiable reference points determine an object’sposition in space (Modell, 2009). The triangulation design facilitates the review andanalysis of evidence from multiple sources (Erlandson et al., 1993) and providesopportunities to confirm or corroborate the qualitative results with the quantitativeresults (Creswell and Plano Clark, 2007). Barbour (1998) highlights differences in the useof triangulation in quantitative and in qualitative studies. She suggests that inquantitative research, any differences found during triangulation may lead to adisconfirmation of the hypothesis whereas differences in qualitative research are viewedas constructive and aid the modification of the resulting data interpretation. Johnstone(2001) regards triangulation as important in justifying her findings and appears toresolve the challenges of triangulation in her study. She triangulates qualitative data(from interviews regarding the work-related consequences of introducing new surgicaltechnology) with quantitative data (including time and motion studies and archivedhospital records). Johnstone takes a broad view of triangulation, which differs markedlyfrom that enunciated by Modell (2009):

To put it simply, the principles of triangulation are essentially the same as the logic employedin legal proceedings in which a defendant is judged guilty or not guilty on the basis ofmultiple types of evidence from multiple sources converging on a finding that is “beyondreasonable doubt”.

Evaluatingclinical trials

77

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

Basing the initial research design on Johnstone (2004) (Figure 2), we determine theanalyses needed to meet the research objectives, develop the study design and select themethod (cell 1), provide a guide for the interactive, circular process of data collection,data analysis and design review that we would follow until no new information emerged,that is, until saturation was reached (Lincoln and Guba, 1985) (cells 2-14). We usequalitative methods to gather data from a number of stakeholder groups and to analysethe data using inductive and/or deductive reasoning (cells 4-7). We anticipate that mixedmethod research would allow us to consider a wide range of data and provideopportunities for triangulation (cells 6, 7 and 9) which includes reviewing and analysingevidence from multiple sources (Creswell, 1994). We plan to merge the qualitative andquantitative data to provide an analysis of the research problem at multiple levels whichwould facilitate reflection and further review of the literature (cells 10-12). We anticipatethat an analysis of the benefits and costs revealed by the merged quantitative (cell 8) andqualitative (cell 13) data will inform our discussion and conclusions (cell 14).

While Johnstone’s research design works for her study, we soon find that our similardesign presents both functional and theoretical challenges, the biggest of which istriangulation. We find at a relatively early stage in the research process (cell 6) that thedata collected does not lend itself to triangulation in the way that we anticipated. Babourand Babour (2003, p. 179) draw us back to the literature to seek another view:

Figure 2.Initial research design

1. Research designand method

2. collect qualitative datafocus groups

semi structured interviewssurveys

3. Collect quantitative dataresults from health outcomes

studyaccounting data

4. Thematic analysis offocus groups, surveys

and interviews

5. Statisticallyanalyse

quantitative data

6. Analyse qualitativedata concurrently:

Stage 1A triangulation

7. Analyse quantitativeand relevant qualitativedata concurrently. Stage

1B triangulation

8. Draw positivist paradigmconclusions

quantified CBA

9. Synthesise outcomes ofanalyses using deductive

reasoning. Stage 2triangulation

10. Evaluateprogress

11. Draw sometentative

conclusions

12. Furtherevaluate

literature onemergentthemes

13. Draw qualitative paradigmconclusions

qualitative CBA

14. Combineddiscussion andconclussions

12. Furtherevaluate

literature onemergentthemes

11. Draw sometentative

conclusions

10. Evaluateprogress

QRAM8,1

78

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

To acknowledge incompatibilities and their source is not to deny the enormous potential ofmixed method designs. However, it is important that we avoid producing a homogenized andbland synthesis, which overlooks incompatibilities, rather than developing solutions,compromises or simply deciding that it is most appropriate to pursue parallel strategies.

5. Developing and executing the final research design5.1 Issues in triangulationWhy is Johnstone able to triangulate in her study when we cannot? A comparisonbetween Johnstone (2001, 2004) and the current study (Table I) offers insights into thepotential impact of the differences between them. The first difference lies in the primaryjustification for selecting a mixed method design. Johnstone (2001, p. 16) states that thevalue of mixed method research is in its “capacity to add rigour and credibility to astudy” and she uses triangulation “to neutralise any bias inherent in a particular datasource, investigator, or method” (p. 17).

We use mixed methods for a different purpose; we seek to understand different aspectsof clinical trials. Johnstone restricts her participants to internal stakeholders and socollects a relatively narrow range of data that is easy to triangulate. The resulting data aretherefore more likely to be interdependent than they might otherwise be and thus may beeasier to triangulate. Our objectives for the qualitative multiple stakeholder perceptionsstrand are broader than Johnstone’s. We elicit data from both internal and externalstakeholders who provide a range of views across many dimensions. It may be moredifficult to triangulate these data with our quantitative dataset. The multiple stakeholderperceptions strand focuses on how individuals perceive the phenomenon (i.e. the benefitsand costs of clinical trials) rather than on the phenomenon itself. We also leave theinterpretation of what constitute benefits or costs up to the respondent stakeholder.

Qualitative data are analysed differently in the two studies. The current study usesphenomenography as a basis for analysing the qualitative results. Phenomenographyidentifies how individuals and groups perceive the phenomenon rather than thephenomenon itself (Ireland et al., 2008) and seeks to distinguish the variations in the wayspeople experience reality (Bowden, 2000). The emphasis is therefore on drawing out bothsimilarities and differences between our data. The broad qualitative dataset provides fewopportunities to triangulate with our relatively narrow quantitative dataset.

Johnstone (2001, 2004) Current study

Setting Health HealthPrimary justification for usingmixed method design

To add rigour and credibility To add perspectives that areunavailable or not evident fromusing one method alone

Questions asked and data used Single question, different data Different questions, differentdata

Informants Internal stakeholders Internal and externalstakeholders

Qualitative data analysis Grounded theory – constantcomparison method

Phenomenography

Between-method triangulation Triangulated quantitative andrelevant qualitative data

Not feasible

Table I.Comparison of Johnstone(2001, 2004) and current

study

Evaluatingclinical trials

79

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

The qualitative and quantitative data that contribute to the evaluation of clinical trials areexpressed in quite different ways. In Johnstone’s study the qualitative and quantitativemethods illuminate the phenomenon under study in similar ways, whereas in the currentstudy the nature of the light shed differs depending on the method. For example,members of the medical staff express emotions such as pride in working within a hospitalthat values research, an insight that is quite different from the economic benefits revealedin the economic outcomes strand. Although Johnstone (2004, p. 266) does find that not allof her data can be triangulated, she has sufficient overlap to “analyse quantitative andrelevant qualitative data together”. Another explanation for the similarity betweenJohnstone’s quantitative and qualitative results is that Johnstone may have “seen whatshe was looking for”. Sale et al. (2002, p. 48) explains:

Achieving similar results may be merely a matter of perception. In order to synthesize resultsobtained via multiple methods research, people often simplify the situation under study,highlighting and packaging results to reflect what they think is happening. The truth is we rarelyknow the extent of disagreement between qualitative and quantitative results because that isoften not reported.

Our struggle to implement the triangulation plan is not uncommon. Bryman (2006, p. 124)reports that while researchers may begin with the goal of triangulation they often findthat “other implications of putting the two sets of data together become apparent whenthe data are analysed.” Hammond and Wiriyapinit (2005, p. 390) clarify the potentialimplications by comparing the findings from one method with those of another andidentifying “consistency (i.e. there was a match between findings), contrast (i.e. findingswere contradictory)” and “complementarity [that] referred to findings derived from onemethod, which added a perspective unavailable, or simply not apparent, within thefindings from a different method”. They warn, therefore, that where there are manyinstances of complementarity, relying on one set of data alone risks giving a “partial,even a misleading, impression” of the phenomena under study (p. 391).

Our data are complementary in the sense that they produce a collage capable ofdelivering an understanding of clinical trials in New Zealand; however, the data are notamenable to producing a meaningful composite. As data relating to many aspects of thestudy can be obtained via only one of the available research methods, our view is thatquantitative and qualitative methods can be combined in the current study only forcomplementary purposes. Given that we had a strong sense from the inception of theproject that a quantitative analysis on its own would lack balance, we seekcomplementarity through a mixed method approach.

Richardson (2000, p. 14) prefers the term “crystallization” to “triangulation” to comparedata sources and highlight complementarity. She chooses a crystal rather than a triangleto represent the relationship between data sources because a crystal has multiple surfaces.It is able to reflect and refract and therefore create multiple images of reality. Whereas thetriangle suggests there is one single triangulated truth, the crystal symbolises thepossibility of multiple truths (Janesick, 2000). This is consistent with Sale et al. (2002, p. 43):

[. . .] the paradigms upon which the methods are based have a different view of reality andtherefore a different view of the phenomenon under study. Because the two paradigms do notstudy the same phenomena, quantitative and qualitative methods cannot be combined forcross-validation or triangulation purposes. However, they can be combined for complementarypurposes.

QRAM8,1

80

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

5.2 A revised research designOur realisation that a collage of results is more meaningful than a composite for thecurrent study leads us to modify the research design to a “simultaneous parallel mixeddesign analysis” (Teddlie and Tashakkori, 2009). Figure 3 sets out the revised and finalresearch design in which the evaluation of clinical trials is common to all three strands.

The left-hand side of Figure 3 shows the sequence of steps in the research process forthe health outcomes strand. The first step in the design is sampling, a process for selectingelements for analysis (for example participants, groups, artefacts and settings) whichmaximise a researcher’s ability to answer research questions (Tashakkori and Teddlie,2003). This strand employs a cohort design (Figure 4) whereby the patients are dividedinto two groups based on their treatment and risk factors (Tay and Tinmouth, 2007).The cases comprise 252 men and women (the clinical trial participants) aged under55 years of age, at high risk of cardiovascular events who were enrolled in two long-termphase-III randomised, controlled clinical trials designed to assess preventativemedication for patients at risk of serious cardiovascular events. CCRep identified thecontrol group by age, gender and ethnicity-matched patients from their decision-supportrecords (Casemixe). The patients were inpatients or attended outpatient clinics at theresearch site at the time of the trials[2].

Figure 3.Final research design

Economic outcomes strand(quantitative)

Multiple stakeholderperceptions strand

(qualitative)

Datacollection

Datacollection

Dataanalysis Data

analysis

Perceivedoutcomes

Economicoutcomes

Inform overallevaluation of clinical trials

Evaluation of clinical trials

Health outcomesstrand

(quantitative, CCRepresearch team)

Datacollection

Dataanalysis

Healthoutcomes

AdoptsampleSampling

SamplingTrial

participants

Evaluatingclinical trials

81

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

The sample of trial participants adopted in the economic outcomes is that selected in thehealth outcomes strand; a sample that would otherwise would have been difficult andexpensive to construct. For the multiple stakeholder perceptions strand, we use purposivesampling methods to select potential informants who are stakeholders as identified inSection 3 above. Our goal is to work with our informants in a way that is of the leastinconvenience to them without sacrificing the integrity of the data. We restrict thenumber of respondents for practical reasons. Qualitative research can be time consumingand often involves collecting vast amounts of data which must then be transcribed,analysed and reported. This requires that we balance conflicting considerations – theneed for adequate respondent data for analysis and the avoidance of collecting“redundant” data (Lillis, 1999), which consumes effort without reward.

Our sampling process also takes into account the difficulties that can be experiencedin finding respondents willing to return surveys or to give up their time for interviewsand focus groups (Abernethy et al., 1999). We randomly select a subgroup of trialparticipants from the health outcomes strand for an in-depth study using four focusgroups and 100 surveys. Respecting the busy workloads of hospital staff and researcherswe initially use volunteer (by invitation) sampling with this group. We interview centraland local government politicians, elected health board members and members of thePharmaceutical Management Agency (PHARMAC) and the New Zealand Medicines andMedical Devices Safety Authority (Medsafe), which are both government organisations.During our interview with the Chief Executive Officer of the Researched MedicinesIndustry Association of New Zealand (RMI) we requested a sample of members of thepharmaceutical industry[3]. We use this sample to interview and survey members of theindustry because there is limited information on web sites and in the phone book.

We collect data on strands in parallel. The strands progress through parallel datacollection and data analysis phases to produce independent outcomes. Employingmultiple collection methods for the multiple stakeholder perceptions strand, such as focusgroups, observation and interviews can improve the validity, reliability and diverseidentification of realities (Golafshani, 2003). It may also promote “crystallization”(Richardson, 2000), as suggested above. We use different methods for each stakeholder,for two reasons:

Figure 4.Cohort sample used inhealth and economicoutcomes strands

TreatmentStudy

population

CMDHBOUTPATIENTS

CASEtwo clinical trials

n = 252

Treatment groupcases

n = 190

Placebo groupcasesn = 62

Control groupn= 504

Controlstandard care

n = 504

Outcomes

QRAM8,1

82

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

(1) to recognise that relying on one method in isolation may give an incompleteview (Barbour, 1998); and

(2) to allow for communication preferences (Hoffman, 2009).

We ask the same questions of each stakeholder group irrespective of the data collectionmethod used to facilitate a comparison of their responses (see the Appendix). Thisuncovers differing perceptions; for example, health board staff members feel that trialparticipants gain the most benefit from the new treatment drugs being offered whereastrial participants identify the additional care and attention as the greatest benefit.

We apply “parallel mixed data analysis” (Teddlie and Tashakkori, 2009), twoseparate and independent processes, to the data analysis. We use:

(1) a qualitative descriptive analysis coupled with a phenomenographical analysisfor the multiple stakeholder perceptions strand; and

(2) a quantitative analysis relying on various data sources for the economicoutcomes strand.

5.3 Multiple stakeholder perceptions strandWe adopt a qualitative descriptive approach, followed by a phenomenographicalanalysis. The qualitative descriptive approach allows the stakeholders to express theirperceptions and stories in their own words (Sandelowski, 2000). We stay close to thetranscribed data in reporting the findings and frequently quote our informants’ views andexperiences so that the accounts are theirs rather than ours. Phenomenography providesa systematic way to separate and describe different ways of experiencing phenomena.

Marton (1986, p. 31) defines phenomenography as:

[. . .] a research method adapted for mapping the qualitative different ways in which peopleexperience, conceptualise, perceive, and understand various aspects of, and phenomena in,the world around them.

It distinguishes itself by focusing on how individuals and groups perceive thephenomenon rather than the phenomenon itself (Ireland et al., 2008). The analysistransforms individual perceptions to conceptions that apply to a larger group.

5.4 Economic outcomes strandBenefit cost analysis (BCA) in the economic outcomes strand has little or no meaningwithout specifying a focal stakeholder that receives the benefit or bears the cost. Takinga broad view of benefits and costs (i.e. accepting that they are not expressed in dollarsalone), the trigger question for all three strands was essentially: “Is the conduct of clinicaltrials a worthwhile activity for CCRep and if so, should we be conducting more trialsthan we are?” While CCRep clearly has a central interest, the current study involves arange of stakeholders. We accordingly present the analysis for the economic outcomesstudy from three perspectives by adapting a spreadsheet-based multiple accountapproach (Campbell and Brown, 2003). The three sections of the spreadsheet reflectBCAs from different but related perspectives, namely:

(1) the CCRep research unit (micro-level);

(2) the CMDHB (meso-level); and

(3) New Zealand society (macro-level).

Evaluatingclinical trials

83

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

Data for the CCRep perspective are drawn from trial profit and loss statements relatingto the two trials. The CMDHB perspective encapsulates the CCRep revenues and costs aswell as CMDHB costs and savings from pharmaceutical and laboratory subsidies,patient treatment programmes, capital costs and infrastructure costs. The societalperspective builds on the CMDHB perspective and takes into account the changes inhealth status and value of lives saved as a result of the clinical trials. Using data showingthe differences between mortality levels in case and control groups reported in the healthoutcomes strand, we calculate quality adjusted life years (QALY) as a measure ofsocietal benefits from clinical trials. QALY takes into account in a single measure gainsmade in both quantity of life (mortality) and quality of life (morbidity) (Dix Smith et al.,2003). Table II indicates data sources for the economic outcomes strand.

The economic outcomes strand links with the health outcomes strand at two points:

(1) in the sample selection (the economic outcomes strand and the health outcomesstrand use the same cohort matched controls); and

(2) during data analysis when we use the health outcomes as a basis for costing inthe economic outcomes strand.

Detailed analysis of the health outcomes strand is outside the scope of this paperbecause the CCrep research team conducted it and will report on it.

6. ConclusionThis paper describes a current research project, its context, the motivation for the mixedmethod research design and illustrates the application of those methods. A blend ofmethods provides a rich and powerful means of evaluating the benefits and costs ofclinical trials. In the multiple stakeholder perceptions strand, we conduct focus groups,interviews and semi-structured surveys to explore the perceptions of a variety ofstakeholders. The economic outcomes strand draws on the health outcomes strand

Data type Details Source

Clinical evidence Health outcomes for clinical trialparticipants and standardisedtreatment

CCREp

Revenue Revenue from trial profit and lossaccounts

CCRep

Direct costs ofclinical trial

Trial profit and loss accounts CCRep

Overhead costs CMDHBHospital outpatientcosts

Chronic care management (CCM)programme costs

CMDHB

Pharmaceuticalcosts

Pharmaceuticals prescribed Pharmaceutical schedule (PHARMAC)

Laboratory costs Costs of outpatient laboratory anddiagnostic tests

Ministry of Health data banks

Quality of life Utility rating in the form of a singlenumber representing the net aggregateimpact of physical, emotional andsocial functioning on quality of life

EQ-5D Tariff; Access Economics(2008) value of statistical lifeTable II.

Data sources – economicoutcomes strand

QRAM8,1

84

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

and from other sources to estimate the benefits and costs of clinical trials. We collect bothquantitative and qualitative data in the interests of robust research findings. We findthat collecting both qualitative and quantitative data is time consuming so we restrictthe quantity of data collected in the multiple stakeholder perceptions strand of the study.In addition, we find that different skills are required to conduct each strand of the study.We overcome this challenge by having a team of researchers who have different butcomplementary skill sets.

While the data do not support triangulation between the qualitative and quantitativemethods, the qualitative data analysis complements the quantitative data analysis.Collectively, the results provide complementary evidence and contribute to acomprehensive understanding of the outcomes of clinical trials in New Zealand.

Stange et al. (2006) suggest that where qualitative and quantitative findings are complex,researchers should publish separate reports for the fruits of the qualitative, quantitative andmixed method endeavours. Hall and Howard (2008, p. 254) support the presentation ofindividual and mixed-paradigm results suggesting that there should be no expectation thatthe two types of results “will confirm, support, or cancel out one another (we are not insearch of a single truth) but rather that multiple perspectives are provided on one subject”.A pragmatic worldview suggests that the results be presented in a way that best suits thesituation and intended audience. With this in mind, it may be appropriate to report theeconomic outcomes and the multiple stakeholder perceptions separately, followed by areport which reveals the power of a complementary treatment of these approaches.

Greene (2005, p. 209) observes that mixed method research “offers greaterpossibilities than a single method approach for responding to decision-makers agenda,as well as to the interests of other legitimate stakeholders.” Mixed method researchrealises these possibilities in the current study, given multiple and diverse stakeholders,including those who need hard data for their decision making and those who wish tounderstand the perceptions of the stakeholders.

Notes

1. Ethics approval for this study was obtained from the Northern Y Regional Ethics Committee;Reference Number NTY/09/04/037. The Tasmanian Social Science Human Research EthicsCommittee Reference Number H10522 and the Manukau Institute of Technology EthicsCommittee.

2. Two groups of 252 patients are selected for the health study to provide at least 80 per centpower (at the 5 per cent significance level) to detect an 8.6 per cent difference (two tailed) inevent rate between groups based on a composite endpoint event rate of 16.8 per cent in theplacebo-treated group in the study over a mean 4.3 years of follow-up (i.e. 16.8-8.6 per cebt).A mixed methods approach to repeated measures is employed to test this hypothesis. Thisrequires a modified ratio of control to increase the power (the ratio is 2:1).

3. The RMI is now known as Medicines New Zealand and has a new chief executive officer.

References

Abernethy, M.A., Chua, W.F., Luckett, P.F. and Selto, F.H. (1999), “Research in managerialaccounting: learning from others’ experiences”, AccountingandFinance, Vol. 39 No. 1, pp. 1-27.

Access Economics (2008), Dementia Economic Impact Report, Alzheimer’s New ZealandIncorporated, Otago.

Evaluatingclinical trials

85

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

Babour, A. and Babour, M. (2003), “Evaluating and synthesizing qualitative research: the need todevelop a distinctive approach”, Journal of Evaluation in Clinical Practice, Vol. 9 No. 2,pp. 179-86.

Barbour, R.S. (1998), “Mixing qualitative methods: qualitative assurance or qualitativequagmire?”, Qualitative Health Research, Vol. 8 No. 3, pp. 352-61.

Bechhofer, F. (1974), “Current approaches to empirical research: some central ideas”, in Rex, J. (Ed.),Approaches to Sociology: An Introduction to Major Trends in British Sociology, Sage, London.

Bowden, J. (2000), “The nature of phenomenographic research”, in Bowden, J.A. and Walsh, E. (Eds),Phenomenography, RMIT University Press, Melbourne, pp. 1-12.

Brannen, J. (2005), “Mixing methods: the entry of qualitative and quantitative approaches intothe research process”, International Journal of Social Research Methodology, Vol. 8 No. 3,pp. 173-84.

Braunholtz, D., Edwards, S. and Lilford, R. (2001), “Are randomized clinical trials good for us(in the short term): evidence of a trial effect”, Journal of Clinical Epidemiology, Vol. 54 No. 3,pp. 217-24.

Bryman, A. (2001), Social Research Methods, Oxford University Press, Oxford.

Bryman, A. (2006), “Paradigm peace and the implications for quality”, International Journal ofSocial Research Methodology, Vol. 9 No. 2, pp. 111-26.

Bryman, A. (2007a), “Barriers to integrating quantitative and qualitative research”, Journal ofMixed Methods Research, Vol. 1, pp. 8-22.

Bryman, A. (2007b), “The research question in social research: what is its role?”, InternationalJournal of Social Research Methodology, Vol. 10 No. 1, pp. 5-20.

Bryman, A., Becker, S. and Sempik, J. (2008), “Quality criteria for quantitative, qualitative andmixed methods research: a view from social policy”, International Journal of SocialResearch Methodology, Vol. 11 No. 4, pp. 261-76.

Campbell, H. and Brown, R. (2003), Benefit-cost Analysis: Financial and Economic AppraisalUsing Spreadsheets, Cambridge University Press, Melbourne.

CMDHB (2009), Health Research Strategy for Counties Manukau District Health Board, CountiesManukau District Health Board, Auckland, available at: www.pdfdownload.org/pdf2html/pdf2html.php?url¼http%3A%2F%2Fwww%2Ecmdhb%2Eorg%2Enz%2Fabout%5Fcmdhb%2FPlanning%2FResearch%2FCMDHB%2Dresearch%2Dstrategy%2Epdf&images¼no (accessed 22 November 2009).

Creswell, J.W. (1994), Research Design: Qualitative and Quantitative Approaches, Sage, ThousandOaks, CA.

Creswell, J.W. (2003), Research Design: Qualitative, Quantitative and Mixed Methods Approaches,2nd ed., Sage, Thousand Oaks, CA.

Creswell, J.W. and Plano Clark, V.L. (2007), Designing and Conducting Mixed Methods Research,Sage, Thousand Oaks, CA.

Dix Smith, M., Berger, M., Bingefors, K., Hedblom, E., Pashors, C. and Torrance, G. (2003), HealthCare Cost, Quality and Outcomes: ISPOR Book of Terms, International Society ofPharmacoeconomics and Outcomes Research, Lawrenceville, NJ.

Doyle, I., Brady, A. and Byrne, G. (2009), “An overview of mixed methods research”, Journal ofResearch in Nursing, Vol. 14 No. 2, pp. 175-85.

Edwards, S.J., Lilford, R.J. and Hewison, J. (1998), “The ethics of randomised controlled trialsfrom the perspectives of patients, the public, and healthcare professionals”, British MedicalJournal, Vol. 317, pp. 1209-12.

QRAM8,1

86

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

Erlandson, D.A., Harris, E.L., Skipper, B.L. and Allen, S.D. (1993), Doing Naturalistic Enquiry:A Guide to Methods, Sage, Newbury Park, CA.

Freeman, R.E. (1984), Strategic Management: A Stakeholder Approach, Pitman, Boston, MA.

Getz, K.A. (2008), “In search of efficiency between trial phases”,AppliedClinicalTrials, March, pp. 1-3.

Golafshani, N. (2003), “Understanding reliability and validity in qualitative research”,The Qualitative Report, Vol. 8 No. 4, pp. 597-607, available at: www.nova.edu/ssss/QR/QR8-4/golafshani.pdf (accessed 12 August 2010).

Greene, J.C. (2005), “The generative potential of mixed methods inquiry”, International Journal ofResearch Method in Education, Vol. 28 No. 2, pp. 207-11.

Hall, B. and Howard, K. (2008), “A synergistic approach: conducting mixed methods researchwith typological and systemic design considerations”, Journal of Mixed Methods Research,Vol. 2, pp. 248-70.

Hammond, M. and Wiriyapinit, M. (2005), “Learning through online discussion: a case oftriangulation in research”, Australasian Journal of Educational Technology, Vol. 21 No. 3,pp. 283-302, available at: www.ascilite.org.au/ajet/ajet21/hammond.html (accessed 12 August2010).

Hoffman, D.M. (2009), “Multiple methods, communicative preferences and the incrementalinterview approach protocol”, Forum Qualitative Sozialforschung/Forum: QualitativeSocial Research, Vol. 10 No. 1, Art. 41, available at: http://nbn-resolving.de/urn:nbn:de:0114-fqs0901419. (accessed 12 November 2009).

Ireland, J., Tambyah, M.M., Neofa, Z. and Harding, T. (2008), “The tale of four researchers:trials and triumphs from the phenomenographic research specialization”, paperpresented at AARE 2008 International Education Conference: Changing Climates:Education for Sustainable Futures, Queensland University of Technology, Brisbane,30 November-4 December, available at: www.aare.edu.au/08pap/ire08373.pdf (accessed12 August 2010).

Janesick, V.J. (2000), “The choreography of qualitative research design: minuets, improvisationsand crystallization”, in Denzin, N. and Lincoln, Y.S. (Eds), Handbook of QualitativeResearch, Sage, Thousand Oaks, CA, pp. 379-400.

Johnson, R.B. and Onwuegbuzie, A.J. (2004), “Mixed methods research: a research paradigmwhose time has come”, Educational Researcher, Vol. 33, pp. 14-26.

Johnstone, P.L. (2001), “The process and organisational consequences of new artefact adoption insurgery”, a thesis submitted for the degree of Doctor of Philosophy, Macquarie GraduateSchool of Management, Macquarie University, Sydney.

Johnstone, P.L. (2004), “Mixed methods, mixed methodology health services research in practice”,Qualitative Health Research, Vol. 14, pp. 259-71.

Jull, A., Wills, M., Scoggins, B. and Rodgers, A. (2005), “Clinical trials in New Zealand – treadingwater in the knowledge wave?”, New Zealand Medical Journal, Vol. 118 No. 1221, available at:www.nzma.org.nz/journal/118-1221/1638/content.pdf (accessed 12 November 2009).

LaFleur, J., Tyler, L.S. and Sharma, R.R. (2004), “Economic benefits of investigational drugservices at an academic institution”, American Journal of Health System Pharmacy, Vol. 61No. 1, pp. 27-32.

Lillis, A.M. (1999), “A framework for the analysis of interview data from multiple field researchsites”, Accounting and Finance, Vol. 39, pp. 79-105.

Lincoln, Y.S. and Guba, E.G. (1985), Naturalistic Inquiry, Sage, Newbury Park, CA.

Evaluatingclinical trials

87

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

Lincoln, Y.S. and Guba, E.G. (2000), “Paradigmatic controversies, contradictions and emergencyconfluences”, in Denzin, N.K. and Lincoln, Y.S. (Eds), Handbook of Qualitative Research,2nd ed., Sage, Thousand Oaks, CA, pp. 163-88.

McDonagh, M.S., Miller, S.A. and Naden, E. (2000), “Costs and savings of investigational drugservices”, American Journal Health Systems Pharmacy, Vol. 57 No. 1, pp. 40-3.

Mark, M. and Shatland, L. (1985), “Stakeholder-based evaluation and value judgements”,Evaluation Review, Vol. 9 No. 5, pp. 605-26.

Marton, F. (1986), “Phenomenography: a research approach to investigating differentunderstandings of reality”, Journal of Thought, Vol. 21 No. 3, pp. 28-43.

Mitchell, R.K., Agle, B.R. and Wood, D.J. (1997), “Toward a theory of stakeholder identificationand salience: defining the principle of who and what really counts”, Academy ofManagement Review, Vol. 22 No. 4, pp. 853-88.

Modell, S. (2009), “In defence of triangulation: a critical realist approach to mixed methods research inmanagement accounting”, Management Accounting Research, Vol. 20, pp. 208-21.

Moffatt, S., White, M., Mackintosh, J. and Howel, D. (2006), “Using quantitative and qualitativedata in health services research – what happens when mixed method findings conflict[ISRCTN61522618]”, BMC Health Services Research, Vol. 6 No. 28, available at: www.biomedcentral.com/1472-6963/6/28 (accessed 12 August 2010).

Morse, J.M. (2000), “Determining sample size”, Qualitative Health Research, Vol. 10 No. 1, pp. 3-5.

O’Cathain, A., Murphy, E. and Nicholl, J. (2007), “Integration and publications as indicators of‘yield’ from mixed methods studies”, Journal of Mixed Methods Research, Vol. 1 No. 2,pp. 147-63.

Richardson, L. (2000), “New writing practices in qualitative research”, Sociology of Sport Journal,Vol. 17 No. 1, pp. 5-20.

Ryan, S. (2009), “Benefits of conducting commercial clinical trials in a NZ public hospital”, paperpresented at the New Zealand Clinical Research Conference, Auckland, 20-21 August.

Sale, J.E.M., Lohfeld, L.H. and Brazil, K. (2002), “Revisiting the quantitative-qualitative debate:implications for mixed-methods research”, Quality & Quantity, Vol. 36, pp. 43-53.

Sandelowski, M. (2000), “Whatever happened to qualitative description?”, Research Nursing andHealth, Vol. 23, pp. 334-40.

Smith, J.K. and Heshusius, L. (1986), “Closing down the conversation: the end of thequantitative-qualitative debate among educational inquirers”, Educational Researcher,Vol. 15, pp. 4-13.

Stange, K., Crabtree, B. and Miller, W. (2006), “Publishing multimethod research”, Annals ofFamily Medicine, Vol. 4 No. 4, pp. 292-4.

Tashakkori, A. and Creswell, J.W. (2007), “The new era of mixed methods”, Journal of MixedMethods Research, Vol. 1, pp. 3-7.

Tashakkori, A. and Teddlie, C. (Eds) (2003), Handbook of Mixed Methods in Social andBehavioural Research, Sage, Thousand Oaks, CA.

Tay, J. and Tinmouth, A. (2007), “Observational studies: what is a cohort study?”, Transfusion,Vol. 47, July, pp. 1115-17.

Teddlie, C. and Tashakkori, A. (2009), Foundations of Mixed Methods Research: IntegratingQuantitative and Qualitative Approaches in the Social and Behavioral Sciences, Sage,Thousand Oaks, CA.

Watson, E. (2006), Pharmaceutical Research and Development in New Zealand – on the Brink ofAbyss, Nazadel Ltd, Auckland.

QRAM8,1

88

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

Personal communication

Ryan, S. (2008), “Overhead calculations will provide value to CCRep, CMDHB, the Ministry ofHealth (MOH) and the Health Research Council”, personal communication betweenLynette Murphy and Dr Stuart Ryan, General Manager CCRep, 1 August.

Further reading

Dolan, P. (2000), “The measurement of health-related quality of life”, in Culyer, A.J. andNewhouse, J.P. (Eds), Handbook of Health Economics, Elsevier, Amsterdam, pp. 1723-55.

Dolan, P., Gudex, C., Kind, P. and Williams, A. (1995), “A social tariff for EuroQol: results from aUK general population survey”, Discussion Paper No. 138, Centre for Health Economics,University of York, York.

Giddings, L.S. (2006), “Mixed-methods research: positivism dressed in drag?”, Journal ofResearch in Nursing, Vol. 11 No. 3, pp. 195-203.

Guest, G., Bunce, A. and Johnson, L. (2006), “How many interviews are enough? An experimentwith data saturation and variability”, Field Methods, Vol. 18 No. 1, pp. 59-82.

Ministry of Health (2005), Laboratory Claims Data Warehouse Data Dictionary Version 2.0,Ministry of Health, Wellington.

Morse, J.M. (2001), “Using shadowed data”, Qualitative Health Research, Vol. 11 No. 3, pp. 291-2.

Murphy, E., Dingwall, R., Greatbatch, D., Parker, S. and Watson, P. (1998), “Qualitative researchmethods in health technology assessment; a review of the literature”, Health TechnologyAssessment, Vol. 2 No. 16, pp. 271-4.

PHARMAC (2005), “Recommended methods to derive clinical inputs for proposals to PHARMACVersion 1B”, 20 July, available at: www.pharmac.govt.nz/2008/05/19/Recommended%20methods%20to%20derive%20clinical%20inputs%20for%20proposals%20to%20PHARMAC%20.pdf/text (accessed 12 August 2010).

Richardson, L. (1991), “Post modern social theory: representational practices”, Socila Theory,Vol. 9, pp. 173-9.

Romney, A., Batchelda, W. and Weller, S. (1986), “Culture as consensus: a theory of culture andinformant accuracy”, American Anthropologist, Vol. 88, pp. 313-38.

Sosulski, M.R. and Lawrence, C. (2008), “Mixing methods for full-strength results”, Journal ofMixed Methods Research, Vol. 2 No. 2, pp. 121-48.

Starks, S.H. and Trinidad, S.B. (2007), “Choose your method: a comparison of phenomenology,discourse analysis, and grounded theory”, Qualitative Health Research, Vol. 17, pp. 1372-80.

Twinn, S. (2003), “Status of mixed methods research in nursing”, in Tashakkori, A. andTeddlie, C. (Eds), Handbook of Mixed Methods in Social and Behavioral Research,Sage, Thousand Oaks, CA, pp. 541-56.

Appendix. Survey questions1. How does the international community benefit from clinical drug trials?2. What are the risks, tradeoffs, disadvantages or costs to the international community in

conducting clinical drug trials?3. How do pharmaceutical companies benefit from clinical drug trials?4. What are the risks, tradeoffs, disadvantages or costs to pharmaceutical companies in

conducting clinical drug trials?5. How do other New Zealanders benefit from clinical drug trials performed in Counties

Manukau?

Evaluatingclinical trials

89

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

6. What are the tradeoffs, disadvantages or costs to other New Zealanders in Counties Manukauconducting clinical drug trials?

7. How does CMDHB benefit from sponsoring clinical drug trials?8. What are the tradeoffs, disadvantages or costs to CMDHB in conducting clinical drug trials?9. How does CMDHB benefit from sponsoring clinical drug trials?

About the authorsLyn Murphy is a Senior Lecturer in Management at Manukau Institute of Technology. In 2010,Lyn Murphy was elected onto the CMDHB. She is a PhD Student enrolled with the Universityof Tasmania. Lyn Murphy is the corresponding author and can be contacted at: lyn.murphy@manukau.ac.nz

William Maguire is a Senior Lecturer in Management Accounting at the University ofTasmania. Before entering academic life, he spent six years in public accounting, commerce andindustry. He has been teaching, researching and consulting for more than 30 years.

QRAM8,1

90

To purchase reprints of this article please e-mail: reprints@emeraldinsight.comOr visit our web site for further details: www.emeraldinsight.com/reprints

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)

This article has been cited by:

1. Parker Lee. 2014. Qualitative perspectives: through a methodological lens. Qualitative Research inAccounting & Management 11:1, 13-28. [Abstract] [Full Text] [PDF]

2. Kym Fraser. 2014. Defeating the ‘paradigm wars’ in accounting: a mixed methods approach is needed in theeducation of PhD scholars. International Journal of Multiple Research Approaches 4299-4325. [CrossRef]

3. Kym Fraser. 2014. Position paper: Defeating the ‘paradigm wars’ in accounting: A mixed-methodsapproach is needed in the education of PhD scholars. International Journal of Multiple Research Approaches8, 49-62. [CrossRef]

Dow

nloa

ded

by O

ND

OK

UZ

MA

YIS

UN

IVE

RSI

TY

At 0

8:09

07

Nov

embe

r 20

14 (

PT)