How to Design a Mixed Methods Study

by John W. Creswell, Ph.D. and Vicki L. Plano Clark, M.S.

University of Nebraska-Lincoln

Andrews University, July, 2004

2

3

How would you combine two types of data?

Qualitative Text Data

This is a sample of a text file of words that might be collected on transcripts through interviews, fieldnotes from observations, or from optically-scanned documents.

Quantitative Numeric Data

234254311223213223322543312243243213243332334441222211143214321322111555233143243213243332135432

4

Objectives of the workshop:

Let’s design a mixed methods study Let’s study how people learn mixed

methods research in this room? (or you can work on your own project and follow along at each step)

Let’s start with a title. Write a title. What data will we collect?

5

What are types of quantitative and qualitative data?

Quantitative data Close-ended

scales Attitudinal/

behavioral scales Behavioral

checklists Census,

attendance records

Qualitative data Open-ended

responses Semi-structured

interviews Semi-structured

observations Records/

documents Videotapes

6

Let’s identify our quantitative and qualitative data collection

7

Now let’s consider some reasons for why we are collecting (and mixing) both forms of data Together quantitative and qualitative data provide both

precise measurement and generalizability of quantitative research and the in-depth, complex picture of qualitative research

To validate quantitative results with qualitative data We do not have an adequate instrument. Thus, we need

to explore views and develop an instrument Our quantitative data provide a general explanation and

we need to follow-up with participants and have them explain the quantitative results

In our experiment, outcomes to be measured are not enough; they need to be complemented by understanding the process of participants

8

Let’s identify our reason for mixing

9

So…

There are good reasons for gathering both forms of data

But…there are certain requirements for this to work best

10

Requirement #1: Now let’s consider whether we have the skills, time, and resources?

We need minimum skills in both qualitative and quantitative data collection. What do we need?

We need time and resources for extensive data collection and analysis. How much time and resources do we need?

11

Write down the skills, time, and resources we will need

12

Requirement #2: The audience(s)

Does our audience appreciate both numbers and stories?

Are they familiar with this design? Do they need to be educated? Are examples of published studies

available in our content area?

13

Let’s identify the audiences

14

But audiences may not recognize it yet because it is so new

Increased use and acceptance of qualitative research from 1990’s to present

The complexity of our research problems today requires understanding trends, differences, as well as individual stories, setting

Individuals advocating for and writing about mixed methods research as a distinct, new procedure (e.g., books)

15

They may think that it is analyzing data separately

Quantitative Data Qualitative Data

Mixing: converging the data or connectingthe data

16

But how do we mix?

Converge data:

Qual QuanResults

Connect data:

Qual Quan Results

17

Why our audience may recognize it

The evidence Books Methodological articles Many published research studies

using it Federal agencies Private foundations

18

19

Other writings, initiatives on mixed methods research: Research studies reported in journals Methodological articles exploring issues

and procedures Website for bringing mixed methods

writers together Conference sessions Handbook of Mixed Methods in Social

and Behavioral Research Private foundation interest; federal

agency interest

20

NIH Guidelines

- Mentioned severalapproaches for combiningqualitative and quantitative research

- Considerations for deciding what model to use (literatureavailable, prior studies, realistic design, expertise)

- Need to describe each method thoroughly

21

Quotes: “Combining qualitative and quantitative

methods has gained broad appeal in public health research. The key question has become not whether it is acceptable or legitimate to combine methods, but rather how they will be combined to be mutually supportive and how findings achieved through different methods will be integrated.” (NIH, Office of Behavioral and Social Science Research, 1999).

22

National Academy of Sciences

Three major research questions inquality educational research:

• What is happening? (qualitative designs)

• Is there a systematic effect? (a quantitative experiment)

• Why or how it is happening? (a qualitative followup)

23

But even if they recognize it, they may not appreciate or understand how to design a mixed methods study

“We are interested in a randomized control trial with a non-experimental approach embedded within it.” (a private foundation officer)

“We accept multi-method studies, but investigators mostly do not sort out the complexity of these projects so that we can understand them.” (a federal projects officer).

24

We need to define mixed methods research for our audiences

Mixed methods research is a design for collecting, analyzing, and mixing both quantitative and qualitative data in a single study or series of studies to understand a research problem.

The purpose of this form of research is that both qualitative and quantitative methods, in combination, provide a better understanding of a research problem or issue than either method alone.

25

Now we could mix within single studies or multiple studies

Single Study:

Results

QualQuan

Multiple Studies (called multimethod research):

Study 1 Study 2 Study 3 Study 4

Quan Qual Qual Quan

26

So how do we design a mixed methods study? The model

Type of mixed methods design

Procedures for:•designing the title•writing the introduction to a study•writing the purpose statement and research questions/hypotheses•data collection•data analysis•writing the mixed methods report•evaluating the mixed methods research

Worldviews, theoretical frameworks, problem and research question, skills, resources

27

What is a worldview? Philosophy about your preferences for how

you learn about something through research

You prefer the quantitative worldview: you are the expert, you decide what needs to be learned, you build in objectivity

You prefer the qualitative worldview: participant is the expert, participant helps you build knowledge, you bring personal bias in

You prefer both the quantitative and qualitative worldview

28

The next steps in planning our study

Let’s write the overall research question for our study

29

Then let’s choose a type of mixed methods study to conduct

What designs are possible?

30

Types of mixed methods designs

I. Triangulation Mixed Methods Design

II. Nested Mixed Methods Design

QUANData and Results

QUANPre-testData and Results

+ QUALData and Results

QUANPost-testData and Results

Interpretation

Qual Process

31

Types of mixed methods designs

III. Explanatory Mixed Methods Design

IV. Exploratory Mixed Methods Design

QUANData and Results

QUALData and Results

qualData and Results

quanData and Results

Follow-up

Building

32

Triangulation Design: Characteristics

Collecting both quantitative and qualitative data

Collecting these data at the same time in the research procedure

Analyzing the quantitative and qualitative data separately

Comparing or combining the results of the quantitative and qualitative analysis

Example: collect survey data (quantitative) and collect individual interviews (qualitative) and then compare the results

33

Triangulation Design: When is it used? When you want to combine the

advantages of quantitative (trends, large numbers, generalization) with qualitative (detail, small numbers, in-depth)

When you want to validate your quantitative findings with qualitative data

When you want to expand your quantitative findings with some open-ended qualitative data (e.g., survey with closed- and open-ended data)

35

Nested Design: Characteristics

Collecting both quantitative and qualitative data Collecting both types of data at the same time Having ONE form of data play a smaller role in the

study than the other form of data Also,

Using one form of data to answer one question; the other form another question

Collecting one form of data at one level of analysis and another at another level of analysis

Example: You conduct an experiment and during the experiment you gather qualitative interview data. The outcomes of the experiment assessed quantitatively address different questions than the process of the experiment explored qualitatively.

36

Nested Design: When is it used? When you do not have time or

resources to commit to extensive quantitative and qualitative data collection

When you want to study the process of an experiment as well as the outcomes

When you want to examine different levels in an organization

37

QuanData collectionPre-test

QuanData collectionPost-test

Process – collectionand analysis of qualitative

data

Nested Research Design

Experiment

Intervention

38

Explanatory Sequential Design: Characteristics Viewing the study as a two-phase project Collecting quantitative data first followed

by collecting qualitative data second Typically, a greater emphasis is placed

on the quantitative data in the study Example: You first conduct a survey and

then follow up with a few individuals who answered positively to the questions through interviews

39

Explanatory Sequential Design: When do you use it?

When you want to explain the quantitative results in more depth with qualitative data (e.g., statistical differences among groups, individuals who scored at extreme levels)

When you want to identify appropriate participants to study in more depth qualitatively

Here is an example of an explanatory design:

Quantitative Data*

Number of cigarettes CES-D6

Qualitative Data*

Semi-structured interviews, audio recorded and transcribed

* Data collected 10 times over the course of a

calendar year for 40 participants

Quantitative Analysis

Graphic plot of CES-D6 scores over time for each participant Graphic plot of cigarettes/day values over time for each participant

Case Selection

Qualitative Analysis

Description of each case

Identification of life events occurring during critical months where

smoking increased or decreased

Thematic analysis of life events for each

case Cross-case thematic

analysis

Interpretation

Why did changes in smoking occur?

Quantitative Data Collection(quan)

Qualitative Data Collection

Interpretation –based on quanand QUAL results

Qualitative Data Analysis(QUAL)

Case Selection Quantitative

Data Analysis(quan)

Selected 5 casesmaximally varyingIdentified criticalmonths in which smoking varied

Creswell et al. (in progress)

+

41

Exploratory Sequential Design: Characteristics Viewing the study as a two-phase project Qualitative data collection precedes

quantitative data collection Typically, greater emphasis is placed on

the qualitative data in the study Example: You collect qualitative diary

entries, analyze the data for themes, and then develop an instrument based on the themes to measure attitudes on a quantitative survey administered to a large sample.

42

Exploratory Sequential Design: When do you use it? To develop an instrument when

one is not available (first explore, then develop instrument)

To develop a classification or typology for testing

To identify the most important variables to study quantitatively when these variable are not known

Phase I Qualitative Research - Year 1

Qualitative Data Collection Unstructured Interviews -50 participants 8 observations at the site16 documents

Qualitative Data Analysis Text Analysis: Using QSR N6

Qualitative FindingsDevelopment of codes and themesfor each site

Phase II Quantitative Research - Year 2

Quantitative Instrument Development

Quantitative Test of the Instrument

Create approximately a 80-iteminstrument plus demographics

Administer survey to 500 individuals

Determine factor structure of items and conduct reliability analysis for scales

Quantitative ResultsDetermine how groups differusing ANOVA test

Sequential Exploratory Mixed Methods Design

44

How will we analyze the quantitative and qualitative data (within the design types)?

Qualitative analysis Text/image data Coding Themes Description Interrelated

themes

Types of analysis: Quantitative

analysis Numeric data Descriptive trend

analysis Hypothesis

testing, effect size, interval estimates

45

Triangulation data analysis

QUANdata collection

QUANdata analysis

QUALdata collection

QUAL data analysis

Results

• Separate QUANand QUAL dataanalysis

• Two options• Data transformation (changeQUAL to QUAN or QUAN to QUAL)• Comparison (keep separate andcompare/contrast)

CountAdj.Count** Row PctColumn Pct

PatientsN=2

PhysiciansN=4

Medical AssistantsN=4

Themes

FamiliarityWith the Form

136.541.95.8

174.2527.43.0

194.7530.75.3

4915.5100.0

Reactions to the Form

2311.522.210.2

10025.048.317.4

6115.329.516.9

18451.75100.0

Use for Managing Depression

6733.538.629.8

17744.2551.030.7

369.010.410.0

28086.75100.0

Changes to the Form

11557.537.551.1

19649.032.034.0

18746.830.551.7

498153.3100.0

Situational Use of the Form

73.58.93.1

8621.554.414.9

5814.536.716.1

15139.5100.0

225112.5100.0

576144.0100.0

36190.3100.0

Table. Example of Data Transformation of Text Units into Numeric Data

47

QuanData collectionPre-test

QuanData collectionPost-test

Qualitative Process

Nested data analysis

Post-test scoresor gain scores

Themes/Codes/Interrelated Themes

Pre-testscores

DataAnalysis

Compare/Describe Results

Quantitative Experiment

Intervention

48

Explanatory sequential data analysis

QUANdata analysis

• Statistical results• Outlier cases• Extreme cases

Qual data collection(purposeful sampling)

• Select cases based on s.d. variables• Select cases to represent outliers• Select cases to represent extreme cases• Select cases to make group comparisons

Qualanalysis

• codes• themes• cases

49

Exploratory sequential data analysis

QUAL data analysis

Quotes

Codes

Themes

Quan data analysisinstrument development

Items on a survey

Variables on a survey

Scales on a survey

50

Let’s identify how we will analyze the data

51

Drawing our Design

Let’s draw a picture of our design Identify the type of design Add in data collection Add in data analysis Show the flow of activities Add in “products” for our

audiences

52

Helpful tips for creating this visual:

1. Give a title to the visual model.2. Choose either horizontal or vertical layout for the model.3. Draw boxes for quantitative and qualitative stages of data

collection, data analysis and interpretation of the study results.

4. Use capitalized (QUAN) or small letters (quan) to designate priority of quantitative and qualitative data collection and analysis.

5. Use single-headed arrows to show the flow of procedures in the design.

6. Specify procedures for each quantitative and qualitative data collection and analysis stage.

7. Specify expected products or outcomes of each quantitative and qualitative data collection and analysis procedure.

8. Make your model simple.9. Size your model to one page.

53

Now let’s rework our purpose statement using some scripts

54

“The purpose of this concurrent mixed methods study is to better understand a research problem by converging both quantitative (numeric) and qualitative (text or image) data. In this approach, ___________ (quantitative instruments) will be used to measure the relationship between the ________ (independent variables) and __________ (dependent variables). At the same time in the study, the __________ (central phenomenon) will be explored using _____________ (qualitative interviews, documents, observations, visual materials) with _________ (participants) at ____________ (the research site).”

Sample Script for a Concurrent Design(Triangulation or Nested)

55

“The purpose of this two-phase, exploratory mixed methods study will be to explore participant views with the intent of using this information to develop and test an instrument with a sample from a population. The first phase will be a qualitative exploration of a _______(central phenomenon) by collecting ___________(data) from ____________ (participants) at _______ (research site). Themes from this qualitative data will then be developed into an instrument (or survey) so that the __________ (theory and research questions/hypotheses) can be tested that ________ (relate, compare) ____________ (independent variable) with __________ (dependent variable) for _________(sample of a population) at _________ (research site).”

Sample Script for a Sequential Exploratory Design

56

“The purpose of this two-phase, explanatory mixed methods study will be to obtain statistical, quantitative results from a sample and then follow-up with a few individuals to probe or explore those results in more depth. In the first phase, quantitative research questions or hypotheses will address the relationship or comparison of __________ (independent) and ________ (dependent) variables with ___________ (participants) at ___________(the research site). In the second phase, qualitative interviews or observations will be used to problem significant _______(quantitative results) by exploring aspects of the ________ (central phenomenon) with_______ (a few participants) at ____________ (research site).”

Sample Script for a Sequential Explanatory Design

57

Criteria for evaluating our plan:

Use appropriate terminology for title and design

Provide a rationale for mixing and include it early in the study (“when you use…”)

Create a mixed methods purpose statement

Identify types of qual and quan data to be collected and qual and quan data analysis steps

Include a visual/procedural diagram of methods with timeline

Use rigorous procedures for the quantitative data collection and analysis

58

Let’s share our drawings of our mixed methods procedures

How to Design a Mixed Methods Study

by John W. Creswell, Ph.D. and Vicki L. Plano Clark, M.S.

University of Nebraska-Lincoln

Andrews University, July, 2004