1 Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask January 10, 2008 Presented by: Yvonne M. Watson, Evaluation

1

Collecting and Analyzing Qualitative Data: All You Wanted To Know, But

Were Afraid To Ask

January 10, 2008

Presented by: Yvonne M. Watson, Evaluation Support DivisionNational Center for Environmental InnovationOffice of Policy, Economics and InnovationU.S. Environmental Protection Agency

and

John McLaughlinMcLaughlin Associates

2

Workshop Objectives

Participants will learn: 1) when to use qualitative data; 2) what data collection methods are available; 3) how to select participants for qualitative data collection; and 4) the steps for analyzing qualitative data.

3

Session Overview

Module 1: Data Collection

I. Overview

II. Qualitative Data Collection Methods: Interviews

Focus Groups

Survey/Questionnaire (Open-ended questions)

Document/File Review

Observation

Module 2: Data Analysis

III. Steps for Analyzing Qualitative Data

IV. Assessing the Rigor of Qualitative Data

Module 3: Appendix, Resources and References

4

Module 1: Data Collection

5

Orientation Exercise

As a group, discuss your perceptions regarding qualitative data versus quantitative data with respect to:

Quality

Collection

Analysis

Utility

6

Quantitative and Qualitative Data

Quantitative Qualitative

Numerical data

Highly structured

Creates precise measures

Relatively easy to analyze

May not explain “why”

Closed

Risk of bias

Text (Descriptions of reactions, opinions, behaviors, experiences)

Structured Unstructured

Creates lots of rich data regarding perceptions

Challenging to analyze

Labor intensive to collect

Risk of bias (evaluator and subject)

(World Bank , Module 6: Data Collection Methods, Slides 20 and 21)

7

Quantitative Data

Answers questions about: How much? How many? How often?

Use quantitative data when you:

• Want to do statistical analysis• Want to be precise• Know exactly what you want to measure• Want to cover a large group or population

Quantitative Methods:

• Examples: Survey questionnaires, tests, checklists, monitoring data.

• Often used to obtain information on outcomes and causal relationships.

8

Qualitative Data

Answers questions which begin with: Why? How? In what way?

Use qualitative data when you:

• Are concerned with opinions, experiences and feelings of individuals producing subjective data.

• Want anecdotes or in-depth information• Are seeking understanding, themes, issues• Are not sure what you want to measure• There is no need to quantify• Are unable to collect quantitative data

Qualitative Methods:

• Examples: Interviews, focus groups, document review, direct observation.

• Often used to obtain information on processes, meanings, in-depth understanding.

9

Levels/Types of Qualitative Information

Different levels/types of information can be gathered from respondents.

Formulate questions that yield information regarding:

• Reactions, feelings and emotions

• Opinions and values

• Knowledge and learning

• Changes in skills

• Behaviors/experiences

• Effectiveness

• Background/history/context

(Hancock 1998)

10

Considerations in Selecting a Data Collection Method

Your evaluation or study question

Stakeholders’ desired sources of data

Resources (Financial and Skills)

Time (available to collect data)

Access to and availability of subjects/respondents

Information Collection Request (ICR)

11

Method Overall Purpose Advantages Challenges

Interviews

To fully understand someone's impressions or experiences, or learn more about their answers to questionnaires

- Get full range and depth of information

- Get targeted information

- Develops relationship with client- Can be flexible with client

-Time consuming/ costly

- Can be hard to compare responses- Interviewer can bias client's responses

- Inaccurate recall

Focus Groups

To explore a topic in depth through group discussion, e.g., about reactions to an experience, understanding common complaints

- Quickly and reliably get common impressions - Can be efficient way to get much range and depth of information in short time

-Can be hard to compare responses- Need good facilitator for safety and closure- Difficult to schedule 6-8 people together

-Inaccurate recall

Direct Observation

To gather accurate information about how a program actually operates, particularly about processes

- Covers events in real-time

- Can adapt to events as they occur

- Covers context of events- Obtain insight into personal behavior and motives

-Can be difficult to interpret observations

-Time consuming/ costly-Can influence behaviors of program participants

Qualitative Data Collection Methods

12

MethodOverall

PurposeAdvantages Challenges

Document Review

To obtain impression of how program operates without interrupting the program; is from review of applications, finances, memos, minutes, etc.

-Get comprehensive and historical information

-Doesn't interrupt program or client's routine in program- Information already exists- Few biases about information

- Broad coverage over time.

- Often takes much time- Info may be incomplete- Need to be clear about what looking for- Data is restricted to what already exists

- Can have reporting biases

-Access might be blocked

Surveys/ Question-

naires

(open-ended

questions)

To quickly and/or easily get lots of information from people in a non threatening way

- Can complete anonymously- Inexpensive to administerto many people- Easy to compare and analyze- Can get lots of data- Many sample questionnaires already exist

- Potentially inaccurate recall/feedback- Wording can bias client's responses- Are impersonal- May need sampling expert- Doesn't get full story

- May need an ICR

Data Collection Methods (con’t)

13

Interviews: Things to Consider

Format

•Structured

•Semi-structured

•Unstructured

Questions

•Open-ended

•Closed-ended

•Sequencing

Location

•In-person

•Telephone

Duration

Selection of Interviewees

Equipment/Supplies

•Recorder (tape or digital)

•Laptop

•Note Paper

Schedule

Interviewer Skills

Resources

•Financial, Staff

14

Interviews: Format

Structured Interview - Interviewer asks a specific set of questions of each respondent in the same way. This allows the interviewer to obtain a uniform set of data from each respondent.

Semi Structured- Includes a series of open ended questions based on the subject of interest to the interviewer but provides flexibility to explore issues in greater detail.

Unstructured Interview – General sets of questions are asked so that subjects respond in a free flowing manner resembling a conversation. The interview is designed to find out more information about a topic.

(Hancock 1998)

15

Interviews: Questions

Open-ended Questions:

Solicit additional information from the respondent and will require more than one or two word responses. Respondents are encouraged to explain their answers.

Advantages:• Respondents can provide more information about a subject.• Researchers can better understand respondents true feelings,

reactions about an issue.• Allows for an unrestricted response

Disadvantages: • Time-consuming • Challenging for respondents that are less articulate

(Hancock 1998)

16

Interviews: Questions (Cont’d)

Close-ended Questions:

Limit interviewee’s responses to a pre-existing set of answers e.g., yes/no, true/ false, or multiple choice with an option for other or a ranking scale response option can be used. Questions can be restrictive and can be answered in a few words.

Advantages:• More easily analyzed• Answers can be assigned a numerical value• Questions can be more specific

Disadvantages:• Can yield incomplete responses• Discourages disclosure• Results could be misinterpreted

17

Interviews: Questions (Cont’d)

Example 1:

Open-ended: Tell me about your relationship with the program’s Project Officer.

Closed-ended: Do you have a good relationship with the program’s Project Officer?

Example 2:

Open-ended?: Can you describe your satisfaction with the program?

Closed-ended?: How satisfied are you with the program?

□ Very satisfied□ Somewhat satisfied□ Dissatisfied

18

Interviews: Location, Duration and Schedule Location

• Decide whether to conduct in-person or telephone interviews.

• Select a time and place that is quiet and free of distractions.

Duration

• Schedule the interview for no more than 1 hour.

Schedule

• Leave ample time to review transcripts and notes after each interview (at least one hour).

19

Interviews: Selection of Interviewees How Should Participants Be Selected?

Snowball sampling: Identify a few members of the community of interest, and then ask them for additional contacts.

Contrasting cases: Select cases with high contrast to learn about what underlies the differences between them.

Typical cases: Select cases that appear to represent the average, normal, typical situation.

Critical cases: Select cases that are considered to be crucial to understanding the study/ evaluation topic or which are assumed to represent the perspective of many other cases.

(Kakoyannis 2007)

20

Interviews: Equipment/Supplies Needed

Equipment/Supplies

• Note paper, recorder (tape or digital) or laptop to record/document responses

Note taking tips

• Take good notes without detracting from the conversation

• Write while maintaining eye contact

• If interviewee says something you want to capture, it is OK to ask them to repeat it or to finish what you are writing before asking the next question.

(World Bank, Module 6: Data Collection Methods, Slides 53 and 54)

21

Interviews: Skills and Resources Needed

Interviewer Skills

• Identify an experienced Interviewer• Interviewer should be aware of any cultural norms: eye

contact, direct questions, gender issues • Stick to the script:

If asking close-ended questions, ask exactly the way written.If asking open-ended questions go with the flow, not too directive.

• Avoid asking yes/no questions. Ask, how, who and why* • Don’t step outside of your role as an interviewer• Good listener

Resources

• Ideally, have a second person to help take notes or use a recorder

*In some instances, when the interviewer consistently asks the respondent why, it may be interpreted as aggressive


22

Focus Groups: Things to Consider

Format

•Group Size•Number of Groups

Questions

•Open-ended•Closed-ended

Location

•In-person•Conference Call

Duration

Selection of Focus Group Participants

Equipment/Supplies

•Recorder (tape or digital)•Laptop•Note Paper

Schedule

Skills

Resources

•Financial, Time, Staff

23

Focus Groups: Format and Questions

Size

• Recommended size of group is 6-10 • Focus group members should have something

in common

Number of Focus Groups

• No rules here. However, more than one is recommended to ensure sufficient information is collected.

Questions

• Start broad and then be specific

(Hancock 1998)

24

Focus Groups: Location, Duration and Schedule

Location

• Comfortable, neutral, safe environment

• Free from distractions and accessible

• Setting: around a table or in a circle

Duration

• Typically 1-2 hours (clear start and stop times)

Schedule

• Piggy back on existing meetings/conferences

• Do not over schedule: 2 or 3 in a day is plenty for one moderator/facilitator.

25

Focus Groups: Selecting Participants

May need to have homogeneous groups with respect to gender, race, social class, managers vs. staff etc.

Cultural norms are important.

Things to Consider:

• What is the geographical spread of your potential participants?

• Are there any specific inclusion criteria for selecting participants

• Where or how could you obtain a list of potential participants?

• Are there any pre-existing groups and what are the advantages and disadvantages of using members?

(World Bank , Module 6: Data Collection Methods, Slide 73), (Hancock 1998)

26

Focus Groups: Equipment, Supplies and Resources Needed

Equipment

• Bring equipment and supplies needed to document/record the focus group. Note paper, recorder (tape or digital), laptop

Resources

• Facilitator and note-taker

Other

• Consider providing food, incentives, childcare, transportation etc., to respondents

27

Focus Groups: Skills Needed

Skilled facilitation is essential.

Facilitator should know the script so focus group appears conversational.

Ensure that everyone is heard.

Ask: “What do other people think?”

State: ”We have heard from a few people, do others have the same views or different views?”

Active listener

Develops and adheres to ground rules

Accept all views while managing differences of opinion.

•So we have different perspectives

Probe for elaboration

•Tell me more.

Manage time

•Closing off discussion and moving to next topic.

Invisible: say as little as possible

•Let conversation flow across the table with minimal direction.

Keep personal views outside the room.

(World Bank, Module 6: Data Collection Methods, Slides 78, 80 and 81

28

Survey/Questionnaire (Open-ended questions): Things To Consider

Format

Type of Questions

•Open-ended

•Closed-ended

Method of Administration

•Self-Administered vs. Guided by Interviewer

•Mail, Telephone, Electronic, In-person

Length

Duration

• Consider 10 – 20 minutes

Confidentiality

Response rate

•May decrease if mailed.

29


Meeting minutes

Organizational mission statements

Letters, records and laws

Memoranda

Correspondence

Official publications and reports

Personal diaries

Photographs and memorabilia

Progress reports

Studies

Collection and examination of documents produced in daily life as a means for better understanding the values of people in the study.

30

Document/File Review: Things To Consider

Type of information

• What data are you looking for? (context, process, outcome, satisfaction)

Accuracy

• Were data accurately recorded? Is it trustworthy? Has it undergone QA?

Access/Availability

• Is permission needed to access files?• Are files in a central location or dispersed

geographically?

Completeness

• Are data available for appropriate years, stakeholders

31

Document/File Review: Things To Consider

Confidentiality

• Can data be shared publicly? Do legal restrictions exist? (e.g., CBI, personnel data)

Informative

• Will data collected from the files help provide information to answer the study question?

Time

• Does the volume of documents/files increase the level of effort needed to complete the review?

32


Advantages:• Unobtrusive • Analysis can easily be replicated because the data are

stable • Documents can allow broader coverage of data by

giving insight into past events that form the context within which the current study is operating in

• Often less expensive and faster than collecting original data

Disadvantages: • Difficult to access and retrieve certain documents• Data gaps exist• Data do not explain why something is occurring/

happening• Data may not be “exactly” what is needed• Selection of documents might be biased if researcher

does not collect a broad range of data

33

Exercise 1:

Selecting a Qualitative Data Collection Method

34

Tribal GAP Data Collection Efforts

Reviewed a sample of files for 111 Tribes in 9 EPA Regions w/ Federally Recognized Tribes

• GAP Accountability Tracking System• Grants Information and Control System• Audit Database• Strategic Goals Reporting System

Reviewed Regional Files (e.g., quarterly reports submitted by Tribes)

Conducted Interviews with GAP Project Officers in 8 Regions

Organized Panel Discussions w/Tribal Representatives

• United South and Eastern Tribes (USET) Impact Week, Arlington, VA

• EPA Region 5, Indian GAP Conference, Chicago, IL• EPA Region 8, Tribal Operations Committee, Denver, CO

35

Module 2: Analyzing Qualitative Data

36

Qualitative Data Analysis

Analysis and interpretation are employed to bring meaning, order, and understanding to the data. (Taylor-Powell and Renner 2003)

The purpose of qualitative data analysis is to describe, interpret, explain and understand data that are collected. (Dey 1993)

37

What is Content Analysis?

A systematic process for identifying themes and patterns in the data, coding and characterizing the themes in order to understand the issue being studied. (Russ-Eft and Preskill 2001)

38

Steps for Analyzing Qualitative Data

Step 1: Focus the analysis

Step 2: Get to know the data

Refocus the analysis if necessary

Step 3: Create Code/Categorize the data

Check validity of codes

Step 4: Identify patterns and themes using codes

Check categorization of coding

Step 5: Interpret the data

Step 6: Conduct member check

(Taylor-Powell and Renner 2003), (McNamara 1998)

39

Step 1: Focus the analysis

Review the purpose of the evaluation

Review the key study questions

• Using the research question as a guide, think about which parts of the text help inform that question

Consider a framework for analyzing the data

Processes – Data are organized to describe an important process

Issues – Data are organized to illuminate key issues (often equivalent of primary evaluation questions)

Questions – Responses to data are organized question by question

Concepts – Data organized by key concepts

(Patton 2007)

40

Step 1: Focus the Analysis

Inductive analysis

• Involves discovering patterns, themes, and categories in one’s data. Findings emerge out of the data, through the analyst’s interactions with the data.

Deductive analysis

• Involves analyzing data according to an existing framework, e.g., the program’s logic model.

Use both

• Build on the strengths of both kinds of analysis. For example, once patterns, themes, and/or categories have been established test the appropriateness of the categories.

(Patton 2007)

41

Step 2: Get to know the data

Transcribe the data

• Listen to audio/recorded tapes

• Read notes and develop a transcript

Read through the transcript first as a whole

• Make brief notes (in the margin) of interesting or relevant information you are seeing in the data

(McNamara 1998)

42

Step3: Create codes and categorize the data

Codes are labels, abbreviations or symbols that are used to identify a particular concept, theme, idea or behavior reflected in the data.

Coding involves breaking down, labeling, comparing and organizing data in order to group them into similar categories.

(Taylor-Powell and Renner 2003)

43

Step3: Create codes and categorize the data Preset Categories:

• Start with a list of themes or categories in advance, and then search the data for these topics.

Emergent Categories:

• Rather than using preconceived themes or categories, you read through the text and find the themes or issues that recur in the data.


44

Step 3: Create codes and categorize data

Review margin notes, and make a list of the different types of information found.

Review the list of data items and categorize them in a way that describes what it is about.

Categorize the code words into similar groups

• As you read, add or modify the descriptive code words so they better reflect the newer data.

• Consider whether they can be linked in some way. Develop major and minor categories if needed

Examine the list of minor and major categories of data. Compare and contrast the categories.


45

Step 4: Identify patterns and themes

Identify common, recurring patterns and themes, ideas, words or phrases

• Look for associations, connections and causal relationships in the themes

Display summaries of data to enhance/illuminate interpretation e.g., compilation sheets, flowcharts, diagrams, matrices;


46

Step 5: Interpret the data

Reflect on the themes and patterns and data collected to make sense of the data and to find meaning and significance

Draw conclusions

If possible relate these to other data sets


47

Step 6: Conduct member check

Share theories and conclusions with respondents to verify the accuracy of your interpretation

48

Exercise 2:

Analyzing Qualitative Data

49

Assessing Rigor of Qualitative Data

Demonstrating data analysis is rigorous is important given criticism and skepticism associated with qualitative data. The rigor of qualitative data may be addressed by assessing:

Reliability (of the methods employed)

Validity (of the interpretation of the data)

• Internal validity (credibility) – Extent to which the findings are credible and the “reality” that is described are credible to the people interviewed.

• External validity (transferability) – Extent findings can be generalized to a larger population of people, settings, or situations.

Objectivity(Lacey and Luff 2001)

50

Strategies for Increasing Reliability and Validity

Reliability

Describe the approach to and procedures for data analysis

Clearly document the process of generating themes, concepts or theories

Validity

Consider and discuss alternative interpretations of the findings

Carefully consider and discuss cases and data that don’t fit overall patterns and themes,

Triangulate the analysis (use of multiple data sources)

(Lacey and Luff 2001), (Patton 2007)

51

Strategies for Increasing Reliability and Validity

Check for representativeness of data

Check of bias

Cross-check data with evidence from other, independent sources

Compare and contrast data

Use extreme (groups of) informants to the maximum.

Do additional research to test the findings of your study.

Respondent validation (Get feedback from your informants.

Triangulate analysis, methods, sourcesModule 23: Analysis of Qualitative Data, International Development Research: http://www.idrc.ca/en/ev-56451-201-1-DO_Tpoic.html, pg. 3-28

52

Increasing Validity through Triangulation

Triangulation Options:

• Check out the consistency of findings generated by different data-collection methods, i.e., methods of triangulation

• Check out the consistency of different data sources within the same method, i.e., triangulation sources

• Use multiple analysts to review findings , i.e. analyst triangulation; and

• Use multiple perspectives or theories to interpret the data, i.e. theory/perspective triangulation

(Patton 2007)

53

Contact

Yvonne M. Watson

202-566-2239

[email protected]

54

Module 3: Appendix, Resources and

References

55

Appendix

Interviews: General Guidelines

Focus Group: General Guidelines

Interviews and Focus Groups: Final Thoughts…

Survey/Questionnaire: General Guidelines

Document/File Review: General Guidelines

56


Define purpose

• Link to study/evaluation objectives

Decide whether you want to ask open-ended or closed-ended questions

Draft interview questions

• Sequence questions so they flow

Prepare introduction and closure

• Purpose of the interview• How and why interviewees were selected• Close with asking whether interviewees have questions or

comments• Thank you and follow-up

Prepare a record of responses

Pre-test the instrument


57


Let interviewees know:• Why they are being interviewed• How they were selected• How the data will be used• Whether it is confidential• How long the interview will take• Whether you might want to talk to them again

Additional touches:• Share interview questions ahead of time.

- No surprises.

• Offer to share a summary of what you understand from the interview

- This might be especially useful to give the interviewee (especially if high ranking official) a greater feeling of control.

- Thank you note afterwards.


58


Every word and idea is valuable.

Take time to write up notes as carefully and in-depth possible.

Do at least a brief clean-up of notes immediately afterwards (leave an hour between interviews).

Write up full notes within a day of the interview: memory decay sets in quickly.


59


Introduction:

• Purpose of focus groups• Sponsor• Why participants were selected• How the information will be used• The ground rules• Overview of the process

Have participants introduce themselves

First question: easy, ice-breaker.

Ask main questions.

Last questions:

• Summary question: most important think that was said here that we should take with us.

• Other comments or questions?


60


Write up impressions immediately after focus group: major issues and major points of discussion.

Compare notes with your partner.

Ideally, the focus group tape will be transcribed verbatim.

If not, listen to the tape afterwards while writing in-depth notes.

• You will be surprised how much you did not hear during the actual focus group.

Leave time to prepare write-up immediately following the focus group.

Capture anything unusual that happened during the focus group.


61

Focus Group: Ground Rules

What is said here, stays here.

Everyone is encourage to participate but note everyone has to answer every question.

Respect different viewpoints.

There are no right or wrong answers.

Only one person speaks at a time.

(World Bank, Module 6: Data Collection Methods, Slide 78 and 79)

62

Interviews and Focus Groups: Final Thoughts…

When gathering data from people:

- Keep it simple, clear, easy, short

- Respect respondents time and intelligence

- Tell them how they were selected and why their participation is important

- Do no harm: keep responses confidential

Consider privacy issues- permission needed to access files?

Is permission needed to record the session?

Are you planning to attribute comments directly to individual interviewees?

63

Survey/Questionnaire

Advantages:

• People are familiar with surveys

• Some respondents prefer surveys to interviews

• Can reach respondents in several geographic locations

Disadvantages:

• Respondents may not complete the survey

• Can’t probe for additional information/details from respondents

• Respondents can misinterpret questions when a set of choices is not available.

• Increased number of open-ended questions may lower the response rate.

(Russ-Eft and Preskill 2001)

64

Survey/Questionnaire: General Guidelines

Include a brief introduction statement

Avoid abbreviations and acronyms

Avoid biases words and phrases

Exercise caution when asking about personal information

Assess personal biases of the interviewer

Ensure only one thought is expressed in each question

(Fink and Kosecoff 1998)

65

Document/File Review: General Guidelines Review the evaluation/study question of interest.

Identify the type of information needed to answer the study question and identify a code for each possible answer.

Determine which documents/files contain the information needed to answer the question.

Develop a data collection form, instrument/matrix or table that will assist you in collecting the specific information needed.

• The instrument should be clear, simple to use and code

Establish procedures for using the instrument

Conduct training to ensure everyone codes the same way.

Review documents/files.

Have a second person review the files.

66

Resources: Publications

Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. 2001. Russ-Eft, D. and H. Preskill. Cambridge, MA: Perseus Publishing.

Real World Evaluation: Working Under Budget, Time, Data, and Political Constraints. Bamberger, M., Rugh, J. and L. Mabry. 2006. Thousand Oaks, CA: Sage Publications.

67

Resources: Websites and Online Texts

International Program for Development Evaluation Training (IPDET) Course Modules. http:///www.worldbank.org/ieg/ipdet/modules.html

University of Wisconsin-Extension Cooperative Extension, Madison, Wisconsin, Program Development & Evaluation, Analyzing Qualitative Data. G3658-12, Taylor-Powell, E. and Renner M. 2003. http://learningstore.uwex.edu/pdf/G3658-12.pdf

Online QDA (Qualitative Data Analysis) http://onlineqda.hud.ac.uk/Intro_QDA/index.php

68

References Hancock B. Trent Focus for Research and Development in Primary Health Care: An

Introduction to Qualitative Research. Trent Focus, 1998

Kakoyannis C. Qualitative Data Collection & Analysis Notes. 2007

Lacey, A. and Luff D. Trent Focus for Research and Development in Primary Health Care: An Introduction to Qualitative Analysis. Trent Focus, 2001

McNamara, C. 1998. Basic Guide to Program Evaluation: Analyzing and Interpreting Information. Available online at:http://www.managementhelp.org/evaluatn/fnl_eval.htm; accessed August 126, 2005).

Patton, M.Q. 2007. 2007 American Evaluation Association Qualitative Methods Workshop, November 5-6, 2007. Baltimore, Maryland.

Russ-Eft, D. and H. Preskill. 2001. Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. Cambridge, MA: Perseus Publishing.

University of Wisconsin-Extension Cooperative Extension, Madison, Wisconsin, Program Development & Evaluation, Analyzing Qualitative Data. G3658-12, Taylor-Powell, E. and Renner M. 2003. http://learningstore.uwex.edu/pdf/G3658-12.pdf

The World Bank Group. Carleton University, IOB/Ministry of Foreign Affairs, Netherlands. International Program for Development Evaluation Training (IPDET), Module 6. Data Collection Methods. Power Point Slides and Narrative Text

The World Bank Group. Carleton University, IOB/Ministry of Foreign Affairs, Netherlands. International Program for Development Evaluation Training (IPDET), Module 8. Data Analysis and Interpretation. Power Point Slides and Narrative Text