Week 3

Qualitative Research and Evaluation Research

QUALITATIVE METHODS

• Best insights may come from ordinary activities: observing, participating, listening and talking

• But, Researchers use these techniques in a more scientific way than found in everyday social interactions

• Researcher must observe keenly, take notes systematically, question strategically and invest more time in reaching conclusions

Characteristics

• Field focused

• Information is verbal

• Analysis is a synthesis of info perceived and interpreted (subjective)

• Often inductive (process generated from specific observations not theory)

• Goal is to search for meaning in situations

Most Popular Qualitative Research Designs

• Participant Observation – researcher develops a sustained relationship with subjects while they go about their normal activities. (roles: covert observer, observer/participant, complete observer)

Elaboration on these to follow later ! • Intensive Interview – open-ended unstructured

questioning to gather in-depth information about subjects feeling, experiences and perceptions

• Focus Groups – unstructured group interviews. A facilitator encourages discussion among participants on topics of interest

• Case Studies – well-orchestrated short story that presents essential info and data on a specific unit: individuals, programs, cities, counties, worksite, business or social organization, cultures etc.

Alternatives in Qualitative Designs

• Traditional Ethnography – participant observation by a single investigator who immerses him/herself in the group for a long period of time (often years) (anthropology)

• Study of a culture or cultures that some group of people share

• Technique – Just being there• Example – Chagnon with Yanamamo or• Elijah Andersons ‘Code of the Street’

(Philadelphia’s Inner City)

• Comparative Analysis – compare same process in similar organizations in different cities, compare same social movement in different countries – i.e. peace movement

• Narrative Analysis – study narratives, read stories and classify into general patterns:

Morrill looked at conflict stories written by 9th graders and classified into action, expressive, moral or rational tales

• Content Analysis – a survey of documents or records, messages or other forms of communication – look at newspapers for Nazi propaganda in WWII

• Is a form of textual analysis – can be quantified and statistically analyzed

• Text is coded and categorized – only lumped with qualitative analysis because categorizing may be subjective

• Can also be used for analyses of TV shows, movies, chat rooms etc. to monitor content

• Ethical Issues in alternative qualitative methods

Use and Misuse of results

Research Integrity (CBS & Bush NG documents)

Conflicts of interest over who own data and different stakeholders

• Grounded Theory – Build or ground theories from continuous observation

• Observations summarized into conceptual categories and categories tested through more observations

• Over time as conceptual categories are refined and linked a theory evolves

The Nightcaps radio show – Why do they listen, Why do they Call, What keeps them coming back, What drives them away etc.

Steps in Qualitative Research Process

• Reflect on setting• Develop Goals• Sample and collect data• Describe the data• Organize the data• Gather and Analyze• Connect different data segments• Corroborate the credibility of connections• Interpret

Focus of Qualitative Research

• Natural behaviors and artifacts• Natural social processes (as they happen

and relatively undisturbed)• See social setting as subject sees it• Avoid artificiality and structured questions• Consider the context in which social

interaction occurs, the interconnected nature of social relations and sequencing of events

• Observe and interact with subjects in their natural setting

• Be prepared to be affected on personal and emotional level

Qualitative Research is especially good for research on previously unstudied processes and unanticipated phenomena (new issues and hard to study groups)

Design # 1 -Participant Observation

• Researcher can take on a continuum of roles: complete observer, partial observer/partial participant, covert observer

1.Complete Observer – observe as things happen but don’t participate. Observer tries to identify who, what why, where and how of activities in settings.

Gradually develops a theory that accounts for what is observed

• Complete Observers presence alters the social situation (reactive effects)

• Smaller setting – larger observer impact• Larger setting - less effect on social processes

2. Participant and Observer• Some active participation in the setting• Participate enough to develop rapport and get a

sense of what subjects experience• Example – Diamond going into nursing home to

observe. Has to take a course to get certified to work there

• Ethical dilemmas arise.

• Have to try to pinpoint effect of your presence

• Need to find some physical space to concentrate on research

• Have to modify role as circumstances require it

3. Covert or complete Participant • Keeps the research secret• Tries to act like the subjects in a social setting• Why Covert?

1. Minimize reactive effect

2. Gain entry into inaccessible settings (Laud Humphreys as ‘watch queen’ in Tea Room Trade, Alfred in Satanists Cult, Goffman working in mental hospital

• Limitations: Can’t openly take notes or record Must write up notes based on memory and at

times when natural to be away from subjects Can’t ask suspicious questions Role playing is difficult – have to keep up role at

all times Serious ethical issues Researches reactions may not be same as

actual members of the group

• Going Native – More involved researcher gets in the social situation the more personal issues become and the greater the risk of getting too personally involved

• Entering the Field – may make social faux pas or may need to use an informant

• Ethical issues in Covert Research

In spite of limitations: if covert research was banned much important information would be lost

Some degree of deception may be justified for the knowledge gained.

Covert researcher must maintain confidentiality of others, keep commitments to others and should not directly lie to others

Necessities of Observation Research

• Gatekeeper – person who grants researcher access to the setting

• Key Informant – a knowledgeable insider who knows the group culture and is willing to share access and insights with the researcher

Tips from Experienced Participant Observers about Covert Observer

• Develop a plausible explanation for yourself and your study

• Maintain support of key individuals in group

• Don’t be too aggressive• Don’t violate implicit norms• Be a reflective listener• Ask sensitive questions only of informants

with whom you have a good relationship

• Let participants know you as a person, but don’t make too much of yourself

• Don’t fake a social similarity with your subjects• Do show a friendly interest in them• Avoid giving or receiving, money or other gifts

without violating norms of reciprocity• Can be active participant, including helping

others, but Don’t go overboard• Avoid taking sides or being used in inter-group

conflicts

Guidelines for Planning Observation Research

• Consider how you want to relate to subjects as people

• Speculate about what personal problems may arise and how you will respond

• Stay Grounded -Keep in touch with other researchers and personal friends outside research area

• Your standard of conduct should make you comfortable as a person while respecting integrity of subjects

• Develop a standard form on which to record info• Observe until saturation point is reached (no new

info is forthcoming)

Design # 2 -Intensive Interviewing

• Can be used alone or as part of Observation technique

• Doesn’t reveal as much as participant observation (what they say, without what they do)

• Goal – develop comprehensive picture of subjects background, attitudes, actions and their understanding of their world

• Researcher probes understandings and engages subject in dialogue (But don’t impose own ideas)

Technique

• Follow a preplanned outline of topics• Let interview flow in a unique direction in

response to subjects experiences and interests• Adapt as necessary• Note nonverbal cues and expressions of feelings

and interests• Be free to follow the data where it leads• Interview until saturation point is reached (No

additional info is forthcoming)

Design # 3 -Focus Groups

• Groups of individuals (usually unrelated) that are formed by a researcher and then led in group discussion of a topic for 1 or more hours

• Researcher (facilitator) asks specific questions and guides discussion

• Resulting info is qualitative and relatively unstructured

• Goal – spontaneous exchange and development of ideas

Characteristics

• Most focus groups involve 7-10 people

• May or may not know one another (could be co-workers)

• Homogeneous groups more convivial and willing to share feelings

• But, Heterogeneous groups may stimulate more ideas

Design # 4 -Case Study

• A case is one unit of analysis• A case study catches the complexity of a single

case: organization, community social group, family, or individual

• A case must be treated as an integrated social unit that must be studied holistically and in its particularity

• Emphasizes nuance, sequentiality of happenings in context, the wholeness

• Creates a thick, rich description that conveys a sense of what it is like from the standpoint of the natural actors in that setting

• Case studies are systematic approaches to Qualitative Research

Tasks: Conceptualization, Connecting info, Corroboration (evaluating alternative explanations) Examining Relationships, Coding and Categorizing, Documentation, Displaying Data, Authenticating Conclusions, Reflection(looking at own actions in study)

• Goal - to catch the complexity of a single case• Means months of fieldwork to get single case

study describing an individual, village, community culture, a program

• Can have smaller cases within overall case – classes within a school

• Layered or nested components – In Street Corner Society (William Whyte) uses single-community case study (Cornerville) which includes stories of several lower income youth

• Case includes all info about each case: Interview data, observations, records or files,

newspaper clippings, impressions and statements of others about the case

• Case Records – pull together and organize the voluminous case data into comprehensive, primary resource package

• Case study takes reader into case situation and experience: person’s life, group’s life, program’s life. You are telling the story.

• In individuals look for epiphanies (moments of major importance)

• In programs, organizations and communities look for critical incidents (crises, transitions, organizational lessons)

• Could do a case study of Lance Armstrong• Could do a case study of Tom Peters at

Harley Davidson (rundown, revamped, revitalized)

• Case studies can be both qualitative and quantitative

• Case studies are good for comparative studies (bars that banned smoking and those that didn’t)

• Date collection for Case is like being a journalist

Want to know: who involved, what they did,what was context or setting, when it happened, why participants did what they did, what caused changes to take place

When to Use Case Study

• When it is the best way to get at the whole story and rich detail

• When you need to understand the process

• When a program or individual is unique

• When outcomes warrant further study

• When something occurs in unpredictable or unusual environment

Data for Individual Cases

• Can include – Interviews with subject and those that know

subject Clinical Records Background and statistical info A life history profile Things person has produced (diaries, photos,

writings, paintings) Personality or other test results

Ethical Issues in Qualitative Field Research

• The very act of research imposes something unnatural on the setting

• Researcher actions may influence or harm subjects

1.Voluntary participation – (covert observation)

2. Subject well-being (Need for debriefing)

3. Identity disclosure – (covert observation)

4. Confidentiality – don’t ID to other subjects, in

reports or publications

Strengths

• Natural setting

• Richer data, verstehen

• Observation of body language

• Can study hard to enter groups

• Can study processes

• Good for exploratory studies

Weaknesses

• Subjective meaning (observer bias)• Ethical issues• Going native• Can’t have representative samples, so can’t

generalize• Can’t replicate• Validity and reliability problems• Limited to certain situations and areas of study• No control• Hard to quantify• Can’t do large populations

EVALUATION RESEARCH

• To know if Social programs, training programs, medical treatments, or other interventions work- have to evaluate the outcomes systematically and fairly

• Evaluation can correct deficiencies• Evaluation Research is not a research method –

it is a type of research like descriptive, exploratory, explanatory, experimental

• Purpose: investigate training, therapies, treatments etc. to access effectiveness

• Often necessary to combine different methods of research to achieve objectives

In the cycle of evaluation we look at:

• The Inputs (the program resources, raw materials, clients and staff)

• The program process (how the program is implemented (how service or treatment is delivered)

• Outputs (the product of the delivery process – indicators that the program is operating)

• Outcomes – Impact of the program• Stakeholders – Individuals and groups who have

some basis of concern with the program (clients, staff, managers, fundors, the public)

Evaluation is a systematic approach to providing feedback

Questions for Evaluation Research

• Is the program needed?

• Can the program be evaluated?

• How does the program operate?

• What is the program’s impact?

• How efficient is the program?

• How costly is the program?

(cost benefit and cost effective)

Components of Evaluation Research

• Needs Assessment – Is there a need? What is the level of need? Develop plans for implementing a program to answer need. Evaluate how well the program satisfied different perspectives of the need.

• Evaluability Assessment – Done in advance to see if a program can be evaluated after implementation

This is a preliminary study designed to check things out and often relies on qualitative methods

• Process Evaluation – investigates the process of service delivery. Helps shape and refine a program when built into initial plan (formative evaluation). Can use a wide range of indicators such as records, surveys, and qualitative descriptions.

• Impact Analysis – Asks the questions – Did the program work and did it have the intended result?

Also known as summative evaluation –measures the extent to which a treatment or other service has an effect

The method of data collection most used for impact evaluation is experiment

However, quasi-experimental design, survey or even qualitative methods can be used.

• Efficiency Analysis – Questions asked – Are the program’s benefits sufficient to offset the program’s costs?

Are the taxpayers or sponsors getting their money’s worth?

What resources are required by the program?

• Efficiency Analysis compares program effects to costs

• Types of Efficiency Analysis

1. Cost-benefit analysis – compares program costs to economic value of program benefits – It must identify the specific costs and benefits that will be studied

2. Cost-effectiveness analysis – compares program costs to actual program outcomes.

May be more important to researcher and clients than to the tax-payer and sponsor. The bottom-line is sometimes considered more important than the benefit to participants.

Who Cares?

• Whose goal matters most? Sponsors or Researcher

• Sponsor may not wish to follow scientific guideline of making results public and encourage researcher to be responsive to stakeholders first

• Researcher cannot passively accept the values and views of Stakeholders as most important

• Researcher need to maintain autonomy

• The Ideal is integrated approach – issues and concerns of both stakeholders and research evaluators are covered

• Bottomline: Not all agencies really want to know if a program works – especially if they need the answer to be yes and it is actually no

Ethics in Evaluation

• Assigning subjects randomly for treatment or benefit for evaluation purposes

• Can confidentiality of evaluation be preserved when the info is owned by sponsors

• Politics can shape evaluation – results may be shared with Stakeholders and policy makers

Health Research Extension Act of 1995

• If research organization receives federal funds it must have a Review Board to assess all research for adherence to ethical practice guidelines

• Criteria are:

1. minimize risks

2. risks must be reasonable in relation to benefits

3. Selection of individuals must be equitable

4. Informed consent must be given

5. Data should be monitored

6. Privacy and confidentiality should be assured.

Since researchers may be required to provide evidence in legal proceedings, subject confidentiality can be a problem.

To lessen possible detrimental program impacts:

• Minimize the number in untreated control group

• Use minimum sample size required to test results

• Test just parts of the new program• Compare treatments that vary in intensity

rather than presence or absence• Vary treatments between settings rather

than among individuals within a setting

Conclusions

• Because evaluation designs are complex important outcomes or aspects of program process may be missed

• Including stakeholders in research decisions may undermine adherence to scientific standards

• Researchers may be pressured to only report positive conclusions

• Findings may be distorted or over simplified because of who report goes to