Dr. Johnson. Other Research Approaches Research Method for Public Administrators Dr. Gail Johnson

Dr. Johnson. www.researchdemystified.org

1

Other Research Approaches

Research Method for Public Administrators

Dr. Gail Johnson


2

Research Complexity

Not all research approaches fit into the Xs and Os framework

Some approaches are really analytic techniques (e.g. cost-benefit analysis)

Some approaches are data collection tools (e.g. survey research)


3

Key Points about Other Approaches Each approach has unique characteristics

and are useful in specific situations Each has strengths and limitations Each has fundamental requirements Researchers often uses a mix of approaches


4

Commonly Used Research Approaches Secondary Data Analysis Evaluation Synthesis Content Analysis Survey Research Case Studies Cost-Benefit Analysis


5

Secondary Analysis of Data

In the planning phase, researchers should determine whether there is data they can use

It is more efficient to use data collected by others if it meets the research requirements and is of high quality. For example: That National Longitudinal

Survey of Youth collected data about drug use.


6

Secondary Analysis of Data

Large research studies gather data that is specifically related to the research questions and often gather other data that might be useful. The Framingham study focused on factors

related to heart disease, tracking 5,000 people from 1983-2003.

Other researchers used that data to look at why people are happy.Researchers Conclusion: happiness is

contagious!


7

Things to Consider

Does the database contain the data needed to answer your research question? If the question is about drug use among high

school seniors, then a study looking at drug use among adults will probably be inappropriate.


8

Things to Consider

How did they collect the data and does it meet research standards? How did they measure their key variables? Did they use documented procedures for

selecting people, developing data collection instruments, and pre-testing?


9

Things to Consider

What quality control procedures were used to ensure the accuracy of the data in the database?What is the error rate?

This means the difference between the data collected and the data in the database.


10

Things to Consider

How hard will it be to gain access to the data? Will the researchers provide all the

documentation, including the data dictionary that specifies every variable and how every value for ever variable is coded? For example, are Men coded 1 and Women coded 2

or is it Women coded 1 and Men coded 2?


11

Evaluation Synthesis

Useful when there are many studies that have been done on a specific topic Nutrition programs for the poor to reduce infant

mortality Early learning programs for disadvantaged

children Impact of race on death penalty sentencing


12

Evaluation Synthesis

An expanded version of using available data Cheaper than original data collection Creates a larger data base, increasing the

potential for drawing conclusions While individual studies often have limited

ability to draw conclusions, if the findings of many small studies form a pattern, conclusions have greater credibility


13

How to Do an Evaluation Synthesis Collect all studies that appear to be relevant Establish criteria to rate the quality of each

of the studies Only those meeting the criteria are included Summarize the key findings and impact

measures Look for similarities, especially among the

highest quality studies


14

Content Analysis

Used to measure the amount and/or message of communication It can be used to analyze content of speeches,

newspaper stories, emails, greeting cards, television, movies, blogs, memos, diaries or journals

Subjects covered in MPA textbooks, amount of violence in Saturday morning cartoons, or images of political women portrayed in news journals


15

Content Analysis

In the narrowest definition, it converts qualitative data into quantitative data: It can count the number of stories about the Iraq

war on major national network news shows to determine whether it changed over time

It can count the number of times presidential candidates spoke about specific policy issues


16

Content Analysis

But it is often used to summarize qualitative data What were the major themes that emerged from

the focus groups with teenaged mothers on welfare?

Is there a discernable political bias in the editorials of the major newspapers (NY times, Washington post, Boston globe, LA times)?


17

Content Analysis: Measurement Challenges Conceptual and operational definitions

How would conceptually define “political bias”?

Do you want to look at support for Republican or Democrat candidates or liberal or conservative policy positions?


18

Content Analysis: Measurement Challenges How would you operationalize “liberal” and

“conservative”—and are they they only political ideologies you want to include?What about libertarians, for example?What about fiscal conservatives who are

liberal on social policy?


19

Content Analysis:Measurement Challenges Conceptual and operational definitions

How would you measure “media images” of women governors?

How would you operationalize “strong” or”weak”, or “masculine” or “feminine”?


20

Content Analysis:Reliability Challenges Coding Reliability: Will everyone code the

same images in exactly the same way?Requires clear and very detailed decision

rulesRequires all team members are trained


21

Assuring Reliability

Jargon term: Inter-rater reliabilityEach team member will code a small sample

of the same materialTheir coding will be compared and if there

are differences, the team will figure out why there were differences and fix the procedures.

Everyone should be coding the same material in the same way.


22

Survey Research

A data collection approach (discussed later under data collection) is used when the researchers want to determine the views of many people

Typically, the exact same questions are asked with limited response options Think of a multiple choice exams (will cover more in

data collection) The structure limits bias of interpretation and is easier

to analyze


23

Survey Research

Open-ended questions—like fill-in-the blank exams—are possible but:Can be burdensome to complete if too

many open-ended questions are askedHard to analyze Bias is a threat.


24

Examples of Survey Research

Opinion polls to a cross-sectional sample of people in your town

Employee attitude survey of everyone at your place of work

Customer survey on the back of a postcard to all who visit the Post Office during the month of February


25

Survey Research

In-personCan be costly to do many in-person

interviewsBut useful when people are not literate

For example: it might be the best way to find out about health care access in rural villages in poor countries


26

Survey Research

MailRequires mail address and participants

who can read Cost of printing and postageRequires follow-up to get people to

complete them


27

Survey Research

Phone. Useful in locations where most people have phones. Works well when the questions are simple (like who

are you going to vote for in the next election) and can be done in less than 15 minutes.

Random digit dialing is used to get a random sample; Even those with unlisted phones are called.

Challenges of cell phones: pollsters are now including cell phones.

But you have to call many, many numbers before you find someone who is willing to take the survey.


28

Survey Research

Email or Web-based Limited to those with access to this technology Requires email addresses Web-based limited to those with access Can work well in work setting or in

associations where everyone has access Web-based applications often create the

database, saving time and money


29

Focus Groups

Another data collection approach that is sometimes presented as a research design approach. (discussed in more detail under data collection)

Small groups (6-12 people), trained facilitator moderates the discussion, few broad open-ended questions, food, pay to participate.


30

Case Studies

Useful in answering descriptive and normative questions

Used in describing “best practices” Can focus on one or several people, groups,

communities, organizations, processes, cities, etc. When it is not possible to gather data from every

person, community, program or organization—Case Studies provide a way to say, “for example.”


31

Case Studies

Case studies are useful when the researchers want to gain an in-depth understanding of a situation, of how a complex process works, or lessons learned about implementing an innovative program

It helps to make a problem or situation concrete: can tell stories of the impact of a program on a few people’s lives

It can capture complexity and variation in a situation


32

Choosing Case Studies

How many? No fixed rules It will depend on the situation and available

resources More is better to capture variation but case

studies are always about taking one or a handful of examples from a much larger universe


33


Basis for selection Convenience: easy for researchers to access Particular aspects: best, typical, worst, or mixed

cases Diversity: mix of locations, people, types of

organizations or perspectives Random: removing researcher bias from

decision


34


Researchers have to explain their rationale for whatever basis they use to select cases

Selection should have “Face validity”: the choices should make sense to the average, reasonable person


35

How to do a Case Study

Case studies can employ a wide variety of methods to gather data and often use several in combination Triangulation: using different methods to gather

information from various perspectives increases the validity of the results

Can use available data, surveys, interviews, observations, analysis of files and records, content analysis of memos, cost-benefit analyses


36

Case Studies: Are They Credible? Case studies are criticized because they

typically have a small scope. This is true. But that is also true of classic experiments. You have to beyond “smallness”—and look at

what they did and how they did it. Researchers should state the limitations of their work and conclusions should stay within the boundaries of their limitations.


37

Case Studies: Do They Make Sense? Case studies are criticized because they are

seen as inherently qualitative. Case studies may include quantitative data and

may in fact be entirely quantitative. While case studies may be qualitative, there is

nothing wrong with doing qualitative research as long as conclusions do not go beyond the limitations of the design.


38

Case Studies in Public Administration Case studies make sense in public

administration where it is difficult to do national studies because of insufficient resources (time, money, staff) Visiting six IRS offices may be far more doable

than visiting 100


39

Case Studies in Public Administration Like any research, quality is built into the

design and procedures used: Relevant research questions Clear and appropriate measures Data collection strategies than minimize error

and bias Honest about limitations Conclusions stay within those limitations


40

Cost-Benefit Analysis

Answers Normative Questions: Do the benefits justify the costs? Do the costs exceed the benefits?

Assumes that the costs and the benefits can be meaningfully measured in $$Dollars$$


41

Cost-benefit Analysis

Is it cost beneficial to get a college degree?What are the costs?

Tuition, books, room and boardOpportunity cost:

The money forgone from working fulltime in order to go to college

The earnings if that money had been invested


42


Is it cost beneficial to get a college degree? What are the benefits?

Difference in income between job with college degree and job without a college degree

Calculated over 20 yearsNeed to take inflation into account

(remember to use constant dollars)


43


Is it cost beneficial to get a college degree? What are benefits that cannot be measured in

dollars? Career opportunities: some jobs are only available

for those with college degrees, even if they do not pay that much

Increased self-esteem Transformative experience: exposure to new ideas Sustains a vibrant democracy


44

Cost-benefit: Public Sector

Policy analysis: what are 5 different approaches to these issues

1. Reduce recidivism for drug users

2. Prevent unwed teenaged parenthood

3. Provide health care to all citizens

4. Prevent financial meltdowns like we saw in the banks and financial industries in Fall 2008

What are assumptions are you making between cause and effect in each scenario?


45


Policy analysis: Pick one policy issue and consider how you

might think about calculating the costs and benefits for each one.

Note: one policy choice is always to do nothing Assuming there was that was the clear winner

in the cost-benefits contest, are dollars the only consideration in the political environment?


46


Values, beliefs, political ideology will all affect the debate

But each side are likely to present their best guess of costs and benefits.

And their estimates are likely to differ


47

Other Costs

Social Costs and Externalities Costs paid by the larger society

Take a look at the externalities of the material’s economy:

http://www.storyofstuff.com/Of course—you might want to question where some

of the numbers come from but it does point out that we tend not to take externalities (meaning someone else pays a price) into account.


48

Other Costs

Externalities: costs paid by someone other than those receiving the benefit Those producing or using goods and services do not

pay the full costs E.g. Corporations pollute but the taxpayers pay for the

cleanup City A benefits from having manufacturing company

but City B down river pay for the price: either higher health costs or costs for cleanup of toxics


49

Other Benefits

Short term benefits are easier to measure than longer-term benefits

Direct benefits are easier to measure than indirect benefits Direct: savings from health promotion program

because less spent on health treatments Indirect: savings from loss of productivity


50

Other Benefits

Intangible benefits: avoidance of pain and grief of infant deaths It is harder to attach a dollar value: if it is your

child, the life of your child is priceless


51

Cost-Benefit Assumptions

Implied balance between costs and benefitsBut: who gains is not necessarily the

same as who pays! Key Question: Do these estimates and

assumptions affect the results in a particular way?


52

Uses of Cost-benefit Analysis

While not as concrete as often portrayed, it is a useful framework for considering policy options.

Can provide a deeper understanding of the many connections between policy options, their potential costs and benefits, externalities, and equity considerations


53

Choosing a Research Approach

The choice is guided by the situation If data is available, then using available data

makes sense If you want to gather people’s opinion, then a

survey makes sense If you want an in-depth understanding of a

program or community, then a case study makes sense


54

Choosing a Research Approach

The choice is guided by the situation If many studies have already been done, then an

evaluation synthesis makes sense If you want to make sense of communications,

then content analysis makes sense Of you want to determine which approach is

more likely to be cost effective, than cost benefit analysis makes sense


55

Choosing a Research Appoach

Research approaches can be combined: A survey might be used to gather data for a

correlation design using statistical controls A case study might include a content analysis

of news clippings, available data and interviews with key community leaders


56

Key Takeaway Lessons

If the researchers have posed a cause-effect question, then they should use a design in the Xs and Os framework

The classic experiment is best for impact questions because it is necessary to eliminate all other rival explanations

Be cautious if they have asked a cause-effect question or implied a cause-effect relationship: Is the research design strong enough to

support the conclusions?


57

Creative Commons

This powerpoint is meant to be used and shared with attribution

Please provide feedback If you make changes, please share freely

and send me a copy of changes: [email protected]

Visit www.creativecommons.org for more information

Documents

Dr. Johnson. Other Research Approaches Research Method for Public Administrators Dr. Gail Johnson