1

PAIR 1005 MATT RYAN

The experimental method

2

Should we

…be control freaks?

…be manipulative?

…treat people who arethe same completelydifferently?

…be a bit random?

3

The magic of the experiment

Intervention/treatment by the researcher in the process of data generation (precise manipulation).

Control in the form of establishing a group to gather data from that have not experienced intervention.

4

Magic

But in laboratory settings control can come through careful management of the variables that are in play.

Random allocation of research subjects so that you can argue that in all features the two groups are the same, only difference is the intervention

‘Similar in all important respects’

5

The logic of experiments

Pre and post and other measurements

Controlling for factors known to have an effect and unknown factors and so isolate impact of intervention variable Control group – nothing happens or placebo

Repeat and replicate and get a firm handle on causality

6

Formally at the head of the hierarchy

For positivists (and even post positivists) experiments are formally at the head of the ‘hierarchy of methods’

Lab and field approaches (and a role for ‘natural’ experiments) Is an artificial environment good or bad?

7

Lijphart (1971) in a seminal article in the American Political Science Review commented on how the purity of the comparison of a treatment and control group, backed up by random assignment, is seen as unrivalled in its capacity to deliver definitive results about causality.

‘The experimental method is the most nearly ideal method for scientific explanation, but unfortunately it can only rarely be used in political science because of practical and ethical impediments’ (Lijphart, 1971: 684-5)

8

Too artificial?

The artificiality of the laboratory setting for social research that provides it with the ability to control extraneous influences also divorces those findings from the social context within which they have meaning. If we find people behave in a certain way in a(n)… experiment can we have any confidence that they will behave in the same way in their everyday lives? The answer is probably not (Peters, CP, 1998: 48).

‘Hawthorne effect’ (novelty problem) and measurement issues leading to ecological fallacy.

9

Just not practical

For most research in the discipline there is little or no opportunity for experimentation-citizens are not likely to submit to very much experimentation on matters as crucial as the selection and management of their governments (Peters, CP, 1998:212).

Can’t change the important things. We can’t exact randomly assign political systems or

welfare systems.

10

Ethical challenges

Treating people like rats? The harm principle

Some experiments may involve misleading participants Blinding!

Avoid or debrief them after the trial.

Justifying the benefit and the need for the experiment

11

The recent rise of experimentation

Always strong in psychology, major growth in economics and now in politics Growing influence of behavioural Economics and ‘Nudge’ theory.

Strong field role in social policy and administration in education, employment, welfare, health and others areas

Great tradition of policy experiments especially in the United States

The rise of lab experiments (students as rats!)

The use of the internet Varying messages Survey question research

The smaller increase in field experiments

12

Experiments can and do go wrong

Too high a risk?

13

Problems in the heat of research

Lack of training among the staff involved in the intervention may create problems.

Decisions made in the heat of the research may prove problematic to the experiment.

Estimating effect size is always a challenge How many folks do you need in your study?

Estimating the effects of system-wide reform can be particularly problematic using randomised trials because of contamination effects.

There can be, for example, particular problems caused by the attrition of participants from programmes.

14

Validity problems?

Internal validity: did the intervention have the impact that the experiment suggested? Sometimes the problem rests in the explication of the construct and sometimes it rests in the measurements and surrogates used to capture it. Response: Care, openness and challenge

External validity: whether the validity of observations about cause and effect hold from the particular study across variation in persons, settings and measurement variables. Response: similar to that of other researchers, namely to argue

for an accumulation of evidence.

15

Why take the risk?

Experiments can support a dynamic between theory and evidence

Enables social scientists to engage with issues that people care about

16

Exploring microfoundations

Offers clarification for our models and theories

Direct benefit in expanding the heuristics available to policy makers and social actors

17

Delivers on relevance

Field experiments require engagement

The need for flexibility and adaptability

A test of practicality

18

Opportunities abound

What we could in the Lab?

What we could do in the field?

But there will always be limits External validity in the lab Confounding factors in the field

19

Variations on experiments

Natural experiments E.g. ballot ordering Quotas - Bhavnani

Quasi-experiments E.g. Fishkin’s deliberative polls?

Design experiments?

20

And so?

Experimenters need to more explicit about their value-added

Consider the prospects for quasi and natural experiments ( design experiments)

Need to accept a mix of methods and continuing role for observational research

The statistical method and the comparative method

21

Experimental ethics

Cases e.g. Milgram, the ‘Strivers’

Withdrawal vs. attrition Ethics of offering incentives?

23

My experience troubleshooting

Fun war stories..

Students

Parishioners

Tenants

24

Experiments frame our thinking about causality using other methods

“Testing relationships between variables by keeping all other factors constant” the experimenters code that other methods try to replicate:

The statistical method aims to ape experimental method by intervening after the data are created ( by holding some variables constant in order to test the impact of other variables) Random selection rather than random allocation.

Comparative analysis Can drive the logic of temporal or spatial comparison when only smaller number of cases available

Can influence case study selection

Rational thought experiments: what if?

BUT SOME REJECT THIS ASSIMILATION OF ALL METHODS TO THE LOGIC OF EXPERIMENTS