POLI10251

Mark Brown

Social Statistics,Ground Floor Humanities Bridgeford Street

Getting Quantitative: using surveys in social research

Statisticians are Cool

Jean-Paul Benzécri: inventor of Multiple Correspondence Analysis

3

Challenging preconceptions..

I hate/can’t do statistics.. quantitative data can often be analyzed with relatively simple techniques – you don’t need to be a statistician.. or even very good at math’s

Quantitative methods are only relevant for ‘Quantitative researchers’ The qualitative versus quantitative debate is unhelpful. In the social sciences evidence comes in numerous forms and you need to be able to work with a variety of data. Many research questions are best answered with a mixed methods approach.

and developing quants skills is good for your CV!

‘QM are the most marketable transferable skill available

Graduates consistently report that ‘my QM skills got me the job’

(HEFCE: Social Science by Numbers)

5

A note on quantitative data

We focus in this lecture on the social survey as one of the most important sources of quantitative data in social science research

But there are other types.. Notably administrative data

collected by many organisations e.g– University collects data on students– NHS collects patient records – Police collect crime statistics

Valuable for monitoring, policy evaluation and research

6

The ubiquitous survey

surveys

government

academia

commercial

political parties

media

measuring characteristics, outcomes, behaviours & attitudes

charities

7

8 out of 10 Cats

British Crime Survey

Surveys making headlines...

Expenditure and Food Survey

Surveys making headlines...

General Household Survey

Expenditure and Food Survey

Health Survey for England

10

What are (quantitative) surveys?

A form of systematic data collection from a well-defined population of interest

They usually– draw a sample– Involve systematic and standardised data

collection: all respondents asked the same thing in the same way and answer using standard categories

– generate quantitative (numeric) data that can be analysed using statistical methods

11

Systematic and standardised data collection: Using tick boxes

Why not just ask respondents to discuss it in their own words?

12

Some potential advantages of the closed format question...

Greater specificity of question and answer (in this case a series of questions measuring attitudes on different aspects of inequality) can generate richer data than just asking ‘what do you think about…?

Answers can be added up to give a quantitative measure of attitudes in a population e.g. What percent of respondents think Government should increase public spending on welfare

Crucially we can compare responses for different groups in the population

e.g. We could look at whether the percent who supported increased spending on welfare varied by age of respondent.. or education level…

13

POLI10251: Increase spending on welfare benefits even if it leads to higher taxes

  POLI10251

Agree strongly  0%

Agree 26%Neither Agree nor Disagree 35%

Disagree 28%

Disagree strongly 10%

Don't know 2%

  100%

  (58 cases)

Agree strongly 

Agree Neither Agree nor Disagree

Disagree Disagree strongly

Don't know0%

5%

10%

15%

20%

25%

30%

35%

40%

14

Making comparisonsCompare POLI0251 with the rest of the nation (BSA 2012)

Increase spending on welfare benefits even if it leads to higher taxes?

  POLI10251 National (2012)

Agree strongly  0% 6%

Agree 26% 29%

Neither Agree nor Disagree 35% 33%

Disagree 28% 27%

Disagree strongly 10% 5%

Don't know 2% 0%

  100% 100%

  (58 cases) (2799 cases)

15

Making comparisonsCompare POLI0251 with the rest of the nation (BSA 2012)

Increase spending on welfare benefits even if it leads to higher taxes?

Agree strongly 

Agree Neither Agree nor Disagree

Disagree Disagree strongly

Don't know0%

5%

10%

15%

20%

25%

30%

35%

40%

POLI10251

National

16

Making comparisons:differences by age of respondent (BSA 2012)

Increase spending on welfare benefits even if it leads to higher taxes?

17 to 34 35 to 54 55+0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

(5) DISAGREE Strongly(4) Disagree(3) Neither(2) Agree(1) Agree strongly

17

Making comparisonsChange over time (BSA 2010 and 2012)

Increase spending on welfare benefits even if it leads to higher taxes?

(1) Agree strongly

(2) Agree (3) Neither (4) Disagree (5) DISAGREE Strongly

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

20102012

18

Would you say that the gap between those with high incomes and those with low incomes is too large, about right or too small?

1983

1984

1985

1986

1987

1989

1990

1991

1993

1994

1995

1997

1998

1999

2000

2001

2002

2003

2004

2006

2007

2008

2009

2010

2012

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

(3) Too small

(2) About right

(1) Too large

19

Getting critical: a question of validity

Always ask how well does a survey question measure the concept of interest (construct validity)

Choice of question wording (and the answer categories provided) are very important.

Consider whether we are ‘collecting data’ or ‘creating’ it

20

The search for pattern

Sure tick box answers oversimplify the diversity of viewpoint but this may be necessary to identify patterns and relationships in the data (a key aim in survey analysis). How would you go about this if respondents all answered in their own words?

Qualitative questions (interviews and maybe some focus groups) would reveal a more complex picture and be better tool for understanding the reasons behind the patterns.

Used together, a powerful research design

21

Survey data is normally collected from a sample of the target population

The Population

The Sample (from which data is collected)

Want to know something about a population?

..It only takes a sample

generalise results back to population (inference)

22

The power of sampling!Mori Final Election Poll 2010

Source: http://www.ipsos-mori.com

Sample <2,000

Population > 40mill !

Representative samples

One of the great strengths of quantitative surveys is that, if well designed, they can generate results that can be generalised from the sample to the wider population

But you can only do this if you have a sample that is representative of the population. Otherwise your results will be biased and potentially misleading

Unfortunately many survey designs fall short and produce biased samples that are not representative of the population

Even more unfortunately many users of surveys (especially in the media) ignore this issue and assume that results from all surveys can be generalised – this is BAD SCIENCE

23

What is a representative sample?

One where the composition of the sample (e.g. the share who are male and female, of different age groups, of rich and poor etc) is the same as in the population

i.e. It resembles a miniature mirror-image version of the population

24population sample

sample includes same share of males and females as in population

Why does it matter?

The things we study in surveys (behaviours, attitudes, etc) will vary according to characteristics of individuals.

E.g. Consider a survey of student use of social media, Let’s suppose females use facebook more than males

If so, a sample with a higher share of females than in the population (below) will over-estimate the true average time spent on face book

25

population sample

share of male v female not representative of population

26

Getting a representative sample – harder than you think

Suppose you were asked to design and carry out a survey to investigate the study behaviour of social science undergraduates in Manchester (looking at hours studied, % of lectures attended etc)

How would you get the sample?

How representative would it be?

Possible sampling strategies(for a sample of 200)

Pros and cons ?

Which one the easiest to get? Which likely to give highest response rate? Which one will give most representative sample? (Which most

susceptible to bias)?27

Strategy 1Identify 1 large first year lecture (>200) and ask lecturer to let you handout questionnaires to the class

Strategy 2Stand with a clipboard at entrance to Arthur Lewis and interview every 10th student (ask screening question first to check a SoSS UG) until you get 200

Strategy 3Get a list of all registered UGs in SoSS from UG office – randomly select 200 from the list – contact sample by email

28

The science of sampling Random (probability) samples

The ability to make ‘statistical inference’ to the population (generalise our results from the sample) really demands the use of random sampling

This is as the name suggests (think National Lottery numbers drawn out of a hat, where everyone has chance of being picked) - also called probability sampling

Strategy 3 on previous slide describes a classic random sample

Strategy 1 and 2 were Non-random samples and subject to bias

Sample Size.. The bigger the better

The other big issue in sample design is sample size – how big does it need to be?

Results from survey analysis will be much more reliable if based on a large number of cases.

This is one of the advantages of using existing large scale surveys like British Social Attitudes ...

...and a frequent weakness of doing your own survey (where the samples are often too small to support meaningful data analysis/generalisation of results)

29

30

And a word on data collection

Need to distinguish between the sample we design and the achieved sample (those in the sample who actually take part in the survey)

Unfortunately non-response is a massive problem in survey research (many surveys struggle to get 50% response)

Can result in serious bias (people who respond are generally different to those who don’t, so sample is unrepresentative)

31

Recap: the value of good description

Standardised measurement + ability to generalise findings to the population (inference) make well designed surveys powerful tools for accurate description of patterns in society

don’t under-value the importance of good description in research – We need to know the nature and extent of differences in society before we can set about asking why they exist or how to tackle them.

… particularly valued in the current climate of evidence-based policy research.

32

Describe and compare – across groups and over time

Source: Scottish Health Survey, Scottish Government

Obesity in Scotland, by age and sex (1995-2005)

33

More than good description: survey analysis can be used to test theory

Survey analysis can be much more than good description

e.g. consider contested theories on what factors are driving an ‘obesity crisis’ Lifestyle.. Related to culture… or deprivation?

Can develop hypotheses from these theories and then test them using survey data. E.g. we could start by crosstabulating obesity levels against income.. or any other variable you think may be important

though be careful… a statistical relationship does not necessarily imply cause and effect e.g. It could be another ‘third factor’, perhaps education level that is separately influencing both income and lifestyle factors related to obesity

Questions of Causality

Lifestyle factors related to obesity

Income

Education level

35

Should I do my own survey? ....or use someone elses.

Doing your own quantitative survey is hard to do well Common problems

– Unrepresentative sample designs – Inadequate sample size – Questionnaires – much harder than you think

The good news is that we have fantastic secondary resources of survey data...

(UK Data Service http://ukdataservice.ac.uk/) – Large representative samples– Rich data on topics you are interested in– Never been more accessible

36

Access to survey data – it’s never been easierand you don’t always need to do your own analysis

Re-purpose existing published tables and charts

Generating your own tables and graphs using survey data on-line

Downloading the dataset and doing your own analysis on your pc – perfectly possible but need training in data analysis (modules in year 2 and 3)

37

Sourcing evidence from British Social Attitudes (BSA) onlinewww.Britsoc.com

38

Sourcing evidence from British Social Attitudes (BSA) onlinewww.Britsoc.com

Tomorrow’s practical class will show you how to access tables of data for use in essays and project work

Please complete on-line registration to use site before you come

Please come to the correct slot 4-5 or 5-6 (see list)