1 ASRP06 Quantitative methods of social research for cross-national comparisons Paul Lambert, 1.2.06

1

ASRP06

Quantitative methods of social research for cross-national

comparisons

Paul Lambert, 1.2.06

2

Quantitative cross-national social research

1) Introduction

2) Three traditions in Qn cross-national research

3) Seven themes in Qn cross-national research

Case study 1: Secondary analysis of cross-national surveys

3

Introduction: Formats of Quantitative Cross-National

research

Some parameters:

cross-national between country

cross-national comparative

Large-N / Small-N Quant / Qual.

4

QDA: Analysis of patterns of relationships between variables in the variable-by-case matrix

[Low # of vars; stats / graphical summaries]

Cases Variables 1 1 17 1.73 A . . . .

2 1 18 1.85 B . . . .

3 2 17 1.60 C . . . .

4 2 18 1.69 A . . . .

. . . . . . . . .

N

5

A convenient distinction

Both micro- and macro- social data can be Large-N and Small-N

Macro-social data Micro-social data

Work and/or report at level of

aggregated unit (country)

Work and/or report at level of

constituent unit (eg individuals)

6

a) Macro-Social QnXR

Each case represents country, & aggregate statistics are compared

Ideal family size

1979 1989 Religiosity ‘81

Denmark 2.31 2.13 2.06

Ireland 3.62 2.79 3.42

Italy 2.11 2.20 2.90

Portugal 2.29 2.23 2.66

UK 2.29 2.14 2.33

(eg from Coleman 1996:39)

7

b) Micro-social QnXR

Cases (eg people) are grouped by country

Case id Country Indv. vars Natl. var

1 1 17 1.73 A 56.2

2 1 18 1.85 B 56.2

3 1 17 1.60 C 56.2

4 2 18 1.69 A 50.8

5 2 18 1.65 C 50.8

6 3 19 1.84 B 260.3

. . . . . .

8

Generic quantitative analysis issues

• Data design – Harkness et al 2003: Pre-harmonisation v’s Ex-

post harmonisation– Bryman 2001(p53) : 4 data collection models

• Data analysis– Selection of alternative techniques – A small number of specific extensions designed

for cross-national analysis (eg, ‘mixed models’)

9

Key feature of QnXR: Country as a categorical variable

Analyse within countries then compare outcomes (‘country-by-country’)

V’s

Analyse data pooled between countries, use countries / country level factors as explanations

(‘pooled’)

10

Country-by-country analysis

Pooled analysis

Macro-social

Large-N Small-N Large-N Small-N

Micro-social Large-N Small-N Large-N Small-N

11

Country as a categorical factor

Often criticised: • Appears to be overly simplistic

However • Same as other QDA factors, eg gender,.. • Critics forget qualified interpretations that good

QDA makes: [these patterns] are associated with categories, all other things being equal.

• Bad QDA: forget controls for relevant other things

12


1) Introduction




13

A popular two-stage story:

Eg: Hantrais and Mangen 96: moves to interpretive methods;Ragin 87: variable v’s case oriented approaches

Early quantitative researchers naively attempted to measure national differences as single variables. They badly misclassified or ignored important national level differences.

Much more thoughtful considerations of complex national contexts are needed, & often

these are more suited to qualitative research methods.

14

This inaccurate simplification implies a false Qn/Ql division:

• Doesn’t reflect variety of current practice in QnXR (& indeed past practice)

• Doesn’t acknowledge multivariate QnXR

• Doesn’t do justice to many carefully conducted / reported QnXR projects

• Tends to over-estimate QlXR capactity

15

A picture of Quantitative cross-national research under this typology:

Crude pooled macro-social analyses (Large-N)

C-by-C

16

Multitude of contemporary social research examples don’t fit this

• There are a great many quantitative country-by-country outputs

• It is unfair to describe all pooled designs as inadequate

• ..though to be fair, many pooled projects are genuinely weak!

17

A fairer typology of QnXR

Crude pooled analyses

Sophisticated pooled analyses

Country-by-country

18

Crude pooled analysesEarly or recent, micro- and macro- research making

claims over country level differences, with: • Insufficient exploration of relevant explanatory

factors• Limited or poor quality variable

operationalisations & discussions• Relevant national contexts not appreciated • False assumptions of good harmonisation

Example: see the illustrated analysis using the ESS

19


Early or recent micro- and macro- research making claims over country level differences, with:

• Sufficient exploration of relevant explanatory factors

• Good quality variable operationalisations and discussions

• Relevant national contexts suitably described• Accurate assumptions of good harmonisation

Example: more applications than is often realised…

20

Country-by-country approaches

Qn analyses within countries, then outcomes evaluated between countries by authors / readers

• Doesn’t require strong assumptions of data harmonisation

• Expertise of report writer covers national context

Examples: Edited books; centrally coordinated projects; end user reviews; …

21


• Attractive method: – offers parsimony of XN summary– uses large scale resources

• Methodology for good conduct necessary– Reliability, validity, implementation, translation– Sample design– Reporting strategy and claims

• Boundary to crude research subjective / contested• Existence often denied by anti-Qn sociologists…

22

Why not be over-cautious?

• C-b-C QnXR seems a safe bet?Doesn’t make claims not justifiedBut doesn’t make much impact either

• Remains need for good pooled research: Offers a parsimonious summary of national

differencesGovt / media with utilise regardless

23


1) Introduction




24

3.1) Data availability

• Massive increases in data resources accessible to social researchers

– Secondary survey datasets– Official statistics resources– Internet provision / communications

• Many data resources under-exploited• Most data originates from survey sources

- but some exceptions

25

3.2) Dataset complexity

• Secondary surveys tend to feature– Many variables and cases– Complex variable operationalisation choices– Complex structuring (eg multiple hierarchies)– Complex weighting / sampling information– Data analysis & management software needs

• Aggregate statistics’ features– Difficulty understanding source derivation

26

3.3) Variable operationalisation

• Single biggest issue in most QnXR conduct – Survey design

– Dataset analysis

– Result reporting

• Models of comparability– Exact equivalence of measures

– Relativistic equivalence of meanings

– Wide literature on ‘reliability’, ‘validity’ of X-N variable measures and aggregate statistics

27

Variable harmonisation ctd

• Choices over key variables allow use of previous literatures (eg H-Z & Wolf 2003).

Eg measures of income; occupation; ethnic group; education; region; crime; health; ..

• Choices over specific analytical variables require new efforts

Eg, attitude harmonisations of Inglehart.

28

3.4) Survey design

Harkness et al 2003:

Ex post facto harmonisation (more widespread, eg Eurostat, IPUMS, LIS)

v’s Coordinated design, sampling, & implementation

(big money projects, eg ESS, ISSP)Latter as preferable – but whilst many projects

attempt this model, far fewer succeed...

29

3.5) Conduct and logistics

• High costs of coordinated surveys• Considerable efforts, and many errors, in ex

post facto harmonisation • Issues of cooperating with colleagues /

diverging academic traditions, eg – different views data access / confidentiality

– Technical / software compatibility

– different organisations involved in survey production

QnXR can be very slow process

30

3.6) Temptation

Cross-national datasets nearly always look simpler than they really are

dangerous temptation to rush into uncritical variable analysis

31

3.7) Prejudice

• Prejudices against quantitative methods pronounced in European sociology, especially wrt cross-national comparisons

– QnXR evidence often ignored– QnXR researchers portrayed as simplistic

• Prejudices favouring quantitative methods often seen in governmental and media organisations

– Mainly: uncritical acceptance of harmonisations

32


1) Introduction




33

Some leading secondary surveys:(see handout for internet links)

ESS ISSP

IPUMS LIS / LES / LWS

Eurobarometer WVS / EVS

ECHP / CHER / PACO EU-SILC

Social Stratification: CASMIN / CCAP / …

Education: PISA / TIMSS

34

European Social Survey

• Annual attitudes / values / social circumstances cross-sections, 2002

• Equivalence of design and survey implementation between countries

• Extensive methodological resources• Free access to data Won the European Union’s 2005 Descartes

Prize for outstanding research!

35

Analysis (see SPSS syntax eg)

• Opens harmonised files from 15 countries in 2002• Select variables measuring attitudes, age, gender

and educational levels• Generate tables of patterns split by countries• Use regression models to evaluate contribution of

mulitiple explanatory factors:– Country specific ‘structural breaks’

– Country effects as variables / interactions

36

Liberal attitudes to homosexuality and their associations with educational level

(national average and Cramer’s V to educ)

% CV % CV

Switzerland 81 10 Israel 59 20

Czech Rep 58 11 Netherlands 88 6

Spain 70 20 Norway 77 13

Finland 62 14 Poland 46 16

UK 75 7 Portugal 71 15

Greece 51 23 Sweden 82 12

Hungary 48 5 Slovenia 52 18

Ireland 82 8

37

Log-regression prediction of liberalism to homosexuality for ESS adults

(value & significance of coefficient estimate) Age-squared -1.72** Interactions:

Low educ -0.31** Low educ*NW 0.19**

High educ 0.35** Female*NW 0.35**

Female 0.21** Female*South -0.15*

North West 1.07** Contrast: medium education male from eastern European country. Southern 0.56**

38

This is ‘crude’ pooled analysis

• Didn’t try out sufficient relevant explanatory factors

• Didn’t check variable choices extensively• Merged variable categories for convenience• Didn’t use survey weights• Didn’t contextualise reporting with sufficient

substantive national background and cross-examinations of data sources and measures

39

..but it could have been sophisticated pooled analysis

• Could have evaluated variable meanings

• Could have studied backgrounds

• Could have added more explanatory factors

• Could have reported more carefully

• .. Research consumption = understanding how well the results were prepared

40

Summary on Quantitative cross-national research

Quant methods contribute to both ‘pooled’ & ‘country-by-country’ comparisons

Crude pooled analyses widely criticised, and many bad examples persist

Sophisticated pooled research can be found, and represents most attractive format of QnXR

Documents

1 ASRP06 Quantitative methods of social research for cross-national comparisons Paul Lambert, 1.2.06