Qualitative work and the testing and development of theory: lessons from a study combining cross-case and

within-case analysis via Ragin’s QCA

Barry Cooper & Judith GlaesserDurham University

Session Title: Challenging the Quantitative-Qualitative Divide: Exploring Causal Analysis

ESRC Research Methods Festival Oxford 2012

This work was supported by an ESRC research fellowship [RES-063-27-0240] awarded to JG.

Our talk will draw on chapter 6 of Cooper, Glaesser, Gomm & Hammersley (2012) Challenging the Qualitative-Quantitative Divide and the paper by Cooper & Glaesser (2012) in FQS. It comprises:• General comments on regularities and causation.• Illustration of the ways in which the set theoretic description

of regularities contrasts with the linear correlational approach.

• Problems in using QCA to gain causal knowledge (analogous to the “correlation does not imply causation” problem).

• Using qualitative data (interviews here) to undertake process-tracing to understand the nature of regularities, i.e. the processes that have generated them, and also to improve predictive models (based on existing theory).

• The substantive context is the sociology of education.

"The basic idea of the covering-law account is very simple: An explanation is a deductive (or statistical) argument that has a description of the explanandum phenomenon as a conclusion and one or more empirically validated general law statements and a set of statements describing particular facts (the initial conditions) as its premise. The core underlying idea is that explanations make the explanandum expected [given the explanans, etc.]. This means that explanation and prediction are more or less the same thing; the only difference is that in the case of explanation we already know the outcome" (HEDSTRÖM & YLIKOSKI, 2010, ARS, pp.54-55).

But, for example: Barometer readings change prior to storms (a regularity-based law). A storm can therefore be “explained” by a barometer’s pressure reading falling. Not obviously explanatory. Common cause is a drop in atmospheric pressure.

So: regularities themselves need explaining.

In survey-based work on social class and educational achievement, conventional approaches to exploring the relationships between theoretically selected variables employ measures of linear correlation, sometime directly, and often as the building blocks of regression models of varying degrees of sophistication. Basically, here, the model is, more of X leads to more of Y.

Our presentation will address an alternative approach, Ragin’s Qualitative Comparative Analysis (QCA), based in set theoretic mathematics rather than the linear algebra that underlies the correlational approach.

While the correlational approach focuses on the relations between “independent” variables and some dependent variable, the set theoretic approach focuses on cases understood as configurations of features, and explores what combinations of features are sufficient and/or necessary for some outcome to be achieved.

Quotes from Pawson (2007), “Causality for Beginners”:

SUCCESSIONISTS locate and identify vital causal agents as ‘variables’ or ‘treatments’. Research seeks to observe the association between such variables by means of surveys or experimental trials. Explanation is a matter of distinguishing between associations that are real or direct (as opposed to spurious and indirect) and of providing an estimate of the size and significance of the observed causal influence(s).

CONFIGURATIONISTS begin with a number of ‘cases’ of a particular family of social phenomenon, which have some similarities and some differences. They locate causal powers in the ‘combination’ of attributes of these cases, with a particular grouping of attributes leading to one outcome and a further grouping linked to another. The goal of research is to unravel the key configurational clusters of properties underpinning the cases and which thus are able to explain variations in outcomes across the family.

Quote from Pawson (2007), “Causality for Beginners”:

GENERATIVISTS, too, begin with measurable patterns and uniformities. It is assumed that these are brought about by the action of some underlying ‘mechanism’. Mechanisms are not variables or attributes and thus not always directly measurable. They are processes describing the human actions that have led to the uniformity. Because they depend on this choice making capacity of individuals and groups, the emergence of social uniformities is always highly conditional. Causal explanation is thus a matter of producing theories of the mechanisms that explain both the presence and absence of the ‘uniformity’.

Now, moving from these general perspectives to something more concrete …

Much theory addressing the relationship between social class and educational achievement has two features that make regression methods based on linear correlation sometimes less valuable than an alternative approach based in set theory that focuses on (i) modelling the conditions necessary and/or sufficient for some outcome and on (ii) causes conceptualised as conjunctions of factors.

For example, Boudon’s model of the primary and secondary effects of social class might be paraphrased as claiming that, in some social classes, early achievement is necessary but not sufficient for later achievement.

Bourdieu’s account of capitals often refers to conjunctions of causes. Educational capital may need to be combined with social capital in order to be sufficient for some occupational destinations to be achieved.

The most common uses of regression do not set out to address these logical and/or causal structures.

In a simple additive (and, for illustration, deterministic) equation:

OUTCOME = 2S + 2T

we can see that even if S is zero T can have an effect, and vice versa.

Of course, regression equations can be written to capture the claims of Boudon and Bourdieu. Re Bourdieu, consider:

DESTINATION = EDUCATIONAL CAPITAL X SOCIAL CAPITAL

Here both capitals must be present to produce a non-zero outcome.

Some of what we discuss can be modelled with such interaction terms, dummy variables, variable transformations, etc. (but not all, we think, see Vaisey, 2007, fn11, on fuzzy set QCA). However, we claim that the set theoretic approach has a more obvious affinity with these types of relation and aim to illustrate this.

We will now contrast correlational and set theoretic accounts of a bivariate relation that varies by type of case.

Initially we will use invented data on the relation between educational achievement and measured ability across three social class groups.

Having shown what the set theoretic approach can capture, we will then apply this approach to real data from the National Child Development Study.

Figure 1: achievement by ability (invented data,194 cases in 3 social classes)

Correlation:

r = 0.601

Note: the size of the shape indicates the number of cases at any point.

Class, ability, achievement: What might a scatterplot of all cases hide?

Figure 2: just the cases in social class 1 (invented data, n=43)

Note:

y = x diagonal line is for later use.

r = 0.739

For class 1, we obtain an upper triangular plot.

Note: Class 1 is the highest social class of the three.

Figure 3: Just class 2 cases (invented data, n=78)

r = 0.739

Figure 4: Just class 3 cases (invented data, n=73)

r = 0.739

While the scatter of points for class 2 has the elliptical form well-described by a linear correlation coefficient, the scatter of points for classes 1 and 3 has a more triangular form not well-suited to a description using this coefficient.

A look at the graph for class 1 shows that the (invented) cases with higher ability achieve highly but that so also do some cases with lower ability.

Ability, in a sense to be described, is sufficient but not necessary for achievement for cases in class 1.

Before we can systematically describe these relationships in terms of sufficiency and necessity, we need to introduce QCA.

Crisp and fuzzy set variants. See Ragin (e.g. 2008) for more detail.

Figure 6: Perfect sufficiency of ability for achievement in some imaginary world

Figure 7: Quasi-sufficiency of ability for achievement in another imaginary world

In Figure 6, cases with “high ability” are a subset of (are included in) the set of “high achievers”. This is equivalent to “high ability” being (logically) a sufficient condition for high achievement.

In Figure 7, the proportion of cases of the condition set that is included in the outcome set can be used as a measure of consistency with perfect sufficiency. This is high here.

Table 3: membership in the sets “high achiever” and “high ability”

Not high achiever High achiever

High Ability Cell 1 Cell 2Not high ability Cell 3 Cell 4

These subset relationships can also be discussed in the context of crosstabulations of membership in sets. In the tradition of correlational analysis, there is a concern with symmetry. For a high positive correlation we would want cases mainly in both of cells 2 and 3 of this table.

Analysing sufficiency moves us away from a concern with symmetry. To test whether high ability is sufficient for high achievement, we only need to look at the first row, containing cells 1 and 2. The crucial thing is that there be no (or very few) cases in cell 1, since these would contradict a claim that being of high ability is sufficient (or quasi-sufficient) for high achievement.

Similar Venn diagrams can be drawn, and arguments made, for necessity, but, in this case, the outcome must be a subset of the condition.

Here the conjoined (intersected) set (in red) of the highly able and the highly ambitious is a subset of the outcome set of high achievers. Hence the two factors conjoined are logically sufficient for high achievement.

Y = (A*B*c) + (A*C*D*E)

In these Boolean equations the symbol * indicates Logical AND (set intersection), + indicates Logical OR (set union), upper case letters indicate the presence of factors, lower case letters indicate their absence.

In this fictional example of causal heterogeneity, the equation indicates that there are two causal paths to the outcome Y. The first, captured by the causal configuration A*B*c involves the conjoined presence in the case of features A and B, combined with the absence of C. The second, captured by A*C*D*E, requires the joint presence of A, C, D and E. Either of these causal configurations is sufficient for the outcome to occur, but neither is necessary, considered alone. The factor A is necessary but not sufficient. The factor C behaves differently in the two configurations.

Example taken from: Mahoney, J. & Goertz, G. (2006): A tale of two cultures: contrasting quantitative and qualitative research. Political Analysis, 14, (3).

Another Boolean algebraic summary of the results of a cross-case analysis:

A*B*C + D*E*f + B*D*F => Outcome

Note that, unlike regression equations, QCA retains skeletal cases – as conjunctions of factors – in its algebraic formulations.

A*B*C is a type of case, as is D*E*f, as is B*D*F.

In variable-based analysis, employing correlation, as every student is warned, correlation does not equal causation (for several reasons). Similar issue arises with QCA. We will return to these later.

Producing these summary equations requires knowledge of “truth tables” and “Boolean minimisation”. Let’s look at these …

QCA employs “truth tables”, where a 1 indicates the factor is present, and 0 it is absent. Here is an invented truth table:

A B C Number Consistency (with sufficiency) for the outcome

Number with the outcome

1 1 1 120 0.925 1111 1 0 110 0.900 991 0 1 100 0.890 891 0 0 120 0.500 600 1 1 100 0.350 350 1 0 100 0.060 60 0 1 800 0.020 160 0 0 720 0.008 6

Table 1: Invented data for O (Outcome) = Function (A,B,C)

Three configurations (shaded) have consistencies high enough to justify a claim that these are quasi-sufficient for the outcome to be attained. They are ABC, ABc and AbC.

(Note: we sometimes drop the *s in A*B*C etc. for the sake of simplicity.)

The result can be minimised thus:

ABC + ABc + AbC => O becomes AB + AC => O, (with an overall consistency with sufficiency of 0.906).

How? Here, e.g., the presence or absence of the factor C makes no relevant difference, given AB, and so we can collapse ABC + ABc to AB.

Similarly, ABC + AbC becomes AC.

“AB + AC => O” tells us a great deal. Given a new case falling into the set “AB + AC”, we could claim that there is a 0.906 chance of this case having the outcome.

These examples all concerned crisp sets (a case is either fully in or out of the set; equivalently, a factor is either present or absent). Chapter 6 of our recent book (Cooper et al, 2012) employs fuzzy sets, allowing partial membership in sets, ranging from zero (fully out) to one (fully in), through such values as 0.25 (more out than in), 0.5 (as much in as out), 0.75 (more in than out).

A good example is “adulthood” (Kosko, 1994). Most judges would agree that an age of ten would rule out adulthood (giving a membership score of zero) and one of 30 would rule it in (giving a membership score of one) but there would be much more discussion about the age range 15 to 21.

Here it would seem inappropriate to allocate a score of either zero or one the only possibilities available in the crisp set context. In fuzzy set based descriptions of cases a score of say 0.9 can be used for the 20 year-old to indicate almost full, but not quite full, membership of the set of `adults'. A nineteen year-old might be allocated a score of 0.8, and so on.

We noted earlier that the scatter of points here represented ability being sufficient (but not necessary) for achievement.

Note that, for each case, the fuzzy ability score is lower than or equal to the fuzzy achievement score (i.e. x < = y). This is in fact the simplest test for (perfect) fuzzy subsethood, i.e., for every case, the condition score must be less than or equal to the outcome score. For real datasets, a simple measure (from 0 to 1) of closeness to perfect sufficiency is the proportion of cases that passes this test.

The invented data for class 1 again.

For necessity the test is reversed (y<=x). The lower triangular plot for class 3 passes this test. Here ability is necessary (but not sufficient) for achievement.

This is a case of perfect necessity. No cases stray over the y=x line.

Again, for real datasets, a simple measure (from 0 to 1) of closeness to perfect necessity is the proportion of cases that is on the correct side of this line or on it.

We can compare these set theoretic measures of sufficiency and necessity for the three classes with the correlation coefficients (which were all equal, recall, at 0.739).The invented data for Class 3 again.

Table 4: set theoretic testing of the ability => achievement relationship

Simple inclusion algorithm

Sufficiency (0-1 scale)

Necessity (0-1 scale)

Result: Ability is …

All classes together 0.459 0.598 Neither sufficient nor necessary

Class 1 1.000 0.023 Sufficient but not necessary

Class 2 0.590 0.539 Neither sufficient nor necessary

Class 3 0.000 1.000 Necessary but not sufficient

While the correlation coefficients were identical by class, the measures of nearness to perfect necessity and sufficiency clearly are not. They have captured important differences between class 1 and class 3.

Now a brief account of an example using real data (see book for the detail):

We explore the relation between

ACHIEVEMENT and measured ABILITY

as it varies by combinations of values of

FATHER’S CLASS, GRANDFATHERS’ CLASS (both), SEX.

We focus on comparing

the correlation between achievement and ability

with

the consistency with sufficiency of ability for achievement.

Our data come from the National Child Development Study (cases born in one week in 1958). We use a sample of 5,272 cases from the NCDS with no missing values on the variables we employ here.  Variables:

Measured ability (variable n920) is taken at age 11.  The measure of achievement, highest qualifications obtained, is taken at age 33 and includes both academic and vocational qualifications (derived variable HQUAL).  Because of the ways each grandfather’s class was recorded we will be using the categories of the Registrar General’s scheme for these, but we will be using an approximation to Goldthorpe’s class scheme for the respondent’s father’s class (at the respondent’s age of 11). Given the illustrative nature of the analysis here, this mixing of categories, though undesirable, is of no consequence, we believe, for our arguments.

Ability and achievement are calibrated as fuzzy sets (see Cooper et al, 2012, chapter 6, for the details).

Consider two contrasting sets of cases (i.e. configurations of factors):

• males with a class of origin towards the top of the social class distribution who also had grandfathers towards the top of this distribution

• females who had fathers and grandfathers towards the bottom of the distribution

We can hypothesise that ability might have tended towards being a sufficient condition for achievement for the former group (but perhaps not a necessary one) and towards being a necessary condition for the latter group (but perhaps not a sufficient one).

If this is so, we should expect an upper triangular plot of achievement against ability in the first case and a lower triangular plot in the second.

The cases here are the 60 males whose father was in Goldthorpe’s class 1 (the upper service class) and whose grandfathers were both in either Registrar General’s social class I or II.

Pearson correlation is 0.505.

Consistency with sufficiency is 0.833.

Consistency with necessity is 0.167.

This approaches the form of an upper triangular plot (i.e. it tends, descriptively, towards ability being sufficient but not necessary for achievement).

The cases here are the 94 females whose own father was in Goldthorpe’s class 7 (the semi and unskilled manual working class) and whose grandfathers were both in either RG class IV or V.

Pearson correlation is 0.472.

Consistency with sufficiency is 0.207.

Consistency with necessity is 0.721.

This approaches the form of a lower triangular plot (i.e. it tends, descriptively, towards ability being necessary but not sufficient for achievement).

The clear tendencies are that:

For the configuration

HIGH_GFs’_CLASS*HIGH_FATHER’S_CLASS*MALE

ability is sufficient, but not necessary, for later achievement.

For the configuration

LOW_GFs’_CLASS*LOW_FATHER’S_CLASS*male

ability is necessary, but not sufficient, for later achievement.

Note: usual language is quasi-sufficient and quasi-necessary in these situations.

* = AND, meaning together with

Lower case => females

HQUAL BY ABILITY*CLASS*MALE*GFS (Ordered by consistency with sufficiency)

Sex Grandfathers both in RG I or II

Goldthorpe Class

Correlations N Measure of consistency with sufficiency (of ABILITY for HQUAL)

N

Male Yes 1 0.505 60 0.833 60

Female Yes 3 0.604 11 0.818 11

Male Yes 7 0.556 9 0.778 9

Male Yes 2 0.539 40 0.700 40

Female Yes 2 0.531 56 0.679 56

Rows omitted for this presentation

Male No 3 0.445 236 0.568 236

Male No 6 0.517 758 0.485 757

Rows omitted for this presentation

Female Yes 4 0.429 37 0.324 37

Female No 6 0.483 820 0.319 818

Female No 5 0.446 147 0.313 147

Female No 7 0.503 463 0.257 460

Female No 4 0.649 110 0.255 110

Types still too heterogeneous? Note: “not in I/II” is rather heterogeneous.

We have now shown why we think QCA, and the ideas of sufficiency and necessity, are useful for describing social regularities.

Also have given examples of configurations of conditions that are quasi-sufficient for some outcome.

But if, say, G*h*I*K => some outcome, what can we say from this re causal relations between the conditions and the outcome?

Now we will address some of the ways in which QCA results might be misinterpreted from the perspective of a concern with causality.

We will take a realist view, wanting to understand the generative processes by which these configurational regularities are produced.

Let’s invent an example …

Let A stand for belonging to a family amongst the social elite in early nineteenth century England. Let B stand for having parents who value literacy. Let C stand for being male, with c then indicating being female. Let the outcome O be achieving literacy.

“AB => literacy” then suggests that gender makes no difference and, indeed, this may be true in the sense that, irrespective of gender, the conjunction of A and B is quasi-sufficient for the outcome. The actual causal routes to the outcome may, nevertheless, of course, differ for ABC and ABc. Boys may receive their education in schools, girls in the home. (The pedagogy may differ by gender. The nature of the outcome “literacy” may therefore also be heterogeneous by gender.)

Minimising configurations (ABC, ABc -> AB here) can deflect us from important differences in causal paths to an outcome.

We explore this and several related issues at length in Cooper & Glaesser (2012).

• The remainder of this talk continues to draw on our project studying educational transitions and achievement in English and German secondary schools.

• British longitudinal data (NCDS, BCS) and German panel survey data (SOEP) have been analysed using a configurational case-based approach (Ragin’s QCA) rather than a variable-based approach (i.e. correlation-based regression methods).

• We will focus on the necessary and/or sufficient conditions for a particular outcome to be achieved.

• However, as noted earlier, like correlational methods, QCA alone cannot ground causal claims.

• To address the causal processes generating the relationships between conjunctions of conditions and educational outcomes, some 80 interviews have been undertaken with young people aged around 17.

• These interviewees were selected on the basis of a typology created via our QCA analyses.

We now move to a QCA analysis, using German SOEP data, of the conjunctions of conditions that are sufficient and/or necessary for being in the Gymnasium at age 17. Our theoretically selected potentially causal conditions are:

Gender: (MALE / male)At least one parent has the Abitur: (ABI_1P / abi_1p)At least one parent in the service class: (SC_1P / sc_1p)Adolescent had Gymnasium recommendation: (GY_REC / gy_rec)

We find (for 790 cases born in the late 1980s) that having the recommendation (GY_REC) is a quasi-necessary condition for the outcome (with a consistency of 0.89).

Also, the following is one solution for quasi-sufficiency (with overall consistency of 0.86):

ABI_1P*GY_REC + MALE*SC_1P*GY_REC => GY_17

Drawing on an earlier paper1 addressing theory development via combining cross-case and within-case analysis, we can now list “typical” and “deviant” cases (from Germany) in regard to both sufficiency and necessity:

1. Glaesser, J. & Cooper, B. (2011) Selecting cases for in-depth study from a survey dataset: an application of Ragin’s configurational methods, Methodological Innovations Online.

Configuration Outcome Type of case Available interviewees (not taken from SOEP dataset)

1 ABI_1P*GY_REC present Typical with regard to sufficiency

Alina, Anna, Lena, Ludwig, RomanDaniela?, Nicole?

2 MALE*SC_1P*GY_REC present Typical w.r.t. sufficiency Jonas, Ludwig3 ABI_1P*GY_REC absent deviant with regard to

sufficiencySamuel

4 MALE*SC_1P*GY_REC absent deviant with regard to sufficiency

None.

5 gy_rec

(i.e. the quasi-necessary condition GY_REC is absent for these cases)

present deviant with regard to necessity

Sibel, Andreas, Julia, Christian, Aynur, Tessa

Table 6: Types of configurations

In Cooper & Glaesser (2012) we discuss three German interviewees, two deviant re necessity and one re sufficiency. Our purpose in focusing on deviant cases is to develop causal understanding and/or theory.

One discussed case is Andreas, deviant re necessity, who is of the type

MALE*ABI_1P*SC_1P*gy_rec.

To make sense of such deviant cases, we aim to find an X2 that might substitute for the quasi-necessary condition X1 (here, the recommendation for Gymnasium).

X2 might, of course, be a conjunction of factors.

Andreas attended a Realschule but is soon to enter the Gymnasium, belatedly achieving this outcome.

The educational careers of his siblings seem key factors. They all attended Gymnasium. Both his parents have the Abitur.

The lack of the recommendation could be calmly overcome given relevant knowledge in his family of the system (see Cooper & Glaesser, 2012, for evidence).

A candidate X2 here (as an alternative to the recommendation) seems to comprise, then, a family where gaining the Abitur is a normal element for both parents and their children conjoined with the organisational possibility of moving from the Realschule to some type of Gymnasium.

In fact, it turns out that having at least one parent with the Abitur can play this role, as the following cross-case test shows:the quasi-necessity of GY_REC + ABI_1P raises consistency with necessity from 0.89 to 0.96.

We argued earlier some regularities can be fully represented in terms of sufficiency and necessity. As we noted, however, regularities alone cannot establish causation. A further step is needed. This might be theoretical interpretation, if strong enough theory exists. Alternatively, within-case data collection and analysis can help gain access to the processes that generate quasi-regularities and the exceptions to them.

In Cooper & Glaesser (2012), we discuss deviant cases, with the goal of understanding why it is, re sufficiency, that some cases with a normally sufficient combination of conditions to gain our outcome did not and why, re necessity, some cases lacking a normal necessary condition nevertheless managed to achieve our outcome.

An important common factor for all three cases was the immediate and extended familial context in terms of what it was “natural” or habitual to do, educationally and occupationally.

We have focused in this paper on discussing “deviant” cases. We are also analysing typical cases in order to asses the extent to which their biographies can be captured by theories such as Bourdieu’s. We will report on this ongoing work in future papers (see Glaesser & Cooper, 2012, BSA, for a start).

References

Cooper, B. & Glaesser, J. (2011). Paradoxes and pitfalls in using fuzzy set QCA: Illustrations from a critical review of a study of educational inequality. Sociological Research Online, 16(3), http://www.socresonline.org.uk/16/3/8.html.

Cooper, B. & Glaesser, J. 2012. Qualitative Work and The Testing and Development of Theory: Lessons from a Study combining Cross-Case and Within-Case Analysis via Ragin's QCA. Forum: Qualitative Social Research 13(2): 4. http://www.qualitative-research.net/index.php/fqs/article/view/1776

Cooper, B.; Glaesser, J.; Gomm, R. & Hammersley, M. (2012) Challenging the Qualitative-Quantitative Divide: Explorations in case-focused causal analysis. Continuum.

Glaesser, J. & Cooper, B. (2011) Selecting cases for in-depth study from a survey dataset: an application of Ragin’s configurational methods, Methodological Innovations Online. 6(2): 52-70.

Glaesser, J. & Cooper, B. (2012) Do rational action theories and Bourdieu’s habitus provide competing or complementary accounts of educational decision-making? British Sociological Association Annual Conference, April 2012, Leeds.

Hedström, Peter & Ylikoski, Petri (2010). Causal mechanisms in the social sciences. Annual Review of Sociology, 36, 49-67.

Kosko, B. (1994) Fuzzy Thinking: The New Science of Fuzzy Logic. Hyperion Books.

Mahoney, J. & Goertz, G. (2006): A tale of two cultures: contrasting quantitative and qualitative research. Political Analysis, 14, (3).

Pawson, R. (2007) Causality for Beginners. eprints.ncrm.ac.uk/245/1/Causality_for_Beginners_Dec_07.doc

Ragin, Charles C. (2008). Redesigning social inquiry. Fuzzy sets and beyond. Chicago: Chicago University Press.

Vaisey, S. (2007) Structure, culture and community: the search for belonging in 50 urban communes. American Sociological Review, 72, 851-873.