Epistemology and Methods

Process-Tracing and Content Analysis

May 19 2009

Process-tracing

• Process-Tracing and Historical Explanation (George & Bennett)

• It’s the Process Stupid (Checkel)

George & Bennett

• What is it about:

• “…to identify the process, one must perform the difficult cognitive feat of figuring out which aspects of the initial conditions observed, in conjunction with which simple principles of the many that may be at work, would have combined to generate the observed sequence of events”

• Applied in many theoretical contexts (rational choice, game theory, institutionalism, constructivism, …)

George & Bennett

Why process-tracing?

• …may help narrow the list of potential causes

– “it might be difficult to eliminate all potential rival explanations (…) especially when human agents are involved (…) for they may be doing their best to conceal causal processes

• …it forces to take equifinality into account (alternative paths through which the outcome could have occurred), thus mapping out various potential causal paths

– with more cases, focus on conditions under which causal paths occur (typological theory)

George & Bennett

Why process-tracing?

• …that intervening variables, if truly part of a causal process, should be connected in particular ways is what allows to reduce the problem of indeterminacy

• …it complements other research methods

• …answers in deviant cases

• …finding omitted variables

• …role in theory development

George & Bennett

Varieties of process-tracing?

• Detailed narrative – historical chronicles

• Use of hypotheses and generalizations– parts of the narratives accompanied with explicit causal

hypotheses /attempting to engage in explaining general patterns

• Analytic explanations– explicit theoretical forms

• More general explanations– Moving up the ladder of abstraction

George & Bennett

Forms of causal processes:

• Linear causality

• Complex causality

• Interacting causal variables / interaction effects (typological theories)

• Path-dependency (locate key decision or branching points)

George & Bennett

To watch out for:

• Assessing alternative hypothesized processes– Your interest in your hypothesis to be valid creates a strong

confirmation bias, and it can overstate the causal weight…• Interruption in the causal path

– Which lead to provisional conclusions• Equifinality

– If you have two alternative explanations / Cumulative effects? Interaction effects?/ Or different interpretation of facts

George & Bennett

Guidelines:

• Identifying and addressing factual errors, disagreements, and misunderstandings

• Identifying all potentially relevant variables and hypotheses

• Comparing various case studies of the same events (using various theoretical perspectives)

• Identifying additional testable and observable implications of competing interpretations

• Identifying the scope conditions…

Checkel

• Process tracing: a new buzzword status

• Bringing theory closer to “politics”

• Making scholarship more policy-relevant

• But watch out for:

• Losing sight of the big picture

• Be aware of data requirements

• Epistemological traps?

Checkel

• Examples:

• in IR: “Democratic Peace” (law-like)

• In EU integration studies: “Bureaucrats Go-native in Brussels” (claim)

• We know little about the process

• Process-tracing could help us for instance to rule out certain alternative explanations (e.g. those send to Brussels are already committed Europeans), help us understand the intervening variables (e.g. that lead to democratic peace)

Checkel

• What is a mechanism?

• “Mechanisms connect things. They are “recurrent processes linking specified initial conditions and a specific outcome” (Mayntz 2003)

• Link between cause and effect

• Mapping the process carefully

• Between the beginning and the end, the researcher looks for a series of theoretically predicted intermediate steps

Checkel

• Does it coincide with prior, theoretically derived expectations about the working of the mechanisms?

• Data is overwhelmingly qualitative (historical memoirs, interviews, press accounts and documents)

• Strong on questions of how and interactions

• Weaker at establishing structural context (constructivist concern)

• Challenge: Significant amount of time and data required

Checkel

• Empirical Application: The Socializing Power of European Institutions

• Persuasion as one mechanism of socialization

• Scope conditions under which it occurred

• “How would I recognize persuasion if I were to walk through the door?”

• Use of multiple data streams (interviews with Committee members over five years, confidential meeting summaries, secondary sources, triangulation)

Checkel

• Process-Tracing: The Good, the Bad and the Ugly

• Lesson 1 (The Good): Helping to Bring Mechanisms Back In

– and the method is here to stay!

• Lesson 2 (More Good): Coming to Grips with First Mover Advantages

– balance theoretical favorites with alternative explanations

• Lesson 3 (Yet More Good): Promoting Bridge Building

– (e.g. between rational choice and social constructivism)

Checkel

• Lesson 4 (The Bad): Weak Theories

– Not conducive to develop parsimonious or generalizable theories

– The mechanism-based approach to theory is “intermediate between laws and descriptions”

– Problematic when it leads to over-determined and, in the worse case, kitchen sink arguments where everything matters

– Attention to research design can minimize the problem

Checkel

• Lesson 5 (More Bad): Proxies Are a Pain for Us, too.

– Persuasion can not be measured directly, so interviews before and after, etc…

• Lesson 6 (Still More Bad): It Takes (lots of ) Time

• Lesson 7 (Yet More Bad): Losing the Big Picture

– Structural context and normative implications

• Lesson 8 (The Ugly): The Dreaded “E” Word

– PT is at odds with the interpretative epistemologies that characterize many forms of constructivism

Content Analysis

• General comments:

• “the systematic counting, assessing, and interpreting of the form and substance of communication”

• Precondition: physical record of communication• Type of communication: books, magazines, newspapers,

documentaries, films, recordings, photographs, meeting minutes, government documents, diplomatic communiqués, cartoons, political advertisement, speeches, letters and diaries, e-mails, blogs,….

Content Analysis

• Steps in using this method:

– Define the population of communications we want to study (this is determined by the research question!)

– Once population is defined, one possibility is sampling (still a large n, sufficient level in generalizing from results)

Content Analysis

• Focus on

– Type of communication – Time period of communication– The size of communication (e.g. op-ed articles)– The frequency of communication (e.g. monthly memos)– The distribution of communication (e.g. home delivered

newspapers)– The location of communication– The parties to the communication (sender, receiver)

Content Analysis

• Define the unit of analysis

• Word (e.g. peace in speeches of presidents)

• Pitfalls:

• Different meanings of the word (e.g. we seek peace vs. we never allow peace): counting the words in context (unipolar vs. bipolar vs. multipolar ways)

Content Analysis

• Item/Theme: the communication itself taken as a whole

• Example: e.g. how many times GB senior referred to Saddam Hussein as Hitler-like prior to 1991 Persian Gulf War

• Full-text search (e.g. NYT) for words “George Bush”, “Saddam Hussein” and “Hitler”

• These words appear within 10 words in either direction

Content Analysis

• Create a “dictionary”: define each and every observation we might make and allocate a particular coding

• What constitutes a reference: e.g. Cuban school books (American, US, Imperialists, Outlaw Regime in Washington, etc…): emphasis, this could measure the saliency…

• Evaluation of references (good or bad, pro or anti, ranking, intensity): assessment, what is the position taken

• E.g. ranking statements to measure “intensity” of newspaper endorsement of candidates

• The role of pre-testing!

Content Analysis

• Special problems:

• Communications are issues for a purpose “All Chinese people believe that the new agricultural policy is a major step….”

• If we want to assess the impact of communication, we need to know whom it reaches…

• What is the degree of our own access to communication (free choice over material we analyze?)

• Is the research sample representative

• Intercoder reliability (human judgments): degree of consensus among coders

Laver/Garry

• Estimating Policy Positions from Political Texts

• The Manifesto Research Group (MRG) / Comparative Manifestos Project (CMP): biggest show on the road (started in the early 1980s)

• Party manifestos are strategic documents written by politically sophisticated party elite with many different objectives in mind

• Usually issues during elections campaigns, reflect party positions

• Provide evidence party position changes over time

Laver/Garry

• MRG: measuring relative emphasis of an issue (salience, not party position!), position # emphasis (attention paid to)

• MRG categories unipolar (e.g. law and order)• MRG categories bipolar (e.g. social service expansion)

• In order to retrieve information on position, additional coding of MRG raw data (e.g. Harmel-Janda Party Change Project (PCP))

• Definition of 19 issues of interests, coding -5 to +5, few experts engage in coding)

• Caveat: Prior knowledge (subjective placement of parties)…

Laver/Garry

• Estimating policy positions / expert coding:

• Data reduction (coding scheme)

• in MRG sentence count for 54 coding categories, in PCP transforming text into scores on 19 policy scales

• New approach for systematic coding: an example of hierarchical decision-making process (Table 1):

• 1 ECONOMY (Role of state in economy)– 11 Economy/+State+ (Increase role)

• 111 Economy/+State+/Budget (Budget)

• 1111 Economy/+State+/Budget/Spending (Public Spending)– 11111 Health

– 11112 Education and Training

– 11113 Housing

– 11114 Transport

– ….

• 1112 Economy/+State+/Budget/Taxes (Increase Taxes)

• 1113 Economy/+State+/Budget/Deficit (Increase Budget Deficit)

Laver/Garry

• Estimating policy positions / computer coding

• Quantitative content analysis

• Dictionary – mechanical criteria

• But need to focus on less ambiguous words or phrases (unipolar)

• Example on “taxes”…it’s possible to lower taxes and to raise taxes, but in manifestos in favor of lowering taxes (question is how often is one wrong in order to correct?)

Laver/Garry

• The validity costs are offset by gains in reliability (Computer coding is reliable) and fewer costs

• Computer codes without knowledge of context (no pre-information…), computer could potentially more easily detect policy changes…

• Laver/Garry on policy positions in UK and Ireland

• Cross-validation of techniques (Computer, revised expert, MRG, expert surveys), high correlation on economic policy, less so on social policy (e.g. MRG technique of saliency…)