Upload
madeline-andrews
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
CODING, MANIPULATION CHECKS, POST-EXPERIMENTAL INTERVIEWS …AND OTHER WAYS QUALITATIVE DATA SNEAK INTO YOUR
LIFE
By: Lauren Boyatzi
November 28, 2011
WHAT IS QUALITATIVE RESEARCH?
Please take a minute… Subjective Atheoretical Unable to show causation Lack of generalizeability Little-to-no a priori knowledge Naturalistic approach Time-consuming/difficult
WHAT IS QUALITATIVE RESEARCH?
There is a “wide consensus that qualitative research is a naturalistic, interpretative approach concerned with understanding the meanings which people attach to phenomena (actions, decisions, beliefs, values, etc.) within their social worlds” (Snape & Spencer, 2003, pg. 3, in “Qualitative
Research Practice: A Guide for Social Science Students and Researchers, edited by Ritchie & Lewis)
WHAT IS QUALITATIVE RESEARCH?Qualitative Research Quantitative Research
Aim is complete, detailed description Aim is to classify features, quantify them, and construct statistical model to explain observations
Zero to little a priori knowledge of phenomena
Much a priori knowledge of phenomena
Recommended during earlier phases of research
Recommended during latter phases of research
Design occurs as study is conducted Study is carefully designed prior to data collection
Data = words, pictures, objects Data = numbersSubjective ObjectiveLess generalizeability, but more rich in detail
More generalizeability but may miss contextual information
Different assumptions (e.g., variables are interwoven, difficult to measure), purpose (e.g., contextualization), approach (e.g., naturalistic), and researcher role (e.g., partiality) than quantitative
Different assumptions (e.g., variables are identifiable and relationships can be measured), purpose (e.g., prediction), approach (e.g., experimentation), and researcher role (e.g., impartiality) than qualitative
HOW APPLICABLE IS QUALITATIVE RESEARCH? Developmental Psychology
Interviews of childhood experiences Observationally coding the quality of parent-child
interactions and/or parental sensitivity Parent-child discourse (quality, open communication)
Culture/Negotiation MURI interviews regarding honor, dignity, face
Terrorism (within Social Psychology) Propaganda videos Motivations of suicide terrorists
Counseling Psychology Amount/type of therapist self-disclosure on patient
outcomes Cognitive Psychology
Accuracy of memory of an event Presence/absence of and elaborateness of a false
memory Legal Psychology
Reasons that jurors (mock or real) provide for verdicts
WHY YOU MAY WANT TO USE QDA (AND) WHAT DOES IT PROVIDE?
Some data are only qualitative (e.g. terrorist propaganda videos) and in order to be analyzed, need to be quantified in some way
We don’t always know what the numbers mean when we collect quantitative data Thus, you should include free response questions
and post experimental interviews to make sure that you and participants are thinking that the numbers mean the same thing Construct validity
Especially good for beginning stages of research and for when awareness of what you’re studying on the part of the participants matters (e.g. social desirability issues, suspicion check)
HOW CAN QDA WORK FOR YOU? (WHAT IS YOUR PURPOSE?) Can be entire project or only a small piece (as with
open-ended questions) Exploratory/pilot
Ex. Understanding basic nature of revenge, why do people get revenge, how often, how do they feel after, etc
(if literature doesn’t provide theory for specifics already) Manipulation check
Ex. Did you think that your partner giving you negative feedback constituted a transgression?
Post experimental interview Ex. During experiment in which you worked with a partner,
did you experience a transgression, if so, how did that make you feel, did you want revenge afterwards, did you consider other reactions to the perceived transgression, etc.?
Test specific questions Ex. Have participants experience a transgression and for
the DV, have open-ended question about how they would like to respond to transgressor
PURPOSE DICTATES WHETHER INTERRATER RELIABILITY/AGREEMENT WITHIN CODING IS A CONCERN
Exploratory/pilot, manipulation check, post experimental interview don’t really require agreement between coders This is mostly for data that you will not publish but
rather informs your research (e.g. focus groups about how well an intervention is working; information used to remove subjects due to suspicion or “drop out” reasons)
Testing hypotheses definitely requires interrater reliability
Look at the standards in your literature and the journals in which you hope to publish Also consider the time/energy you can commit
SEVERAL APPROACHES TO INTER-RATER RELIABILITY/AGREEMENT AND CODING
Train two people to 95% agreement, then release each to code half of the document(s) least rigorous; probably okay for a lower impact
journal Train two people, have each code separate
parts of the document as well as some of the same parts to assess IRR on the latter part “medium” level of rigor
Train two (or more) people, have each code all of the document(s); assess IRR on every document most rigorous; best for top tier journal
(there are variations of these)
ASSESSING INTER-RATER RELIABILITY AND AGREEMENT
Assigning Stimuli to Nominal Categories interrater agreement - % agree/total
best for present/not (0 or 1) type of codes Content validity ratio (CVR) – multiple raters rating a
single dichotomous stimulus Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology,
28, 563-575.
Cohen’s kappa – two raters rating multiple stimuli **Most commonly used Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and
Psychological Measurement, 20, 37-46.
Fleiss’ kappa – extension of Cohen’s kappa for more than two coders
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-383.
Intraclass correlation coefficients (ICC) – interrater agreement for multiple judges
There are many versions and they differ (Shrout & Fleiss, 1979)
ASSESSING INTER-RATER RELIABILITY AND AGREEMENT
Assigning Stimuli to Continuous Constructs rwg(1) –interrater agreement index for continuous
constructs Not limited to two coders
James, L.R. Demaree, R.J., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
James, L.R. Demaree, R.J., & Wolf, G. (1993). Rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.
Standard deviation (measure of observed variability) Schmidt, F. L., & Hunter, J. E. (1989). Interrater reliability coefficients cannot be computed when
only one stimulus is rated. Journal of Applied Psychology, 74, 638-370.
…Or you can skip coding altogether and go with a “QDA program,” e.g. Atlas.ti nVivo LIWC
WHAT YOU CAN DO WITH THE DATA ONCE YOU HAVE IT Examine frequencies
E.g. “How often is revenge mentioned in a recall task asking about a transgression?”
Measure correlation between two constructs E.g. “Within an interview about past
transgressions, how often are revenge and honor mentioned?” Use Phi coefficient
Network analysis – how words are connected to other words Same technique as social network analysis Use programs AutoMap and Ora to analyze Fun fact: I heard that this is how Enron was
busted (words were connected to other words that they shouldn’t have been)
CONCLUSIONS
Both qualitative and quantitative research can have same methodological rigor (and same assumptions, purpose, approach,
research role/objectivity, etc) What you do differently (or the same)
depends on your purpose and research question
Qualitative data collection and analysis is for everyone! Improves research by making it richer/stronger
Can be used in field, lab, and regardless of area/topics
IMPORTANT POINTS FROM DISCUSSION
Interrater reliability should be assessed at the level at which you make your conclusions
You can/should assess coder “drifting” which can occur: If one coder drifts from manual and other coders If two coders are reliable with each other but drift from
manual Qualitative data can provide compelling examples
(e.g. quotes or video) for your paper/talks You can/should publish your coding manual so that
other researchers in your field understand your criteria for different constructs May alleviate simple interpretation disagreements