Corpus-based dictionary of emotional instability for use …ymarton/pub/sentiment/shukicohen... · Web viewAlmost invariably, the word pretty was used as an intensifier (e.g. “pretty

Running head: MEASUREMENT OF NEGATIVITY BIAS

Measurement of Negativity Bias in Personal Narratives Using Corpus-Based Emotion Dictionaries

Shuki J. Cohen

New York University, Department of Psychology

Corresponding Author:

Shuki J. Cohen

Psychology Department

John Jay College of Criminal Justice

445 W 59th St rm# 2402

New York, NY 10019

Ph: (646) 557-4627 Fax: (212) 237-8930

e-mail: [email protected]

Yale University School of Medicine

Department of Psychiatry

34 Park St. Suite #B-38

New Haven, CT 06519

1

Measurement of Negativity Bias 2

Abstract

This study presents a novel methodology for the measurement of negativity bias using positive

and negative dictionaries of emotion words applied to autobiographical narratives. At odds with

the cognitive theory of mood dysregulation, previous text-analytical studies have failed to find

significant correlation between emotion dictionaries and negative affectivity or dysphoria. In the

present study, an a priori list dictionary of emotion words was refined based on the actual use of

these words in personal narratives collected from close to 500 college students. Half of the

corpus was used to construct, via concordance analysis, the grammatical structures associated

with the words in their emotional sense. The second half of the corpus served as a validation

corpus. The resulting dictionary ignores words that are not used in their intended emotional

sense, including negated emotions, homophones, frozen idioms etc. Correlations of the resulting

corpus-based negative and positive emotion dictionaries with self-report measures of negative

affectivity were in the expected direction, and were statistically significant, with medium effect

size. The potential use of these dictionaries as implicit measures of negativity bias and in the

analysis of psychotherapy transcripts is discussed.

Keywords: Text-Analysis, Dysphoria, Negativity Bias, Personal Narratives, Corpus Linguistics

Footnotes: The author wishes to thank Patrick E. Shrout and the late Carol Feldman and

Joan Welkowitz for their advice in conducting this research. Also, many thanks to the numerous

research assistants who assisted with various parts of this research, especially to: Julia Betensky,

Venessa Bikhazi ,Candis Conover, Heather Holahan, Melanie Fox, Tyson Fur, Jonathan

Kirschner, Jen Leeder, Julie McCarroll, Joanna Murstein, Jonathan Shaffer and Sam Zun.


Measurement of Negativity Bias in Personal Narratives Using Corpus-Based Emotion Dictionaries

Introduction

Bias in the processing of emotional information has been widely established as one of the

most fundamental features of psychological disorders as of individual personality. Extensive

research has repeatedly demonstrated that people high on neuroticism or negative affectivity

selectively perceive, attend to, interpret and recall negatively charged or ambiguous events more

negatively than normal controls (Watson & Clark, 1984; Clark et al., 1999; Rusting, 1999;

Abramson et al., 1989; Ingram, 1984; Rector et al., 1998, Rude et al., 2003). In cognitive

psychology, as well as in its psychotherapeutic applications, the cognitive overrepresentation of

negative information is conceptualized within the framework of depressive self-schema (e.g.

Beck, 1976; Markus, 1977; Segal & Ingram, 1994; Nasby & Kihlstrom, 1986). Evidence for the

existence of depressogenic self-schema among both dysphoric subjects as well as clinically

depressed patients has been demonstrated in experiments involving attention, recall, lexical

decision, mood priming, and other cognitive capacities (for reviews see Isen, 1984; Mathews &

MacLeod, 1994; Rector et al., 1998; Rusting, 1998; Scher et al., 2004).

Negativity bias in the self-schema has been mostly studied using self-report

questionnaires (Shaw & Dobson, 1981; Dobson & Breiter, 1983; Oliver & Baumgart, 1985;

Beck et al., 1988; Clark, 1988; Haaga et al., 1991; Ingram et al., 1995; Chien & Dunner, 1996;

Glass & Arnoff, 1997; Alloy et al., 1999; Winters et al., 2002). Self-reports have mostly

demonstrated excellent psychometric properties and are straightforward, practical and

inexpensive to administer. Further, self-reports have demonstrated theory-consistent


correspondence with cognitive content and schematic organization of emotional material.

However, self-report questionnaires are also frequently criticized due to their susceptibility to the

introspective limits of the respondent and to response bias, including impression management

and self-deception or defensiveness (e.g. Winters & Neale, 1985; Polaino & Senra, 1991;

Shedler et al., 1993; Gotlib et al., 1995; Lindeman & Verkasalo, 1995; Paulhus & John, 1998;

Stone et al., 2000; Wenzlaff & Wegner, 2000; Rude et al., 2001; Nosek, 2005). Moreover,

cognitive theory regards the depressogenic schema as a stable cognitive style that is minimally

affected by states of mood or depressive symptomatology, but numerous self-reports have failed

to detect negativity bias in remitted patients under normal mood conditions (e.g. Persons &

Miranda, 1992; Segal & Ingram, 1994; Haeffel et al., 2005; Fresco et al., 2006).

One vehicle that was suggested to circumvent the high face validity of self-reports is the

analysis of verbal behavior. Studies that have used autobiographic speech samples to assess the

negative emotional bias of the speaker rest on the premise that a) the ability of the speaker to

control word choice is limited due to the rapidity of fluent speech and b) that speech may afford

an almost instantaneous account of the cognitive content of the mind as it reaches awareness,

thus making it the fastest method of introspection. Negativity bias in the emotional content of

spontaneous speech has shown consistent associations with indices of mental health. For

example, Gottschalk and Gleser (1969 p. 106) found significant correlations between their

measure of verbal anxiety and the emotional instability subscale of the 16PF, and Gottschalk

(1997) found significant correlations between the same scale and self-reported anxiety. Other

studies that used manual-based content analysis of spontaneous speech have also reported

significant correlations with the cognitive negativity bias of the speaker (e.g. Viney, 1983;


Watkins & Rush, 1983; Peterson, 1992; Cacioppo et al., 1997; Davison et al., 1997; Hurlburt,

1997; Hartman-Hall & Haaga, 1999)

Manual-based content analysis of spontaneous speech has been criticized for being time-

consuming, training-intensive and susceptible to rater drift and other rater-based biases (e.g.

Hurlburt, 1997; Pennebaker et al., 2003). In contrast, computerized text-analysis can process

large amounts of transcribed speech in considerably shorter time (Gottschalk & Bechtel, 1982;

Pennebaker et al., 2003; Alpers et al., 2005). Moreover, several studies that used computerized

text-analysis of multiple categories of words have demonstrated superiority of this technique

over human raters in assessing mental health (Rosenberg et al., 1990; Pennebaker et al., 1997;

Pennebaker & King, 1999).

This study reports the construction and initial validation of an index of negativity bias

based on context-based dictionaries for positive and negative emotional language. These

dictionaries can be plugged into text-analytical software to obtain a metric representing the

positive and negative emotional content of the speech sample. The construction process is

described briefly, while a more detailed account is now under review. The advantage of these

dictionaries, as well as the importance of contextualized dictionary entries for the computerized

detection of emotional language are then discussed.

Method

General Overview:


The construction of both positive and negative dictionaries followed several phases. In

Phase 1, three research assistants independently detected candidates for emotion words in

transcripts of 240 narratives obtained from undergraduate students using standard elicitation

technique. The resulting list was consolidated and then contrasted and completed using other

text-analytical dictionaries In Phase 2, the context in which each of the candidate emotion words

occurred in the narrative was examined using concordances specific to each word. This phase

detected word combinations that are reliably associated with the use of the word in its intended

emotional meaning. Conversely, this phase also detected idioms or word combinations in which

the candidate words were not used in their emotional sense. To avoid the ‘dilution’ of the

proposed dictionary, this list was subtracted from the final metric. Overall, each positive or

negative dictionary is comprised of a list of words, word stems or idioms that constitute the

emotionality metric, and an ‘anti-dictionary’ of word combinations that should be subtracted

from the dictionary to ensure its purported emotional meaning. Finally, to test the concurrent

validity of the proposed dictionaries, their association with two common self-report measures of

psychological distress and depression were examined. The Global Severity of symptom Index

(GSI) of the Symptom Checklist-90 (SCL-90) was used as a measure of general psychological

distress, and the Beck Depression Inventory (BDI) was used as a measure of dysphoria..

Participants:

Participants were 483 undergraduate students (345 female and 138 male) from a large

private urban northeastern university who received partial course credit for their participation.


All participants were self-identified as native speakers of North American English. The average

age of the subjects was 19.8 years.

The participants were recorded telling a story concerning a recent disagreement they had

with a significant other, and then filled out a battery of self-report questionnaires covering

various aspects of their mental well-being and symptoms.

Personal narratives:

For the elicitation of speech samples, research assistants approached the participant with

the following request:

“I would like for you to tell me in your own words the details of a recent disagreement you had

with somebody who is close to you emotionally. It may be a small or big disagreement, as long

as you feel comfortable sharing it and it is with somebody who is or was close to you at the time.

You have five minutes to tell your story, and it will be recorded on the machine in front of you.

Five minutes is a long time for a story, so you may want to go into details. Also, I will not be

able to help you along or lead you with questions. I will stop the machine when the five minutes

are up. Do you have any questions? (subjects questions were answered at this point as succinctly

as possible, although almost invariably the subject replied negatively to this question). Ok, let me

know when I can start recording” (subjects to the experimenter when they were ready to

proceed).

The elicitation requests were delivered verbatim to avoid variability in the demand

characteristics of the task. The lack of verbal interaction with the subject was also designed to


standardize the task and preserve the unique narrative structure of each speaker. The stories were

then transcribed according to a common standard in the field (Mergenthaler & Stinson, 1992),

using a 3-stage process to ensure quality control.

Computerized Text-analysis:

The text-analysis software was written by the author, and was comprised of a set of

computerized routines, or modules – each allowing the manipulation and analysis of the text in a

manner tailored to the task at hand. Like existing text-analytical packages, the software couls

search for whole words, word combinations, stems of words and word suffixes. The software

could also construct a concordance showing the immediate context of a given word, and

presented the results as both counts and frequency of the searched items (For a solid linguistics

and methodological background, including previous studies utilizing this technique see Sinclair,

1991). The software was designed as a programmer’s tool, rather than a user-ready package, to

ensure its accuracy and avoid the blind suppression of error messages – a common practice in

commercial text-analytical packages.

The Development of the Dictionary of Positive and Negative Emotional Language:

Phase 1:

A group of 3 research assistants-- all native speakers of US English-- inspected

independently the narratives of the first 240 subjects for positive and negative emotion language.


The definition of emotion used in this stage was broad, to foster maximal inclusion. The words

could include behaviors associated with emotions (e.g. yelling, crying, fighting, break* up, slam

(door), hang up (phone) etc), emotional states (e.g. happy, upset, awkward, surprised, and

cognitive words frequently associated with emotional states (e.g. discouraging, appreciate, agree,

understand etc.)

Words, word stems or word combinations that were selected by two or more research

assistants were retained. This list was then merged with a comprehensive list of emotion words

from other, mostly text-analytical, dictionaries (Zuckerman et al., 1965; Stone et al., 1966;

Anderson, 1968; Dahl & Stengel, 1978; Sweeney & Whissell, 1984; Whissell et al., 1986;

Marchitelli, 1983; Marchitelli & Levenson, 1992; Pennebaker et al., 2001). The resulting master-

list comprised of 1,613 positive words and 1,927 negative words. After eliminating duplicate

words, words that have not been uttered by at least two speakers in the training corpus

(corresponding to frequency of below 1/10,000 words), and valenced words that connote

emotions but don’t represent them (e.g. noise, anniversary etc.), the final list of word candidates

contained 211 negative words and 161 positive words.

Phase 2:

In this phase, a concordance was constructed for each word in the list described in the previous

phase. The concordance presented the rater with the context in which the word appeared,

approximately two sentences before and after the word was uttered in the training corpus. Based

on this concordance a set of exclusion or inclusion criteria was crafted for the use of each word

in its intended emotional sense. In cases where no consistent grammatical structures involving


the word could be established, and no inclusion or exclusion criteria could be discerned, the

word was deemed ambiguous in its emotional sense and was eliminated from the dictionary. For

example, whereas words like pretty, like or kind, were included in the positive emotion

dictionary of Phase 1, Phase 2 showed that their usage is too variable and is rarely consistently

emotional to merit inclusion. Almost invariably, the word pretty was used as an intensifier (e.g.

“pretty big”), the word like was used mostly as a conversational filler (e.g. “and I was, like,

totally shocked”), and the word kind was used as a qualifier or a hedge (e.g. “it was kind of

surprising to see him there”). The concordance for these words showed that they were used to

convey emotional meaning in less than 2% of the cases. Acknowledging the vast ‘dilution’ that

could result from including these words in the positive emotional dictionary, they were deleted at

this stage. However, conjugations of these words that reliably and consistently denoted

emotional meaning (e.g. liked, prettier etc.) were retained. Similarly, negations of emotion words

were added to the opposite-valence dictionary. For example, out of the 47 occurrences of the

word happy in the training corpus, 17 (36%) were negated, and of the 16 occurrences of the word

bother in the training corpus 11 (69%) were negated (e.g. “it didn’t bother me”). Including these

words in isolation could have resulted in considerable ‘noise’ in assessing the emotionality of the

narrative.

This phase represents a methodological hybrid between traditional computerized text-

analysis and the more nuanced corpus-based concordance analysis. As such, the process

described above has the potential of being more sensitive to local context and word sense.

General Severity Index (GSI):


The Symptom Checklist -90- Revised (SCL-90-R; Derogatis et al., 1976; Derogatis &

Savitz, 2000) is one of the most commonly used and extensively studied instruments for the

assessment of psychological distress. The SCL-90-R evaluates the degree of perceived emotional

distress caused by 90 common psychologically-relevant symptoms. Despite its theoretical

multidimensionality, the relatively low internal consistency of the scale has popularized the use

of its overall score, the Global Severity Index (GSI) as a measure of general psychological

distress (e.g. Cyr et al., 1985; Derogatis & Savitz, 2000).

Beck Depression Inventory (BDI):

The Beck Depression Inventory (BDI; Beck et al., 1961) is arguably the most commonly-

used instrument for the assessment of dysphoria and screening for depression (e.g. Ritterband &

Spielberger, 1996). The subject is asked to rate the severity level of 21 common symptoms of

depression. The instrument has demonstrated adequate psychometric properties in numerous

settings and populations (for review see Beck et al., 1988). However, while the instrument was

designed as a measure of depression, a growing body of research has established its low

specificity, as evidenced by its moderate to high associations with measures of general

psychological symptomatology (e.g. Endler et al., 1998; Hill et al., 1986). This compromised

discriminant validity may be particularly pertinent for college population (see review in Gotlib,

1984).

Results


Split-half reliability of the emotional dictionaries

Both Phase 1 and Phase 2 were based solely on the first 240 narratives. Therefore, it was

important to establish the split-half reliability of the corpus by comparing the emotional tone of

these narratives (i.e. the training corpus) with the rest of the stories. To accomplish that, the two

parts of the narrative corpus were compared using t-tests applied to the proportion of positive

and negative emotion words identified through the dictionaries. The proportion of emotion words

was defined as the number of emotion markers identified by the dictionary divided by the

number of words in the story. The training and the validation corpora did not differ in the

proportion of emotion words (Positive Emotions: t(239)=-0.59, p=0.555; Negative Emotions

t(239)=1.56, p=0.121, using 2-tail t-test under the heteroscedasticity assumption), thus

establishing the homogeneity of the corpus.

Concurrent Validity Checks for the Dictionary of Positive and Negative Emotional Language:

The association between the proportion of emotion words captured by the dictionaries

and self-report measures of dysphoria and psychological distress is shown in Table 1. Table 1

also shows the associations of dysphoria and distress with the proportion of emotion words as

captured by two leading text-analytical software packages – the Linguistic Inquiry and Word

Count (Pennebaker & Francis, 1996; Pennebaker et al., 2001) and the General Inquirer (Stone et

al., 1966).

For the General Inquirer scales, the positive and negative emotion dictionaries proposed

here were compared to the “negativ” and “positiv” scales from the Harvard-IV


Psychosociological Dictionary, which are comprised of 2,291 and 1,915 words of negative and

positive outlook, respectively. The corresponding dictionaries of the LIWC were the positive and

negative emotion dictionaries. As Table 1 shows, the correlations between the corpus-based

emotion dictionaries and the GI and LIWC dictionaries are significantly different from zero, thus

conferring a modest degree of concurrent validity to the proposed corpus-based dictionaries.

However, the magnitude of the correlations ranges between small to moderate, with no

statistically significant correlation between the corpus-based positive emotion dictionary and the

LIWC positive emotion dictionary. In contrast, the corpus-based positive emotion dictionary

exhibits statistically significant correlation with the respective dictionary in the GI program.

However, the effect size of this association is small, and the dictionaries share only 6% of their

variance.

Construct Validity Checks for the Dictionary of Positive and Negative Emotional Language:

A crucial test for the construct validity of the proposed corpus-based dictionary is its

correlation with measures of psychological distress or dysphoria. As mentioned above, the

construct of psychological distress, with its related manifestations, namely depression or

dysphoria, is associated with preferential processing of negative information (for reviews see

Clark et al., 1999; Blaney, 1986; Segal & Ingram, 1994; Rusting, 1999; Rector et al., 1998;

Martin, 1985; Teasdale, 1999; Beck, 2002; Scher et al., 2004) has not been demonstrated reliably

in most previous computerized text-analytical studies.

Table 1 shows the correlations between the two measures of psychological distress (BDI

and GSI) and the proportion of positive and negative emotion words based on the corpus-based


dictionary. As an adjunct to the analysis of the concurrent validity of the corpus-based emotion

dictionaries, Table 1 also shows the correlations of the proposed dictionaries with measures of

distress as compared with the two other text-analytical programs mentioned above.

As shown in Table 1, the corpus-based negative and positive emotions correlated

significantly, and in the expected direction, with measures of distress and dysphoria, while the

two other programs did not exhibit such a correlation. The exception to this general finding is the

LIWC’s negative emotion scale, which correlates significantly with the BDI. To assess the

statistical significance of the difference between the correlations of the self-reports with the 3

text-analytical systems, a procedure developed by Olkin & Finn (1990, 1995) for testing the

equality of dependent correlations was used. This technique was preferable to the common

methods of comparing correlation magnitude based on Fisher-test, since the correlations to be

compared were derived using the same sample, hence violating the independence assumption

inherent in the Fisher test. Table 2 the 95% Confidence intervals for the differences between the

correlations of the text-analytical scales and the outcome measures, as calculated by Olkin &

Finn (1990,1995) procedure. The analysis has shown that the difference between the correlations

is statistically significant. Therefore, the frequency of emotion words detected by the corpus-

based dictionary was significantly more associated with measures of dysphoria and distress than

the two other text-analytical programs.

Discussion

This study examined a novel approach to the assessment of negativity bias, based on

computerized text-analysis of personal narratives that considers context information.


Incorporating concordance-based contextual information into the automatic identification of

emotion words is an attractive alternative to extant text-analytical methodology, which currently

relies almost solely on decontextualized word lists. In lieu of amassing a large number of

emotion words or their synonyms, and tallying their occurrence in personal narratives, this study

showed that including simple usage-based rules to preferentially tally only words that are used in

their emotional sense robustly augmented the validity of the text-analytical dictionaries. The

construct validity for the proposed dictionaries was tested by examining their correlations with

common measures of general psychological distress (GSI) and dysphoria (BDI) on normal

undergraduate population. Both corpus-based positive and negative emotion dictionaries

demonstrated improved construct validity as compared to the two most studied text-analytical

programs, namely the LIWC and the General Inquirer.

Several methodological steps were taken to maximize the ecological validity of the

corpus-based emotional dictionaries. Firstly, the personal narratives in the corpus used to

construct the dictionaries concerned a recent disagreement with a significant other. This type of

narrative is highly frequent in psychotherapy, as well as general social settings (Horowitz, 1979;

Biber et al., 1998; McAdams, 2001; Bohanek et al., 2005). Further, Ample research has shown

that negative bias in processing emotional information is most consistently demonstrated when

the information pertains to the self or a significant other, but not as consistently when it concerns

generic person (Martin et al., 1983; Kuiper & Higgins, 1985; Bargh & Tota, 1988; Kuiper et al.,

1988; Mangan & Hookway, 1988; Baldwin, 1992; Cacioppo et al., 1997; Collins & Feeney,

2004). Secondly, the relatively passive role of the experimenter in the elicitation of the narrative

contributed to narratives that are similar in structure and content to those that are usually told to a

generic stranger. These narratives also look remarkably similar to those found in transcripts of


first sessions of psychodynamic psychotherapy or psychodiagnostic interviews. From the

psycholinguistic and socio-linguistic point of view, this level of interaction minimize the effects

of familiarity, speech accommodation, politeness and other external influence on the subject’s

speech (e.g. Bell, 1984; Hudson, 1996; Shepard et al., 2001; Giles, 2001). Thirdly, the speakers’

unawareness of our interest in emotional language was crucial to the elicitation of emotional

narratives while minimizing the demand characteristics of the situation. To that aim, nowhere in

the description of the experiment to the subject was there any mention of emotions. Akin to a

clinical discourse, the request to talk about a disagreement with a significant other (rather than a

topic of the speaker’s choice) was designed to minimize simultaneously both the social

desirability demand to share with strangers harmlessly positive stories and the demand to

mention or elaborate on emotions per se. Further, avoiding a direct request to speak about

negative emotions arguably contributed to the validity of the corpus, as speakers could choose

the emotional elaboration level with which they felt most comfortable with a stranger. Indeed,

the corpus demonstrated a wide range of emotional elaboration levels, including a wide variety

of psycholinguistic devices to minimize emotionality: narrating the factual chain of events,

minimizing the emotional impact of the event, or even reframing it as a positive experience.

The results of this study underline the usefulness of theory-driven emotion dictionaries

for text-analysis of speech. The substantial correlations of the dictionaries with measures of

psychological distress make them a natural tool for exploring the link between psychopathology

and negative information processing bias using naturalistic speech as a medium. As mentioned

above, previous attempts to link text-analytical indices of negativity bias with emotional or

clinical presentation yielded mixed results. In general, theory-guided computerized text-

analytical scales failed so far to demonstrate convincing correlations with either self-report


questionnaires or clinical interviews (Bohanek et al., 2005; Pennebaker & King, 1999; Williams

et al., 2003; Stirman & Pennebaker, 2001), especially considering the robust correlations

between human ratings of speech emotionality and mental health indices (e.g. (Davison et al.,

1997; Lee & Peterson, 1997; Gottschalk & Gleser, 1969; Viney, 1983; Ruiz-Caballero &

Bermudez, 1995; Rusting, 1999; Klein et al., 1986; Cacioppo et al., 1997; Hurlburt, 1980;

Weintraub, 1989; Demorest et al., 1999; Luborsky & Crits-Christoph, 1990; Hermans, 1995;

Pennebaker et al., 2003). In contrast, those computerized text-analytical systems that have

achieved adequate construct validity when compared with clinically-validated measures are not

theoretically guided, but rather based on empirical, criterion-keyed or bottom-up principles

(Bucci & Freedman, 1981; Campbell & Pennebaker, 2003; Fertuck et al., 2004; Mergenthaler &

Bucci, 1999; Oxman et al., 1988; Pennebaker et al., 2003; Rosenberg et al., 1990; Rosenberg et

al., 1994; Segal et al., 1993; Spencer & Spencer, 1993; Stone, 1997). The dictionaries reported

here may be the only theory-based measures to demonstrate robust correlations with indices of

dysphoria and general psychological distress, comparable to those of human ratings of speech

emotionality.

The results of this study also underline the crucial contribution that contextual

information can make to the validity of text-analytical indices of emotionality. Unlike previous

text-analytical systems, this study identified contextual constraints on the text-analytical

dictionaries using concordance analysis of emotional markers that was applied to the corpus of

narratives (For an important previous attempt, albeit more limited and syntax-bound rather than

usage-based, to disambiguate emotional sense of words in computerized text-analysis see Kelly

& Stone, 1975). Concordance analysis revealed substantial sources of “noise” in the

decontextualized word-lists used currently in text-analytical programs. For example, while the


words pretty, like and kind are listed in most text-analysis dictionaries as positive emotion

words, concordance analysis showed that over 97% of their occurrence is not consistent with

their intended positive emotional meaning. Rather, the word pretty is used as a quantifier, and

like and kind (especially as the phrase “kind of”, which constitutes about 1425/1435 or 99.3% of

all occurrences of kind in our sample), are used as fillers, filled pauses or hedges. Concordance

analysis also found that although speakers tend to prefer the affirmative sense of emotion words

rather than their negation, it is not rare to find up to 75% of the word occurrences in its negated

sense – which may introduce an unacceptable amount of sense misidentification to the text-

analytical endeavor.

In fact, analysis of the current corpus has found that the negation of words like accept, agree,

trust, comfortable and ready is substantial enough to add to the reliability of the dictionary of

negative emotions, while the negation of words like bother contributes similarly to the reliability

of the positive emotion dictionary.

The results of this study may shed a new light on findings made with traditional text-

analytical systems. For example, using the LIWC text-analytical program, low to no correlations

were found between measures of emotional dysregulation and the frequency of emotion words

used by the informants. Thus, Pennebaker & King (1999) found that in a sample of 841

undergraduate students, self-report measures of Neuroticism correlated 0.16 with negative

emotion words and -0.13 with positive emotion words in their personal essays. Both correlations

were statistically significant and both exhibited the expected direction, but their effect size was

not consistent with the overwhelming array of models that identify negativity bias as the core

bias underlying Neuroticism or emotional dysregulation. Similar meager magnitude of

correlation was found between negative emotion words in narratives concerning the September


11 terrorism attack and self-reported neuroticism of the speakers, while no statistically

significant correlation was established between speakers’ Neuroticism and positive emotion

word frequency (Williams et al., 2003). Consistent with these results, in comparing poetry

samples from poets who committed suicide to those who did not, Stirman & Pennebaker (2001)

found no statistically significant effect of negative or positive emotion word frequency in the

poetry excerpts and the suicidality of the poet.

Somewhat surprisingly, the above-mentioned studies, among numerous others, found

associations between measures of emotional distress and personal pronouns that were equal or

even higher in their magnitude than those between emotional distress and negativity bias in the

use of emotion words. For example, Pennebaker & King (1999) found a correlation of 0.13

between Neuroticism and first person pronoun, while Stirman & Pennebaker (2001), using

ANOVA methodology, found that elevated first person markers and lower first person plural

markers are associated with risk for suicide in poetry excerpts. In a similarly naturalistic speech

samples, Pennebaker and Lay (2002) found that New York City’s Mayor Rudy Guilani’s use of

first-person singular, and not of negative emotion words, was consistently greater during times of

personal distress. Several other studies using the LIWC found that pronouns were also

meaningfully related to recovery from traumatic events, while emotion words were either

unpredictive of grief response following trauma or exhibiting surprising patterns. Thus, Boals &

Klein (2005) found that pronoun use in narratives of romantic breakup was highly associated

with a self-report measure of grief, unlike the use of negative or positive emotions, and in a text-

analytical study of the evolvement of chat-rooms conversations concerning princess Diana

shortly after her death, negative emotions had not changed, while positive emotions were used in

significantly higher level than control chats. In contrast, first person pronouns have changed


steadily and predictably, with grieving writers using collective speech (i.e. high level of plural

pronouns and low levels of singular pronouns) to individualized speech (Stone & Pennebaker,

2002).

The superior stability, consistency and relative robustness of the association between

indices of emotion dysregulation and pronouns as compared to their association with emotion

words led researchers to conclude that social-psychological and sociological theories of mental

health may be more pertinent than emotion regulation theories of mental health. For example,

Stirman & Pennebaker (2001) privileged Durkheim’s social integration theory of suicide over

hopelessness theories of suicide, citing the clear association between higher first person singular

markers and lower first person plural markers in poets who committed suicide, compared to no

main effect of negative or positive emotion on suicidality. Similarly, theories of grief highlighted

the collectivistic vs. individualized aspects of the phenomenon as well as the cognitive search for

meaning, and downplayed the role of emotional experience as expressed by emotional language.

With LIWC being the sole software of choice for text-analysis from 1999 onwards,

numerous publications have since ‘confirmed’ the privileged role of personal pronouns in

emotion dysregulation, above and beyond the expression of emotions. This impressive body of

consistent findings created a sense of consensus regarding the relatively marginal role of

expressive emotionality in emotion dysregulation, thus contributing to the phenomenological rift

between clinical theory and practice, which focus on emotional experience, and social

psychological theories of emotion dysregulation that favor constructs as self-involvement, social

embeddedness, and repressive coping style as the most defensible empirically validated

processes underlying emotion dysregulation (other LIWC-based studies that reported the same


pattern include: Campbell & Pennebaker, 2003; Gill, 2003; Oberlander & Gill, 2004; Bohanek et

al., 2005; Simmons et al., 2005; Mairesse et al., 2007, among others).

The robustly augmented associations between the corpus-based text-analytical

dictionaries reported here and measures of mental distress urge caution in accepting uncritically

the above mentioned body of research, however extensive and diverse. As Table 1 shows,

introducing context to the internal dictionary changes dramatically its ability to detect

emotionality, leading to significantly higher correlations with mental health indices (see Table 2

for statistical significance of difference). Further, since emotions are usually attributed to

persons, with several emotions being attributable or relevant to one person, it comes as no

surprise that pronouns are correlated with emotional processes or states. Further, the grammatical

rules concerning pronouns ensure their relative ‘inertness’ to context compared to emotions.

While we have a host of linguistic qualifiers to emotions (e.g. “very happy”, “kind of happy”,

“not at all happy”, etc.), no similar contextual operators exist that could change the basic

meaning of pronouns. Therefore, basic grammatical rules may explain both the involvement of

pronouns in emotional narratives as well as its consistency compared to those of emotion words.

However, using corpus-based emotional dictionary may reveal different relative significance of

emotional language compared to pronouns, when narratives about emotional processes or states

are concerned.

The cumulative experience with other text-analytical studies adds to the caution with which the

results of this study should be interpreted. As corpus-based scales, the proposed dictionaries may

be of limited generalizability due to a plethora of factors, including: corpus-size, verbal

exclusivity, speech genre and speaker demographics.


Despite the fact that the corpus in this study included close to 500 participants and

380,000 words, past socio-linguistics research has demonstrated inconsistencies in the estimation

of word frequencies that persisted up to corpus-size of xxx words. To assess the stability of the

corpus, the preliminary construction of the dictionaries was based on keyword concordances of

only half of the narratives, while its validation was conducted using the other half of the corpus.

Added stability to corpus-based measures comes from the use of dictionaries, or aggregates of

words, to estimate emotionality.

Text-analytical scales that are based on orthographic transcription of speech can only

detect emotionality if it is explicitly verbalized. Thus, emotional information conveyed by non-

verbal means such as intonation and prosody, however significant, is left undetected. Similarly

undetectable is the avoidance of explicitly verbalized emotionality (e.g. irony, sarcasm,

idiosyncratic figurative language etc.), which can also serve as an effective tool for expressing

emotions (e.g. Labov, 1982). In the dictionaries proposed here, laugher and cry were the only

non-verbal emotionality markers that were audible and unambiguous, and their inclusion indeed

contributed to the accuracy of the dictionaries. Since non-verbal emotional cues are used more in

familiar, informal and interactive settings (e.g. Brown & Levinson, 1987; Biber, 1995; Clark,

1996; Hudson, 1996; Baugh, 2001; Coupland, 2001; Eckert & Rickford, 2001), the current

elicitation conditions may have minimized their effect and thus contributed to the dictionaries

sensitivity. The proposed dictionaries, conversely, may prove less sensitive in different

interaction settings.

Lastly, following speech variations, the sensitivity of corpus-based measures may vary

with the demographic characteristics of the speakers, including geographic background (e.g.

Coupland, 2001; Ash, 2003), race (e.g. Mufwene et al., 1998), Socio-economic status, (e.g.


Labov, 1972; Wolfram & Fasold, 1974), age and generation (e.g. Snow & Hoefnagel-Hoehle,

1978; Bloom, 1994; Eckert & Rickford, 2001; Tagliamonte & D'Arcy, 2007), and gender

(Coates, 1998; Cohen, in press).

Although concerns regarding the exhaustiveness and generalizability of corpus-based

linguistic measures are the first set of unknowns to study, the current dictionaries have already

demonstrated the importance of context, however narrow, in determining the sense in which an

emotion word is being used. The robust, highly statistically significant and valence-consistent

correlations between the corpus-based emotion dictionaries and measures of psychological

distress and dysphoria has the potential of rekindling the search for theory-driven assessment of

emotion dysregulation. Better estimation of emotional experience also holds a promise for a

more accurate evaluation of the relative contribution that self-involvement, social embeddedness,

referential activity and other constructs that rely on the context-independent nature of speech

particles (e.g. Spencer & Spencer, 1993; Bucci, 1997, 2001; Stirman & Pennebaker, 2001;

Campbell & Pennebaker, 2003; Pennebaker et al., 2003; Fertuck et al., 2004) play in emotion

dysregulation.


References

Abramson, L. Y., Metalsky, G. I., & Alloy, L. B. (1989). Hopelessness depression: A theory-

based subtype of depression. Psychological review, 96(2), 358-372.

Alloy, L. B., Abramson, L. Y., Whitehouse, W. G., Hogan, M. E., Tashman, N. A., & Steinberg,

D. L. et al. (1999). Depressogenic cognitive styles: Predictive validity, information

processing and personality characteristics, and developmental origins. Behaviour

Research & Therapy, 37(6), 503-531.

Alpers, G. W.,Winzelberg, A. J., Classen, C., Roberts, H., Dev, P., Koopman, C., and Taylor, C.

B. (2005). Evaluation of computerized text analysis in an Internet breast cancer support

group. Computers in Human Behavior, 21, 343–358

Anderson, N. H. (1968). Likableness ratings of 555 personality-trait words. Journal of

Personality & Social Psychology, 9(3), 272-279.

Ash, S. (2003). A national survey of North American dialects. American speech, suppl. 88, 57-

73.

Baldwin, M. W. (1992). Relationship schemas and the processing of social information.

Psychological Bulletin. 112, 461-484.


Bargh, J. A., & Tota, M. E. (1988). Context-dependent automatic processing in depression:

Accessibility of negative constructs with regard to self but not others. Journal of

Personality & Social Psychology, 54(6), 925-939.

Baugh, J. (2001). A dissection of style-shifting. In P. Eckert & J. R. Rickford (Eds.), Style and

sociolinguistic variation. (pp. 109-118). Cambridge, MA: Cambridge University Press.

Beck, A.T. (1976). Cognitive therapies and the emotional disorders. New York: International

University Press.

Beck, A. T. (2002). Cognitive models of depression. In R. L. Leahy, & E. T. Dowd (Eds.),

Clinical advances in cognitive psychotherapy: Theory and application (pp. 29-61). New

York, NY, US: Springer Publishing Co.

Beck, A. T., Epstein, N., & Harrison, R. (1983). Cognitions, attitudes and personality dimensions

in depression. British Journal of Cognitive Psychotherapy, 1(1), 1-16.

Beck, A. T., Steer, R. A., & Garbin, M. G. (1988). Psychometric properties of the beck

depression inventory: Twenty-five years of evaluation. Clinical psychology review, 8(1),

77-100.

Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for

measuring depression. Archives of General Psychiatry, 4, 561-571.

Bell, A. (1984). Language style as audience design. Language in society, 13(2), 145-204.

Biber, D. (1995). Dimensions of register variation: A cross-linguistic comparison. Cambridge:

Cambridge University Press.

Biber, D., Conrad, S., & Reppen, R. (1998). Corpus Linguistics: Investigating language

structure and use. Cambridge, MA: Cambridge University Press.

Blaney, P. H. (1986). Affect and memory: A review. Psychological bulletin, 99, 2, 229-246.


Bloom, P. (1994). Overview: Controversies in language acquisition. In P. Bloom (Ed.),

Language acquisition: Core readings (pp. 5-15). Cambridge, MA: MIT Press.

Boals, A. & Klein, K. (2005). Word Use in Emotional Narratives about Failed Romantic

Relationships and Subsequent Mental Health. Journal of Language and Social

Psychology, 24, 252-268

Bohanek, J. G., Fivush, R., & Walker, E. (2005). Memories of positive and negative emotional

events. Applied Cognitive Psychology, 19(1), 51-66.

Brown, P., & Levinson, S. C. (1987). Politeness: Some universals in language usage.

Cambridge, MA: Cambridge University Press.

Bucci, W. (1997). Psychoanalysis and cognitive science: A multiple code theory. New York,

NY, US: Guilford Press.

Bucci, W. (2001). Pathways of emotional communication. Psychoanalytic Inquiry, 21, 1, 40-70.

Bucci, W., & Freedman, N. (1981). The language of depression. Bulletin of the Menninger

Clinic, 45, 4, 334-358.

Cacioppo, J. T., von Hippel, W., & Ernst, J. M. (1997). Mapping cognitive structures and

processes through verbal content: The thought-listing technique. Journal of Consulting &

Clinical Psychology, 65, 6, 928-940.

Campbell, R. S., & Pennebaker, J. W. (2003). The secret life of pronouns: Flexibility in writing

style and physical health. Psychological Science, 14, 1, 60-65.

Chien, A. J., & Dunner, D. L. (1996). The tridimensional personality questionnaire in

depression: State versus trait issues. Journal of Psychiatric Research, 30, 21-27.

Clark, D. A., Beck, A. T., & Alford, B. A. (1999). Scientific foundations of cognitive theory and

therapy of depression. New York, NY, US: John Wiley & Sons, Inc.


Clark, H. H. (1996). Using language. Cambridge, MA: Cambridge University Press.

Clark, D. A. (1988). The validity of measures of cognition: A review of the literature. Cognitive

Therapy and Research, 12, 1-20.

Coates, J. (ed.) (1998) Language and Gender: A Reader. Blackwell: Oxford, Malden Masc.

Cohen, S. J. (In press). Gender differences in speech temporal patterns detected using lagged co-

occurrence text-analysis of personal narratives. Journal of Psycholinguistics Research

Collins, N. L., & Feeney, B. C. (2004). Working models of attachment shape perceptions of

social support: Evidence from experimental and observational studies. Journal of

Personality & Social Psychology, 87, 3, 363-383.

Coupland, N. (2001). Language, situation, and the relational self: Theorizing dialect-style in

sociolinguistics. In P. Eckert & J. R. Rickford (Eds.), Style and sociolinguistic variation

(pp. 185-210). Cambridge, MA: Cambridge University Press.

Cyr, J. J., McKenna-Foley, J. M., & Peacock, E. (1985). Factor structure of the SCL-90-R: Is

there one? Journal of personality assessment, 49, 6, 571-578.

Dahl, H., & Stengel, B. (1978). A classification of emotion words: A modification and partial

test of de rivera's decision theory of emotions. Psychoanalysis & Contemporary Thought,

1, 2, 269-312.

Davison, G. C., Vogel, R. S., & Coffman, S. G. (1997). Think-aloud approaches to cognitive

assessment and the articulated thoughts in simulated situations paradigm. Journal of

Consulting & Clinical Psychology, 65, 6, 950-958.


Demorest, A., Crits-Christoph, P., Hatch, M., & Luborsky, L. (1999). A comparison of

interpersonal scripts in clinically depressed versus nondepressed individuals. Journal of

Research in Personality, 33, 3, 265-280.

Derogatis, L. R., Rickels, K., & Rock, A. F. (1976). The SCL-90 and the MMPI: A step in the

validation of a new self-report scale. British Journal of Psychiatry, 128, 280-289.

Derogatis, L. R., & Savitz, K. L. (2000). The SCL-90-R and Brief Symptom Inventory (BSI) in

primary care. In M. E. Maruish (Ed.), Handbook of psychological assessment in primary

care settings (pp. 297-334). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Dobson, K. S., & Breiter, H. J. (1983). Cognitive assessment of depression: Reliability and

validity of three measures. Journal of Abnormal Psychology, 92, 107–109.

Eckert, P., & Rickford, J. R. (Eds.). (2001). Style and sociolinguistic variation. Cambridge, MA:

Cambridge University Press.

Endler, N. S., Denisoff, E., & Rutherford, A. (1998). Anxiety and depression: Evidence for the

differentiation of commonly co-occurring constructs. Journal of Psychopathology &

Behavioral Assessment, 20, 2, 149-171.

Fertuck, E. A., Bucci, W., Blatt, S. J., & Ford, R. Q. (2004). Verbal representation and

therapeutic change in anaclitic and introjective inpatients. Psychotherapy: Theory,

Research, Practice, Training, 41, 1, 13-25.

Fresco, D. M., Heimberg, R. G., Abramowitz, A., & Bertram, T. L. (2006). The effect of a

negative mood priming challenge on dysfunctional attitudes, explanatory style, and

explanatory flexibility. British Journal of Clinical Psychology, 45,2, 167-183.


Giles, H. (2001). Sociopsychological parameters of style shifting. In P. Eckert & J. R. Rickford

(Eds.), Style and sociolinguistic variation (pp. 211-219). Cambridge, MA: Cambridge

University Press.

Gill, A. (2003). Personality and Language: The Projection and Perception of Personality in

Computer-Mediated Communication. Ph.D. thesis, University of Edinburgh.

Glass, C. R., & Arnoff, D. B. (1997). Questionnaire methods of cognitive self-statement

assessment. Journal of Consulting and Clinical Psychology, 65, 911-927.

Gotlib, I. H. (1984). Depression and general psychopathology in university students. Journal of

Abnormal Psychology, 93, 1, 19-30.

Gotlib, I. H., Lewinson, P. M., & Seeley, J. R. (1995). Symptoms versus a diagnosis of

depression: Differences in psychosocial functioning. Journal of Consulting and Clinical


Gottschalk, L. (1997). The unobstrusive measure of psychological traits and states. In C. W.

Roberts (Ed.), Text analysis for the social sciences: Methods for drawing statistical

inferences from texts and transcripts (pp. 117-129). Mahwah, NJ: Lawrence Erlbaum.

Gottschalk, L. A., & Bechtel, R. J. (1982). The measurement of anxiety through the computer

analysis of verbal samples. Comprehensive Psychiatry, 23, 4, 364-369.

Gottschalk, L. A., & Gleser, G. C. (1969). The measurement of psychological states through the

content analysis of verbal behavior. Oxford, England: U. California Press.

Haaga, D. A., Dyck, M. J., & Ernst, D. (1991). Empirical status of cognitive theory of

depression. Psychological Bulletin, 110, 2, 215-236.

Haeffel, G. J., Abramson, L. Y., Voelz, Z. R., Metalsky, G. I., Halberstadt, L., Dykman, B. M., et

al. (2005). Negative cognitive styles, dysfunctional attitudes, and the remitted depression


paradigm: A search for the elusive cognitive vulnerability to depression factor among

remitted depressives. Emotion, 5, 343 – 348.

Hartman-Hall, H. M. & Haaga, D. A. F. (1999) Content Analysis of Cognitive Bias:

Development of a Standardized Measure. Journal of Rational Emotive and Cognitive

Behavior Therapy, 17, 2, 105-114

Hermans, H. J. M. (1995). From assessment to change: The personal meaning of clinical

problems in the context of the self-narrative. In R. A. Neimeyer, & M. J. Mahoney (Eds.),

Constructivism in psychotherapy. (pp. 247-272). Washington, DC, US: American

Psychological Association;

Hill, A. B., Kemp-Wheeler, S. M., & Jones, S. A. (1986). What does the beck depression

inventory measure in students? Personality & Individual Differences, 7, 1, 39-47.

Horowitz, L. M. (1979). On the cognitive structure of interpersonal problems treated in

psychotherapy. Journal of Consulting & Clinical Psychology, 47, 1, 5-15.

Hudson, R. A. (1996). Sociolinguistics. New York, NY: Cambridge University Press.

Hurlburt, R. T. (1980). Validation and correlation of thought sampling with retrospective

measures. Cognitive Therapy & Research, 4, 2, 235-238.

Hurlburt, R. T. (1997). Randomly sampling thinking in the natural environment. Journal of

Consulting and Clinical Psychology, 65, 941–949.

Ingram, R. E. (1984). Toward an information-processing analysis of depression. Cognitive

Therapy & Research, 8, 5, 443-477.

Ingram, R. E , Kendall, P. C , Siegle, G., Guarino, J., & McLaughlin, S. (1995). Psychometric

properties of the Positive Automatic Thoughts Questionnaire. Psychological Assessment,

7, 495-507.


Isen, A. M. (1984). Toward understanding the role of affect in cognition. In R. S. Wyer & T. K.

Srull (Eds.), Handbook of social cognition. (pp. 179-236). Hillsdale, NJ: Lawrence

Erlbaum Associates.

Kelly, E. F., & Stone, P. J. (1975). Computer recognition of English word senses. Amsterdam,

Netherlands: North Holland Publishing Company.

Klein, M. H., Mathieu-Coughlan, P., & Kiesler, D. J. (1986). The experiencing scales. In L. S.

Greenberg, & W. M. Pinsof (Eds.), The psychotherapeutic process: A research

handbook. Guilford clinical psychology and psychotherapy series (pp. 21-71). New York,


Kuiper, N. A., & Higgins, E. T. (1985). Social cognition and depression: A general integrative

perspective. Social Cognition. Special issue: Depression, 3, 1, 1-15.

Kuiper, N. A., Olinger, L. J., & MacDonald, M. R. (1988). Vulnerability and episodic cognitions

in a self-worth contingency model of depression. In L. B. Alloy (Ed.), Cognitive

processes in depression. (pp. 289-309). New York, NY, US: Guilford Press.

Labov, W. (1972). Sociolinguistic patterns. Philadelphia: University of Pennsylvania Press.

Labov, W. (1982). Speech Actions and Reactions in Personal Narrative. In Analyzing Discourse:

Text and Talk, D. Tannen (ed.). Washington D.C.: Georgetown University Press.

Lee, F., & Peterson, C. (1997). Content analysis of archival data. Journal of Consulting &

Clinical Psychology, 65, 6, 959-969.

Lindeman, M., & Verkasalo, M. (1995). Personality, situation, and positive – negative

asymmetry in socially desirable responding. European Journal of Personality, 9, 125 –

134.


Luborsky, L., & Crits-Christoph, P. (1990). Understanding transference: The core conflictual

relationship theme method. New York, NY, US: Basic Books, Inc.

Mairesse, F., Walker, M., Mehl, M. & Moore, R. (2007). Using linguistic cues for the automatic

recognition of personality in conversation and text. Journal of Artificial Intelligence

Research, 30, 457–500.

Mangan, G. L., & Hookway, D. (1988). Perception and recall of aversive material as a function

of personality type. Personality & Individual Differences, 9, 2, 289-295.

Marchitelli, L. (1983). The language of the emotions: Couples in conversation. Unpublished

M.A., University of California at Berkeley, Berkeley.

Marchitelli, L. & Levenson, R.W. (1992). When couples converse: the language and physiology

of emotion. Paper presented at the Society for Psychophysiological Research, San Diego,

CA.

Markus, H. (1977). Self-schemata and processing information about the self. Journal of

Personality & Social Psychology, 35, 2, 63-78.

Martin, M. (1985). Neuroticism as predisposition toward depression: A cognitive mechanism.

Personality & Individual Differences, 6, 3, 353-365.

Martin, M., Ward, J. C., & Clark, D. M. (1983). Neuroticism and the recall of positive and

negative personality information. Behaviour Research & Therapy, 21, 5, 495-503.

Mathews, A., & MacLeod, C. (1994). Cognitive approaches to emotion and emotional disorders.

Annual Review of Psychology, 45, 25-50.

Mergenthaler, E., & Bucci, W. (1999). Linking verbal and non-verbal representations: Computer

analysis of referential activity. British Journal of Medical Psychology, 72, 3, 339-354.


Mergenthaler, E., & Stinson, C. H. (1992). Psychotherapy transcription standards. Psychotherapy

Research, 2, 2, 125-142.

Mufwene, S.S., Rickford, J.R., Bailey, G., & Baugh, J. (Eds.). (1998). African-American

English: Structure, history and use. London: Routledge.

Nasby, W., & Kihlstrom, J. F. (1986). Cognitive assessment of personality and psychopathology.

In R. E. Ingram (Ed.), Information processing approaches to clinical psychology. (pp.

217-239). San Diego, CA, US: Academic Press, Inc.

Nosek, B. A. (2005). Moderators of the relationship between implicit and explicit evaluation.

Journal of Experimental Psychology, 134, 565- 584.

Oberlander, J. and Gill, A. (2004). Language generation and personality: two dimensions, two

stages, two hemispheres? In Papers from the AAAI Spring Symposium on Architectures

for Modeling Emotion: Cross-Disciplinary Foundations, 104– 111.

Oliver, J. M., & Baumgart, E. P. (1985). The Dysfunctional Attitude Scale: psychometric

properties and relation to depression in an unselected adult population. Cognitive Therapy

and Research, 11, 25–40.

Olkin, I., & Finn, J. D. (1990). Testing correlated correlations. Psychological Bulletin, 108, 2,

330-333.

Olkin, I., & Finn, J. D. (1995). Correlations redux. Psychological Bulletin, 118, 1, 155-164.

Oxman, T. E., Rosenberg, S. D., Schnurr, P. P., & Tucker, G. J. (1988). The language of altered

states. Journal of Nervous & Mental Disease, 176, 7, 401-408.

Paulhus, D. L., & John, O. P. (1998). Egoistic and moralistic biases in self-perception: The

interplay of self-deceptive styles with basic traits and motives. Journal of Personality, 66,

1025 – 1060.


Pennebaker, J. W., & Francis, M. E. (1996). Cognitive, emotional, and language processes in

disclosure. Cognition & Emotion, 10, 6, 601-626.

Pennebaker, J. W., & Francis, M. E. (1999). Linguistic inquiry and word count: LIWC [Software

Manual]. Mahwah, NJ: Erlbaum publishers.

Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic inquiry and word count:

LIWC 2001 [Software Manual]. Mahwah, NJ: Erlbaum publishers.

Pennebaker, J. W., & Graybeal, A. (2001). Patterns of natural language use: Disclosure,

personality, and social integration. Current Directions in Psychological Science, 10, 3,

90-93.

Pennebaker, J. W., & King, L. A. (1999). Linguistic styles: Language use as an individual

difference. Journal of Personality & Social Psychology, 77, 6, 1296-1312.

Pennebaker, J. W., & Lay, T. C. (2002). Language use and personality during crises: Analyses of

mayor rudolph giuliani's press conferences. Journal of Research in Personality, 36, 3,

271-282.

Pennebaker, J. W., & Stone, L. D. (2003). Words of wisdom: Language use over the life span.

Journal of Personality & Social Psychology, 85, 2, 291-301.

Pennebaker, J. W., Mayne, T. J., & Francis, M. E. (1997). Linguistic predictors of adaptive

bereavement. Journal of Personality & Social Psychology, 72, 4, 863-871.

Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003). Psychological aspects of natural

language use: Our words, our selves. Annual Review of Psychology, 54, 547-577.

Pennebaker, J. W., Zech, E., & Rimé, B. (2001). Disclosing and sharing emotion: Psychological,

social, and health consequences. In M. S. Stroebe, & R. O. Hansson (Eds.), Handbook of


bereavement research: Consequences, coping, and care. (pp. 517-543). Washington, DC,

US: American Psychological Association

Persons, J. B., & Miranda, J. (1992). Cognitive theories of vulnerability to depression:

Reconciling negative evidence. Cognitive Therapy and Research, 16, 485–502.

Peterson, C. (1992). Explanatory style. In C. P. Smith (Eds.), Handbook of thematic content

analysis (pp. 376-382). New York: Cambridge.

Polaino, A., & Senra, C. (1991). Measurement of depression: Comparison between

self-reports and clinical assessments of depressed outpatients. Journal of Psychopathology and

Behavioral Assessment, 13, 313-324.

Rector, N. A., Segal, Z. V., & Gemar, M. (1998). Schema research in depression: A canadian

perspective. Canadian Journal of Behavioural Science.Special Canadian perspectives on

research in depression, 30, 4, 213-224.

Ritterband, L. M., & Spielberger, C. D. (1996). Construct validity of the Beck Depression

Inventory as a measure of state and trait depression in nonclinical populations.

Depression and Stress, 2, 123-145.

Rosenberg, S. D., Blatt, S. J., Oxman, T. E., & McHugo, G. J. (1994). Assessment of object

relatedness through a lexical content analysis of the TAT. Journal of personality

assessment, 63, 2, 345-362.

Rosenberg, S. D., Schnurr, P. P., & Oxman, T. E. (1990). Content analysis: A comparison of

manual and computerized systems. Journal of personality assessment, 54, 1, 298-310.

Rude, S., Covich, J., Jarrold, W., Hedlund, S., & Zentner, M. (2001). Detecting depressive

schemata in vulnerable individuals: questionnaires versus laboratory tasks. Cognitive

Therapy and Research, 25, 103–116.


Rude, S. S., Valdez, C. R., Odom, S., & Ebrahimi, A. (2003). Negative cognitive biases predict

subsequent depression. Cognitive Therapy & Research, 27, 4, 415-429.

Ruiz-Caballero, J. A., & Bermúdez, J. (1995). Neuroticism, mood, and retrieval of negative

personal memories. Journal of General Psychology, 122, 1, 29-35.

Rusting, C. L. (1998). Personality, mood, and cognitive processing of emotional information:

Three conceptual frameworks. Psychological bulletin, 124, 2, 165-196.

Rusting, C. L. (1999). Interactive effects of personality and mood on emotion-congruent memory

and judgment. Journal of Personality & Social Psychology, 77, 5, 1073-1086.

Scher, C. D., Segal, Z. V., & Ingram, R. E. (2004). Beck's theory of depression: Origins,

empirical status, and future directions for cognitive vulnerability. In R. L. Leahy (Ed.),

Contemporary cognitive therapy: Theory, research, and practice. (pp. 27-44). New York,


Segal, H. G., Westen, D., Lohr, N. E., & Silk, K. R. (1993). Clinical assessment of object

relations and social cognition using stories told to the picture arrangement subtest of the

WAIS--R. Journal of personality assessment, 61, 1, 58-80.

Segal, Z. V., & Ingram, R. E. (1994). Mood priming and construct activation in tests of cognitive

vulnerability to unipolar depression. Clinical psychology review, 14, 7, 663-695.

Shaw, B. F., & Dobson, K. S. (1981). The cognitive assessment of depression. In T. Merluzzi, C.

Glass, & M. Genest (Eds.), Cognitive assessment. New York: Guilford Press.

Shedler, J., Mayman, M., & Manis, M. (1993). The illusion of mental health. American

Psychologist, 48, 1117 – 1131.


Shepard, C. A., Giles, H., & Le Poire, B. A. (2001). Communication accomodation theory. In P.

W. Robinson & H. Giles (Eds.), The new handbook of language and social psychology

(pp. 33-56). New York, NY: John Wiley & Sons.

Simmons, R. A., & Gordon, P. C., Chambless, D. L. (2005). Pronouns in marital interaction:

What do ‘you’ and ‘I’ say about marital health. Psychological Journal, 16, 932-936.

Sinclair, J. (1991). Corpus, Concordance, Collocation. Oxford, UK: Oxford University Press.

Snow, C. E., & Hoefnagel-Hohle, M. (1978). The critical period for language acquisition:

Evidence from second language learning. Child Development, 49, 4, 1114-1128.

Spencer, L. M., & Spencer, S. M. (1993). Competence at work: Models for superior

performance. New York:Wiley

Stirman, S. W., & Pennebaker, J. W. (2001). Word use in the poetry of suicidal and nonsuicidal

poets. Psychosomatic medicine, 63, 4, 517-522.

Stone, A. A., Turkkan, J. S., Bachrach, C. A., Jobe, J. B., Kurtzman, H. S., & Cain, V. S. (Eds.).

(2000). The science of self-report: Implications for research and practice. Mahwah, NJ:

Lawrence Erlbaum Associates.

Stone, L. D., & Pennebaker, J. W. (2002). Trauma in real time: Talking and avoiding online

conversations about the death of Princess Diana. Basic and Applied Social Psychology,

24, 3, 173-183

Stone, P. J. (1997). Thematic text analysis: New agendas for analyzing text content. In C. W.

Roberts (Ed.), Text analysis for the social sciences: Methods for drawing statistical

inferences from texts and transcripts. (pp. 35-53). Mahwah, NJ: Lawrence Erlbaum

Associates.


Stone, P. J., Dunphy, D. C., & Smith, M. S. (1966). The general inquirer: A computer approach

to content analysis. Oxford, England: M.I.T. Press.

Sweeney, K., & Whissell, C. (1984). A dictionary of affect in language: I. establishment and

preliminary validation. Perceptual & Motor Skills, 59, 3, 695-698.

Tagliamonte, S. A., & D'Arcy, A. (2007). Frequency and variation in the community grammar:

Tracking a new change through the generations. Language Variation and Change, 19, 2,

199-217.

Teasdale, J. D. (1999). Emotional processing, three modes of mind and the prevention of relapse

in depression. Behaviour Research & Therapy. Special Issue: Cognitive Behaviour

Therapy: Evolution and prospects.A festschrift in honour of Dr S.Rachman, Editor of

Behavior Research and Therapy, 37(Suppl 1), S53-S77.

Viney, L. L. (1983). The assessment of psychological states through content analysis of verbal

communications. Psychological bulletin, 94, 3, 542-563.

Watkins, J., & Rush, A. J. (1983). The Cognitive Response Test. Cognitive Therapy and

Research, 7, 425-436.

Watson, D., & Clark, L. A. (1984). Negative affectivity: The disposition to experience aversive

emotional states. Psychological bulletin, 96, 3, 465-490.

Weintraub, W. (1989). Verbal behavior in everyday life. New York, NY, US: Springer

Publishing Co;

Wenzlaff, E. M., & Wegner, D. M. (2000). Thought suppression. Annual Review of Psychology,

51, 59–91.


Whissell, C. M., Fournier, M., Pelland, R., & Weir, D. (1986). A dictionary of affect in

language: IV. reliability, validity, and applications. Perceptual & Motor Skills, 62, 3,

875-888.

Williams, K. M., Paulhus, D. L., & Nathanson, C. (2003, May). Personality Correlates of

Emotional Reactions to 9/11. Poster presented at the 83rd Annual Convention of the

Western Psychological Association, Vancouver, BC, Canada.

Winters, N., Myers, K., & Proud, L. (2002). Ten-year review of rating scales, III: Assessing

suicidality, cognitive style, and self-esteem. Journal of American Academy of Child &

Adolescent Psychiatry, 41, 1150 – 1181.

Winters, K. C., & Neale, J. M. (1985). Mania and low self-esteem. Journal of Abnormal


Wolfram, W., & Fasold, R. W. (1974). The study of social dialects in American English.

Englewood Cliffs, NJ: Prentice-Hall.

Zuckerman, M., Lubin, B., & Robins, S. (1965). Validation of the multiple affect adjective check

list in clinical situations. Journal of consulting psychology, 29, 6, 594.


Table 1

Pearson correlations between the 3 computerized text-analysis systems and various measures of mental health

Dictionary 1 2 3 4 5 6 7

N=483

1. General Severity Index (GSI) --

2. Beck Depression Inventory (BDI)

.76** --

3. Corpus-based negative emotion .41** .41** --

4. Corpus-based positive emotion -.27** -.26** -.17** --

5. General Inquirer negative emotion

.08 .05 .28** -.06 --

6. General Inquirer positive emotion

-.04 -.01 .05 .24** -.04 --

7. LIWC negative emotion .04 .11* .46** -.05 .50**

.00 --

8. LIWC positive emotion .02 .01 .06 .05 -.03 .30**

.01

Note: *) p<0.05; **) p<0.01

41

Table 2

95% Confidence intervals for the differences between the correlations of the text-analytical

scales and the outcome measures, as calculated by Olkin & Finn (1990,1995) procedure.

Correlation difference Correlations

with GSI

Correlations

with BDI

N=483

Corpus-based negative emotion -

General Inquirer negative emotion

0.331±0.005 0.362±0.005

Corpus-based negative emotion – LIWC

negative emotion

0.369±0.004 0.303±0.004

Corpus-based positive emotion -

General Inquirer positive emotion

-0.228±0.006 -0.251±0.006

Corpus-based positive emotion – LIWC

positive emotion

-0.293±0.007 -0.272±0.007

Documents

Corpus-based dictionary of emotional instability for use …ymarton/pub/sentiment/shukicohen... · Web viewAlmost invariably, the word pretty was used as an intensifier (e.g. “pretty