Language Sciences 34 (2012) 702–714

Contents lists available at SciVerse ScienceDirect

Language Sciences

journal homepage: www.elsevier .com/ locate/ langsci

Circularity effects in corpus studies – why annotations sometimesgo round in circles

Manfred Consten a,⇑, Annegret Loll b

a Institute of Germanic Linguistics, University of Jena, D-07737 Jena, Germanyb Cologne, Germany

a r t i c l e i n f o

Article history:Available online 28 May 2012

Keywords:AnaphorAnnotationCircularityCorpus studyDefinitenessReference

0388-0001/$ - see front matter � 2012 Elsevier Ltdhttp://dx.doi.org/10.1016/j.langsci.2012.04.010

⇑ Corresponding author.E-mail addresses: manfred.consten@uni-jena.de

1 Raw data are data which have not undergone tranonly in original audio/video tapes; transcriptions ofauditive mode into a written mode. However, raw dTranscriptions must not cause a loss of information recan be considered as raw data input for further annodiscussed in Section 4.2.

a b s t r a c t

Linguistic corpus research mainly deals with annotated data rather than raw data. Thiscontribution investigates the status of annotated corpus data in empirical linguistics.

We argue that annotators should be regarded as co-producers of data; annotationsdepend on certain theoretical categories, hence they are theory-laden. Annotation catego-ries differ with respect to different (structural and functional) levels of description and dif-ferent degrees of canonisation, e.g. annotating a corpus item as a noun at a structural levelis a highly canonised decision in most cases whereas the allocation of a cognitive-func-tional annotation category like expression with identifyable referent is subject to specific the-ories that often lack established definitions. As a minimal requirement, annotated datahave to allow the reconstruction of the original raw data and annotations should be con-strained by guidelines in order to avoid that the annotator’s decisions are arbitrary.

Annotation problems resulting from the close relation between annotation categoriesand their theoretical prerequisites are exemplified using a newspaper corpus study anda study on a second-language acquisition corpus, both studies dealing with anaphora asa discourse-functional phenomenon.

It is shown that the problems discussed have their origins in two circles: the first oneresults from the interplay of deductive and inductive procedures that causes an impactof theory on annotation; the second circle originates from the relations between languagestructures and their discourse functions, the latter failing to be observable independentlyfrom the structural features of the utterance.

� 2012 Elsevier Ltd. All rights reserved.

1. Introduction

In empirical linguistics, corpus studies are a widespread method employed not only for the exploration of phenomena,but also for the testing of hypotheses. The annotation of raw data (i.e. the assigning of theoretically acquired categoriesto raw data segments)1 is an essential step in the process of conducting corpus studies. This contribution will show by examplesome problems arising in the course of this process.

. All rights reserved.

(M. Consten), lolla@smail.uni-koeln.de (A. Loll).sformations or manipulations by researchers. Strictly speaking, oral speech is represented as real raw dataoral speech are no raw data in a strict sense since they are data which have been transformed from anata is a relative term. It depends on the object of investigation which data are considered as raw data.levant for the investigation, and they must not add any information. Only on that condition, transcriptionstation steps at which additional information will be generated. These requirements are met in the study

M. Consten, A. Loll / Language Sciences 34 (2012) 702–714 703

The article is organised as follows: in Section 1 we will postulate the interaction of both induction and deduction inempirical research. Section 2 characterises properties of data: it is shown that empirical research mainly deals with anno-tated data rather than raw data, a process in which annotators should be regarded as quasi co-producers of data.

Annotations are located at various levels that differ with respect to the theoretical complexity and different degrees ofcanonisation of the annotation categories and of the theories which these categories are based on. Annotated data haveto fulfil the postulate of reconstructability (i.e., raw data must be reconstructable from annotated data) in order to guaranteetransparent annotations.

In Section 4, two corpus studies are introduced: both show methodical problems, mainly with respect to the annotation ofsome doubtful data.

These problems are discussed more closely in Section 5.1 and revealed as problems of circularity. Here, two circles arediscovered: the first circle has its origin in the interplay of deductive and inductive procedures (especially the impact of the-ory on annotation); the second circle has its origin in the characteristic of functional theories (i.e. theories reflecting the lan-guage function in human communication) which aim to specify relations between certain structures and certain functions:in some cases, functional categories (in the sense of categories of natural language use) cannot be specified and annotatedindependently from the structural features of the utterance.

In Section 5.2, characteristics of functional categories and their annotation are highlighted. Strategies based on plausibil-ity often turn out to be the only way to avoid arbitrary annotations, and provide for reconstructability as claimed inSection 3.

Sadly, these considerations fail to resolve the circles described in Section 5.1.

2. The place of empirical data in linguistics: induction and deduction

In the recent history of linguistics, there has been one important topic in the debate between the opposing communitiesof Generative Grammarians and corpus linguists: the question of whether introspective data as they are preferred by gen-erativists are empirical data at all (Kertész and Rákosi, 2008, p. 27).

We do not intend to pursue this topic, which has already been discussed extensively (cf. Lehmann, 2004; Geeraerts, 2006,as well as the contributions in Sternefeld, 2007, and Stefanowitsch and Gries, 2007).

If (pure) empirical data is defined as ‘‘data based on pure observation without any theoretical impact’’, it will be of greaterinterest whether purely empirical data – apart from raw data – actually exist in linguistics and what problems arise in empir-ical linguistics from a questionable state of data. Certain problems of data annotation in corpus studies do not seem to beaccounted for by the exponents of purely empirical data analyses.

Geeraerts (2006) may serve as an example of such a position. At first, he reports the viewpoint of those who claim thatpurely empirical research is impossible:

‘‘Isn’t any attempt to reduce the role of introspection and intuition in linguistic research contrary to the spirit of CognitiveLinguistics, which stresses the semantic aspects of the language – and the meaning of linguistic expressions is the least tan-gible of linguistic phenomena. Because meanings do not present themselves directly in the corpus data, will introspectionnot always be used in any cognitive analysis of language?’’ (Geeraerts, 2006, p. 42)

It has to be added that the present contribution does not only deal with meaning but also with referential structures,which contain mental objects, exceed the levels of language structure and, thus, are even less tangible than semanticstructures.

Geeraerts, then, turns against the sceptic position and claims that introspective and intuitive elements are part of theoryformation only, and not part of data work.

‘‘Empirical research does not imply that intuition and interpretation have no role to play in linguistic research: rather, itimplies that interpretation is but one step in the empirical cycle of successful research.’’ (Geeraerts, 2006, p. 45)

Probably this view is due to the common assumption that empirical work consists in either inductive or deductive use ofdata: ‘‘In the context of empirical scientific research, a datum serves either as the basis for the inductive construction of ahypothesis or as the test for a theorem arrived at deductively.’’ (Lehmann, 2004, pp. 180–181)

With this notion, however, it remains unclear how theorems which are tested deductively arise. Some sort of induction isrequired anyway – at least induction is provided by utterances linguists happen to pick up in every day life.

Therefore, we prefer a different notion of empirical work, assuming that data are used at inductive as well as deductivestages in a cyclic way.2

A sketch of this notion is shown as Fig. 1.An empirical cycle consists of a process of the following-up of inductive and deductive steps3:

2 For a differentiation of cycle versus circle see Rákosi (this issue), and Kertesz/Rakosi (2009, pp. 718–719). Following Rescher (1976), they define a cyclicprocess as a fruitful process that will lead back to a starting point (e.g. a theory) but at a higher level of insight (here: the initial theory is now confirmed andpossibly refined by data). A circle, on the other hand, is a kind of fallacious argumentation. We will discuss circular argumentation in 4.

3 Similar to the debate about the status of introspective data, the question of whether a strict deductive or a strict inductive approach is adequate forlinguistics was the subject of a debate between generative and corpus linguists: Chomsky generally rejects inductive work whereas it is consideredindispensable by corpus linguists (for a closer discussion see Kertész/Rákosi, 2008, pp. 26–27). This debate, however, seems to be quite ridiculous, since alllinguists are capable of at least one language that will always inductively influence their theory formation.

Inductive

Deductive

theory-

formation

hypothesis-

testing

Theory

Hypothesis

Data

Fig. 1. Empirical cycle.

704 M. Consten, A. Loll / Language Sciences 34 (2012) 702–714

In a first, inductive step, a theory is generated on the basis of conspicuities in the data.This theory is tested deductively by the data in a second step; more precisely, hypotheses are developed on the basis of

the theory and are tested using the data again.These hypotheses are then accepted or rejected. As a result, refined hypotheses are developed and tested again.4

There is one important consequence resulting from this cyclical structure: working empirically, data are needed both intheory formation and in theory-derived hypotheses testing.

In both processes, the same data can be employed; namely in those cases where the first observation has already beenmade with the same annotated corpus used for the further stage of hypotheses testing as well.

At least, both the data used for theory formation and the data used for hypotheses testing often have their origins in thesame raw data: conspicuities in raw data result in the formation of a theory; then an annotation of the same raw data isconducted, or completed, or refined for the purpose of testing this theory.

With merely inductive work, researchers would be able to gain theories from data, but these theories wouldhave a vague and arbitrary status if they lack a deductive stage in which they are confronted with the data onceagain.

3. Manageable data in empirical linguistics: properties and requirements

In Section 1 it was pointed out that data are used at both deductive and inductive stages of empirical linguisticresearch, thus data are part of a cycle during which – in most cases – annotated data have to be derived fromraw data. This section deals with the question of how to get from raw data to data amenable to empiricalinvestigation.

According to Haspelmath (2009, p. 158), the raw data the studies presented in Section 3 are based on are categorised aspure linguistic (in terms of verbal) data – as opposed to ‘‘meta linguistic’’ data, such as grammaticality judgement question-naires, and non-linguistic data, such as acting out and gestures. Within the category of ‘‘pure linguistic’’ data, our data arewritten corpus data (see Section 4.1, a category not accounted for in Haspelmath, 2009) or ‘‘stimulated narration’’ (see Sec-tion 4.2, as opposed to elicited data).

In oral corpora (as used in Section 4.2), raw data are available as video or audio tapes that have to be transformed into awritten representation for the purposes of investigation (where a phonetical transcription is on a lower level than an ortho-graphical transcription).

Lehmann (2004, p. 206) refers to this process as a ‘‘change of mode. This is the first step towards the reification (or hypos-tatization) of the epistemic object of linguistics.’’ When investigating a written corpus (see Section 4.1), the phonetical tran-scription stage is skipped and the investigation starts immediately at the orthographical level.

4 There are slight differences to Geeraert’s description of the empirical cycle, cf.: ‘‘Empirical research necessarily combines inductive and deductivereasoning: On the one hand, you work in a bottom-up way from data to hypothesis, but on the other hand, those hypotheses will also be derived top-down fromthe theoretical perspective you adopt in thinking about your data’’ (Geeraerts, 2006, p. 24). ‘‘Empirical research involves an empirical cycle in which severalrounds of data gathering, testing of hypotheses, and interpretation of the results follow each other. [. . .] we started with an anecdotal observation, tried tointerpret it, turned the possible interpretations into alternative hypotheses, tested the hypotheses, and so arrived at a new, more secure and better establishedobservation. But this observation was only the starting-point for a new attempt at interpretation which led to a new hypothesis [. . .], which led to new researchefforts’’ (Geeraerts, 2006, pp. 24–25). Firstly, Geeraerts involves an act of interpretation at several steps. Since interpretation is a term which needs a closerdiscussion, we avoid it at this general stage of explanation. Secondly, he skips the step from rejecting an old hypothesis to creating a new, refined hypothesis.Instead, the process immediately leads to a refined observation. These differences, however, do not conform to the overall idea of a necessary follow-up ofdeductive and inductive steps in empirical research.

Raw data

Structural levels:

Functional level

Grammatical representation

Morpho-lexical level

Orthographical transcription

Phonetical transcription

C A N O N I S A T I O N

I N T E R P R E T A T I O N

Fig. 2. Representation levels of annotated data.

M. Consten, A. Loll / Language Sciences 34 (2012) 702–714 705

The next level of representation is the morpho-lexical level, at which the morphemes which the utterance contains arenoted. This level is followed by the level of grammatical representation at which grammatical categories and relations aretagged.5

We subsume these levels as ‘structural levels’. Unlike Lehmann (2004),6 we claim a functional level beyond the structurallevels. At the functional level, categories concerning the referential structure of a text or discourse are located; this referentialstructure serves as the base for the textual coherence and information structure and as a link to the conceptual relations whichspeakers address (cf. Section 5.1).7 An overview of all levels is given in Fig. 2.

All levels of representation function as levels of annotation as well. The latter follow up each other in the process of anno-tating; at least this holds with regard to the higher levels; e.g., it is not necessary to conduct a phonetical transcription for thepurpose of a grammatical analysis whereas annotating lexical categories is a prerequisite for annotating phrasal structures.For the functional level, however, rudimental structural–grammatical annotations (such as simple phrasal structures) arerequired as a base.

It is important to remark that the process of annotating always produces data (Lehmann, 2004, p. 207). The content of theannotation levels can be treated as data only if reconstructability is given: i.e., for all levels of annotation the claim holds thatderived data can be ‘related back’ to raw data (cf. Lehmann, 2004, pp. 207–208, who rightly argues that reconstructability is afundamental precondition for the status of linguistics as an empirical science). This means that raw data can be recon-structed from the derived (i.e. annotated data) with respect to the categories of annotation.8

Raw data are – when they stem from spontaneous speech – to a large extent theory free data (cf. Lehmann, 2004, p. 207).At the annotation levels derived from raw data, reconstructability is provided for by making transparent which theoreticalassumptions have an impact on the annotations. This prerequisite may be fulfilled because the theoretical assumptions arecanonised and it is assumed that the theories and their potential impact on the annotations are known – or, if not, the pre-requisite has to be fulfilled by an explicit explanation.

The higher the level of annotation, the grade of canonisation of the involved theories decreases and the influence of inter-pretation increases (see Fig. 2).9 What is more: at higher levels, the ‘‘standard methodological procedures’’ (Lehmann, 2004, p.208) that should guarantee the reconstructability of derived to raw data will inescapably be based on the same theories that aretested by that very data. That is, first a theory serves as a framework for annotating data; then, the same theory is tested bymeans of those data that have been generated on the basis of this theory.

Problems resulting from this especially concern the annotation of functional level categories, which do not only implyassumptions about language structure but also about human cognition. This will be discussed in Section 5.1.

As the influence of interpretation increases, theories increasingly allow multiple possibilities of annotation for one and thesame datum; consequently, neither a consistent annotation nor reconstructability can be guaranteed. In these cases, re-searches will have to make use of heuristics in order to make theoretical concepts manageable for annotators; see Section 5.2.

5 For Lehmann, all corpus data regardless of which level of annotation they feature are ‘‘primary data’’: ‘‘Primary linguistic data are (original or derived)representations of specific speech events with their spatio-temporal coordinates [. . .]. Secondary data are [. . .] sentences in written representation that lackspatio-temporal coordinates [. . .], they are being used as types rather than as tokens, but come along with a claim of being usable in some actual speechsituation, thus, a claim of being potential primary data’’ (Lehmann, 2004, p. 184).

6 Lehmann (2004) presents a categorisation which is similar, though lacking any categories beyond the ‘‘semantic level‘‘, and with Lehmann semantic level istaken to mean simply ‘‘translations in various languages‘‘ (2004, p. 206). Thus, Lehmann does not seem to assume a discourse-functional level of reference.

7 Thus, ‘functional category’ is not meant in the sense of Generative Grammar which defines functional categories as syntactic categories without any lexicalcorrelate, but as a category reflecting language function in human communication.

8 E.g., in an oral corpus study dealing with grammatical or functional categories of annotation, phonetical features of the raw data do not necessarily have tobe reconstructable.

9 Cf. Lehmann (2004, p. 200): ‘‘At the higher levels [. . .] the recognition of patterns and the correct assignment of a given token to a pattern involveprocedures of interpretation [. . .].’’

a(ah

706 M. Consten, A. Loll / Language Sciences 34 (2012) 702–714

The following section will relate the theoretical topics discussed so far to the reality of research. The problems arisingwith the studies introduced in Section 4 are looked at more closely and a solution is attempted in Section 5.

4. Empirical data in the reality of research

4.1. Newspaper corpus: complex or nominal based anaphor?

Complex anaphors are nominal expressions that refer to situational and other complex entities and establish them ascondensed discourse objects. Therefore, they are important means of information flow and textual coherence. In the studypresented in this section,10 different kinds of specifications which complex referents undergo in the course of the anaphoricso-called complexation process were investigated, and some hypotheses on relations between grammatical characteristicsand textual functions of complex anaphors were tested. For this purpose, a newspaper corpus11 was annotated with respectto structural as well as functional features of anaphors. The first task annotators had to fulfil was to decide which of thenoun phrases in the corpus are complex anaphors. This stage of pre-annotation resulted in insights into the quantitativedistribution of complex anaphors with respect to different grammatical types of noun phrases. So, the question relevantfor this section does not concern features of complex anaphors but how to distinguish complex anaphors from similarconstructions.

4.1.1. How to distinguish complex anaphor and NP-based anaphorIn general, anaphors reactivate referents that have been introduced by NP-antecedents (cf. Consten and Schwarz-Friesel,

2008) as in (1)12:

10

by11

sy12

wi13

nai0) Lnaold

(1)

The sthe DThe

ntacticIn th

thoutIn thephor: (etzte W

lysis fofor le

Last week, [an accident] happened here. My colleague saw it when she was looking out of the window.

Complex anaphors, in contrast, establish referents such as events, processes and states that have been introduced by pro-positionally structured antecedents such as sentences or even longer text segments (Consten et al., 2007, 2009):

(2)

[Accidents happen here again and again.] This/this fact shows that the Police should do something about thesituation urgently.

The anaphors in (1) and (2) clearly refer to different kinds of referents, i.e. a singular event in (1) and an iteration of sim-ilar events in (2).

With other examples, however, the difference between complex anaphors and NP-based anaphors seems to be an analyticand not a referential one; thus it is an ambiguity from the annotator’s but not from the language user’s point of view:

(10)

eTaea

i

x

[Last week, an accident happened here.] My colleague saw this when she was looking out of the window.

The anaphor in (1) is not a complex anaphor since the event-referent has been introduced by an NP and not by the wholefirst sentence, more precisely: the anaphor in (1) will be analysed as an NP-based anaphor since for various theoretical andempirical reasons personal pronouns like it are not regarded as sufficient for complex anaphoric use (Consten et al., 2007, p.95). Unlike it, the demonstrative pronoun this in (10) is considered a typical means of complex anaphora (Consten et al.,2007).13

Given the situation that (1) and (10) were part of a text corpus analysed to explore the distribution of personal versusdemonstrative pronouns in complex anaphora, an argumentation like this would be circular since personal pronouns areannotated as NP-based anaphors for no other reason than the theoretical claim that personal pronouns cannot be complexanaphors; demonstrative pronouns are annotated as complex anaphors due to the theoretical claim that demonstratives area typical means of complex anaphora. Yet, the very aim of the corpus study was to confirm or reject these theoretical claims.On the other hand, annotators would not have any independent criterion to decide whether the accident-NP or the wholesentence counts as antecedent. There is no referential difference between (1) and (10); an event-referent is established in(1) as well since the noun accident bears as a part of its literal meaning an event structure similar to the meaning of the wholesentence.

tudy on complex anaphors has been conducted in the framework of the research group ‘‘KomplexTex – textual function of complex anaphors’’, fundedutsche Forschungsgemeinschaft, Bonn, code SCHW 509/6.iGer-Corpus consists of approx. 50,000 sentences/900,000 tokens from the daily newspaper Frankfurter Rundschau; it is pre-annotated morpho-lly.following examples, anaphoric expressions are underlined. Antecedents and ambiguities in their interpretation are marked by brackets. Examples

ny reference are constructed.original German data, there is sometimes the chance of an analytical disambiguation by gender congruence between the NP-antecedent and the) Letzte Woche ist hier [ein UnfallMASK SG] passiert. Meine Kollegin hat ihnMASK SG gesehen. . . Pronominal complex anaphors are always neutrum singular:oche ist hier [ein UnfallMASK SG] passiert. Meine Kollegin hat dasNEUTR SG gesehen. . . Thus, for grammatical reasons you can exclude a complex anaphoric

r (i) and an NP-based anaphoric analysis for (i0). This neither holds when the potential antecedent NP happens to be neutrum singular; nor does itical anaphors: (ii) [Letzte Woche ist hier [ein Unfall] passiert.] Meine Kollegin hat das Geschehnis gesehen. . ., see (100).

M. Consten, A. Loll / Language Sciences 34 (2012) 702–714 707

In case of lexical anaphors (like in (100)), annotators would even lack a grammatical congruence based criterion.

14

funarehaargtexTh

15

intthsefraref

(100)

The coded bysponta

d beenumenttworld

e transc

Indireroduce

e referemanticme baserents

[Last week, [an accident] happened here.] My colleague saw the occurrence. . .

Here, a decision for complex or NP-based anaphor would be made by spontaneous determinants that hardly can be con-trolled, or it would be pure arbitrariness.

4.1.2. Problems of annotation

(3)

[Etwa 25 Kilogramm Natur-Urankonzentrat sind bei [einem Unfall] im thüringischen Wismut-Zwischenlager

Seelingstädt freigesetzt worden.] Der Zwischenfall ereignete sich bereits am 1. Juli [. . .] (TiGerKorpus 2275f)

[About 25 kilograms of yellowcake have been released during [an accident] at the Wismut interim storage area

Seelingstädt, Thuringia.] The incident already took place on July, 1st.

The anaphor der Zwischenfall can be analysed both as a complex anaphor (then the whole preceding sentence is its anteced-ent) and as NP-based anaphor (with an accident serving as its antecedent).

(4)

[[Israelischer Angriff im Gaza-Streifen] fordert mindestens 13 Tote, darunter viele Zivilisten.] General spricht von

‘‘unabdingbarer Operation’’ gegen Hamas, für palästinensischen Unterhändler zielt dieses ‘‘Massaker’’ auf dieFriedensmission von Solana. (‘‘Massaker’’ im ‘‘Terrornest’’, taz, 08.10.2002)

[[Israeli attack in the Gaza Strip] claims the lives of 13 people, many civilians being among them.] A General calls

[it/this] an ‘‘indispensable operation’’ against Hamas; from a Palestinian negotiator’s point of view this ‘‘massacre’’is aimed at Solana’s peace operation.

Like in (3) and (100), the anaphor can be a complex anaphor (which seems to be the preferable analysis due to the demon-strative determiner, see Section 4.1.1), but it can be an NP-based anaphor with the antecedent Israelischer Angriff imGaza-Streifen as well. Like in the previous examples, the referential content would be the same with both of the analyses.Consequently, there are no independent criteria for such a decision; at most one could argue that due to its lexical meaningdieses ‘‘Massaker’’ is more plausible to be read as a complex anaphor with an antecedent including the reference to the num-ber of people killed.

These are fine-grained differentiations, however, and they would not be sufficient to construct reading tests that would behelpful to resolve the analytical ambiguity. Again, the ambiguity is an artefact of linguistic analysis and it does not reflect anyproperties of the natural reference process. Thus, it cannot be resolved by any data gained from natural language users.

This problem is approached by an annotation guideline for practical handling: Whenever a complex anaphoric reading isplausible (plausible in a sense that has to be discussed in Section 5.2), the datum counts as a complex anaphor and is anno-tated. In the course of the annotation, a feature ‘‘analytical ambiguous’’ (versus referentially ambiguous) is annotated in or-der to document the arbitrary character of the annotation. The theoretical status of such a guideline will be discussed inSection 5.2.

4.2. Longitudinal acquisition corpus: how to specify referential features separate from determiner use?

Loll (2007) has investigated the determiner use and its annotation in a longitudinal corpus of second-language acquisitionof German (cf. Consten and Loll, 2009).14

The aim of this study was to survey to what extent the learner uses definite determiners to mark identifiable referents asrequired in the German target-language. It turned out soon that with so-called indirect anaphors15 referent properties couldnot be assigned independently. In particular, the annotation of the referent feature ‘identifiability’, which was intended to serve

rpus was compiled during 1998 to 1999 in the framework of the research group ‘‘Second Language Acquisition of German by Russian Learners‘‘,Max-Planck-Institute for Psycholinguistics, Nijmegen, supervised by Ursula Stephany, Cologne, and Wolfgang Klein, Nijmegen. The data cited hereneous speech interviews; the speaker is a monolingual Russian girl named Nastja. At the time of the interviews cited here, she was 9 years old andliving in Germany for 15–16 months. The annotation in (5) and (6) is cited from Loll (2007). Only the annotation features relevant for our

ation are cited; these are: IREF = the referent of the NP is identifiable/NIREF = the referent is not identifiable; EINF = the referent is introduced into themodel for the first time (Germ. EINFührung)/WAUF = the referent is anaphorically resumed, including indirect anaphors (Germ. WiederAUFnahme).ription follows the CHILDES-standard: [�] deviant, eh@fp filled pause, [/] repetition, [//] self repair. Cf. Stephany et al., 2001. Deviances and self repairs

ct anaphors (underlined) are anaphors without any explicitly co-referring antecedent expression. They relate to textual anchors (in brackets) thatrelevant referential information into the textworld model. There are different kinds of relationship between the referent of the indirect anaphor and

nt of the anchor, e.g. (a) part-whole (meronymic, [The house] needs some repairs. The roof [of the pre-mentioned house] seems to be rotten), (b) based onroles dependent from the verb ([She shot her husband.] The gun was found later in her car. (The gun filling the instrument role allocated by shoot)), (c)ed (When we entered [the restaurant], the waiter pretended not to see us.) (Schwarz, 2000a; Consten, 2004; Schwarz-Friesel, 2007). Consequently,of indirect anaphors are identifiable due to textual coherence relations, which results in the use of definite determiners with indirect anaphoric NPs.

708 M. Consten, A. Loll / Language Sciences 34 (2012) 702–714

as an independent variable, was not possible without a circular dependency on the referential expression chosen by the speak-er; cf. (5) and its description in Section 4.2.1.

4.2.1. Indirect anaphor or first-mention use?

Der Arzt can be analysed as an indirect anaphor (WAUF = resuming) which is conceptually anchored by prementioning thecase of illness. Since indirect anaphors are definite NPs, we are justified in counting the use of the definite determiner in derArzt as target-language. On the other hand, the use of an indefinite NP would be perfectly interpretable as well, see the con-structed example (50) and the indefinite NP einem Elektriker in (6):

The paired annotation features EINF/NIREF (first mention use/non-identifiable) versus WAUF/IREF (resumption/identifi-able) are meant to be features of the referent; they should be annotated independently from the way a referent is verbalised.In (500) the original datum (definite NP) is shown with an alternative annotation, which is – given the independence of ref-erent features and the article used – also possible. The same argument holds for the annotation for the constructed example(5000) in contrast to (50).

As with (100), the alternative annotations (5)/(500) and (50)/(5000) simply establish an alternative analysis having no bearingon any non-arbitrary base. The variables WAUF (resuming a referent)/EINF (introducing a referent) as well as IREF (identi-fiable referent)/NIREF (non-identifiable referent) turn out not to reflect the psychological reality of speech processing asplanned in the design of the study: it does not really make a difference if the speaker in (5) refers to ‘the medic who med-icated the prementioned sick girl’ (i.e., the speaker makes use of an indirect anaphor) or if she refers to ‘a(ny) medic who isnot known yet’ (i.e., she makes use of an indefinite NP). Anyway, a new referential entry is introduced into the textworldmodel, and this new entry can be coherently integrated into the established referential structure.16

‘‘In einer recht großen Grenzzone zwischen indirekter Anaphorik und Neueinführung von Referenten muss die Arti-kelwahl somit als eine Art stilistischer Variation angesehen werden, die sich einer präzisen Erklärung durch die hieruntersuchten Variablen entzieht.’’ (Loll, 2007, p. 21) (‘‘Within a quite wide border zone between indirect anaphora andfirst-mention use, the choice of determiners has to be regarded as a kind of stylistic variation which fails to be explainedby the variables investigated here.’’)

The continuation of the corpus section cited in (5) provides for some more examples: the sick girl has been established asan ongoing discourse topic; hence the expressions referring to her are valid anchors for potential indirect anaphors.

16 For this reason, Schwarz (2000b) describes indirect anaphors as thematic and rhematic at the same time, i.e. maintaining text coherence but establishingnew referents. See also Consten/Schwarz-Friesel (2008, pp. 284–286).

M. Consten, A. Loll / Language Sciences 34 (2012) 702–714 709

Der Vater in (6) is a clear case of indirect anaphora. In German, relational nouns like father do not need to be specified bypossessive determiners. Thus, der Vater (with a child prementioned) is a common example for a conceptually based indirectanaphor.

Concerning the definite NPs der Lehrer und der Bäcker, the problem of circularity arises again. This will be discussed in thenext section.

4.2.2. Interaction of uniqueness and indirect anaphora?In Loll (2007, p. 18), the definite determiner use in dem Lehrer ‘‘the teacherDAT’’ and dem Bäcker ‘‘the bakerDAT’’ is explained

by the uniqueness (basically defined by Russell, 1905, and Hawkins, 1978) of the referent within its domain of reference. A

710 M. Consten, A. Loll / Language Sciences 34 (2012) 702–714

discussion on uniqueness in reference-semantics (Löbner, 1985; Larson and Segal, 1995; Lyons, 1999, pp. 7–12)17 has re-sulted in the insight that uniqueness is not a feature of a referent itself, but a feature of the situation in which an object is re-ferred to. That means, annotators should have – like normal language users – situational knowledge which is independent fromthe discourse in order to evaluate the discourse. With respect to the data, annotators would have to know whether in the villagewhere the story takes place there is only one teacher and only one baker. In this case, this would be a doubly difficult task sincethe speaker verbalises her own mental representation of another author’s fictional narration, and it is not very likely that thecrucial uniqueness-features are specified in any version of the narration. In other words, there is no referential reality the speak-er’s determiner use could be evaluated with; annotators do not have any other means to evaluate the discourse but the dis-course itself.18 Thus, we can only state that it is ‘plausible’ to evaluate the definite determiner use with these NPs asmeeting the rules of the target language since it is ‘plausible’ that in a small village there is only one teacher and only one baker.This assumption results in the cited annotation of the respective NPs as first mentioning (EINF) a new but identifiable (IREF),since unique, referent. Thus, plausible here means ‘‘not substantially inconsistent with conceptual knowledge’’ – this is, ofcourse, a quite vague criterion allowing for a number of different annotations. We will come back to this topic in Section 5.2and now discuss one of the other different but equally plausible annotations.

At least der Lehrer could be analysed as an indirect anaphor (see footnote 14) anchored by ein Mädchen (5), that is, the NPis read as ‘‘the girl’s teacher’’. This would be a conceptually based indirect anaphor grounded on a link in the long-term mem-ory between the concepts GIRL (as an instantiation of child), SCHOOL (as an institution for children) and TEACHER (as someone whoworks at a school).

The use of the definite determiner could be explained more convincingly by an addition of the features ‘anaphoricalanchorage’ and ‘uniqueness’. On the one hand, the anaphorical anchorage restricts the reference domain in favour of unique-ness: the feature of uniqueness would be given, then, if there is more than one teacher in the village, but there is only oneteacher who teaches the girl. Thus, anaphorical anchorage makes the uniqueness-reading more plausible. On the other hand,uniqueness may be required to allow definiteness since indirect anaphors are not always and necessarily definite NPs. Moreprecisely, indirect anaphors can be indefinite when their referent is an unspecified object within a group of objects that allrelate comparably to the referent of the anchor expression, cf. (7):

17

18

kn19

no

(7)

For aCf. Lo

owledThe p

t weak

Wir entdeckten [ein Auto] am Straßenrand. Der Motor war noch warm, aber ein Reifen war platt.

We found [a car] at.the roadside.The motor was still warm, but a tire was flat.

(‘‘. . .but it had a flat tire’’/‘‘one of its tyres was flat’’)a.

a German ein can be an indefinite determiner (‘‘a’’) as well as a quantor (‘‘one’’).

Undeniably, the motor- and the tire-referent relate to the car-referent in the same way, i.e. meronymically. Consequently,both of the NPs der Motor and ein Reifen have to be regarded as indirect anaphors, the latter, however, being indefinite due tothe fact that the NP does not specify which of the (presumably) four tyres is referred to. The same is true for a reference to‘‘one of the girl’s teachers’’ which is expected to be carried out with an indefinite NP.

The same might hold for das Fenster: it is annotated as an indirect anaphor (resumption, WAUF). The indirect anaphoricanchorage can easily be explained as based on the conceptual chain BAKER ? BAKERY ? SHOP WINDOW OF THE BAKERY. Here, again,uniqueness is another referential precondition for definite determiner use. If there were more than one shop window, theindefinite NP ein Fenster (see footnote 19) would be obligatory.

To conclude, these annotation details show three problems: (1) annotators do not have independent criteria to evaluatethe discourse; they are reliant on the discourse itself – probably bearing circular conclusions – and their conceptual knowl-edge, which is a vague and almost ungovernable parameter; (2) this can result in different annotations that are comparably‘plausible’, cf. the baker and the teacher as unique referents versus the shop window as an anaphorically anchored referent;(3) there can be interdependencies between the annotated criteria (such as uniqueness and anaphoric anchorage) that arenot reflected in the clear-cut, dichotomic annotation features that are needed for statistical analyses.

As with the study presented in Section 4.1, annotation possibilities are constrained by a guideline for practical handling:Whenever the determiner chosen by the speaker is plausibly interpretable (in a sense discussed in Section 5.2), referentialfeatures are annotated according to this interpretation; e.g. if the speaker uses a definite noun phrase the referent of whichcan plausibly regarded as anchored (though indirectly and weakly) in the discourse, the features ‘‘identifiable referent’’ and‘‘anaphorical resuming’’ will be annotated19; i.e. the determiner use is interpreted as adequate whenever this seems to bepossible.

‘‘Es kann jedoch nicht ausgeschlossen werden, daß die Sprecherin eigentlich etwas anderes intendierte und der Artikelnicht zielsprachlich gewählt wurde’’. (‘‘However, the possibility that the speaker actually had another intention unddid not choose the determiner according to the rules of the target language cannot be eliminated’’) (Loll, 2007, p. 19).

brief summary, see Consten (2004, pp. 48–51).ll (2007, p. 7) for the same problem of circularity with respect to ‘common ground’ (terminus by Hawkins, 1978): how will annotators know about the

ge shared by speakers and hearers if not by the discourse itself?roblem of interdependent criteria such as uniqueness and indirect anaphorical anchorage cannot be solved by such a guideline. This, however, doesen the results in Loll (2007) since both uniqueness and anaphorical anchorage match with definite determiner use.

M. Consten, A. Loll / Language Sciences 34 (2012) 702–714 711

5. Circularity effects in empirical linguistics

5.1. Why and at which point of investigation do circularity effects occur?

The studies presented in Section 4 allow the conclusion that circularity effects arise with certain (especially functionallyoriented) investigations. In this Section 5, we will discuss more closely why and at which stage of research these effectsoccur.

The annotation categories used in Sections 4.1 and 4.2 are not mere structural but referential–semantical ones; conse-quently, they are categories at the functional level.

Mostly, data are pre-annotated (if at all) with respect to structural categories only. Reasons for this are obvious:As shown in Section 3, structural annotations are prerequisites for functional ones; structural annotations in many cases

can be conducted automatically; and there are many theories that restrict themselves to structural aspects of languageanyway.

The most important reason is that the annotation of functional categories20 depends to a larger extent on theoreticalrequirements and definitions which are specific to the respective investigation; i.e., in many cases, the categories of investiga-tion are not created (or at least precisely specified) until the formation of the specific theory that is to be tested using thesecategories.

Thus, we may act on the assumption that in general functional categories are less canonised than structural ones. Mostresearchers deal with the basic structural categories as if these categories were undoubted and, so to speak, God-given. Infact, structural categories are based on and depend on theories as well; these theories, however, show a higher grade ofcanonisation. Structural categories used for a specific investigation are not independent from theory at all, but in many casesthey depend on a more generalised theoretical framework that is not scrutinised by the majority of the research community.

The studies presented in Section 4 investigate corpora which are both pre-annotated with respect to parts of speech cat-egories. Categories of this kind are said to be easily manageable as long as they are used to describe indo-european lan-guages, although they are questionable: if the current linguistic models had not been created on the basis of indo-european languages, the canonised part of speech categories would likely have been different ones.

Canonised and non-canonised annotation categories have to be differentiated gradually; this can be shown with the TiGercorpus (used for study in Section 4.1). It is pre-annotated not only with respect to parts of speech but also with respect tophrasal categories. The latter depend to a larger extent on a certain theory; within the range of phrasal categories, a categorylike noun phrase is more canonised and less theory-laden than determiner phrase or a generative tree position like specifier ofcomplement phrase (SpecCP).

One reason for this gradual fading of canonisation may be that theories on a more complex level involve a higher degreeof abstraction and interpretation; i.e. the gap between a theoretical concept and the respective observable items increases:although the labelling of a word as noun is subject to theoretical assumptions as well (as we have argued above), even a non-expert is intuitively able to assign the category label noun to a certain item because of the easily graspable relation betweenan observable item and the assigned category (one category matches one item). With noun phrases, there is one categorylabel for a group of words which are interpreted as a unit at an underlying (not an observable) level of speech. Still thereis, however, a noun as a concrete nucleus of the more abstract concept noun phrase. A category like SpecCP even lacks sucha concrete nucleus, being highly dependent on interpretative assumptions within a special theoretical framework.

Examples like these show that increasing complexity at the structural level involves an increasing impact of interpreta-tion and a fading acceptance of theories and the categories related to them.

The examples discussed so far hold for increasing complexity on different structural levels. Functional categories arehigher-levelled than structural ones (see Fig. 2 in Section 3) and are based on structural categories; therefore, with functionalcategories the problem will arise to a larger extent.

Functional theories do not aim for formal models that have to be nothing else except self-consistent, but they have tomake assumptions about the human mind on a high level of abstraction. Categories concerning cognitive structures and lan-guage use in principle depend on theoretical prerequisites to a larger extent than structural ones. As with structural catego-ries, there are different grades of canonisation: a concept of anaphora is widely spread and accepted in all semantical and textlinguistic theories (though with slightly different definitions); notions like complex anaphor (as used in Section 4.1) and indi-rect anaphor (Section 4.2), on the other hand, are much more specific to certain theoretical approaches.

The empirical testing of cognitively oriented theories requires a follow-up of a structural annotation (arrow I in Fig. 3) andan annotation with respect to functional categories (arrow II); the latter is often conducted during a separate step in additionto the structural annotation (as in the studies presented in Section 4).

As we have argued above, this step is especially theory-laden to a great extent; i.e. the selection of annotation categoriesis closely dependent on the theoretical framework of the study.

Consequently, some assumptions of the respective theory will have an impact on the annotation (arrow III in Fig. 3). Thisprocess is exemplified in Section 4.1 as the basic question of how to define complex anaphor and in Section 4.2 as the question

20 See footnote 6. Functional category is meant as a category concerning the communicative function of language, i.e. natural language use and the user’smental states.

V

RAW DATA S+F-DATA S-DATA

THEORY

structure function

I II

IIIIV

VI

Fig. 3. Circularity effects during the annotation process.

712 M. Consten, A. Loll / Language Sciences 34 (2012) 702–714

of what phenomena should be covered by the notion of anaphorical resumption. So, the theory is tested by data which havepartly been generated by the theory itself (arrow IV).21

This is a circle that seems to be unsolvable at least with respect to higher-levelled theories.As shown above, this problem in principle arises for research at a mere structural level as well (dashed arrows V and VI).

However, this fact does not seem to attract that much attention since structural theories are canonised to a larger extent (aswe have argued above and in Section 3).22

With functional theories, the problem increases since functional theories characteristically aim to match structures tofunctions; i.e. these theories are of the type x has function Y.

In many cases (like in the studies in Section 4), annotators do not have any independent criterion to assume the existenceof a postulated function; they are reliant on nothing other than the data themselves and their conceptual knowledge. Then,annotators have to conclude from structure to function, although such a conclusion is part of the theory that should be testedby means of the annotated data instead of being a prerequisite for annotation.

Lehmann (2004, p. 196) remarks that such a process is basically a circular one: ‘‘The procedure in which a linguist pro-duces data on which he constructs a theory which he then tests by these data is, of course, circular. The data do not actuallyfulfil any control function.’’

Stating the problem more precisely, in this section it has been shown that the specific point of circularity in this process isthe theory-laden annotation; this holds especially with respect to functional categories (in Fig. 3 the respective circle is builtup by the arrows III, II, and IV) but in a certain way with respect to mere structural categories as well (arrows V, I, and VI).

In the light of these problems, the studies presented in Section 4 make use of annotation guidelines in order to guaranteeat least a consistent annotation and the reconstructability of the data (as postulated in Section 3). These guidelines and theinevitable impact of conceptual knowledge on annotations will be discussed in Section 5.2 in terms of the criterion ofplausibility.

5.2. How to deal with annotational circularity effects: guidelines for annotators

Section 5.1 has shown why and at which point of investigation circularity effects such as those reported in Section 4 oc-cur: hypotheses of a theory are (partly) tested with derived (annotated) data that have been generated in the framework ofthe same theory. Functional theories are assigned two properties: normally, they are less canonised than structural theories,and they are mostly based on structural categories since they link structural and functional features. The latter, which areestablished as independent variables in the studies presented here, turn out to be indeterminable by any independentnon-arbitrary criteria. Consequently, they either depend on those structural features that should serve as dependent vari-ables, or they are fixed arbitrarily by the annotators. These studies make use of simple heuristics in order to achieve anno-tations that are at least reconstructable (see Section 3) and consistent, i.e. the risk that different annotators will producedifferent annotations for the same data is minimised due to strict annotation guidelines.

Inconsistencies occur if there are several possible intentions or referential structures the speaker might have intended toexpress – or if intentions and possibly intended referential structures are not identifiable at all; in most cases these differentpossibilities will emerge with different annotation possibilities so that annotators are in doubt about the appropriateannotation.

A situation like this will arise especially with theories matching structural to functional features since speakers’ inten-tions are a fundamental part of such theories: they do not represent formal constructs but real properties and differentia-tions of conceptual representations and natural language processing – such as the state of referents in a mental textworldmodel (cf. Sections 4.1 and 4.2) and the speakers’ assumptions about the identifiability of referents in the ongoing discourse(cf. Section 4.2). Theoretical concepts on these topics are hardly ever clear-cut enough to be easily transformable into an

21 In Section 2 it has been argued that annotation is a kind of theory-dependent production of data as well.22 Take as an example a purely structural theory claiming that German genitive will vanish (‘‘structure �will be replaced by structure y’’). One hypothesis

related to this theory could be that in contemporary German oral language the genitive is replaced by the dative. Data will be annotated with respect to case inorder to test the hypothesis. Both hypothesis and data, however, are based on the same (strongly canonised) theory that German has four cases.

M. Consten, A. Loll / Language Sciences 34 (2012) 702–714 713

annotation guideline. Adequate annotations require assumptions on speakers’ internal states and intentions that should beindependent from discourse.23 In most cases, however, there are no other criteria for such assumptions than the discourse itselfand the annotator’s conceptual knowledge.24

The heuristics shown in both Sections 4.1 and 4.2 crucially are based on the concept of plausibility. According to Luhmann, aplausible decision is a decision which makes sense25 to the decider without any reasons; moreover, the decider expects that it willmake sense to other people as well. However, a plausible decision is a choice between alternatives, whereas evidence means that notonly the decision itself but also the exclusion of alternatives makes sense (Luhmann, 1980, p. 49, cit. in Meißner, 2007, p. 91).26

Thus, the annotation heuristics reported in Section 4 restrict the annotators’ possibilities of reaching a decision wherethere is no fixed, reasonable criterion to rule out alternatives. Heuristics of this kind will not burden annotators with thequestion of which possibility is the most plausible one but they give a clear instruction of the sort ‘‘if X is among a set ofplausible choices, take X’’.27 This results in consistent annotated data that are capable of a statistical evaluation since the im-pact of the annotators’ individuality is minimised. Nevertheless, the impact of arbitrariness remains, but by heuristics it is raisedto a supra-individual level. This means that the annotated data are at least comparable since they were all subject to the sameannotation principles and therefore reconstructable, which in Section 3 is mentioned as an important claim for empirical data.

Thus, researchers have to be aware that in many cases annotations require an annotator’s judgement. Such a judgement stillimplies introspective elements (see Kertész and Rákosi, 2008, pp. 33–34) albeit it is guided and constrained by heuristic rules.

Consequently, annotators play a crucial role as co-producers of data in the process of annotating. The postulation thatempirical data must not ‘‘depend essentially on the observer’’ (Hempel, 1952, p. 22, cited in Kertész and Rákosi, 2008, p.22) or more strictly ‘‘a datum must have a basis outside of and independent from the researcher’’ (Lehmann, 2004, p.181) becomes questionable.

6. Conclusion

This contribution has raised the question of whether empirical studies can be based on functional data (i.e. data anno-tated with respect to functional categories) at all in the light of the emerging problems we have discussed:

(1) annotations are always theory-laden (which actually holds for annotations at all levels but entails problems especiallyfor annotations at the functional level);

(2) annotations at the functional level in most cases cannot be conducted without accounting for and allowing an impactfrom the structural levels;

(3) annotations at the functional level mostly concern (or even have to express) internal mental states of language users,which are inaccessible to annotators;

(4) consequently, annotations are often based on the annotator’s interpretations, which can be constrained only by crite-ria such as plausibility. Thus, annotators participate to a large extent in the generation of data: a database outside ofand independent from the researcher does not exist.

To sum up, the question remains whether empirical studies on cognitive, functional topics can make use of really inde-pendent variables. At the very least, it is desirable that considerations like these will be fruitful for theoretical approachesand their formation of functional concepts.

References

Consten, M., 2004. Anaphorisch oder deiktisch? Zu einem integrativen Modell domänengebundener Referenz. Tübingen, Niemeyer (LA 484).Consten, M., Loll, A., 2009. Indirekte Anaphern – ein Zirkularitätsproblem zwischen Grammatik und Pragmatik. In: Brdar-Szabó, R., Knipf-Komlósi, E., Péteri,

A. (Eds.), An der Grenze zwischen Grammatik und Pragmatik. Peter Lang, M. Frankfurt, pp. 323–330.Consten, M., Schwarz-Friesel, M., 2008. Anapher. In: Hoffmann, L. (Ed.), Deutsche Wortarten. Mouton de Gruyter, Berlin/New York, pp. 265–292.Consten, M., Knees, M., Schwarz-Friesel, M., 2007. The function of complex anaphors in texts. In: Schwarz-Friesel, M., Consten, M., Knees, M. (Eds.), Anaphors

in Texts. Benjamins, Amsterdam, pp. 81–102.

23 The problem increases in cases of acquisition corpora where the speakers’ intentions may remain unclear even in the context of the discourse: ‘‘Mostconspicuous here is the problem of interpreting child language data, where the hermeneutic intuition of the researcher may fail and possibilities ofmetalinguistic interaction with the native speaker are limited.’’ (Lehmann, 2004, p. 195)

24 Conceptual knowledge (world knowledge) is the source annotators make use of in deciding whether a reading is plausible. Thus, our notion of plausibility isin accordance with Meißner (2007, p. 88) who relates plausibility to a knowledge ‘‘das nur sozial und historisch gebunden einleuchtet’’; i.e. that makes senseonly within a certain social and historical domain. Our notion of plausibility is narrower than the one given in Kertész/Rákosi (2009, p. 713), where the fullrange of possible cognitive objects that can serve to make assumptions plausible (such as sense perception, theories and inferences) is accounted for. Thisdifference may be due to the different contexts the notion of plausibility is discussed in: Kertész/Rákosi (2009) mainly deal with plausible inferences andargumentation whereas we claim plausibility as a necessary feature of annotation heuristics that are not arguments themselves but mere tools of gainingsystematic, reliable data.

25 Making sense is our translation from ‘‘einleuchtend’’ in the original German text.26 ‘‘Während nämlich mit einem evidenten Wahrheitskriterium Mögliches von Unmöglichen [sic] (im Sinne von Unwahrheit) getrennt wird, kann über

Plausibiltät Wirkliches von Möglichem unterschieden werden.‘‘ Using criteria of truth based on evidence, the Possible is separated from the Impossible (in thesense of the Untrue) whereas using plausibilty the Real is separated from the Possible. (Meißner, 2007, p. 95)

27 In Section 4.1, X is a complex-anaphorical reading; in Section 4.2, X is the assumption that the speaker has used a determiner according to the rules of thetarget language.

714 M. Consten, A. Loll / Language Sciences 34 (2012) 702–714

Consten, M., Knees, M., Schwarz-Friesel, M., 2009. Complex anaphors. In: Zlatev, J., Johansson Falck, M., Lundmark, C., Andrén, M. (Eds.), The Impact ofOntology, Cotext and Conceptual Knowledge, Studies in Language and Cognition. Cambridge Scholars Publishing, Newcastle, pp. 285–302.

Geeraerts, D., 2006. Methodology in cognitive linguistics. In: Kristiansen, G., Michel, A., Dirven, R., Ruiz de Mendoza Ibánez, F. (Eds.), Cognitive Linguistics:Current Applications and Future Perspectives. Mouton de Gruyter, Berlin/New York, pp. 21–49.

Haspelmath, M., 2009. Welche Fragen können wir mit herkömmlichen Daten beweisen? Zeitschrift für Sprachwissenschaft 28, 157–162.Hawkins, J., 1978. Definiteness and Indefiniteness. Croom Helm, London.Hempel, C.G., 1952. Fundamentals of Concept Formation in Empirical Science. Chicago University Press, Chicago.Kertész, A., Rákosi, C., 2008. Daten und Evidenz in linguistischen Theorien: Ein Forschungsüberblick. In: Kertész, A., Rákosi, C. (Eds.), New approaches to

linguistic evidence/Neue Ansätze zu linguistischer Evidenz. Peter Lang, M. Frankfurt, pp. 21–60.Kertesz, A., Rakosi, C., 2009. Cyclic versus circular argumentation in the conceptual metaphor theory. Cognitive Linguistics 20 (4), 703–732.Larson, R., Segal, G., 1995. Knowledge of Meaning. An Introduction to Semantic Theory. MIT-Press, Cambridge.Lehmann, C., 2004. Data in linguistics. The linguistic. Review 21 (2004), 175–210.Löbner, S., 1985. Definites. Journal of Semantics 4, 279–326.Loll, A., 2007. Determinierer im Erwerb des Deutschen als Zweitsprache – eine Fallstudie. (Arbeitspapier N.F. 52). Institut für Linguistik, Köln. <www.uni-

koeln.de/phil-fak/ifl/asw/forschung/forschung_frames_d.html>.Luhmann, N., 1980. Gesellschaftliche Struktur und semantische Tradition. In: Luhmann, N. (Ed.), Gesellschaftsstruktur und Semantik. Studien zur

Wissenssoziologie der modernen Gesellschaft. Suhrkamp, M. Frankfurt Band 1, pp. 9–71.Lyons, C., 1999. Definiteness. CUP, Cambridge.Meißner, S., 2007. Wahrheit oder Plausibilität? In: Langner, R., Luks, T., Schlimm, A., Straube, G., Thomaschke, D. (Eds.), Ordnungen des Denkens, Debatten

um Wissenschaftstheorie und Erkenntniskritik. Lit Verlag, Berlin, pp. 87–96.Rákosi, C., this issue. The fabulous engine: strengths and flaws of psycholinguistic experiments.Rescher, N., 1976. Plausible Reasoning. Van Gorcum, Assen, Amsterdam.Schwarz, M., 2000a. Indirekte Anaphern in Texten. Niemeyer, Tübingen (LA 413).Schwarz, M., 2000b. Textuelle Progression durch Anaphern – Aspekte einer prozeduralen Thema-Rhema-Analyse. In: Dölling, J., Pechmann, T. (Eds.),

Prosodie – Struktur – Interpretation. Linguistische Arbeitsberichte 74. Institut für Linguistik, Leipzig, pp. 111–126.Schwarz-Friesel, M., 2007. Indirect anaphora in text: a cognitive account. In: Schwarz-Friesel, M., Consten, M., Knees, M. (Eds.), Anaphors in Text. Benjamins,

Amsterdam, pp. 3–20.Stefanowitsch, A., Gries, S, (Eds.), 2007. Grammar without grammaticality (Special Issue of Corpus Linguistics and Linguistic Theory).Stephany, U., Bast, C., Lehmann, K., 2001. Computer-Assisted Transcription and Analysis of Speech. (Arbeitspapier N.F. 41). Institut für Linguistik, Köln.

<www.uni-koeln.de/phil-fak/ifl/asw/forschung/ap/childes.pdf>Sternefeld, W. (Ed.), 2007. Data in generative grammar. Theoretical Linguistics 33 (3).