No Sentence Is Too Confusing To Ignore

  • Published on

  • View

  • Download

Embed Size (px)


  • Proceedings of the 2010 Workshop on NLP and Linguistics: Finding the Common Ground, ACL 2010, pages 6169,Uppsala, Sweden, 16 July 2010. c2010 Association for Computational Linguistics

    No sentence is too confusing to ignore

    Paul CookDepartment of Computer Science

    University of TorontoToronto, Canada

    Suzanne StevensonDepartment of Computer Science

    University of TorontoToronto, Canada


    We consider sentences of the form NoX is too Y to Z, in which X is a nounphrase, Y is an adjective phrase, and Zis a verb phrase. Such constructions areambiguous, with two possible (and oppo-site!) interpretations, roughly meaning ei-ther that Every X Zs, or that No X Zs.The interpretations have been noted to de-pend on semantic and pragmatic factors.We show here that automatic disambigua-tion of this pragmatically complex con-struction can be largely achieved by us-ing features of the lexical semantic prop-erties of the verb (i.e., Z) participating inthe construction. We discuss our experi-mental findings in the context of construc-tion grammar, which suggests a possibleaccount of this phenomenon.

    1 No noun is too adjective to verb

    Consider the following two sentences:

    (1) No interest is too narrow to deserve its ownnewsletter.

    (2) No item is too minor to escape his attention.

    Each of these sentences has the form of No X is tooY to Z, where X, Y, and Z are a noun phrase, ad-jective phrase, and verb phrase, respectively. Sen-tence (1) is generally taken to mean that every in-terest deserves its own newsletter, regardless ofhow narrow it is. On the other hand, (2) is typi-cally interpreted as meaning that no item escapeshis attention, regardless of how minor it is. Thatis, sentences with the identical form of No X is tooY to Z either can mean that every X Zs, or canmean the oppositethat no X Zs!1

    1Note that in examples (1) and (2), the nouns interest anditem are the subjects of the verbs deserve and escape, respec-

    This verbal illusion (Wason and Reich, 1979),so-called because there are two opposite inter-pretations for the very same structure, is of in-terest to us for two reasons. First, the con-tradictory nature of the possible meanings hasbeen explained in terms of pragmatic factors con-cerning the relevant presuppositions of the sen-tences. According to Wason and Reich (1979)(as explained in more detail below), sentencessuch as (2) are actually nonsensical, but peoplecoerce them into a sensible reading by revers-ing the interpretation. One of our goals in thiswork is to explore whether computational lin-guistic techniquesspecifically automatic corpusanalysis drawing on lexical resourcescan helpto elucidate the factors influencing interpretationof such sentences across a collection of actual us-ages.

    The second reason for our interest in this con-struction is that it illustrates a complex ambigu-ity that can cause difficulty for natural languageprocessing applications that seek to semanticallyinterpret text. Faced with the above two sen-tences, a parsing system (in the absence of spe-cific knowledge of this construction) will presum-ably find the exact same structure for each, giv-ing no basis on which to determine the correctmeaning from the parse. (Unsurprisingly, whenwe run the C&C Parser (Curran et al., 2007) on (1)and (2) it assigns the same structure to each sen-tence.) Our second goal in this work is thus to ex-plore whether increased linguistic understandingof this phenomenon could be used to disambiguatesuch examples automatically. Specifically, we usethis construction as an example of the kind ofdifficulties faced in semantic interpretation whenmeaning may be determined by pragmatic or otherextra-syntactic factors, in order to explore whether

    tively. In this construction the noun can also be the object ofthe verb, as in the title of this paper which claims no sentencecan/should be ignored.


  • lexical semantic features can be used as cues toresolving pragmatic ambiguity when a complexsemantico-pragmatic model is not feasible.

    In the remainder of this paper, we present thefirst computational study of the No X is too Y toZ phenomenon, which attempts to automaticallydetermine the meaning of instances of this seman-tically and pragmatically complex construction. InSection 2 we present previous analyses of thisconstruction, and our hypothesis. In Section 3,we describe the creation of a dataset of instancesthat verifies that both interpretations (every andno) indeed occur in corpora. We then analyzethe human annotations in this dataset in more de-tail in Section 4. In Section 5, we present the fea-ture model we use to describe the instances, whichtaps into the lexical semantics and polarity of theconstituents. In Section 6, we describe machinelearning experiments and classification results thatsupport our hypothesis that the interpretation ofthis construction largely depends on the semanticsof its component verb. In Section 7 we suggest thatour results support an analysis of this phenomenonwithin construction grammar, and point to somefuture directions in our research in Section 8.

    2 Background and our proposal

    The No X is too Y to Z construction was investi-gated by Wason and Reich (1979), and discussedmore recently by Pullum (2004) and Liberman(2009a,b). Here we highlight some of the mostimportant properties of this complex phenomenon.Our presentation owes much to the lucid discus-sion and clarification of this topic, and of the workof Wason and Reich specifically, by Liberman.

    Wason and Reich argue that the compositionalinterpretation of sentences of the form of (1) and(2) is every X Zs. Intuitively, this can be under-stood by considering a sentence identical to sen-tence (1), but without a negative subject: This in-terest is too narrow to deserve its own newslet-ter, which means that this interest is so narrowthat it does not deserve a newsletter. This ex-ample indicates that the meaning of too narrowto deserve its own newsletter is so narrow thatit does not deserve a newsletter. When this neg-ative too assertion is compositionally combinedwith the No interest subject of sentence (1), it re-sults in a meaning with two negatives: No inter-est is so narrow that it does not deserve a newslet-ter, or simply, Every interest deserves a newslet-

    ter. Wason and Reich note that in sentences suchas (1), the compositional every interpretation isconsistent with common beliefs about the world,and thus refer to such sentences as pragmatic.

    By contrast, the compositional interpretation ofsentences such as (2) does not correspond to ourcommon sense beliefs. Consider an analogous(non-negative subject) sentence to sentence (2)i.e., This item is too minor to escape his attention.It is nonsensical that This item is so minor thatit does not escape his attention, since being moreminor entails more likelihood of escaping atten-tion, not less. The compositional interpretation of(2) is similarly nonsensicali.e., that No itemis so minor that it does not escape his attention;Such sentences are thus termed non-pragmaticby Wason and Reich, who argue that the com-plexity of the non-pragmatic sentencesarising inpart due to the number of negations they containcauses the listener or reader to misconstrue them.According to their reasoning, listeners choose aninterpretation that is consistent with their beliefsabout the worldnamely that no X Zs, in thiscase that No item escapes his attentioninsteadof the compositional interpretation (Every itemescapes his attention).

    While Wason and Reich focus on the compo-sitional semantics and pragmatics of these sen-tences, they also note that the non-pragmatic ex-amples typically use a verb that itself has someaspect of negation, such as ignore, miss, and over-look. This property is also pointed out by Pullum(2004), who notes that avoid in his example ofthe construction means manage to not do some-thing. Building on this observation, we hypothe-size that lexical properties of the component con-stituents of this construction, particularly the verb,can be important cues to its semantico-pragmaticinterpretation. Specifically, we hypothesize thatthe pragmatic (every interpretation) and non-pragmatic (no interpretation) sentences will tendto involve verbs with different semantics. Giventhat verbs of different semantic classes have differ-ent selectional preferences, we also expect to seethe every and no sentences associated with se-mantically different nouns and adjectives.

    3 Dataset

    3.1 Extraction

    To create a dataset of usages of the constructionno NP is too AP to VPreferred to as the tar-


  • get constructionwe use two corpora: the BritishNational Corpus (Burnard, 2000), an approxi-mately one hundred million word corpus of late-twentieth century British English, and The NewYork Times Annotated Corpus (Sandhaus, 2008),approximately one billion words of non-newswiretext from the New York Times from the years19872006. We extract all sentences in these cor-pora containing the sequence of strings no, is too,and to separated by one or more words. We thenmanually filter all sentences that do not have noNP as the subject of is too, or that do not have toVP as an argument of is too. After removing dupli-cates, this results in 170 sentences. We randomlyselect 20 of these sentences for development data,leaving 150 sentences for testing.

    Although we find only 170 examples of thetarget construction in 1.1 billion words of text,note that our extraction process is quite strict andmisses some relevant usages. For example, we donot extract sentences of the form Nothing is too Yto Z in which the subject NP does not contain theword no. Nor do we extract usages of the relatedconstruction No X is too Y for Z, where Z is an NPrelated to a verb, as in No interest is too narrowfor attention. (We would only extract the latter ifthere were an infinitive verb embedded in or fol-lowing the NP.) In the present study we limit ourconsideration to sentences of the form discussedby Wason and Reich (1979), but intend to con-sider related constructions such as thesewhichappear to exhibit the same ambiguity as the targetconstructionin the future.

    We next manually identify the noun, adjective,and verb that participate in the target constructionin each sentence. Although this could be done au-tomatically using a parser (e.g., Collins, 2003) orchunker (e.g., Abney, 1991), here we want to en-sure error-free identification. We also note a num-ber of sentences containing co-ordination, such asin the following example.

    (3) These days, no topic is too recent orspecialized to disqualify it from museumapotheosis.

    This sentence contains two instances of the tar-get construction: one corresponding to the noun-adjective-verb triple topic, recent, disqualify, andthe other to the triple topic, specialized, disqual-ify. In general, we consider each unique noun-adjective-verb triple participating in the target con-struction as a separate instance.

    3.2 Annotation

    We used Amazon Mechanical Turk (AMT, to obtain judge-ments as to the correct interpretation of each in-stance of the target construction in both the devel-opment and testing datasets. For each instance, wegenerated two paraphrases, one corresponding toeach of the interpretations discussed in Section 1.We then presented the given instance of the targetconstruction along with its two paraphrases to an-notators through AMT, as shown in Table 1. Ingenerating the paraphrases, one of the authors se-lected the most appropriate paraphrase, in theirjudgement, where can in the paraphrases in Ta-ble 1 was selected from can, should, will, and .Note that the paraphrases do not contain the ad-jective from the target construction. In the case ofmultiple instances of the target construction withdiffering adjectives but the same noun and verb,we only solicited judgements for one instance, andused these judgements for the other instances. Inour dataset we observe that all instances obtainedfrom the same sentence which differ only with re-spect to their noun or verb have the same inter-pretation. We therefore believe that instances withthe same noun and verb but a different adjectiveare unlikely to differ in their interpretation.


    Read the sentence below.

    Based on your interpretation of that sen-tence, select the answer that most closelymatches your interpretation.

    Select I dont know if neither answer isclose to your interpretation, or if you arereally unsure.

    That success was accomplished in large part totight control on costs , and no cost is too smallto be scrutinized .

    Every cost can be scrutinized.

    No cost can be scrutinized.

    I dont know.

    Enter any feedback you have about this HIT. Wegreatly appreciate you taking the time to do so.

    Table 1: A sample of the Amazon MechanicalTurk annotation task.


  • We also allowed the judges to optionally enterany feedback about the annotation task which insome casesdiscussed in the following sectionwas useful in determining whether the judgesfound a particular instance difficult to annotate.2

    For each instance of the target construction weobtained three judgements from unique workerson AMT. For approximately 80% of the items,the judgements were unanimous. In the remainingcases we solicited four additional judgements, andused the majority judgement. We paid $0.05 perjudgement; the average time spent on each annota-tion was approximately twenty seconds, resultingin an average hourly wage of about $10.

    The development data was also annotated bythree native English speaking experts (compu-tational linguists with extensive linguistic back-ground, two of whom are also authors of this pa-per). The inter-annotator agreement among thesejudges is very high, with pairwise observed agree-ments of 1.00, 0.90, and 0.90, and correspondingunweighted Kappa scores of 1.00, 0.79, and 0.79.The majority judgements of these annotators arethe same as those obtained from AMT on the de-velopment data, giving us confidence in the reli-ability of the AMT judgements. These findingsare consistent with those of Snow et al. (2008) inshowing that AMT judgements can be as reliableas those of expert judges.

    Finally, we remove a small number of itemsfrom the testing dataset which were difficult toparaphrase due to ellipsis of the verb participatingin the target construction, or an extra negation inthe verb phrase. We further remove one sentencebecause we believe the paraphrases we providedare in fact misleading. The number of sentencesand of instances (i.e., noun-verb-adjective triples)of the target construction in the development andtesting datasets is given in Table 2. 160 of the 199testing instances (80%) have the every interpre-tation, with the remainder having the no inter-pretation.

    4 Analysis of annotation

    We now more closely examine the annotations ob-tained from AMT to better determine the extent to

    2In other cases the comments were more humourous. Inresponse to the following sentence If youve ever yearnedto live on Sesame Street,...