Mining and Modeling Relations between Formal and Informal ...zfli/pubs/informal_emnlp08_slides.pdf · Mining and Modeling Relations between Formal and Informal Chinese ... A general

Mining and Modeling Relations between

Formal and Informal Chinese Phrases

from Web Corpora

Zhifei Li and David Yarowsky

Johns Hopkins University

Tuesday, August 18, 2009

Translation: Sorry, i got to go, bye bye

Person-A:!"#$ %&'()*, ++



Person-A:!"#$ %&'()*, ++



Person-A:!"#$ %&'()*, ++



Person-A:!"#$ %&'()*, ++

Person-B:



Person-A:!"#$ %&'()*, ++

Person-B: bye bye



Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C:



Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88



Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88



???

Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88


Language from Mars?


???

Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88


Language from Mars?

88 = bye-bye =


???

Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88

++


Language from Mars?

88 = bye-bye =


???

Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88

++


Language from Mars?

88 = bye-bye =


???

Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88

BaBa BaiBai

++


Language from Mars?

88 = bye-bye =


???

Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88

BaBa BaiBai

++


Language from Mars?

88 = bye-bye =


???

Our task: mining formal-informal relations

from web

Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88

BaBa BaiBai

++


Language from Mars?

88 = bye-bye =


???

Our task: mining formal-informal relations

from web

Identify informal phrases

Find their formal equivalents

Person-A:!"#$ %&'()*, ++

Person-B: bye bye

Person-C: 88

BaBa BaiBai

++


Why is this task important?

Enormous amount of informal text on the web

Online-chat, Bulletin Board System (BBS), Web

Blog, and so on

New informal phrases are created everyday

A general text-processing system does not work

well on informal text

machine translation

information retrieval

Our method can be used for text normalization


Outline

Rule-driven intuition

Data-driven intuition

Our approach

a log-linear model combining both rule- and data-

driven intuitions

Experimental results


Data Collection

We manually collect about 900 examples

used for both training and testing using cross-validation

We then classify the examples into different

classes based on our knowledge in Chinese and

linguistics


Homophones


Homophones

Similar pronunciation but different meaning or

written-form


Homophones


written-form

Formal Informal Percentage

!! (BanZhu) [system administrator] !! (BanZhu) [bamboo] 4%

!! (XiHuan)[like] "! (XiFan)[gruel] 4%

!! (BaiBai)[bye-bye] 88 (BaBa) 21%


!!!! (MeiGuoJunDui)[american army] !! (MeiJun)[american army] 4%

!! (GeGe)[elder brother] GG 12%

!!! (NuPengYou)[girl friend] GF 7%

For illustrative purposes, we can present thetransformation path showing how the informalphrase is obtained from the formal phrase. In par-ticular, the transformation path for this category is“Formal ! PinYin ! Informal (similar or samePinYin as the formal phrase)”.

2.2 Abbreviation and AcronymA Chinese abbreviation of a formal phrase is ob-tained by selecting one or more characters from thisformal phrase, and the selected characters can be atany position in the formal phrase (Li and Yarowsky,2008; Lee, 2005; Yin, 1999). In comparison, anacronym is a special form of abbreviation, whereonly the first character of each word in the formalphrase is selected to form the informal phrase. Table?? presents three examples belonging to this cate-gory. While the first example is an abbreviation, andthe other two examples are acronyms.

The transformation path for the second exam-ple is “Formal ! PinYin ! Acronym”, and thetransformation path for the third example is “For-mal! English! Acronym”. Clearly, they differ inwhether PinYin or English is used as a bridge.

2.3 TransliterationA transliteration is transcribing a word or text writ-ten in one writing system into another writing sys-tem. Table ?? presents examples belonging to thiscategory. In the first example, the Chinese infor-

mal phrase “!" (FenSi)[a chinese food]” can bethought as a transliteration of the English phase“fans” as the pronunciation of “fans” is quite sim-ilar to the PinYin “FenSi”.

The transformation path for this category is “For-mal! English! Chinese Transliteration”.

2.4 OthersDue to the inherently informal and flexible nature ofexpressions in informal genre, the formation of aninformal phrase can be very complex or ad-hoc. Forexample, an informal phrase can be generated by ap-plying the above transformation rules jointly. Moreimportantly, many relations cannot be described us-ing a simple set of rules. Table ?? presents threesuch examples, where the first two examples aregenerated by applying rules jointly and the third ex-ample is created by decomposing the Chinese char-acters in the formal form. The statistics collectedfrom the 904 examples tells us that about 45% ofthe relations belonging to this category. This mo-tivates us to use a data-driven method to automat-ically discover the relations between informal andformal phrases.

3 Data Co-occurrenceIn natural language, related words tend to appear to-gether (i.e., co-occurrence). For example, Bill Gatestends to appear together with Microsoft more of-ten than expected by chance. Such co-occurrence


Homophones


written-form


















Homophones


written-form


















Homophones


written-form


















Homophones


written-form


















Homophones


written-form


















Homophones


written-form


















Homophones


written-form


















Homophones


written-form

Formal


















Homophones


written-form

Formal


















Homophones


written-form

Formal

















Pin Yin


Homophones


written-form

Formal

















Pin Yin


Homophones


written-form

Formal

















Pin Yin Homophone


Abbreviations and Acronyms















































































































Formal



















Formal



















Formal Pin Yin



















Formal Pin Yin



















Formal Pin Yin Acronym




















Formal




















Formal




















Formal English




















Formal English




















Formal English Acronym




















Formal English Acronym


















Transliterations


Transliterations


!! (GeMi)[fans] !! (FenSi)[a chinese food] 2 %

!! (XieXie)[thank you] 3Q (SanQiu)


!!!!! (XiLaLiFenSi)[fans of Hilary] !! (XiFan)[gruel]45%

!!!!! (AoBaMaFenSi)[fans of Obama] !! (AoFan)

!! (ChaoQiang)[super strong] "!"! (ZouZhaoGongXu)

may imply the existence of a relationship, and is ex-ploited in formal-informal relation discovery underdifferent conditions.

3.1 Data Co-occurrence in DefinitionsIn general, for many informal phrases in popular use,there is likely to be an explicit definition somewherethat provides or paraphrases its meaning for an unfa-miliar audience. People have created dedicated def-inition web-pages to explain the relations betweenformal and informal phrases. For example, the firstexample in Table ?? is commonly explained in manydedicated definition web-pages on the Internet. Onthe other hand, in some formal text (e.g., researchpapers), people tend to define the informal phrasebefore it is used frequently in the later part of thetext. The second example of Table ?? illustrates thisphenomena. Clearly, the definition text normallycontain salient patterns. For example, the first ex-ample follows the “informal!formal!!!” defi-nition pattern, while the second example follows thepattern “formal (informal)”. This gives us a reliableway to seed and bootstrap a list of informal phrasesas will be discussed in Section 4.1.

3.2 Data Co-occurrence in Online ChatInformal phrases appear in online chat very often forinput convenience or just for fun. Since differentpeople may have different ways or traditions to ex-press semantically-equivalent phrases, one may findmany nearby data co-occurrence examples in chattext. For example, in Table 2, after a series of mes-

sage exchanges, person A wants to end the conver-sation and types “""” (meaning “bye-bye”), per-son B later includes the same semantic content, butin a different (more or less formal) expression (e.g.“88”).

...Person A: !"#!#!!#!$Person B: 88Person A: ""

Table 2: Data Co-occurrence in Online Chat for Relation("", 88) meaning “bye-bye”

...Person A: !"#!#!!#!$,""Person B: 88

3.3 Data Co-occurrence in News ArticlesFor some formal-informal relations, since both ofthe informal and formal phrases have been used inpublic very often and people are normally awareof these relations, an author may use the informaland formal phrases interchangeably without bother-ing to explain the relations. This is particularly truein news articles for some well-known relations. Ta-ble 3 shows an example, where the abbreviation “"!!” (meaning “winter olympics”) appears in thetitle and its full-form “""!"!” appears in the


Transliterations














...Person A: !"#!#!!#!$,""Person B: 88



Transliterations














...Person A: !"#!#!!#!$,""Person B: 88



Transliterations














...Person A: !"#!#!!#!$,""Person B: 88



Transliterations














...Person A: !"#!#!!#!$,""Person B: 88



Transliterations

Formal














...Person A: !"#!#!!#!$,""Person B: 88



Transliterations

Formal














...Person A: !"#!#!!#!$,""Person B: 88



Transliterations

Formal English














...Person A: !"#!#!!#!$,""Person B: 88



Transliterations

Formal English














...Person A: !"#!#!!#!$,""Person B: 88



Transliterations

Formal English Transliteration














...Person A: !"#!#!!#!$,""Person B: 88



Other phenomena


Other phenomenaFormal Informal Percentage













...Person A: !"#!#!!#!$,""Person B: 88
















...Person A: !"#!#!!#!$,""Person B: 88
















...Person A: !"#!#!!#!$,""Person B: 88
















...Person A: !"#!#!!#!$,""Person B: 88
















...Person A: !"#!#!!#!$,""Person B: 88


informal and formal text, a text-processing systemtrained on formal text does not typically work wellon informal genres. For example, in a machinetranslation system (Koehn et al., 2007), if the bilin-gual training data does not contain the word “!!” (the second example in Table 1), it leaves theword untranslated. On the other hand, if the word“!!” does appear in the training data but it hasonly a translation “gruel” as that is the meaning inthe formal text, the translation system may wronglytranslate “!!” into “gruel” for the informal textwhere the word “!!” is more likely to mean“like”. Therefore, as a text-normalization step, itis desirable to transform the informal text into itsstandard formal equivalent before feeding it into ageneral-purpose text-processing system. Unfortu-nately, there are many processes for generating in-formal expressions in common use today. Suchtransformations are highly flexible/diverse, and newphrases are invented on the Internet every day due tomajor news events, popular movies, TV shows, ra-dio talks, political activities, and so on. Therefore,it is of great interest to have a data-driven methodthat can automatically find the relations between in-formal and formal expressions.

In this paper, we present a novel method for dis-covering and modeling the relationship between in-formal Chinese expressions found in web corporaand their formal equivalents. Specifically, we im-plement a bootstrapping procedure to identify alist of candidate informal phrases. Given an indi-vidual informal phrase, we retrieve contextual in-stances from the web using a search engine (in thiscase, www.baidu.com), generate hypotheses of for-mal equivalents via this data, and rank the hypothe-ses using a conditional log-linear model. In the log-linear model, we incorporate as feature functionsboth rule-based intuitions and data co-occurrencephenomena (either as an explicit or indirect defini-tion, or through formal/informal usages occurring infree variation in a discourse). We test our system onmanually collected test examples2, and find that the(formal-informal) relationship discovery and extrac-tion process using our method achieves an averageprecision of more than 60%. This work has applica-

2The training and test examples are freely available athttp://www.cs.jhu.edu/!zfli.

tions for text normalization in many general-purposetext-processing systems including machine transla-tion.

To the best of our knowledge, our work is thefirst published machine-learning approach to pro-ductively model the broad types of relationships be-tween informal and formal expressions in Chinese.

2 Formal to Informal: Phenomena andExamples

In this section, we describe the phenomena and pro-vide examples of the relations between formal andinformal expressions in Chinese (we refer to therelation as formal-informal phrases hereafter, evenin the case of single-word expressions). We man-ually collected 908 formal-informal relations, andclassified these relations into four categories. Wecollected these pairs by investigating multiple web-pages where the formal-informal relations are man-ually compiled, and then merged these seed relationsand removed duplicates. In this way, the 908 exam-ples should give good coverage on the typical cat-egories in the formal-informal relations. Also, thedistribution of the categories found in the 908 exam-ples should be representative of the actual distribu-tion of the formal-informal relations occurring in thereal texts. Table ?? presents these categories and ex-amples in each category. In the last column, the tablealso shows the relative frequency of each category,computed based on the 908 examples. Recall thatwe represent Chinese words in the format: Chinesecharacters (optional PinYin equivalent in parenthe-ses and optional English gloss in brackets).!!!!!!

"""!!!"""!!!

2.1 Homophone

In general, a homophone is a word that is pro-nounced the same as another word but differs inmeaning and/or written-form. Here, we use the word“homophone” in a loose way. In particular, we re-fer an informal phrase as a homophone of a formalphrase if its pronunciation is the same or similar tothe formal phrase. In the three examples belongingto the homophone category in Table ??, the first ex-ample is a true homophone, while the other two areloose homophones. The third example represents a















...Person A: !"#!#!!#!$,""Person B: 88








In this section, we describe the phenomena and pro-vide examples of the relations between formal andinformal expressions in Chinese (we refer to therelation as formal-informal phrases hereafter, evenin the case of single-word expressions). We man-ually collected 908 formal-informal relations, andclassified these relations into four categories. Wecollected these pairs by investigating multiple web-pages where the formal-informal relations are man-ually compiled, and then merged these seed relationsand removed duplicates. In this way, the 908 exam-ples should give good coverage on the typical cat-egories in the formal-informal relations. Also, thedistribution of the categories found in the 908 exam-ples should be representative of the actual distribu-tion of the formal-informal relations occurring in thereal texts. Table ?? presents these categories and ex-amples in each category. In the last column, the tablealso shows the relative frequency of each category,computed based on the 908 examples. Recall thatwe represent Chinese words in the format: Chinesecharacters (optional PinYin equivalent in parenthe-ses and optional English gloss in brackets).!!!!!!!!!!!!

2.1 Homophone

In general, a homophone is a word that is pro-nounced the same as another word but differs inmeaning and/or written-form. Here, we use the word“homophone” in a loose way. In particular, we re-fer an informal phrase as a homophone of a formalphrase if its pronunciation is the same or similar tothe formal phrase. In the three examples belongingto the homophone category in Table ??, the first ex-ample is a true homophone, while the other two areloose homophones. The third example represents amajor sub-class where the informal phrase is a num-








2.1 Homophone









"""!!!"""!!!

2.1 Homophone
















...Person A: !"#!#!!#!$,""Person B: 88









2.1 Homophone









2.1 Homophone









2.1 Homophone









2.1 Homophone









"""!!!"""!!!

2.1 Homophone



Other phenomena

Web users are very very creative!





!!! !!! !!! !!!

!!!!!!


Definitions in Real Text

In practice, the corresponding formal

equivalent can sometimes be found near the

informal phrases

direct definitions

indirect associations


Direct Definitions

Definition Text Relation

GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")

Category Precision (%)Top-1 Top-10 Top-50 Top-100

Homophone Same PinYin 31.6 47.4 68.4 73.7Similar PinYin 15.0 35.0 45.0 50.0Number 31.6 64.2 84.2 90.5

Abbreviation Chinese abbreviation 11.8 35.3 41.2 41.2Acronym PinYin Acronym 39.3 82.1 91.1 92.9

English Acronym 3.1 6.3 9.4 28.1

Transliteration 10.0 20.0 20.0 20.0

Average 26.1 53.4 66.3 72.3

Table 5: Rule-driven Features only: Precision on Chinese Formal-informal Relation Extraction






Average 51.4 71.1 81.1 82.7

Table 6: Data-driven Features only: Precision on Chinese Formal-informal Relation Extraction


Direct Definitions


GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7



Direct Definitions


GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7



Direct Definitions


GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7


Pattern: informal,formal -./


Direct Definitions


GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7




Direct Definitions


GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7




Direct Definitions


GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7




Direct Definitions


GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7



Pattern: formal(informal)


Indirect Associations in Online Chat

Formal Informal Fraction (%)

!! (GeMi)[fans] !! (FenSi)[a chinese food] 2.3



!!!!! (XiLaLiFenSi)[fans of Hilary] !! (XiFan)[gruel]44.8









...Person A: !"#!#!!#!$,""Person B: 88

















...Person A: !"#!#!!#!$,""Person B: 88

















...Person A: !"#!#!!#!$,""Person B: 88

















...Person A: !"#!#!!#!$,""Person B: 88


(++$88)
















...Person A: !"#!#!!#!$,""Person B: 88


(++$88)

Bye bye


Indirect Associations in News Articles



Title !!!!!!!!!!!!!"!!!!"

Text !!!!!2!9"!(!!"!!!)!20"!!!!!!!!!!!!!!"!##!#!10"!8""!!!"""""#!"!!"#!!!!



Title !!!!!!!!!!!!!"!!!!"

Text !!!!!2!9"!(!!"!!!)!20"!!!!!!!!!!!!!!"!##!#!10"!8""!!!"""""#!"!!"#!!!!



Title !!!!!!!!!!!!!"!!!!"

Text !!!!!2!9"!(!!"!!!)!20"!!!!!!!!!!!!!!"!##!#!10"!8""!!!"""""#!"!!"#!!!!



Title !!!!!!!!!!!!!"!!!!"

Text !!!!!2!9"!(!!"!!!)!20"!!!!!!!!!!!!!!"!##!#!10"!8""!!!"""""#!"!!"#!!!!



Title !!!!!!!!!!!!!"!!!!"

Text !!!!!2!9"!(!!"!!!)!20"!!!!!!!!!!!!!!"!##!#!10"!8""!!!"""""#!"!!"#!!!!



(01234$024)

Winter Olympic Games

Title !!!!!!!!!!!!!"!!!!"

Text !!!!!2!9"!(!!"!!!)!20"!!!!!!!!!!!!!!"!##!#!10"!8""!!!"""""#!"!!"#!!!!


Mining Relations from the Web



Identifying informal phrases




For each informal phrase, identify its formal

equivalent:





equivalent:

Step-1: retrieving data from the web





equivalent:


Step-2: generating candidate hypotheses (i.e.,

formal phrases)





equivalent:


Step-2: generating candidate hypotheses (i.e.,

formal phrases)

Step-3: ranking hypotheses


Identifying informal phrases: bootstrapping



Seeds



Seeds

Data



Extract

Seeds

Data



Extract

Seeds

Data



Extract

Seeds

Data

Patterns



Extract Identify

Seeds

Data

Patterns



Extract Identify

Seeds

Data

Patterns



Extract Identify

Seeds

Data

Patterns

New phrases



Extract Identify

Seeds

Data

Patterns

New phrases


Seed:!(GF,567)


GF,567-./

89,:;-./

<,=>-./

Seed:!(GF,567)


GF,567-./

89,:;-./

<,=>-./

Seed:!(GF,567)


GF,567-./

89,:;-./

<,=>-./

Seed:!(GF,567)

Pattern:

informal,formal-./


GF,567-./

89,:;-./

<,=>-./

Seed:!(GF,567)

Pattern:

informal,formal-./

New relations:


GF,567-./

89,:;-./

<,=>-./

Seed:!(GF,567)

Pattern:

informal,formal-./

New relations:

(89, :;) (gruel, like)


GF,567-./

89,:;-./

<,=>-./

Seed:!(GF,567)

Pattern:

informal,formal-./

New relations:

(89, :;)

(<, =>) (this, they)

(gruel, like)


GF,567-./

89,:;-./

<,=>-./

Seed:!(GF,567)

Pattern:

informal,formal-./

New relations:

(89, :;)

(<, =>) (this, they)! (gruel, like)


Step-1: Retrieving data from the Web

Given an informal phrase, we retrieve data

from the Web

search the informal phrase using a search engine

(e.g., www.baidu.com)

just search the informal phrase

search the informal phrase with domain information

use a search engine dedicated to informal text, e.g.,

blogsearch.baidu.com

download the relevant webpages returned by the

search engine


http://www.baidu.com








Step-2: Generating Candidate Hypotheses

Data-driven hypothesis generation

we extract n-grams occurring within a certain

distance from the informal phrase






GF,567-./






GF,567-./






GF,567-./






GF,567-./ 1-gram,567-./






GF,567-./ 1-gram,567-./

2-gram,556677--../






GF,567-./ 1-gram,567-./

2-gram,556677--../

3-gram,5656767-7-.-./






GF,567-./ 1-gram,567-./

2-gram,556677--../

3-gram,5656767-7-.-./


Step-3: Ranking Hypotheses: Log-

linear Models

Score of a hypothesis

s(x, y) =K!

i=1

!i(x, y)! !i



linear Models

Probability of a hypothesis


s(x, y) =K!

i=1

!i(x, y)! !i

P!"(y|x) =1

Z(x, !")es(x,y)



linear Models



s(x, y) =K!

i=1

!i(x, y)! !i

P!"(y|x) =1

Z(x, !")es(x,y)

Regularized Conditional Log-likelihood

LLR(!") =N!

j=1

log P!"(yj |xj)!||!"||2

2#2



linear Models



s(x, y) =K!

i=1

!i(x, y)! !i

P!"(y|x) =1

Z(x, !")es(x,y)

!"! = arg max!"

LLR(!")

Regularized Conditional Log-likelihood

LLR(!") =N!

j=1

log P!"(yj |xj)!||!"||2

2#2

Training


Rule-driven Feature Functions

LD-PinYin(x,y)

Levenshtein distance between PinYin of x and y

LEN-D-PinYin(x,y)

Difference in the number of PinYin characters

between x and y

Is-PinYin-Acronym(x,y)

is x a PinYin acronym of y?

Is-CN-Abbreviation(x,y)

is x a Chinese abbreviation of y?



LD-PinYin(x,y)


LEN-D-PinYin(x,y)


between x and y





4.4 Ranking Hypotheses: ConditionalLog-linear Model

Log-linear models are known for flexible incorpora-tion of features into the model. Each feature func-tion reflects a hint/intuition that can be used to rankthe hypotheses. In this subsection, we develop aconditional log-linear model that incorporates boththe rule and data intuitions as feature functions.

4.4.1 Conditional Log-linear ModelGiven an informal phrase (say x) and a candidate

formal phrase (say y), the model assigns the pair ascore (say s(x, y)), which will be used to rank thehypothesis y. The score s(x, y) is a linear combina-tion of the feature scores (say !i(x, y)) over a set offeature functions indexed by i. Formally,

s(x, y) =K!

i=1

!i(x, y)! !i (1)

where K is the number of feature functions definedand !i is the weight assigned to the i-th feature func-tion (i.e., !i). To learn the weight vector "!, we firstdefine a probability measure,

P!"(y|x) =1

Z(x, "!)es(x,y) (2)

where Z(x, "!) is a normalization constant. Now, wedefine the regularized log-likelihood (LLR) of thetraining data (i.e, a set of pairs of (x, y)), as follows,

LLR("!) =N!

j=1

log P!"(yj |xj)"||"!||2

2#2(3)

where N is the number of training examples, and theregularization term ||!"||2

2#2 is a Gaussian prior with avariance #2 (Roark et al., 2007). The optimal weightvector "!! is obtained by maximizing the regularizedlog-likelihood (LLR), that is,

"!! = arg max!"

LLR("!) (4)

To maximize the above function, we use a limited-memory variable method (Benson and More, 2002)that is implemented in the TAO package (Benson etal., 2002) and has been shown to be very effective invarious natural language processing tasks (Malouf,2002).

During test time, the following decision rule isnormally used to predict the optimal formal phrasey! for a given informal phrase x,

y! = arg maxy

s(x, y). (5)

4.4.2 Feature FunctionsAs mentioned before, we incorporate both the

rule- and data-driven intuitions as feature functionsin the log-linear model.

Rule-driven feature functions: Clearly, if a pair(x, y) matches the rule patterns described in Table 2,the pair has a high possibility to be a true formal-informal relation. To reflect this intuition, we de-velop several feature functions as follows.

• LD-PinYin(x, y): the Levenshtein distance onPinYin of x and y. The distance betweentwo PinYin characters is weighted based onthe similarity of pronunciation, for example,the weight w(l, n) is smaller than the weightw(a, z).

• LEN-PinYin(x, y): the difference in the num-ber of PinYin characters between x and y.

• Is-PinYin-Acronym(x, y): is x a PinYinacronym of y? For example,Is-PinYin-Acronym(GG,!!)=1,Is-PinYin-Acronym(GG,!!)=0.

• Is-CN-Abbreviation(x, y): is x a Chinese ab-breviation of y? For example,Is-CN-Abbreviation(!!,!!!")=1,Is-CN-Abbreviation(!!,!!!")=0.

Data-driven feature functions: As described inSection 3, the informal and formal phrases tends toco-occur in the data. Here, we develop several fea-ture functions to reflect this intuition.

• n-gram co-occurrence relative frequency: wecollect the n-grams that occur in the data withina window of the occurrence of the informalphrase, and compute their relative frequencyas feature values. Since different orders ofgrams will have quite different statistics, wedefine 7 features in this category: 1-gram, 2-gram, 3-gram, 4-gram, 5-gram, 6to10-gram,and 11to15-gram. Note that the order n of an-gram is in terms of number of Chinese char-acters instead of words.



LD-PinYin(x,y)


LEN-D-PinYin(x,y)


between x and y









s(x, y) =K!

i=1

!i(x, y)! !i (1)


P!"(y|x) =1

Z(x, "!)es(x,y) (2)


LLR("!) =N!

j=1

log P!"(yj |xj)"||"!||2

2#2(3)



"!! = arg max!"

LLR("!) (4)



y! = arg maxy

s(x, y). (5)














s(x, y) =K!

i=1

!i(x, y)! !i (1)


P!"(y|x) =1

Z(x, "!)es(x,y) (2)


LLR("!) =N!

j=1

log P!"(yj |xj)"||"!||2

2#2(3)



"!! = arg max!"

LLR("!) (4)



y! = arg maxy

s(x, y). (5)











Data-driven Feature Functions

n-gram co-occurrence relative frequency

how often x and y appear together?

Feature on a definition pattern

Does co-occurrence of x and y follow a definition

pattern?

Feature on the number of relevant webpages

search the web using the pair of x and y

get the number of webpages returned


Results on Bootstrapping Informal Phrases

Size of seed set 130Size of candidate set 3000Size of test set 750Recall 30%

Table 8: Recall of Test Set on a Candidate Set Extractedby a Bootstrapping Procedure

helpful than the rule-driven features (e.g, 25.3% ab-solute improvement in 1-best precision), while thecombination of these features does boost the perfor-mance of any individual feature set (e.g., 10.4% ab-solute improvement in 1-best precision over the caseusing data-driven features only).

We also carried out experiments (see Table 8)in the bootstrapping procedure described in Section4.1. In particular, we start from a seed set having130 relations. We identify the frequent patterns fromthe data retrieved from the web for these seed exam-ples. Then, we use these patterns to identify manymore new possible formal-informal relations. Afterthe first iteration, we select the top 3000 pairs of re-lations matched by the patterns. The recall of a man-ually collected test set (having 750 pairs) on these3000 pairs is around 30%, which is quite promisinggiven the highly noisy data.

6 Related Work

Automatically extracting the relations between full-form Chinese phrases and their abbreviations is aninteresting and important task for many NLP appli-cations (e.g., machine translation, information re-trieval, etc.). Recently, Chang and Lai (2004),Chang and Teng (2006), Lee (2005) and Li andYarowsky (2008) have investigated this task. Specif-ically, Chang and Lai (2004) describes a hiddenmarkov model (HMM) to model the relationship be-tween a full-form phrase and its abbreviation, bytreating the abbreviation as the observation and thefull-form words as states in the model. Using aset of manually-created full-abbreviation relationsas training data, they report experimental results ona recognition task (i.e., given an abbreviation, thetask is to obtain its full-form, or the vice versa).Chang and Teng (2006) extends the work in Changand Lai (2004) to automatically extract the relationsbetween full-form phrases and their abbreviations,

where both the full-form phrase and its abbrevia-tion are not given. Clearly, the method in (Changand Lai, 2004; Chang and Teng, 2006) is super-vised because it requires the full-abbreviation rela-tions as training data. Li and Yarowsky (2008) pro-pose an unsupervised method to extract the relationsbetween full-form phrases and their abbreviations.Their method exploits the data co-occurrence phe-nomena in the newswire text, as we have done inthis paper. Moreover, they augment and improve astatistical machine translation by incorporating theextracted relations into the baseline translation sys-tem. Other interesting work that addresses a similartask as ours includes the work on homophones (e.g.,Lee and Chen (1997)), abbreviations with their def-initions (e.g., Park and Byrd (2001)), abbreviationsand acronyms in the medical domain (?), translitera-tion (e.g., (Knight and Graehl, 1998; Wu and Chang,2007)), and so on.

While all the above work deals with the rela-tions occurring within the formal texts, we considerthe formal-informal relations that occur across bothformal and informal texts, and we extract the rela-tions from the web corpora, instead from just formaltexts. Moreover, our method is semi-supervised inthe sense that the weights of the feature functionsare tuned in a supervised log-linear model using asmall number of seed relations while the generationand ranking of the hypotheses are unsupervised byexploiting the data co-occurrence phenomena.

7 Conclusions

In this paper, we have first presented a taxonomy ofthe formal-informal relations occurring in Chinesetexts. We have then proposed a novel method fordiscovering and modeling the relationship betweeninformal Chinese expressions (including colloqui-alisms and instant-messaging slang) and their formalequivalents. Specifically, we have proposed a boot-strapping procedure to identify a list of candidateinformal phrases in web corpora. Given an infor-mal phrase, we retrieved contextual instances fromthe web using a search engine, generated hypothe-ses of formal equivalents via this data, and rankedthe hypotheses using a conditional log-linear model.In the log-linear model, we incorporated as featurefunctions both rule-based intuitions and data co-


Precisions on Relation Extraction




English Acronym 21.9 46.9 56.3 59.4Transliteration 20.0 40.0 50.0 50.0Average 61.8 77.1 83.1 84.7

Table 7: Both Data and Rule-drive Features: Precision on Chinese Formal-informal Relation Extraction























GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7


(using rule-driven features)









GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7



GF!!!!!!!! (!!!, GF)

!!!!!!!!""#!" (!")"!!!!!!……

(!""#!",!")






Average 26.1 53.4 66.3 72.3







Average 51.4 71.1 81.1 82.7


(using rule-driven features)

(using data-driven features)


Summary

We proposed an approach to extract relations

between informal and formal Chinese phrases

from Web corpora

Our approach combines both rule and data-

driven features

The combinations of both rule and data-driven

features outperforms the case just using only

rule or data-driven features


Training and test examples are online at

http://www.cs.jhu.edu/~zfli




Thank you!


Documents

Mining and Modeling Relations between Formal and Informal ...zfli/pubs/informal_emnlp08_slides.pdf · Mining and Modeling Relations between Formal and Informal Chinese ... A general