Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 974–982,Chiang Mai, Thailand, November 8 – 13, 2011. c©2011 AFNLP

A Character-Level Machine Translation Approach for Normalizationof SMS Abbreviations

Deana L. PennellThe University of Texas at Dallas

800 W. Campbell RoadRichardson, TX 75080

deana@hlt.utdallas.edu

Yang LiuThe University of Texas at Dallas

800 W. Campbell RoadRichardson, TX 75080

yangl@hlt.utdallas.edu

Abstract

This paper describes a two-phase methodfor expanding abbreviations found in in-formal text (e.g., email, text messages,chat room conversations) using a machinetranslation system trained at the charac-ter level during the first phase. In thisway, the system learns mappings betweencharacter-level “phrases” and is muchmore robust to new abbreviations than aword-level system. We generate transla-tion models that are independent of theway in which the abbreviations are formedand show that the results show little degra-dation compared to when type-dependentmodels are trained. Our experiments ona large data set show our proposed systemperforms well when tested both on isolatedabbreviations and, with the incorporationof a second phase utilizing an in-domainlanguage model, in the context of neigh-boring words.

1 Introduction

Text messaging (SMS) is a rapidly growing formof alternative communication for cell phones. Thispopularity has caused safety concerns leadingmany US states to pass laws prohibiting textingwhile driving. The technology is also difficult forusers with visual impairments or physical hand-icaps to use. We believe a text-to-speech (TTS)system for cell phones can decrease these prob-lems to promote safe travel and ease of use for all.

Text normalization is the usual first step forTTS. Text message lingo is also similar to the chat-speak that is prolific on forums, blogs and chat-rooms. Screen readers will thus benefit from suchtechnology, enabling visually impaired users totake part in internet culture. In addition, normal-izing informal text is important for various lan-

guage processing tasks, such as information re-trieval, summarization, and keyword, topic, sen-timent and emotion detection, which are currentlyreceiving a lot of attention for informal domains.

Normalization of informal text is complicatedby the large number of abbreviations used. Someprevious work on this problem used phrase-basedmachine translation (MT) for SMS normalization;however, a large annotated corpus is required forsuch a supervised learning method since the learn-ing is performed at the word level. By definition,this method cannot make a hypothesis for an ab-breviation it did not see in training. This is a se-rious limitation in a domain where new words arecreated frequently and irregularly.

We propose a two-phase approach. In the firstphase, an MT model is trained at the character-level rather than the word- or phrase-level, allow-ing recognition of common abbreviation patternsregardless of the words in which they appear. Wedecode the hypotheses in the second phase using alanguage model for the final prediction.

2 Related Work

Text normalization is an important first stepfor any text-to-speech (TTS) system and hasbeen widely studied in many formal domains.Sproat et al. (2001) provides a good resource fortext normalization and its associated problems.Spell-checking algorithms are mostly ineffectiveon this type of data because they do not account forthe phenomena in informal text. Some prior workinstead focused on single typographic errors usingedit distance (Kukich, 1992), perhaps combinedwith pronunciation modeling, such as (Toutanovaand Moore, 2002).

One line of research views normalization as anoisy channel problem. Choudhury et al. (2007)describe a supervised noisy channel model usingHMMs for SMS normalization. Cook and Steven-son (2009) extend this work to create an unsuper-

974

vised noisy channel approach using probabilisticmodels for common abbreviation types and choos-ing the English word with the highest probabilityafter combining the models. Deletion-based ab-breviations were addressed in our past work usingstatistical models (maximum entropy and condi-tional random fields) combined with an in-domainlanguage model (LM) (Pennell and Liu, 2010;Pennell and Liu, 2011). Liu et al. (2011) extendthe statistical model to be independent of abbrevi-ation type with good results.

Whitelaw et al. (2009) used a noisy channelmodel based on orthographic edit distance usingthe web to generate a large set of automaticallygenerated (noisy) pairs to be used for training andfor spelling suggestions. Although they use theweb for collection, they do not focus on informaltext but rather on unintentional spelling mistakes.Beaufort et al. (2010) combine a noisy channelmodel with a rule-based finite-state transducer andgot reasonable results on French SMS, but havenot tried their method on English text. Han andBaldwin (2011) first determine whether a givenout-of-vocabulary (OOV) word needs to be ex-panded or is some other type of properly-formedOOV. For those predicted to be ill-formed, a con-fusion set of possible candidate words is generatedbased on a combination of lexical and phoneticedit distance and the top word is chosen in con-text using LM and dependency parse information.

Machine translation (MT) techniques trained atthe word- or phrase-level are also common. Trans-lation of SMS from one language to another ledBangalore et al. (2002) to use consensus trans-lations to bootstrap a translation system for in-stant messages and chat rooms where abbrevia-tions are common. Aw et al. (2006) view SMSlingo as if it were another language with its ownwords and grammar to produce grammatically cor-rect English sentences using MT. Q. and H. (2009)trained an MT system using three on-line SMSdictionaries for normalizing chat-like messages onTwitter. Kobus et al. (2008) incorporate a sec-ond phase in the translation model that maps char-acters in the texting abbreviation to phonemes,which are viewed as the output of an automaticspeech recognition (ASR) system. They use a non-deterministic phonemic transducer to decode thephonemes into English words. The technical paperof Raghunathan and Krawczyk (2009) details anexplicit study varying language model orders, dis-

tortion limit and maximum phrase length allowedby the MT system during decoding. Contractor etal. (2010) also uses an SMT model; however, inan attempt to get around the problem of collectingand annotating a large parallel corpus, they auto-matically create a noisy list of word-abbreviationpairs for training using some heuristics. As far aswe know, we are the first to use an MT system atthe character-level for this task.

3 Data

Due to the lack of a large parallel corpus suitablefor our study, we are building a corpus using statusupdates from twitter.com. Twitter allows users toupdate status messages by sending an SMS mes-sage to a number designated for the Twitter ser-vice. To ensure that our corpus is representativeof the domain we are modeling, we use Twitter’smeta-data to collect only messages sent via SMS.Some examples of highly abbreviated messagesfrom our corpus are shown below.(a) Aye.oops,dat was spose to be a

txt

(b) Rndm fct bout wife: n thepast 10 yrs I can cnt on onehand the num Xs she’s 4gotn tounlock my car door

(c) OMG I LOVE YOU GUYS. You pwn:) !!!

(d) i need to qo to bedd qottawakee up at 7am for school....

(e) heard it again! xD 3 TIMES.asng i nvr hear!

3.1 Choosing Messages for AnnotationAn annotator’s time is wasted if he is presentedwith many messages containing no abbreviations,or with sentences all containing the same, verycommon abbreviations. A scoring system wasthus devised using the following metrics:

1. Word Count Index. A low word count indexindicates that a message is close to the meanmessage length. Messages with fewer thanfive words are removed from consideration.We calculate the index as |N−E(N)|/σ(N),where N is the number of words in the mes-sage and mean and standard deviation are cal-culated over the entire corpus.

2. Perplexity Scores. Two perplexity scores arecalculated against character-level language

975

models. A low perplexity of the messagecompared to standard English text indicatesthat the message is less likely to be in a for-eign language or jibberish. Similarly, a lowperplexity of the message compared to ourcorpus indicates that the message is more rep-resentative of the domain. A sentence is re-moved if in either case the perplexity value isgreater than a threshold (1000 in our study).

3. OOV Count. This is a simple count of thenumber of out of vocabulary (OOV) words inthe message compared to an English dictio-nary, which we denote NOOV . This metrichelps guarantee that we select messages con-taining many OOV words. We remove thesentence completely when NOOV = 0.

4. OOV Percentages. This metric consists oftwo scores: the first is NOOV /N ; the secondis a non-duplicate OOV percentage, wherewe remove all repeated words and then re-calculate the percentage. If the first score isgreater than 0.5 but the second is not, we re-move the message from consideration.

5. OOV Frequency Scores. For each OOV to-ken (including emoticons) we find the fre-quency of the token across the entire corpus.This ensures that we annotate those abbrevi-ations that are commonly used.

A sorted list is generated for each metric. Thefinal score for each sentence is a weighted aver-age of its position in each list, with more weightgiven to the non-duplicate OOV percentage andless weight given to the OOV frequency scores.The sentences are ranked in a penultimate listbased on this final score. Finally, a post process-ing step iterates through the list to remove sen-tences introducing no new OOV words comparedto higher-ranked sentences. Messages were anno-tated in the order of rank.

3.2 AnnotationFive undergraduate students were hired for the an-notation task. In total, 4661 twitter status mes-sages were annotated. Of these, 74 were given tomultiple annotators so that we can calculate inter-annotator agreement. All 74 of these messageswere also annotated by the first author as a stan-dard for comparison. There were 7769 tokens an-notated as an abbreviation by at least one annotatorwith 3761 (48%) unique to a single annotator.

We calculate pairwise agreement for whether ornot a token is an abbreviation for all tokens givento each pair, including those that both agreed werenot abbreviations. The Fleiss’ Kappa κ = 0.891for these pairwise values is quite high so the num-ber of annotators can be reduced without negativeeffects to the project as long as they are familiarwith the domain.

4 Normalization Approach

We describe a two-phase approach for SMS nor-malization. The first phase uses a character-levelMT system to generate possible hypotheses foreach abbreviation. The second uses a languagemodel (LM) to choose a hypothesis in context.

Typically, an SMT system translates a sentencefrom one language to another. An alignment steplearns a mapping of words and phrases betweenthe two languages using a training corpus of paral-lel sentences. During testing, this mapping is usedalong with LMs to translate a sentence from onelanguage to another. While researchers have usedthis method to normalize abbreviations (Bangaloreet al., 2002; Aw et al., 2006; Kobus et al., 2008; Q.and H., 2009; Contractor et al., 2010), it is not ro-bust to new words and leads to poor accuracy inthis domain where new terms are used frequentlyand inconsistently.

Our system differs in that we train the MTmodel at the character-level during the first phase;that is, rather than learning mappings betweenwords and phrases we instead map characters toother characters in what can be a many-to-manymapping. For example, the ending “-er” (as in“teacher” or “brother”) is often abbreviated withthe single character ‘a’. Characters may also bemapped to symbols (“@” for “at”), numbers (“8”for “ate”) or nothing at all (deletions).

Formally, for an abbreviation a :c1(a), c2(a), ..., cm(a) (where ci(a) is the ith

character in the abbreviation), we use an MTsystem to find the proper word hypothesis:

w = argmax p(w|a) (1)

= argmax p(w)p(a|w)= argmax p(c1(w), ...cn(w))

×p(c1(a), ..., cm(a)|c1(w), ...cn(w))where ci(w) is a character in the English wordw, p(c1(w), ...cn(w)) is obtained using a charac-ter LM, and p(c1(a), ..., cm(a)|c1(w), ...cn(w)) isbased on the learned phrase translation table.

976

Pairs of words from our annotated data are usedfor training: the original token (which may or maynot be an abbreviation) and its corresponding En-glish word. We removed tokens which the annota-tors were unable to translate and those marked assound effects (e.g., ahhhhh, zzzzz, blech, hrmpf,etc.). Punctuation was removed from the remain-ing data, excluding that found in emoticons (whichwe treat as words) and apostrophes in commoncontractions and possessive forms. To facilitatecharacter-level training, we insert a space betweeneach character and replace any spaces with the un-derscore character.

Because training is done at the character-level,each hypothesis (w) for an abbreviation (a) is asequence of characters, which may or may not bea valid word. To see why, examine the partialphrase-table shown in Figure 11; using this table,“hab” could be translated to “hib”, “habulary” or“habou” as well as the word “have”. It is also verylikely that a hypothesis will appear many timesin the hypothesis list due to different segmenta-tion (two characters may be generated by a sin-gle phrase mapping or by two different mappings,one for each character). We generate 20 distincthypotheses for each token and then eliminate hy-potheses that do not occur in the CMU Lexicon.

Until this point, we have only normalized a sin-gle abbreviation without context. In a realisticsetting contextual information is available to helpwith translation. We thus introduce a second phaseusing a word-level LM to disambiguate hypothe-ses when context is available. This is the typicalnoisy channel model used for speech recognitionor MT decoding and is comparable to equation 1.

Determining the standard English sentence,W = w1w2...wn, from a given informal sentence,

1Aside from the possible translations, the phrase tablealso shows five values for each word. They correspond tothe inverse phrase translation probability φ(f |e), the inverselexical weighting lex(f |e), the direct phrase translation prob-ability φ(e|f), the direct lexical weighting lex(e|f) and thephrase penalty (always exp(1) = 2.718), where e and f arethe English and foreign phrases, respectively. We do not cur-rently make use of these values.

A = a1a2...an, can be formally described as:

W = argmaxP (W |A) (2)

= argmaxP (W )P (A|W )

≈ argmax∏

P (wi|wi−n+1...wi−1)

×∏

P (ai|wi)

= argmax(∑

logP (wi|wi−n+1...wi−1)

+∑

logP (ai|wi))

where the approximation is based on the assump-tion that each abbreviation depends only on thecorresponding word (we are not considering one-to-many mappings in this study), and a word isdependent on its previous (n− 1) words. In otherwords, this probability is represented by a tradi-tional n-gram LM.

The abbreviation score in Equation 2, P (ai|wi),represents the likelihood that abbreviation ai is de-rived from word wi, and can be obtained from:

p(ai|wi) ∝p(wi|ai)p(wi)

(3)

where p(wi|ai) is the abbreviation model (AM)score from the character-level MT system, andp(wi) is from the character LM we used in MTdecoding. In this study, we just use the score fromthe MT system as the likelihood score, withoutdividing by the character-level LM contribution.This is equivalent to using both character- and aword-level LMs during decoding.

Equation 2 assumes that the AM and LM shouldbe weighted equally, but in actuality one modelmay prove to be more helpful than the other. Forthis reason, we allow the terms from equation 2 tobe weighted differently, yielding the final equation

W = argmax(α∑

logP (wi|wi−n+1...wi−1)

+β∑

logP (ai|wi)) (4)

where α and β are determined empirically.

5 Experiment 1: Evaluation on IsolatedAbbreviations

We use the Moses MT system (Koehn et al., 2007)in all our experiments, including Giza++ for align-ment. The score assigned to a translation is de-rived from four models: the phrase translation ta-ble, the language model, the reordering model,

977

Figure 1: An excerpt from a phrase table showing possible translations when the character sequence “ab”is found in a message.

and a word length penalty. We use Moses’ de-fault weights for these models, but plan to ad-just the weights in the future to determine eachmodel’s contribution to a good translation. Foreach abbreviation we generate 20 distinct hypothe-ses. We finally remove non-word hypotheses us-ing the CMU lexicon. Note that occasionallyMoses generates a positive score (impossible fora real log-probability). When this occurs, we usethe value -0.1 instead, indicating that this pair isvery likely.

Note that our accuracy is bounded from aboveby the percentage of pairs in testing that appear inthe 20-best hypotheses generated by Moses. Forthis reason, we first wish to find a setup to maxi-mize the upper bound on our performance.

5.1 Experimental Variations

There are different factors that may affect the per-formance of our system for individual words: thecharacter-level LM used for decoding, the way themodels are trained, and the data used to train thosemodels. We thus evaluate different experimentalvariations, described below.

Training/Testing ConfigurationTo test the effect of using more specific versusmore general data when training the MT system,we used the following training sets:

1. General Setup: Training and testing are both

Formation Method ExampleDeletions ppl (people)

Substitutions 2nite (tonight)Repetitions yeeeeesssss (yes)

Swaps tounge (tongue)Insertions borded (bored)

Table 1: Examples of major abbreviation types.

done using all of the annotated data, includ-ing words that are already in standard form inthe message.

2. Single-word Abbreviation Setup: In train-ing, we remove those abbreviations corre-sponding to multiple words (for example,“brb” representing “be right back”) and thosewords already in standard form. Again wetest the system using all of the annotated data.

3. Type-dependent Setup: We train a separatemodel for each major abbreviation type (seeTable 1) except insertions. The type of eachabbreviation in the test set has been annotatedso we use its respective model for testing.We assumed that knowledge about the ab-breviation formation method would increasethe model’s ability to translate the abbrevia-tion. Currently there is no known procedurefor automatically determining the abbrevia-tion type without having prior knowledge ofits translation, so this setup is just for compar-ison purpose to evaluate how well the moregeneral methods are performing.

Language Model (LM) OrderWe wanted to determine how the order of thecharacter-level LM used by Moses affected per-formance. A smaller order model may not capturelarger phrases or long distance dependencies, butthe smaller model may generalize better due to thesparseness of many high-order phrases.

We explore different character LMs used in de-coding to generate word hypotheses, includingcharacter-based 3-, 5-, and 7-gram LMs. Theseare trained on a subset of 16,543,813 messagesfrom the Edinburgh Twitter corpus (Petrovic etal., 2010) containing no OOV words compared toan English dictionary. Each model is used in theabove training/testing configurations.

978

Order 3 Order 5 Order 7Top-1 Top-20 Top-1 Top-20 Top-1 Top-20

All (14611) 59.30 81.32 62.62 81.49 63.30 81.95Abbreviations (2842) 11.75 53.24 30.33 62.95 33.04 63.83Standard Form (10629) 76.89 94.39 75.86 91.55 75.91 91.81Deletions (1406) 8.68 52.63 31.58 62.87 34.71 63.51Substitutions (620) 9.35 37.91 22.58 51.94 25.32 52.58Repetitions (510) 23.14 81.37 32.55 79.02 34.90 79.80Swaps (106) 33.02 70.75 69.81 86.79 66.04 90.57Insertions (55) 0.00 27.27 14.55 49.09 20.00 56.36Combination (140) 0.71 23.57 21.43 43.57 25.00 43.57

Table 2: Accuracy using the General setup (%).

Order 3 Order 5 Order 7Top-1 Top-20 Top-1 Top-20 Top-1 Top-20

All (14611) 75.97 91.23 75.75 92.50 75.57 92.53Abbreviations (2842) 14.53 63.16 36.45 72.31 37.40 72.38Standard Form (10629) 92.47 98.79 86.31 97.93 85.84 97.95Deletions (1406 ) 11.38 64.15 39.90 74.04 40.61 74.11Substitutions (620) 11.13 48.23 25.48 60.80 27.42 61.45Repetitions (510) 28.24 84.31 37.45 82.75 38.24 82.55Swaps (106) 35.85 91.51 76.42 96.23 73.58 95.28Insertions (55) 0.00 36.36 18.18 54.55 20.00 58.18Combination (140) 1.43 33.57 25.00 59.29 27.14 57.14

Table 3: Accuracy using the Single-word setup (%).

5.2 Results

The results from different training/testing config-urations and LM orders are shown in Tables 2, 3and 4. We use a cross-validation setup where wetrain the MT system using the annotations fromfour annotators and test on the data from the fifth.The mean score from the five runs is reported here.In all three tables, we show top-1 and top-20 accu-racy. For top-1 accuracy, the system is consideredcorrect if and only if the top hypothesis given byMoses exactly matches the standard English form.If the correct standard English form is listed any-where in the 20 hypotheses generated by Moses,we count it correct in top-20 accuracy. The top20-accuracy represents the possible gain we canachieve by re-ranking the lists using context.

We break down the results by abbreviation typeto determine any performance difference. The topportion of Tables 2 and 3 shows results for all14611 unique pairs in the annotated data, includ-ing words already in standard form and abbrevi-ations corresponding to multiple words. The fol-lowing two rows show results on abbreviations of

single words and words already in standard form,respectively. The lower portion breaks down theresults by abbreviation type. The rows listing spe-cific types include those abbreviations using onlythat abbreviation method. The final row lists re-sults for those abbreviations formed using multi-ple methods. The type-dependent setup (Table 4)reports results for each type when using the modeltrained for that specific type.2

Removing multi-word abbreviations and wordsalready in standard form from training generallyincreases accuracy. The type-dependent modelusually outperforms the single-word model, as ex-pected. Comparing Tables 3 and 4 shows thatperformance decrease using the type-independentsingle-word model is usually small, helping avoidcostly human annotation of types in the future.Contrary to expectations, the type-independent

2We do not train a separate model for the insertions dueto the small number of abbreviations of that type in our data.Looking at those marked as insertions, most of them appearto be typographical errors caused by “fat fingering” a keypad(e.g. “jusyt” for “just”), although a few appear stylistic innature, e.g.,“confrused” for “confused”.

979

Order 3 Order 5 Order 7Top-1 Top-20 Top-1 Top-20 Top-1 Top-20

Deletions (1406) 11.45 64.08 38.98 73.33 40.11 73.47Substitutions (620) 12.26 55.48 27.58 62.58 27.74 63.06Repetitions (510) 28.63 84.51 39.02 83.33 39.61 83.53Swaps (106) 35.85 97.17 77.36 97.17 77.36 97.17

Table 4: Accuracy using the type-dependent training and testing (%).

model outperformed the type-dependent model forthe deletion type. Examining the abbreviationsformed by combining at least two methods helpsexplain why this occurs. Of the 140 abbreviationscombining abbreviation types, 102 contain at leastone deletion. Adding these abbreviations in train-ing yields additional examples for the model tolearn from.

There is a large increase in accuracy betweenusing a 3-gram versus a 5-gram model. How-ever, the jump from 5-gram to 7-gram is muchsmaller, and in some cases it even decreases per-formance. We thus recommend using a 5-gramcharacter model and training using only single-word abbreviations.

6 Experiment 2: Incorporation ofContextual Information

In many cases, there are many word hypothesesfor a given abbreviation. Without contextual infor-mation the top hypothesis is static. In some casesthe system can use contextual information to sug-gest the correct word even when it may not be themost likely choice using the AM alone. To evalu-ate the effect of context on normalization, we cre-ate a test set containing trigrams using one wordof context on either side of each abbreviation. Ifan abbreviation falls at the beginning or end ofa message we use <s> and </s> for the left orright context, respectively. We replace OOV con-text words with their annotated standard forms.

The top performing setup from Section 5 is usedto build the AM, which is then combined with abigram LM for decoding using equation 2. TheLM uses Kneser-Ney smoothing and was trainedon the same portion of the Edinburgh corpus asused for the character-level models. We do notinclude the <unk> token in the LM because itsprobability is quite high and preliminary tests (notincluded here) show a degradation in performancewhen it is utilized. Instead, we use a log probabil-ity of -99 for all unknown terms seen in testing.

LM-only Full Partial Section 5.23

13.5 69.7 69.3 36.45

Table 5: Accuracy on tests using context (%).

We performed three types of tests in order to de-termine the best setup for decoding using context.

1. LM-only. This system serves as a baselinefor comparison. We eliminate the abbrevia-tion model completely and use only the LMfor decoding. In this case, all unigrams in theLM are considered to be possible candidatesfor each abbreviation.

2. Full. We use the LM in combination withthe abbreviation model using equation 2 withα = 1.0 and β = 0.1 (found empirically).

3. Partial. Due to the small weight found forthe abbreviation model in the previous setup,we wish to show that the abbreviation scoreis helpful for decoding. We use the abbrevi-ation model to generate the candidate wordsfor each abbreviation, but we do not use theabbreviation score in decoding.4

We again performed cross validation by firsttraining an AM using the data from four of the an-notators and then testing on the trigrams formedfrom the final annotator’s data. In total there were4415 context trigrams used in testing. This islarger than the 2842 abbreviations shown in theprevious section because the previous tests usedunique abbreviations, which may in reality corre-spond to multiple words and unique contexts. Ta-ble 5 shows the normalization results. As a ba-sis of comparison, the state-of-the-art Jazzy spell-

3Using single-word training and 5-gram character LM.4We eliminate the AM term from equation 4, yielding

W = argmaxwi∈S(∑

logP (wi|wi−n+1...wi−1) wherethe candidate set S is generated by the MT system. Equiv-alently, we set β = 0.

980

Top-1 Top-3 Top-10 Top-20Jazzy (Idzelis, 2005) 49.86 53.13 54.78 55.44

(Choudhury et al., 2007) 59.9 – 84.3 88.7(Cook and Stevenson, 2009) 59.4 – 83.8 87.8

(Liu et al., 2011)a 58.09 70.96 – –(Liu et al., 2011)b 62.05 75.91 – –

This Work 60.39 74.58 75.57 75.57

Table 6: System comparison on the 303-term test set from (Choudhury et al., 2007).aLetterTran.bLetterTran + Jazzy.

checker (Idzelis, 2005) achieves 38% accuracy onthis dataset.

It is clear that the LM-only setup is not suf-ficient for this task and that our AM is neces-sary. The Full and Partial setups perform simi-larly; while the Full model consistently performsslightly better for all annotators, it is probably nota significant difference. We hope that by optimiz-ing Moses’ parameters we can obtain higher per-formance from the AM.

Compared to the 1-best results from Section 5.2we see that incorporating contextual informa-tion yields significant gain. Our performance isbounded from above by the percentage of correctsolutions that appear in our candidate lists. For thecontextual task, 77.2% of the correct words appearin the lookup lists after the AM is applied, show-ing that we can still improve the decoding pro-cess. Fully 91.7% of the correct words appear inthe LM, meaning that the AM is still eliminatingmany correct answers from consideration. We areinvestigating methods to address these shortcom-ings for our future work.

7 Comparison to Past Work

Although there has been very little work on thistask until quite recently, multiple studies havebeen done using the small 303-term dataset firstused by (Choudhury et al., 2007). For thisreason, we run our system on this small dataset. We also use the state-of-the-art Jazzy spell-checker (Idzelis, 2005) as a baseline.

System comparisons are shown in Table 6. Theresults shown for other systems (except Jazzy) arethose reported in their respective papers; we didnot re-implement their systems. Our system per-forms comparably to the work of (Liu et al., 2011)on this dataset. Although we outperform both(Choudhury et al., 2007) and (Cook and Steven-

son, 2009) in top-1, they outperform our system attop-10 and top-20. With better re-ranking, theirsystems have the potential to outperform ours.One of our goals is thus to obtain better coverage.

8 Conclusions and Future Work

In this paper we presented a two-phase approachusing character-level machine translation for in-formal text normalization. When combined withcontextual information at the word level, our re-sults are state-of-the art, but there is still someroom for improvement.

Although they are far more common, ab-breviations formed by deletion and substitutionmethod have worse performance compared tothose formed by repetition or swap. It may beuseful to create systems specific to those forma-tion types, such as the deletion system from (Pen-nell and Liu, 2010). Provided that these sys-tems are not trained using the MT approach, itis likely that they contain complementary (ratherthan redundant) information. Combining hypothe-ses from multiple sources may increase perfor-mance on these abbreviation types.

Although we incorporate contextual informa-tion in Section 6, the setup is not entirely real-istic. We test only on abbreviations, assumingthat we know which words are abbreviations andwhich are not. A simple heuristic of only ap-plying our system to OOV words (compared to adictionary) may not perform well. Proper nounsare often OOV and should not be expanded, butwe would like to expand “off” (“office”) or “cat”(“category”), which are also dictionary words. Wealso assume there are not other abbreviations tobe expanded in the context. In reality, many userswrite highly abbreviated messages, posing greaterdifficulties for sentence-level decoding. We willaddress sentence-level decoding in future work.

981

ReferencesAiTi Aw, Min Zhang, Juan Xian, and Jian Su. 2006.

A phrase-based statistical model for SMS text nor-malization. In COLING/ACL, pages 33–40, Sydney,Australia.

Srinivas Bangalore, Vanessa Murdock, and GiuseppeRiccardi. 2002. Bootstrapping bilingual data usingconsensus translation for a multilingual instant mes-saging system. In 19th International Conference onComputational Linguistics, pages 1–7, Taipei, Tai-wan.

R. Beaufort, S. Roekhaut, L.A. Cougnon, and C. Fa-iron. 2010. A hybrid rule/model-based finite-stateframework for normalizing SMS messages. In Pro-ceedings of the 48th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 770–779, Uppsala, Sweden. Association for Computa-tional Linguistics.

Monojit Choudhury, Rahul Saraf, Vijit Jain, AnimeshMukherjee, Sudeshna Sarkar, and Anupam Basu.2007. Investigation and modeling of the structureof texting language. International Journal of Docu-ment Analysis and Recognition, 10:157–174.

Danish Contractor, Tanveer A. Faruquie, andL. Venkata Subramaniam. 2010. Unsuper-vised cleansing of noisy text. In Proceedings ofthe 23rd International Conference on Computa-tional Linguistics: Posters, COLING ’10, pages189–196, Stroudsburg, PA, USA. Association forComputational Linguistics.

Paul Cook and Suzanne Stevenson. 2009. Anunsupervised model for text message normaliza-tion. In NAACL HLT Workshop on ComputationalApproaches to Linguistic Creativity, pages 71–78,Boulder, CO.

B. Han and T. Baldwin. 2011. Lexical normalisationof short text messages: Makn sens a# twitter. InProceedings of the 49th Annual Meeting of the Asso-ciation for Computational Linguistics: Human Lan-guage Technologies. Association for ComputationalLinguistics.

Mindaugas Idzelis. 2005. Jazzy: The java open sourcespell checker.

Catherine Kobus, Francois Yvon, and GeraldineDamnati. 2008. Normalizing SMS: Are twometaphors better than one? In 22nd InternationalConference on Computational Linguistics, pages441–448, Manchester, UK.

Philipp Koehn, Hieu Hoang, Alexandra Birch, ChrisCallison-Burch, Marcello Federico, Nicola Bertoldi,Brooke Cowan, Wade Shen, Christine Moran,Richard Zens, Chris Dyer, Ondrej Bojar, AlexandraConstantin, and Evan Herbst. 2007. Moses: opensource toolkit for statistical machine translation. InProceedings of the 45th Annual Meeting of the ACLon Interactive Poster and Demonstration Sessions,

ACL ’07, pages 177–180, Stroudsburg, PA, USA.Association for Computational Linguistics.

Karen Kukich. 1992. Technique for automaticallycorrecting words in text. ACM Computing Surveys,24(4):377–439.

F. Liu, F. Weng, B. Wang, and Y. Liu. 2011. Insertion,deletion, or substitution? normalizing text messageswithout pre-categorization nor supervision. Proc. ofACL-HLT.

Deana Pennell and Yang Liu. 2010. Normalization oftext messages for text-to-speech. In ICASSP, pages4842–4845, Dallas, Texas, USA.

Deana Pennell and Yang Liu. 2011. Toward text mes-sage normalization: Modeling abbreviation genera-tion. In ICASSP, Prague, Czech Republic, May.

Sasa Petrovic, Miles Osborne, and Victor Lavrenko.2010. The edinburgh twitter corpus. In NAACL-HLT Workshop on Computational Linguistics in aWorld of Social Media, pages 25–26, Los Angeles,California, USA.

Carlos A. Henrıquez Q. and Adolfo Hernandez H.2009. A ngram-based statistical machine translationapproach for text normalization on chat-speak stylecommunications. In Content Analysis for the Web2.0 (CAW2.0 2009), Madrid, Spain.

K. Raghunathan and S. Krawczyk. 2009. Cs224n: In-vestigating sms text normalization using statisticalmachine translation.

Richard Sproat, Alan Black, Stanley Chen, ShankarKumar, Mari Ostendorf, and Christopher Richards.2001. Normalization of non-standard words. Com-puter Speech and Language, 15(3):287–333.

Kristina Toutanova and Robert C. Moore. 2002. Pro-nunciation modeling for improved spelling correc-tion. In ACL, pages 144–151, Philadelphia, Penn-sylvania.

C. Whitelaw, B. Hutchinson, G.Y. Chung, and G. El-lis. 2009. Using the web for language independentspellchecking and autocorrection. In Proceedingsof the 2009 Conference on Empirical Methods inNatural Language Processing: Volume 2-Volume 2,pages 890–899. Association for Computational Lin-guistics.

982