Language Identification of Search Engine Queries

Hakan Ceylan Yookyung KimDepartment of Computer Science Yahoo! Inc.University of North Texas 2821 Mission College Blvd.Denton,TX,76203 Santa Clara,CA,95054hakan@unt.edu ykim@yahoo-inc.com

ACL 2009

outline

• Introduction• Data Generation• Language Identification• Conclusions and Future Work

Introduction(1)

• Decide in which language a given text is written

• It is heavily studied• It is critical importance to search engines for

queries• Challenges : lack of any standard or publicly

available data set

Introduction(2)

• A case where a correct identification of language is not necessary.

example : query ”homo sapiens” , a user enter this query from Spain. Add a non-linguistic feature to system

Introduction(3)

Data Generation(1)

• Data set : Constructed by the queries with clicked urls From : Yahoo! Search Engine for each language Time : three months time period

Data Generation(2)

• Preprocess : remove any numbers or special characters or

extra spaces. lowercase all the letters of the queries. Calculating the frequencies of the urls for

each query.• A web page is 474 words on the average• Identify the language for web page using one of

the existing methods.

Data Generation(3)

• Using Table 1(T1) and Table 2(T2) to store the above information

T1 : [ q , u , fu ] T2 : [ u , l ] q : query u : a unique url u : url l : language identified for u fu : the frequency of u

• Combine T1 and T2 into T3 T3 : [ q , l , fl , cu,l ]

l : a language fl : the count of clicks for l cu,l : the count of unique urls in language l

Data Generation(4)

• It has many noise. 1. A query maps to more than one language. solve : Giving a weight wq,l for each query to a language set a threshold parameter W if wq,l < W then remove this query

2.navigational query example : ACL 2009

Data Generation(5)

Solve : set two threshold parameter F and U if Fq > F or Uq < U then remove this query• Algorithm

Data Generation(6)

• How to turn our parameter dependent on the size of data set (Silverstein et al.,1999) W = 1 , F = 50 , U = 5

• How many query will be filter 5%~10% of the queries

• Pick 500 queries randomly and annotate them by human

Category-1: If the query does not contain any foreign terms. Category-2: If there exists some foreign terms but the query would still be expected to bring web pages in the same language. Category-3: If the query belongs to other languages, or all the terms are foreign to the annotator.

Data Generation(7)

• How much of this multi-linguality parameter selection eliminate? result : Category-1 : 47.6% Category-1+2 : 60.2%

Language Identification(1)

• Implement three models use a different existing feature

1.statistical model 2.knowledge based model 3.morphological model• EuroParl Corpora• Combine all three models in a machine learning

framework using a novel approach• Add a non-linguistic

Language Identification(2)

• Test set-3500 human annotated queries

Statistical model

• Character based n-gram feature (n=1 to 7)• Vocabulary from training corpus(EuroParl)• Generate a probability distribution from these

count• Above work can use SRILM Toolkit with

Kneser-Ney Discounting and interpolation

Knowledge based model

• Word based n-gram feature (n=1)• Vocabulary from training corpus(EuroParl)• Generate a probability distribution from these

count

Morphological model

• Gather the affix information from corpora in an unsupervised(Harald Hammarstr¨om 2006)

• Give a score for each affix

Language Identification(3)

• Performance

Decision tree classification

• Each model can complement the other in certain cases

• Train data : automatically annotated data set• Feature : confidence score• Use the Kurtosis measure

Decision tree classification

• An example : query “the sovereign individual” and statistical model identifies it as English k = 7.6 > = = ( 4.47 + 1.96 ) so this query’s confidence score is “en-HIGH”• Implement DT classifier by the Weka Machine

Learning Toolkit (Witten and Frank,2005)

Decision tree classification

• Outperform all the models for each size on average

Decision tree classification

Mli,lj : language li misclassified by the system as lj

non-linguistic feature

• Non-linguistic feature is the language information of the country

• It helps the search engine in guessing the language

example : query “how to tape for plantar fasciits”(it is labelled as Category-2) It is classified to Porteguese query

non-linguistic feature

• Increase test set size to 430 queries

Conclusions

• A completely automated method to generate a reliable data set

• Built a decision tree classifier that improves the results on average

• Built a second classifier that takes into account the geographical information of the users

Feature Work

• To improve the accuracy of data generation• More careful examination in parameter values• To extend the number of languages in data set• Consider other alternatives to the decision

tree framework