Natural Language Processing

NATURAL LANGUAGE PROCESSING

a developer’s perspective to

- Vanya Seth & Dharmendra Prasad

WHAT IS NATURAL LANGUAGE PROCESSING?

WHAT IS NATURAL LANGUAGE PROCESSING?

NLP is a tool to devise ways to interpret human understandable languages and extract meaning out of it, which can be used by other humans (translators), machines or software systems for achieving greater objectives.

Why do we need this? Not everyone speaks the same language (MT) Not all the languages are written the same way (MT) We would like to automate tasks which are tedious (IE) We desire more accuracy and less human errors!

(Automation)

FEW INTERESTING PROBLEMS

1. Spam Detection2. Spelling correction3. Parts of Speech Tagging4. Named Entity Recognition

1. Co reference Resolution2. Information Extraction3. Sentiment Analysis4. Machine Translation

1. Answering Questions2. Summarization 3. Paraphrasing4. Dialog

WHY IS NLP A COMPLEX PROBLEM

Ambiguity is pervasiveHeadline: Republicans Grill IRS Chief Over Lost Emails

Meanings: Republicans harshly question the chief about the emails Republicans cook the chief using email as the fuel

New ways of writing Twitter hashtags All Capitals Abused notations (U for You, FB for facebook, @ for At) New words (Retweet, Unfriend etc) Emoticons ( , and many others)

HOW DO WE DEAL WITH THIS PROBLEM?

We learn, we remember and we conquer What tools we use?

We require the knowledge about the language We require the knowledge about the world We need a way to combine knowledge sources

How we do this? Probabilistic models built upon language data for

inferring language properties P(“fragrant” -> “rose”) is high P(“awful” -> “love”) is low

TEXT PROCESSING – THE BASE OF NLP

What we mostly do in NLP is “Text Processing”. Normalizing the text in one way or the other.

Word Tokenization, Text Search, Sentence Segmentation, Pattern Recognition, Disambiguating Words etc…

Text Processing is Important, one important tool is

Regular Expressions (http://regexpal.com)

Word Tokenization (http://sentiment.christopherpotts.net/tokenizing.html)

how many words are there in the text

what is the size of the vocabulary and so on..

LANGUAGE MODELING

probabilistic approach

WHAT ARE WE TRYING TO SOLVE?

Sentence completion

How are _____?The weather is ______.

Phrase Rearrangement

little had Mary a lambhappy everyone during is holidays

How can we solve these problems?

STEP BY STEP - LANGUAGE MODELING

Sentence completion – Probability(upcoming word)

Phrase rearrangement – Probability(occurrence of a sentence)

For both of these, we need a machinery, which is called “language model”

In simple words – a language model is a black box, which has prior knowledge about the language(s) and for any question we consult the LM for an appropriate answer

Formally – A language model is a model which computes either P(upcoming word) or P(occurrence of a sentence)

WHAT DOES THE LANGAUGE MODEL CONTAINS?

This ideally depends on what kind of answer you want from the model.

An answer to the question:

P(upcoming word) - list of pairs(phrase – probability), a phrase with highest probability (or any word based on the complexity of the algorithm)

P(phrase rearrangement) – list of pairs (sentence – probability), a sentence with highest probability (or any sentence based on the complexity of the algorithm)

How do we get these probabilities?

HOW TO CALCULATE THE PROBABILITIES

Goal: Calculating the Probability of a sequence of words

P("The tiger is a fierce animal") =P(The) * P(tiger | The) * P(is | The tiger) * P(a | The tiger

is) * * P(fierce | The tiger is a) * P(animal | The tiger is a

fierce)

This is the joint probability of the sequence of words by the chain rule of probability.

P(animal | The tiger is a fierce) = Count(The tiger is a fierce animal) Count(The tiger is a fierce)

A PRACTICAL APPROACH : MARKOV’S ASSUMPTION

It would be mostly sufficient to assume that P(animal| the tiger is a fierce) =

P(animal| fierce) or P(animal| a fierce)Hence, P(the tiger is a fierce animal) = P(is| the tiger) *

P(a| tiger is) * P(fierce| is a) * P(animal| a fierce)

EXERCISE 1 : UNIGRAMS

Vocabulary: Cold, You, They, How, Are, The, Weather, Hot, Is, Was

Strategy 1: Assign probabilities to all the words depending on the number of times it occurred in the corpus

How are Cold? How are You? How are They? How are How?

All these sentences are equally likely.

This model is called unigram model

EXERCISE 1 : IMPROVING WITH BIGRAMS

Strategy 2: Assign probabilities based on some context. Look for one previous word.

is cold , is hot, is gone are gone, are you, are they was cold, was hot they are, the weather, you are

Complete the sentence : The weather is ________

Three possible options are The weather is hot The weather is cold The weather is gone

This model is called bigram model

EXERCISE 1 : GETTING BETTER WITH TRIGRAMS

Strategy 3: Start looking at more than one previous words

weather is cold , tea is hot, they are gone, how are you, where are they water was cold, food was hot

Complete the sentence : The weather is ________Possible options is : The weather is cold.

Complete the sentence : How are ________?Possible options is : How are you?

This model is called trigram model

EXERCISE 1 : HOW FAR CAN WE DO THIS?

The computer I installed in the chemistry laboratory _______

Bigrams may result in :

The computer I installed in the chemistry laboratory equipment

Trigrams may result in :

The computer I installed in the chemistry laboratory apparatus

Long Distance Dependencies**

Huge corpus with common n grams.

“The computer I installed in the chemistry laboratory crashed”

DEALING WITH UNSEEN NGRAMS

A 2000 words vocabulary has 2000*2000 = 4,000,000 possible bigrams

A piece of document with 16000 tokens produces around 5000 unique bigrams

This means 5000/4000000 or 99.875% of the bigrams are never seen.

This will assign zero probability to all the unseen bigrams and hence, if we calculate the probability of a sentence with a new bigram, our model will return a value zero.

To avoid this we use a technique called smoothing. The simplest of all is the Laplace’s Smoothing or the Add One Smoothing

SMOOTHING TECHNIQUES

Laplace’s Smoothing: It suggests that, consider that you saw everything one more time. This solves the problems of unseen bigrams or ngrams.

Applying Laplace’s smoothing we will calculate probability as below:

Without Smoothing: P(wi| wi-1) = c(wi| wi-1)

c(wi-1)

Add One Smoothing: P(wi| wi-1) = c(wi| wi-1) + 1

c(wi-1) + |V|

Why adding a |V| in the denominator?

Seeing each word one more time means seeing all the unique words one more time and the total number of unique words is |V| so the denominator increases by |V|

IMPORTANT LINKS FOR THE SECTION

Online Available Corpus

SRI Language Model ToolKit : http://www.speech.sri.com/projects/srilm/

Google Books Ngram Viewer: https://books.google.com/ngrams

Google Ngram Corpus: http://googleresearch.blogspot.in/2006/08/all-our-n-gram-are-belong-to-you.html

SPELLING CORRECTION

SPELLING TASKS

Typing is a manual process & not everyone types correctly. Spelling mistakes occur frequently. There are two spelling tasks in such scenarios

Error Detection Error Correction

Auto Correct hte -> the

Suggest one correction theate-> theatre

Suggest list of correction leding -> leading, lending

TYPES OF SPELLING ERRORS

Non Word Errors opportnity -> opportunity graffe -> giraffe

Real Word Errors Typographical Errors

In dog we trust -> god

Cognitive Errors (Homophones) good buy -> bye withdraw cache -> cash

FIXING NON WORD ERRORS

Non Word Errors – very easy to correct

Detection Have a word list from the dictionary of the language Check the word in the dictionary, if the word is not present, the word is

an error.

Correction Find candidate words which are similar to the error Choose the best candidate based on any algorithm or model discussed

next.

FIXING REAL WORD ERRORS

Real Word Errors – tough to fix

Detection There is no error detection possible, because even the error is a real

word hence it is impossible to test the word against the given dictionary.

Correction Find candidate words which are similar (pronunciation, spelling) to a

word in the sentence Do this for all the words in the sentences Choose the best candidate based on any algorithm or model discussed

next.

FINDING CANDIDATES - MINIMUM EDIT DISTANCE

N M EN A M E

H T ET H E

A C R E S SA C R E S

INSERT

TRANSPOSE

DELETE

NOISY CHANNEL MODEL – BASIC

The noisy channel is a probabilistic model which represents the real world conditions.

We run a lot of guesses through the channel and the one which matches the most to the noisy word is our correct word.

NOISY CHANNEL MODEL – BASIC CONT..

What are we trying to find?

For an observation x(the noisy word), we are looking for a word w (from the vocabulary) which maximizes the probability of the word using this noisy channel.

ŵ = argmax P(w|x) w ϵ V

= argmax P(x|w) * P(w) / P(x) -> Bayes Rule w ϵ V

P(x) is constant for all the w in the vocabulary, because x is the observation.

P(x|w) is called the channel model and P(w) is called the language model. 

NOISY CHANNEL MODEL – EXAMPLEx = acress

Possible candidates:

Error (x) Correction (w)

Correct Letter

Error Letter

Error Type

acress actress t - Deletionacress cress - a Insertionacress caress ca ac Transpositi

onacress access c r Substitutio

nacress across o e Substitutio

nacress acres - s Insertionacress acres - s Insertion

NOISY CHANNEL MODEL – EXAMPLE CONT…

Words Taken from Corpus of Contemporary English (400,000,000 words)

Clearly the probability of across is the highest and our model will suggest the correction ‘across’ in a unigram language model

Word Frequency P(word)actress 9448 0.00002362cress 220 0.00000055caress 686 0.00000171access 35310 0.00008827across 105559 0.00026389acres 12874 0.00003218

NOISY CHANNEL MODEL – ITS NOT YET OVER

We just talked about the probability of the words (the language model), which is one factor contributing to the correction.

The other factor which is the channel model, still needs to be consulted for a better correction task.

The channel model consults a tool called Confusion Matrix for finding out the likelihood of a type of error:

Types of Error: Insertion, Deletion, Transposition & Substitution

Each confusion matrix tells the possibility of a given type of error.

CONFUSION MATRICES

NOISY CHANNEL MODEL – EXAMPLE CONT…

Words Taken from Corpus of Contemporary English (400,000,000 words)

Clearly the probability of across is the highest and our model will suggest the correction ‘across’ in a unigram language model

Word Correct letter

Error Letter

x|w P(x|word)

P(word) Result10e-9

actress

t - c|ct 0.0001170

0.00002362

2.7

cress - a a|# 0.0000014

0.00000055

0.00078

caress ca ac ac|ca 0.0000016

0.00000171

0.0028

access c r r|c 0.0000002

0.00008827

0.019

across o e e|o 0.0000093

0.00026389

2.8

acres - s es|e 0.0000321

0.00003218

1.0

acres - s ss|s 0.0000342

0.00003218

1.0

IMPORTANT LINKS FOR THE SECTION

Wikipedia - Common English Spelling mistakes : https://en.wikipedia.org/wiki/Commonly_misspelled_English_words

Birkbeck Spelling Error Corpus : http://ota.ox.ac.uk/headers/0643.xml

Peter Norvig List of Error: http://norvig.com/ngrams/spell-errors.txt

TEXT CLASSIFICATION

SPAM DETECTION - CLASSIC CASE OF TEXT CLASSIFICATION

SPAM DETECTION - CLASSIC CASE OF TEXT CLASSIFICATION

Spam like attributes Undisclosed Recipients Prize!! No name, lucky draw Suspicious URL

WHAT IS TEXT CLASSIFICATION

It is a process of classifying given documents into various classes. For e.g.:

An e-mail is a SPAM or NOT? A product review is POSITIVE or NEGATIVE?

And so on…

We need a classification model, which works on the given inputs and produces the desired output.

Inputs: a document d a fixed set of classes C = {c1, c2, c3, …., cn}

Desired Output:

a predicted class c ϵ C such that the document belongs to that class c.

CLASSIFICATION METHODS

Hand Coded Rules

Based on Combination of words for e.g. a black listed sender, words like Viagra, dollars, impress a girl.

Supervised Machine LearningNaïve BayesSupport Vector MachinesLogistic RegressionKNN (K Nearest Neighbor)

SENTIMENT ANALYSIS

What is a Sentiment? It is an attitude, affectively colored beliefs or dispositions towards

objects and persons liking, loving, hating, valuing, desiring

What is the task of Sentiment AnalysisDetecting attitude holder of the attitude, target of the attitude, type of

the attitude.

Types of Analysis Simplest is to assign a binary value to a sentence or document Slightly complex is to rate on a scale of 1-10 Toughest is to detect the target and source

EXAMPLE OF SENTIMENTS FOR A PERFUME

1. The fragrance is just awesome, I love it

2. Keeps you going all day long, this is the best perfume

3. Seriously do you call it a perfume? It’s awful.

4. Thanks for this wonderful fragrance in the classy bottle. Great!!

5. I feel like being cheated after buying this piece of crap

6. If you are reading this because it is your darling fragrance, please wear it at home exclusively, and tape the windows shut.

Attitude – awesome, best, classy, awful, crap, cheated, thanksTarget – perfume, bottle, fragrance, Holder – purchaser, user

We are basically extracting the opinion.

WHY IS SENTIMENT ANALYSIS IMPORTANT

1. We prefer to watch movies after reading the reviews

2. We prefer to buy products after reader the reviews

3. We prefer to invest in stocks after understanding the market sentiments

4. We want a profitable asset

5. We don’t want to get cheated with odd surprises

6. We believe that we can predict future (election result, market outcome)

Now all the above, more or less can be solved using the tool called ‘Sentiment Analysis’.

HOW TO? – A BASELINE ALGORITHM

Input : Test documents Tokenize the test documents Feature Extraction ( either all the tokens or a group of relevant

tokens) Classification using any of the classifier

Naïve Bayes Classifiers Maximum Entropy Classifiers Support Vector Machine Classifier

TELL ME MORE - NAÏVE BAYES CLASSIFIER

Our goal:

For a document d and set of classes C, we need to calculate probability of all the classes given the document.

P(c|d) = P(d|c)*P(c)

P(d)The class which maximizes P(c|d) is the class where the document

belongs. P(d|c) – conditional probability of the document given the class P(c) – prior probability of the class

NAÏVE BAYES CLASSIFIER CONT…

What is the practical meaning of prior P(c) and likelihood P(d|c) ?

P(c) means the probability of occurrence of the class, i.e. how often the class occurs in the corpus.

P(d|c) means the probability of the occurrence of some set of features(of certain length) given a class, i.e. P(x1, x2, x3, … xi | c). The joint probability of all the features given a class.

Now calculating the joint probability requires enormous amount of parameters to be calculated, may be of the order |x|n and these many for each class.

That would require enormous amount of training samples which is mostly not available.

This looks complicated, we must try out simplifications!!

SIMPLIFYING THE NAÏVE BAYES CLASSIFIER

Simplifying Assumptions Position of the word in the document doesn’t matter – Bag of Words

Feature probabilities given a class are independent - Conditional Independence

This simplifies our model to

P(x1, x2, x3… xi | c) = P(x1|c) * P(x2|c) * P(x3|c)* … *P(xi | c)

And hence, the probability of a class given a document reduces to

P(c|d) = P(c) * P(x1|c) * P(x2|c) * P(x3|c) *… * P(xn|c)

A STEP BY STEP DEMONSTRATION

Calculating the priors: P(India) = 3/5 P(Pakistan) = 2/5

Calculating the conditional probabilities :P(Delhi | India) = 0.15789473 P(Lahore | Pakistan) = 0.25P(India | India) = 0.15789473 P(Islamabad | Pakistan) = 0.1875P(Mumbai | India) = 0.2631579 P(Pakistan | Pakistan) = 0.1875P(Chennai| India) = 0.10526316 P(Hyderabad | Pakistan) = 0.125P(Hyderabad | India) = 0.15789473 P(India | Pakistan) = 0.0625P(Lahore | India) = 0.05263158 P(Mumbai | Pakistan) = 0.0625P(Islamabad | India) = 0.05263158 P(Chennai | Pakistan) = 0.0625

Vocabulary Size: 8

P(w|c) = count(w,c) + 1

count(c) +|V|

A STEP BY STEP DEMONSTRATION CONT…Priors: P(India) = 3/5 P(Pakistan) = 2/5

Conditional probabilities :P(Lahore | India) = 0.05263158 P(Lahore | Pakistan) = 0.25P(Hyderabad | India) = 0.15789473 P(Hyderabad | Pakistan) = 0.125P(Chennai| India) = 0.10526316 P(Chennai | Pakistan) = 0.0625P(Islamabad | India) = 0.05263158 P(Islamabad | Pakistan) = 0.1875

Test Doc : Lahore Hyderabad Chennai Islamabad P(test doc | India) = P(India) * P(Lahore | India) * P(Hyderabad | India) *

P(Chennai | India) * P(Islamabad | India)= 0.6 * 0.0526 * 0.1578 * 0.1052 * 0.0526 =

0.0000275

P(test doc | Pakistan) = P(Pakistan) * P(Lahore | Pakistan) * P(Hyderabad | Pakistan) * P(Chennai | Pakistan) * P(Islamabad | Pakistan)

= 0.4 * 0.25 * 0.125 * 0.0625 * 0.1875 = 0.0001464

CHALLENGES IN SENTIMENT ANALYSIS

Tokenization Issues Data is available online in HTML, XML and various other mark up

languages

Twitter names, hash tags etc pollute the data

Phone numbers, short forms, new words, emoticons, phone numbers etc..

Extracting Features Handling negations

I didn’t like this movie vs I really liked this movie

Which words to use? Choosing the words (I, this, movie etc do not belong to the set of

words which contribute to attitude)

SENTIMENT LEXICONS

The words which matter in the sentiments. Its better to train our models on these lexicons instead of the complete list of words in the training documents.

Here are few links for the lexicons:

The General Inquirer : http://www.wjh.harvard.edu/~inquirer/inquirerbasic.xls http://www.wjh.harvard.edu/~inquirer/homecat.htm

LIWC(Linguistic Inquiry and Word Count) http://www.liwc.net

Negative emotions (bad, weird, hate, problem, crap) Positive Emotions ( love, wonderful, magnificent, lovely)

Bing Liu's page on Opinion Mining https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html

HOW GOOD IS OUR CLASSIFIER?Not all classifiers we build is a state of art system. We must refine and fine

tune it after building.

What are the parameters to judge a classifier?Contingency Matrix

What is the accuracy of the system?Accuracy = TP + TN TP+TN+FP+FN

THE PROBLEM WITH THE ACCURACY PARAMETER

Task : Build a classifier to identify food items on web pages

Most of the tokens on a web page won’t be the name off a food item. Let us say that there are 1000 words and only 10 words are names of food item.

Let us consider that our classifier is a bogus one and it always returns a false for each word it encounters.

This means the accuracy of our system = TP + TN = 990/1000 = 99% TP+TN+FP+FN

Hence, our 99% accurate system is not able to do what we needed, i.e. detecting food items.

HOW TO FIX THIS?

We definitely need a better parameter to judge our model. So we define two parameters PRECISION and RECALL

Precision, is the percentage of selected items that are correct.Recall, is the percentage of correct item that the system was able to

select.

PRECISION = TP / ( TP + FP ) = 0/0 = UNDEFINEDRECALL = TP / (TP + FN) = 0/10 = 0

So, these parameters somewhere fairly judge our classifiers. Here, the recall is zero and the precision too is zero.

ADVANTAGES OF PRECISION & RECALL

The figures on the last slide didn’t give much insight into the roles of these two parameters.

Slightly better classifier - capable of selecting the true food itemsPrecision = 10/(10 + 20) = 33%Recall = 10/(10+10) = 50%

Hence these parameters fairly judge the classifier, so a combination of these two measures can be a fair evaluation criteria. It is also called the F measure.

WHAT NEXT?

Discriminative Language modelsMaximum EntropySupport Vector Machines

Other Advanced ApplicationsNamed Entity RecognitionPOS TaggingMachine Translation & Probabilistic Parsing

CFGsLanguage Grammars

Areas of researchInformation Retrieval (Query Based and Generic)Question & AnsweringSummarization

Dharmendra Prasad admin@techieme.in

Thank You