Sentence Modeling• Representation of sentences is the heart of Natural Language

Processing • A sentence model is a representation and analysis of semantic

content of a sentence for classification or generation• The sentence modeling task is at the core of many tasks such as

sentiment analysis, paraphrase detection, entailment recognition, summarization, discourse analysis, machine translation, grounded language learning and image retrieval• The aim of sentence modeling is a feature function that guides the

process by which features of a sentence are extracted.

One Dimensional Convolution

Vector of weights Size: m Filter

Input SequenceSentence

M-gram Dot product

Produces a sequence c

Narrow ConvolutionSize of c :

s – m + 1

It requires thats ≥ m

Wide Convolution

Size:s + m - 1

Wide Convolution• Size of c

s+m-1

• No requirement on s or m• Out of range values are taken to be 0• Result of narrow convolution is subsequence of result of wide

convolution

Advantages of Wide Convolution• Guarantees that a valid non empty c will always be produced• All weights in the filter reach the entire sentence• Holds no limit on the size of m or s

Time Delay Neural Network• A key feature for TDNN’s are the ability to express a relation between

inputs in time.• The sequence s is viewed as having a time dimension and the

convolution is applied over the time dimension.

Max TDNN

Properties of Max TDNN• Sensitive to order of the words• Does not depend on external language specific features• Largely uniform importance to the signal from each of the words

• Range of feature detectors is limited• Higher order and long range feature detectors cannot be incorporated• Multiple occurrences of features and the sequence ignored• Pooling factor: s-m+1

k-Max Pooling

k – Max Pooling• Given a value k and a sequence P of length p, k-max pooling selects

the subsequence p-max of the k highest values of p.

• The order of the values in p-max corresponds to the original order in p.

k-Max Pooling• k most active features• Features may be number of positions apart• Preserves the order of the features• But is insensitive to their specific positions• Can detect multiple occurrences of feature

What should k be?• Why not let it decide for itself?

Dynamic k-Max Pooling

Suppose length of sentence = 18L = 3Ktop = 3

K1 = 12K2 = 6K3 = 3

Multiple Feature Maps

Convolution K-max Pooling Layer

Non linear function

Feature Map

Second Order

Feature Map

To increase the number of learnt feature detectors of a certain order, multiple feature maps may be computed in parallel at the same layer.

Folding• Feature detectors in different rows are independent of each other.

Properties of Sentence Model• The subsequence of n-grams extracted by the pooling operation

induces invariance to absolute positions, but maintains their order and relative positions.

• DCNN feature graphs have a global range of the pooling operations

• DCNN has internal input dependent structure and does not rely on externally provided parse trees.

Experiments

Sentiment Prediction in Movie Reviews• Concerns prediction of sentiment of movie reviews in Stanford

Sentiment Treebank• Output is binary in experiment 1 and “negative, somewhat negative,

neutral, somewhat positive, positive” in experiment 2• Binary: MultiClass:

Question Type Classification• TREC question dataset• Six Different Question Types

Twitter Sentiment Prediction with Distant Supervision• Large dataset of tweets• Tweet is labelled positive or negative automatically based on

emoticon• Tweets are preprocessed

Conclusion• Dynamic CNN defined, which uses Dynamic k-max Pooling• Feature Graph captures word relation of varying size• High performance on sentiment prediction and question classification

without requiring external features