BSSML16 L4. Association Discovery and Topic Modeling

D E C E M B E R 8 - 9 , 2 0 1 6

BigML, Inc 2

Poul Petersen CIO, BigML, Inc.

Association DiscoveryFinding Interesting Correlations

BigML, Inc 3Association Discovery

Association Discovery

• Algorithm: “Magnum Opus” from Geoff Webb • Unsupervised Learning: Works with unlabelled

data, like clustering and anomaly detection. • Learning Task: Find “interesting” relations

between variables.


Unsupervised Learning

date customer account auth class zip amountMon Bob 3421 pin clothes 46140 135Tue Bob 3421 sign food 46140 401Tue Alice 2456 pin food 12222 234Wed Sally 6788 pin gas 26339 94Wed Bob 3421 pin tech 21350 2459Wed Bob 3421 pin gas 46140 83The Sally 6788 sign food 26339 51


Clustering

Anomaly Detection

similar

unusual


Association Rules


zip = 46140amount < 100

Rules:

Antecedent Consequent

{customer = Bob, account = 3421}{class = gas}


Use Cases

• Market Basket Analysis

• Web usage patterns

• Intrusion detection

• Fraud detection

• Bioinformatics

• Medical risk factors


Magnum Opus

• Select measure of interest: Levarage, Lift, etc• System finds the top-k associations on that

measure within constraints • Must be statistically significant interaction between

antecedent and consequent• Every item in the antecedent must increase the

strength of association


Association Metrics

Instances

AC

Coverage

Percentage of instances which match antecedent “A”


Association Metrics

Instances

AC

Support

Percentage of instances which match antecedent “A” and Consequent “C”


Association Metrics

Coverage

Support

Instances

AC

Confidence

Percentage of instances in the antecedent which also contain the consequent.


Association Metrics

CInstances

A C

A

Instances

C

Instances

A

Instances

AC

0% 100%

Instances

AC

Confidence

A never implies C

A sometimes implies C

A always implies C


Association Metrics

Independent

AC

C

Observed

A

Lift

Ratio of observed support to support if A and C were statistically independent.

Support == Confidence p(A) * p(C) p(C)


Association Metrics

C

Observed

A

Observed

AC

< 1 > 1

Independent

A C

Lift = 1

Negative Correlation No Association Positive

Correlation

Independent

A C

Independent

A C

Observed

A C


Association Metrics

Independent

AC

C

Observed

A

Leverage

Difference of observed support and support if A and C were statistically independent.

Support - [ p(A) * p(C) ]


Association Metrics

C

Observed

A

Observed

AC

< 0 > 0

Independent

A C

Leverage = 0

NegativeCorrelation No Association Positive

Correlation

Independent

A C

Independent

A C

Observed

A C

-1…


Use Cases

GOAL: Discover “interesting” rules about what store items

are typically purchased together.

• Dataset of 9,834 grocery cart transactions

• Each row is a list of all items in a cart at checkout

BigML, Inc 17

Association Discovery Demo #1


Use Cases

GOAL: Find general rules that indicate diabetes.

• Dataset of diagnostic measurements of 768 patients.

• Each patient labelled True/False for diabetes.

BigML, Inc 19

Association Discovery Demo #2


Medical RisksDecision Tree

If plasma glucose > 155 and bmi > 29.32 and diabetes pedigree > 0.32 and insulin <= 629 and age <= 44

then diabetes = TRUE

Association Rule

If plasma glucose > 146 then diabetes = TRUE

BigML, Inc 21

Poul Petersen CIO, BigML, Inc.

Topic ModelingDiscovering Meaning in Text

BigML, Inc 22Topic Modelling

Unsupervised LearningFeatures

Inst

ance

s

• Learn from instances

• Each instance has features

• There is no label

Clustering Find similar instances

Anomaly Detection Find unusual instances

Association Discovery Find feature rules


Topic ModelText Fields

• Unsupervised algorithm

• Learns only from text fields

• Finds hidden topics that model the text

• How is this different from the Text Analysis that BigML already offers?

• What does it output and how do we use it

• Unsupervised… model?

Questions:


Text Analysis

Be not afraid of greatness: some are born great, some achieve greatness, and some have greatnessthrust upon 'em.

great: appears 4 times

Bag of Words


Text Analysis

… great afraid born achieve … …

… 4 1 1 1 … …

… … … … … … …

Be not afraid of greatness: some are born great, some achieve greatness, and some have greatnessthrust upon ‘em.

Model

The token “great” occurs more than 3 times

The token “afraid” occurs no more than once

BigML, Inc 26

Topic Model Demo #1


TA vs TMText Analysis Topic Model

Creates thousands of hidden token counts

Token counts are independently uninteresting

No semantic importance

No measure of co-occurrence

Creates tens of topics that model the text

Topics are independently interesting

Semantic meaning extracted

Support for bigrams


Generative Modeling

• Decision trees are discriminative models • Aggressively model the classification boundary

• Parsimonious: Don’t consider anything you don’t have to

• Topic Models are generative models • Come up with a theory of how the data is generated

• Tweak the theory to fit your data


Generating Documents

cat shoe zebra ball tree jump pen asteroid

cable box step cabinet yellow

plate flashlight…

shoe asteroid flashlight pizza…

plate giraffe purple jump…

Be not afraid of greatness: some are born great, some achieve greatness…

• "Machine" that generates a random word with equal probability with each pull.

• Pull random number of times to generate a document.

• All documents can be generated, but most are nonsense.

word probabilityshoe ϵ

asteroid ϵflashlight ϵ

pizza ϵ… ϵ


Topic Model• Written documents have meaning - one way to

describe meaning is to assign a topic.

• For our random machine, the topic can be thought of as increasing the probability of certain words.

Intuition:

Topic: travel



plate flashlight…

airplane passport pizza …

word probabilitytravel 23,55 %

airplane 2,33 %mars 0,003 %

mantle ϵ… ϵ

Topic: space



plate flashlight…

mars quasar lightyear soda

word probabilityspace 38,94 %

airplane ϵmars 13,43 %

mantle 0,05 %… ϵ


Topic Model


Topic: "1"



plate flashlight…

word probability

travel 23,55 %

airplane 2,33 %

mars 0,003 %

mantle ϵ

… ϵ

Topic: "k"



plate flashlight…

word probability

shoe 12,12 %

coffee 3,39 %

telephone 13,43 %

paper 4,11 %

… ϵ

…Topic: "2"



plate flashlight…

word probability

space 38,94 %airplane ϵ

mars 13,43 %

mantle 0,05 %

… ϵ

airplane passport pizza …


• Each text field in a row is concatenated into a document

• The documents are analyzed to generate "k" related topics that can model the documents

• Each topic is represented by a distribution of term probabilities

BigML, Inc 32

Topic Model Demo #2


Use Cases

• As a preprocessor for other techniques

• Bootstrapping categories for classification

• Recommendation

• Discovery in large, heterogeneous text datasets


Topic Distribution• Any given document is likely a mixture of the

modeled topics…

• This can be represented as a distribution of topic probabilities

Intuition:

Will 2020 be the year that humans will embrace space exploration and finally travel to Mars?

Topic: travel



plate flashlight…

word probabilitytravel 23,55 %

airplane 2,33 %mars 0,003 %

mantle ϵ… ϵ

11%

Topic: space



plate flashlight…

word probabilityspace 38,94 %

airplane ϵmars 13,43 %

mantle 0,05 %… ϵ

89%

BigML, Inc 35

Topic Model Demo #3


Batch Topic Distribution

Unlabelled Data

Centroid Label

Unlabelled Data

topic 1 prob

topic 3 prob

topic k prob

Clustering Batch Centroid

Topic Model

Text Fields

Batch Topic Distribution

…

BigML, Inc 37

Topic Model Demo #4


Tips

• Setting k • Much like k-means, the best value is data specific

• Too few will agglomerate unrelated topics, too many will partition highly related topics

• I tend to find the latter more annoying than the former

• Tuning the Model • Remove common, useless terms

• Set term limit higher, use bigrams

Data & Analytics

BSSML16 L4. Association Discovery and Topic Modeling