School of Computer Science Carnegie Mellon University Athens University of Economics & Business Patterns amongst Competing Task Frequencies: S u p e r

School of Computer ScienceCarnegie Mellon University

Athens University of Economics & Business

Patterns amongst Competing Task Frequencies:

S u p e r – L i n e a r i t i e s , &t h e A l m o n d-D G m o d e l

Danai KoutraB.Aditya Prakash

Vasileios KoutrasChristos Faloutsos

PAKDD, 15-17 April 2013, Gold Coast, Australia

© Danai Koutra (CMU) - PAKDD'13 2CMU AUEB

Questions we answer (1)

Patterns: If Bob executes task x for nx times,

how many times does he execute task y?

Modeling: Which 2-d distribution fits 2-d clouds of points?

# of

# of

‘Smith’ (100 calls, 700 sms)


Questions we answer (2)

Patterns: If Bob executes task x for nx times,

how many times does he execute task y?

Modeling: Which 2-d distribution fits 2-d clouds of points?

# of

# of


Let’s peek...… at our contributions

Patterns:• power laws between competing tasks• log-logistic distributions for many tasks

Modeling:Almond-DG distribution for 2-d real datasets

Practical Use:spot outliers; what-if scenarios

ln(comments)

ln(t

weets

)












RoadmapData

Observed Patterns

Related Work

Proposed Distribution

Goodness of Fit

Conclusions


Data 1: Tencent Weibo•micro-blogging website in China• 2.2 million users• Tasks extracted

Tweets Retweets Comments Mentions Followees


Data 2: Phonecall Dataset• phone-call records• 3.1 million users• Tasks extracted:

Calls Messages Voice friends SMS friends Total minutes of phonecalls


RoadmapData

Observed Patterns

Related Work


Goodness of Fit

Conclusions


Pattern 1 - SuRF: Super Linear Relative Frequency (1)ln

(tw

eets

)

ln(retweets)

‘Smith’ (1100 retweets, 7 tweets)

Logarithmic Binning Fit [Akoglu’10]•15 log buckets•E[Y|X=x] per bucket•linear regression on conditional means

0.23


Pattern 1 – SuRF (2)

ln(t

weets

)

ln(comments)

Corr coeff: ++Intuition:2x tweets, 4x comments0.304



ln(t

weets

)

ln(mentions)

Corr coeff: ++Intuition:

0.33



ln(f

ollo

wees

)

ln(retweets)


0.25



Super-linear relationship: more calls, even more minutes

ln(calls_no)

ln(t

ota

l_m

ins

)

1.18



Pattern 1 – SuRF (6a)

ln(calls_no)

ln(v

oic

e_f

rien

ds)

2x friends, 3x phonecalls


Pattern 1 – SuRF (6b)

Telemarketers?

ln(calls_no)

ln(v

oic

e_f

rien

ds)


Pattern 1 – SuRF (7)ln

(sm

s_fr

ien

ds)

ln(sms_no)

2x friends, 5x sms


Contributions revisited (1)



Practical Use:spot outliers; what-if scenarios.

ln(comments)

ln(t

weets

)


Pattern 2: log-logistic marginals (1)

NOT power law

ln(retweets)

ln(f

req

uen

cy)

Marginal PDF


ln(comments)

ln(f

req

uen

cy)

Marginal PDF

NOT power law



power law

ln(mentions)

ln(f

req

uen

cy)

Marginal PDF




Patterns: We observe • power law relationships between competing

tasks• log-logistic distributions for many tasks

Modeling:We propose the Almond-DG distribution for fitting

2-d real world datasets Practical Use:

spot outliers; what-if scenarios.


RoadmapData

Observed Patterns


Problem Definition

Almond-DG

Background: copulas

Goodness of Fit

Conclusions


Problem definitionGiven: cloud of points

Find: a 2-d PDF, f(x,y), that captures (a) the marginals

(b) the dependency

# of #

of

# of

# of


Solutions in the Literature?

•Multivariate Logistic [Malik & Abraham, 1973]•Multivariate Pareto Distribution [Mardia, 1962]• Triple Power Law [Akoglu et al., 2012] bivariate distribution for modeling reciprocity in phonecall networks


Solutions in the Literature?

•Multivariate Logistic [Malik & Abraham, 1973]•Multivariate Pareto Distribution [Mardia, 1962]• Triple Power Law [Akoglu et al., 2012] bivariate distribution for modeling reciprocity in phonecall networksBUT none of them captures the

2-d marginals

AND

dependency / correlation!!!


RoadmapRelated Work

Data

Observed Patterns


Problem Definition

Almond-DG

Background: copulas

Goodness of Fit

Conclusions




(b) the dependency

# of

# of

# of

# of


STEP 1: How to model the marginal distributions?

• A: Log-logistic!

• Q: Why?• A: Because it•mimics Pareto• captures the top concavity•matches reality

ln(retweets)ln

(fre

quency

)

Marginal PDF


Reminder:Log-logistic (1)

• The longer you survive the disease, the even longer you survive• Not memoryless• 2 parameters: scale (α) and shape (β)

BACKGROUND

a=1 β=


Reminder:Log-logistic (2a)

• In log-log scales, looks like hyperbola

BACKGROUND

a=1 β=


Reminder:Log-logistic (2b)

• In log-log scales, looks like hyperbola

BACKGROUND

a=1 β=Blank out the top

concavity - power law


Fact:Log-logistic (3)

• linear log-odd plots

BACKGROUND

Prob(X<=x)Prob(X>x)

real

Theory

ln(mentions)

ln(o

dds) α =

2.07β = 1.27




(b) the dependency

# of #

of

# of

# of✔

✔


STEP 2: How to model the dependency?

• A: we borrow an idea from survival models, financial risk management, decision analysis

COPULA!


• Modeling dependence between r.v.’s (e.g., X = # of , Y = # of )

BACKGROUNDCopulas in a nutshell

© Danai Koutra (CMU) - PAKDD'13CMU AUEB

• Model dependence between r.v.’s (e.g., X = # of , Y = # of )

• Create multivariate distribution s.t.: the marginals are preserved the correlation (+, -, none) is captured

BACKGROUNDCopulas in a nutshell

38

# of

# of


STEP 2: Which copula?

• A: among the many copulas• Blah• Blah• Blah• Gumbel’s copula


Applications ofGumbel’s copula

Modeling of:• the dependence between loss and

lawyer’s fees in order to calculate reinsurance premiums• the rainfall frequency as a joint

distribution of volume, peak, duration etc.• …

BACKGROUND


Gumbel’s copula:Example 1

BACKGROUND

• Uniform marginals• No dependence

# of

# of



BACKGROUND

• Skewed marginals• No correlation

# of

# of



BACKGROUND

• Skewed marginals• ρ = 0.7

# of

# of




(b) the dependency

# of

# of

# of

# of

✔✔


Proposed Continuous Distribution: ALMOND

where θ = ( 1 – ρ )-1 captures the dependence ρ = Spearman’s coefficient

ρ=0 ρ=0.4 ρ=0.7 ρ=0 ρ=0.2 ρ=0.7 α = ? β = ? α = ? β = ?


- DG

Proposed Discrete Distribution: ALMOND-DG

If (X,Y) ~ ALMONDthen (floor(X), floor(Y)) ~ ALMOND-DG where X>=1 and Y>=1.

i.e., we discretize the values of ALMOND, and reject the pairs with either X=0 or Y=0.



Patterns: We observe • power laws between competing tasks• log-logistic distributions for many tasks


Practical Use:spot outliers; what-if scenarios.


RoadmapRelated Work

Data

Observed Patterns


Goodness of Fit

Conclusions


Synthetic Data Generation

Parameter Estimation•Traditionally: MLE, MOM log-logistic•Proposed: log-odd plot 2 parametersintercept + slope of the line•Copula-based generation 1 parameterdependence θ

Evaluation is hard even for 1-dskewed distributions!!!

[Chakrabarti, 2006]ln(mentions)

ln(o

dds)


Goodness of Fit (1a)ln

(fre

qu

en

cy)

ln(comments)

Marginal PDF

ln(mentions)

ln(f

req

uen

cy)

Real data - Synthetic data

1✔


Goodness of Fit (1b)Contour plots Conditional Means (SuRF)

Synthetic data

Real data

ln(m

en

tions)

ln(m

en

tions)

ln(m

en

tions)

ln(m

en

tions)

ln(comments)

ln(comments)

ln(comments)

ln(comments)

2✔ 3✔


Goodness of Fit (2a)

Real data - Synthetic data

ln(f

req

uen

cy)

ln(retweets)

Marginal PDF

ln(tweets)

ln(f

req

uen

cy)1✔


Goodness of Fit (2b)

Real data

Synthetic data

Contour plots Conditional Means (SuRF)

ln(retweets)

ln(t

weets

)ln

(tw

eets

)

ln(retweets)

ln(retweets)

ln(retweets)ln

(tw

eets

)ln

(tw

eets

)

32✔ ✔


RoadmapRelated Work

Data

Observed Patterns


Goodness of Fit

Conclusions


Conclusions

Patterns: Discovery of• power law between competing tasks• log-logistic distributions for many tasks

Modeling:Almond-DG, that explains (i) super-linearity, (ii) marginals and (iii) conditionals in real 2-d data

Practical Use:anomaly detection; what-if scenarios.

ln(comments)

ln(t

weets

)


Thank you!

[email protected]

- DG


Backup slides

• Likely question areas: ideas glossed over, shortcomings of methods or results, and future work

• Why Gumbel? It fits, it has been used in the past + parsimonious (theta, alpha, beta)

Documents

School of Computer Science Carnegie Mellon University Athens University of Economics & Business Patterns amongst Competing Task Frequencies: S u p e r