Research © 2008 Yahoo! Statistical Challenges in Online Advertising Deepak Agarwal Deepayan Chakrabarti (Yahoo! Research)

Research © 2008 Yahoo!

Statistical Challenges in Online Advertising

Deepak Agarwal

Deepayan Chakrabarti(Yahoo! Research)


Online Advertising

• Multi-billion dollar industry, high growth

– $9.7B in 2006 (17% increase), total $150B

• Why this will continue?

– Broadband cheap, ubiquitous

– “Getting things done” easier on the internet

– Advertisers shifting dollars

• Why does it work?

– Massive scale, automated, low marginal cost

– Key: Monetize more and better, “learn from data”

– New discipline “Computational Advertising”


What is “Computational Advertising”?

New scientific sub-discipline, at the intersection of – Large scale search and text analysis

– Information retrieval

– Statistical modeling

– Machine learning

– Optimization

– Microeconomics


Online advertising: 6000 ft Overview

Ad

vert

iser

s

Ad Network

Ads

Content

Pick ads

User

Content Provider

Examples:Yahoo, Google,

MSN, RightMedia, …


Outline

• Background on online advertising

– Sponsored Search, Content Match, Display, Unified marketplace

• The Fundamental Problem

• Statistical sub-problems:

– Description

– Existing methods

– Challenges


Different flavors

Online Advertising

Revenue Models

Advertising Setting

Misc.

CPM CPC CPA

Display Content Match

Sponsored Search

Ad exchanges


Revenue Models

CPM CPC CPA

Ad

vert

iser

s

Ads

Content

Pick ads

User

Cost Per iMpression

$$

$Content Provider

Ad Network


Revenue Models

CPM CPC CPA

Ad

vert

iser

s

Ads

Content

Pick ads

User

Cost Per Click

$$

$Content Provider

Ad Networkclick


Revenue Models

CPM CPC CPA

Ad

vert

iser

s

Ads

Content

Pick ads

User

Cost Per Action

$$

$Content Provider

Ad Networkclick

Adver

tiser

land

ing

page


Revenue Models

• Example: Suppose we show an ad N times on the same spot

• Under CPM: Revenue = N * CPM

• Under CPC: Revenue = N * CTR * CPC

CPM CPC CPA

Click-through Rate(probability of a click given an impression)

Depends on auction

mechanism


Auction Mechanism

• Revenue depends on type of auction– Generalized First-price:

• CPC = bid on clicked ad

– Generalized Second-price: • CPC = bid of ad below clicked ad (or the reserve price)

• CPC could be modified by additional factors

• [Optimal Auction Design in a Multi-Unit Environment: The Case of Sponsored Search Auctions] by Edelman+/2006

• [Internet Advertising and the Generalized Second Price Auction…] by Edelman+/2006


Revenue Models

• Example: Suppose we show an ad N times on the same spot

• Under CPM: Revenue = N * CPM

• Under CPC: Revenue = N * CTR * CPC

• Under CPA: Revenue = N * CTR * Conv. Rate * CPA

CPM CPC CPA

Conversion Rate(probability of a user conversion on the advertiser’s landing page

given a click)


Revenue Models

CPM

website traffic

CPC

website traffic +ad relevance

Revenue dependence

CPA

website traffic +ad relevance +landing page quality

Relevance to advertisers

Prices and Bids

Ease of picking ads


Background

Online Advertising

Revenue Models

Advertising Setting

Misc.

CPM CPC CPA


Sponsored Search

Ad exchanges


Advertising Setting

Ad

vert

iser

s

Ad Network

Content

Pick ads

User

Content Provider

Ads

• What do you show the user?

• How does the user interact with the ad system?


Advertising Setting


Sponsored Search


Advertising Setting


Sponsored Search

Pick ads


Advertising Setting

• Graphical display ads

• Mostly for brand awareness

• Revenue model is typically CPM


Sponsored Search


Advertising Setting


Sponsored Search

Content match ad


Advertising Setting


Sponsored Search

Pick ads

Text ads

Match ads to the content


Advertising Setting

• The user intent is unclear

• Revenue model is typically CPC

• Query (webpage) is long and noisy


Sponsored Search


Advertising Setting


Sponsored Search

Search Query

Sponsored Search Ads


Advertising Setting


Sponsored Search

Pick ads

Text ads

Search Query

Match ads to the query


Advertising Setting

• User “declares” his/her intention

• Click rates generally higher than for Content Match

• Revenue model is typically CPC (recently some CPA)

• Query is short and less noisy than Content Match


Sponsored Search


Summary

• Different revenue models

– Depends on the goal of the advertiser campaign

• Brand awareness

– Display advertising

– Pay per impression (CPM)

• Attracting users to advertised product

– Content Match, Sponsored Search

– Pay per click (CPC), Pay per action (CPA)


Background

Online Advertising

Revenue Models

Advertising Setting

Misc.

CPM CPC CPA


Sponsored Search

Ad exchanges


Unified Marketplace

• Publishers, Ad-networks, advertisers participate together in a singe exchange

• Publishers put impressions in the exchange; advertisers/ad-networks bid for it

• CPM, CPC, CPA are all integrated into a single auction mechanism


Overview: The Open Exchange

Transparency and value

Has ad impression to sell --AUCTIONS

Bids $0.50Bids $0.75 via Network…

… which becomes $0.45 bid

Bids $0.65—WINS!

AdSenseAd.com

Bids $0.60


Unified scale: Expected CPM

• Campaigns are CPC, CPA, CPM

• They may all participate in an auction together

• Converting to a common denomination is a challenge


Outline




– Description


– Challenges


Outline




– Sponsored Search and Content Match


– Description


– Challenges


Display Advertising


Display Advertising

• Main goal of advertisers: Brand Awareness

• Revenue Model: Primarily Cost per impression (CPM)

• Traditional Advertising Model:

1. Ads are targeted at particular demographics (user characteristics)

1. GM ads on Y! autos shown to “males above 55”

2. Mortgage ad shown to “everybody on Y! Front page”

2. Book a slot well in advance– “2M impressions in Jan next year”

– These future impressions must be guaranteed by the ad network


Display Advertising

• Fundamental Problem: Guarantee impressions to advertisers

3

24

2 2

1

1

Young US

FemaleY! Mail

1. Predict Supply:

• How many impressions will be available?

• Demographics overlap

2. Predict Demand:

• How much will advertisers want each demographic?


Display Advertising


3

24

2 2

1

1

Young US

FemaleY! Mail

1. Predict Supply

2. Predict Demand

3. Find the optimal allocation

• subject to supply and demand constraints


Display Advertising


1. Predict Supply

2. Predict Demand

3. Find the optimal allocation, subject to constraints

• Optimal in terms of what objective function?


Allocation through Optimization

• Optimal in terms of what objective function?

– E.g. Maximize value of remaining inventory • Cherry-picks valuable inventory, saves it for later

– Fairness• “Spreads the wealth” subject to constraints

sisupply demand

dj

xij


Example

324

2 2

1

1

Young US

FemaleY!

Mail

US & Y(2)

Supply Pools

DemandUS, Y, nFSupply = 2Price = 1

US, Y, FSupply = 3Price = 5

Supply Pools

How should we distribute impressions from the supply pools to satisfy this

demand?


Example (Cherry-picking)

• Cherry-picking: Fulfill demands at least cost

US & Y(2)

Supply Pools

DemandUS, Y, nFSupply = 2Price = 1

US, Y, FSupply = 3Price = 5


demand?

(2)


Example (Fairness)

• Cherry-picking: Fulfill demands at least cost

• Fairness:Equitable distribution of available supply pools US & Y

(2)

Supply Pools

DemandUS, Y, nFSupply = 2

Cost = 1

US, Y, FSupply = 3

Cost = 5


demand?

(1)

(1)


Objective functions

jV

jy

yV

j

j

jjj

pool of Value:

poolfor inventory remaining :

Maximize

.0

)~/log(

Minimize :function Objective

.V valueof

function decreasinglly monotonica becan general,In

allocation alproportion1 ..

;)/(~

j

:

k

jkj

jkjkk

k

j

j

SSjjjwkwjjjk

xxx

w

wge

wxXdXwxxkj

Fairness""


Display Advertising


1. Predict Supply

2. Predict Demand

3. Find the optimal allocation, subject to constraints– Pick the right objective function

• Further issues:

– Risk Management: Supply and demand forecasts should have both mean and variance

– Forecast aggregation: Forecasts may be needed over multiple resolutions, in time and in demographics


Display Advertising


1. Predict Supply

2. Predict Demand

3. Find the optimal allocation, subject to constraints– Pick the right objective function

• Forecasting accuracy is critical!

– Overshoot under-delivery of impressions unhappy advertisers

– Undershoot loss in revenue


Outline




– Sponsored Search and Content Match


– Description


– Challenges


Sponsored Search and Content Match

• Given a query:

– Select the top-k ads to be shown on the k slots to maximize total expected revenue

• What is total expected revenue?


Example (Content Match)

Ad Position 1

Ad Position 2

Ad Position 3


Example (Content Match)


Reminder: Auction Mechanism

• Revenue depends on type of auction– Generalized First-price:

• CPC = bid on clicked ad

– Generalized Second-price: • CPC = bid of ad below clicked ad (or the reserve price)

• CPC could be modified by additional factors

• Total expected revenue = revenue obtained in a given time window

• [Optimal Auction Design in a Multi-Unit Environment: The Case of Sponsored Search Auctions] by Edelman+/2006

• [Internet Advertising and the Generalized Second Price Auction…] by Edelman+/2006



• Given a query:


• What affects the total revenue?

– Relevance of the ad to the query

– Bids on the ads

– User experience on the ad landing page (ad “quality”)

– Expected total revenue is some function of these.



• Given a query:


• Fundamental Problem:

– Estimate relevance of the ad to the query


Ad Relevance Computation


Overview

• Information Retrieval (IR)

– Techniques

– Challenges

• Machine Learning using Click Feedback

• Online Learning


IR-based ad matching

• “Why not use a search engine to match ads to context?”– Ads are the “documents”

– Context (user query or webpage content) is the “query”

• Three broad approaches:– Vector space models

– Probabilistic models

– Language models

• Open-source software is available:

– Lemur (www.lemurproject.org)



• Vector space models:

– Each word/phrase in the vocabulary is a separate dimension

– Each ad and query is a point in this vector space

– Example: cosine similarity

• Probabilistic models

• Language models


• Q1: How can we score the goodness of an ad for a context?

• Cosine similarity:

• Advantages:

– Simple and easy to interpret

– Normalizes for different ad and context lengths


Ad vectorQuery vector



• Vector space models

• Probabilistic models:

– Predict, for every (ad, query) pair, the probability that the ad is relevant to the query

– Example: Okapi BM25

• Language models



• Okapi BM25:


Term Frequency

in ad

Parameters

Norm. document

length

Inverse Document Frequency

Term Frequency

in query



• Okapi BM25:

• Advantages:– Different terms are weighted differently

– Tunable parameters

– Good performance


Term Frequency

in ad

Norm. document

length

Term Frequency

in query



• Vector space models

• Probabilistic models

• Language models:

– Ads and queries are generated by statistical models of how words are used in the language

– What statistical models can be used?

– How do we translate query and ad generation probabilities into relevance?



• What statistical models can be used?

– Bigram model

– Multinomial model• Given any ad or query, we can compute the parameter

setting most likely to have generated the document

Term Frequency

Term probability (model parameters)Total length



How do we translate query and ad generation probabilities into relevance?

Method 1

• Compute most likelyquery and ad params

• Generate ad usingquery params

• High probability high relevance

QueryQuery

params

AdAd

params




Method 2


• Generate query usingad params

• High probability high relevance

QueryQuery

params

AdAd

params




Method 3


• Compute KL-divergencebetween params

• Low KL-divergence high relevance

QueryQuery

params

AdAd

params



• New methods to combine syntactic and semantic information

• For example, “A Semantic Approach to Contextual Advertising” by Broder+/SIGIR/2007– Words only provide syntactic clues

– Classify ads and queries into a common taxonomy

– Taxonomy matches provide semantic clues


Overview


– Techniques

– Challenges


• Online Learning


Challenges of IR-based ad matching

• Word matches might not always work


Woes of word matching

Extract Topical info

Increases coverage,more relevant match



• Word matches might not always work

• Works well for frequent words, what about rare words? Long tail, big revenue impact.– Remedy: Add more matching dimensions (phrase,…)

• Static, does not capture effect of external factors– E.g. high interest in basketball page due to an event;

dies off after the event

– Click feedback a powerful way of capturing such latent effects; difficult to do it through relevance only

• Relevance scores may not correspond to CTR; does not provide estimates of expected revenue



• Heterogeneous corpus (query, ads). Single tfidf scores not applicable.

• In content match, queries long and noisy

• Partial feedback does not work

– Not scalable

• Ads are small, relevance of landing page difficult to determine (video, image, text)


Machine Learning using Click Feedback


Overview



– Advantages and Challenges of Click Feedback

– Feature-based models• Description

• Case Studies

– Hierarchical Models

– Matrix Factorization and Collaborative Filtering

– Challenges and Open Problems

• Online Learning


Learning from Click Feedback

• Learning relevance from partial human-labeled training data

– Attractive but not scalable

• Users provide us direct feedback through ad clicks

– Low cost and automated learning mechanism

– Large amounts of feedback for big ad-networks

• Estimation problem:

– Estimate CTR = Pr(click| query, ad, user)


Learning from Clicks: Challenges

• Noisy labels– Clicks (unscrupulous users gaming the system)

– Negatives (not clear; I never click on ads )

• Sparseness– (query, ad) matrix has billions of cells; long tail

• Too few data points in large number of cells; MLE has high variance

• Goal is to learn the best cells, not all cells

• Dynamic and seasonal effects– CTRs evolve; subject to seasonal effects

• Summer, Halloween,..

• Palin ads popular yesterday, not today


Challenges continued

• Selection bias– We never showed watch ads on golf pages

• Positional bias, presentation bias– Same ad performs differently at different positions

• Slate bias– Performance of ad depends on other ads that were

displayed


Overview





• Case Studies




• Online Learning


Feature based approach

• Query, Ad characterized by features– Query: bag-of-words, phrases, topic,…

– Ads: bag-of-words, keywords, size,…

• Query feature vector: q

• Ad feature vector: a

• Pr(Click|Q,A) = f(q,a;θ)

• Example: Logistic regression– log-odds(Pr(Click|Q,A)) = q’ W a

– W estimated from data


Feature based models: Challenges

• Challenges– High dimensional, need to regularize (Priors)

– De-bias for positional and slate effects

– Negative events to be weighted appropriately

• Go through case studies reported in literature


Predicting Clicks: Estimating the Click-through rates of new ads: Richardson et al, WWW 2007

• Estimate CTR of new ads in Sponsored search

• Log-odds(CTR(ad)) = wifi(ad)

• Features used:

– Bid term CTRs of related ads (from other accounts)• CTRs of all other ads with keyword “camera”

– Appearance, attention, advertiser reputation, landing page quality, relevance of bid terms to ad, bag-of-words in ad.

• Does not capture interactions between (query, ad), main focus is to estimate CTR of new ads only

• Negative events down-weighted based on eye-tracking study


Combining relevance with Click Feedback, Chakrabarti et al, WWW 08

• Content Match application

• CTR estimation for arbitrary (page, ad) pairs

• Features :

– Bag-of-words in query, ads; relevance scores from IR

– Cross-product of words: Occurs in both page and ad

• Learn to predict click data using such features

• Prediction function amenable to WAND algorithm

– Helps with fast retrieval at serve time


Proposed Method

• A logistic regression method model for CTR

CTR Main effect for page

(how good is the page)

Main effect for ad

(how good is the ad)

Interaction effect

(words shared by page and ad)

Model parameters


Proposed Method

• Mp,w = tfp,w

• Ma,w = tfa,w

• Ip,a,w = tfp,w * tfa,w

• So, IR-based term frequency measures are taken into account


Proposed Method

• Two sources of complexity

– Adding in IR scores

– Word selection for efficient learning


Proposed Method

• How can IR scores fit into the model?

– What is the relationship between logit(pij) and cosine score?

– Quadratic relationship

Cosine scorelo

git(

p ij)


Proposed Method

• How can IR scores fit into the model?

• This quadratic relationship can be used in two ways

– Put in cosine and cosine2 as features

– Use it as a prior


Proposed Method

• Word selection

– Overall, nearly 110k words in corpus

– Learning parameters for each word would be:• Very expensive

• Require a huge amount of data

• Suffer from diminishing returns

– So we want to select ~1k top words which will have the most impact


Proposed Method

• Word selection

– Data based:• Define an interaction measure for each word

• Higher values for words which have higher-than-expected CTR when they occur on both page and ad


Experiments

Recall

Pre

cisi

on

25% lift in precision at 10% recall


Overview





• Case Studies




• Online Learning


Regelsen and Fain, 2006

• Estimate CTR of terms by “borrowing strength” at multiple resolutions

• Hierarchical clustering of related terms

– Clustering advertiser keyword matrix

• Estimating CTR at finer resolutions by using information at coarser resolutions

– Weighted average, more weight to finer resolutions

– Weights selected heuristically, no principled approach


Estimation in the “tail”

• A more principled approach to “Estimating Rates of Rare Events at Multiple Resolutions” [KDD/2007]

• Contextual Advertising

– Show an ad on a webpage (“impression”)

– Revenue is generated if a user clicks

– Problem: Estimate the click-through rate (CTR) of an ad on a page

• Most (ad, page) pairs have very few impressions, if any,

• and even fewer clicks

Severe data sparsity


Estimation in the “tail”

• Use an existing, well-understood hierarchy

– Categorize ads and webpages to leaves of the hierarchy

– CTR estimates of siblings are correlated

The hierarchy allows us to aggregate data

• Coarser resolutions

– provide reliable estimates for rare events

– which then influences estimation at finer resolutions


System overview

Retrospective data[URL, ad, isClicked]

Crawl URLs

Classify pages and ads

Rare event estimation using

hierarchy

a sample of URLs

Impute impressions, fix sampling bias


Sampling of webpages

• Naïve strategy: sample at random from the set of URLsSampling errors in impression volume AND click

volume

• Instead, we propose:

– Crawling all URLs with at least one click, and

– a sample of the remaining URLs

Variability is only in impression volume


Imputation of impression volume

Ad classes

Pag

e cl

asse

s

sums to #impressions on ads of this ad class

[column constraint]

sums to ∑nij + K.∑mij

[row constraint]

sums toTotal impressions

(known)

#impressions = nij + mij + xij

Clicked pool

Sampled Non-clicked

pool

Excess impressions(to be imputed)



Level 0

Level i

Page hierarchy Ad hierarchy

• Region= (page node, ad node)

• Region Hierarchy A cross-product of the page

hierarchy and the ad hierarchy

Page classes Ad classes

Region



sums to

[block constraint]

Level i

Level i+1


Imputing xij

Level i

Level i+1

Iterative Proportional Fitting [Darroch+/1972]

• Initialize xij = nij + mij

• Iteratively scale xij values to match row/col/block constraint

• Ordering of constraints: top-down, then bottom-up, and repeat

blockPage classes Ad classes


Imputation: Summary

• Given

– nij (impressions in clicked pool)

– mij (impressions in sampled non-clicked pool)

– # impressions on ads of each ad class in the ad hierarchy

• We get

– Estimated impression volume Ñij = nij + mij + xij

in each region ij of every level


System overview

Retrospective data[page, ad, isclicked]

Crawl Pages

Classify pages and ads

Rare event estimation using

hierarchy

a sample of pages

Impute impressions, fix sampling bias


Rare rate modeling

1. Freeman-Tukey transform: – yij = F-T(clicks and impressions at ij)

≈ transformed-CTR

– Variance stabilizing transformation: Var(y) is independent of E[y] needed in further modeling


SijSparent(ij)

Rare rate modeling

2. Generative Model (Tree-structured Markov Model)

yij yparent(ij)

covariates βij variance Vij

Unobserved “state”

variance Wij

Vparent(ij)

βparent(ij)

Wparent(ij)


Rare rate modeling

• Model fitting with a 2-pass Kalman filter:

– Filtering: Leaf to root

– Smoothing: Root to leaf

• Linear in thenumber of regions


Tree-structured Markov model

rVrWRootWRootSrpaSrwrWNrw

rwrpaSrS

rSd(r).rd

rVrSrdT

ruNry

/on Depends :Smoothing

.0;)( indep );,0(~)(

smoothing) (requireregion per one effects, random: levelat covariatesfor t vector coefficien:)(

),)((~

Model Markov

)(

1 1 1

'

)(

'

/),Corr( ;)(

;/rd

i

l

i

l

i

iiir

rdrrr

WWllWSVar

WWNVV


Scalable Model fitting Multi-resolution Kalman filter

1994) Rubin, and(Liu algorithm ECME :componets Variance

downtree , Uptree:steps Two

n computatio )3regionchildren O(# region,parent each At

regionsparent ofnumber on Dependsregions ofnumber in thelinear y"essentiall" Algorithm

2002) Cressie, and (Huang algorithmfilter Kalman -

:

smoothingfiltering

}r{S states ofPosterior


Multi-Resolution Kalman filter: Mathematical overview

parent from info usingchildren on info Update

)(

parentfor available info recombine

children fromn informatio Combine

parentfor child ofon contributiCollect

);1)(),(( ;)(

equations; state Invert the

updatesBayesian standard using nodes leaf ofposterior Update

)(

step Smoothing

:step Filtering

downtree

rdrdcorrrBrrSrBrpa

S

uptree


Experiments

• 503M impressions

• 7-level hierarchy of which the top 3 levels were used

• Zero clicks in

– 76% regions in level 2

– 95% regions in level 3

• Full dataset DFULL, and a 2/3 sample DSAMPLE


Experiments

• Estimate CTRs for all regions R in level 3 with zero clicks in DSAMPLE

• Some of these regions R>0 get clicks in DFULL

• A good model should predict higher CTRs for R>0 as against the other regions in R


Experiments

• We compared 4 models

– TS: our tree-structured model

– LM (level-mean): each level smoothed independently

– NS (no smoothing): CTR proportional to 1/Ñ

– Random: Assuming |R>0| is given, randomly predict the membership of R>0 out of R


Experiments

TS

Rando

m

LM, N

S


Experiments

Enough impressions little “borrowing”

from siblings

Few impressions Estimates depend more on siblings


Related Work

• Multi-resolution modeling

– studied in time series modeling and spatial statistics [Openshaw+/79, Cressie/90, Chou+/94]

• Imputation

– studied in statistics [Darroch+/1972]

• Application of such models to estimation of such rare events (rates of ~10-3) is novel


Summary

• A method to estimate

– rates of extremely rare events

– at multiple resolutions

– under severe sparsity constraints

• The method has two parts

– Imputation incorporates hierarchy, fixes sampling bias

– Tree-structured generative model extremely fast parameter fitting


Overview





• Case Studies




• Online Learning


Collaborative Filtering

• Collaborative filtering– Similarity based methods

)()(

/iNj

ijujiNj

ijuisrsr

Rating (CTR) for query u of ad i

Ad-ad similarity matrix

Local neighborhood of ad i


Collaborative Filtering

• Collaborative filtering– Similarity based methods

– Possible adaptation

– Challenges: • Learning similarity

• Simultaneously incorporating query and ad similarities

)()(

/iNj

ijujiNj

ijuisrsr

)()(/

);,()(odds-log

aNjqjqj

aNjqjqa

qaqa

szsz

zfp θaq

Feature-based model

Collaborative filtering model


Matrix Factorization

• Matrix Factorization– Each query (ad) is a linear

combination of latent factors

– Solve for factors, under someregularization and constraints

r

kakqkqa

vufp1

)()(odds-log θa;q,

Factor coefficients for query

Factor coefficients

for ad


Matrix Factorization

• Matrix Factorization

• Bi-clustering

– Predictive Discrete latent factor models, Agarwal and Merugu, KDD 07.

r

kakqkqa

vufp1

)()(odds-log θa;q,

cluster ad:(a)cluster;Query :)(

);,()(odds-log)(),(

q

zaqfpaqqa


Overview





• Case Studies




• Online Learning


Challenges of Feature-based models

• Learns from clicks but still misses context in many instances as in relevance based approach

• Introducing features that are too granular makes it hard to learn CTR reliably

• Does not capture the dynamics of the system

• Training cost is high

• Slow prediction functions inadmissible due to latency constraints



• Other methods– Boosting, Neural nets, Decision Trees, Random Forests, ……

• Local models– Mixture of experts: Fit local, think global

• Hierarchical modeling with multiple trees– User interest, query, ad,..

– Each tree is different

– How to perform smoothing with multiple disparate trees?

L

1kk

A)Q,|click(A)Q,|P(clickk

P



• Combining cold start with warm start together main challenge in collaborative filtering based methods

• We believe, solving basic issues more challenging

– Positional bias

– Selection bias

– Correlation in ads on a slate

– Dynamic CTR; seasonal variations


Online learning


Overview



• Online Learning


Online learning for ad matching

• All previous approaches learn from historical data

• This has several drawbacks:

– Slow response to emerging patterns in the data• due to special events like elections, …

– Initial systemic biases are never corrected• If the system has never shown “sound system dock” ads

for the “iPod” query, it can never learn if this match is good

– System needs to be retrained periodically



• Solution: Combining exploitation with exploration

– Exploitation: Pick ads that are good according to current model

– Exploration: Pick ads that increase our knowledge about the entire space of ads

• Multi-armed bandits

– Background

– Applications to online advertising



Background: Bandits

Bandit “arms”

p1 p2 p3(unknown payoff

probabilities)

• “Pulling” arm i yields a reward:

• reward = 1 with probability pi (success)

• reward = 0 otherwise (failure)


Background: Bandits

• Goal: Pull arms sequentially so as to maximize the total expected reward

– Estimate payoff probabilities pi

– Bias the estimation process towards better arms

Bandit “arms”

p1 p2 p3(unknown payoff

probabilities)


Background: Bandits

• An algorithm to sequentially pick the arms is called a bandit policy

• Regret of a policy = how much extra payoff could be gained in expectation if the best arm is always pulled

– Of course, the best arm is not known to the policy

– Hence, the regret is the price of exploration

– Low regret implies that the policy quickly converges to the best arm

• What is the optimal policy?


Background: Bandits

• Which arm should be pulled next?– Not necessarily what looks best right now, since it might have

had a few lucky successes

– Seems to depend on some complicated function of the successes and failures of all arms

argmax g(s1, f1, s2, f2, …, sk, fk) ?

Number of successes

Number of failures


Background: Bandits


• Consider a bandit which

– has an infinite time horizon, but

– future rewards are geometrically discountedRtotal = R(1) + γ.R(2) + γ2.R(3) + … (0<γ<1)

• Theorem [Gittins/1979]: The optimal policy decouples and solves a bandit problem for each arm independently

argmax {g1(s1, f1), g2(s2, f2), …, gk(sk, fk)}

argmax g(s1, f1, s2, f2, …, sk, fk) ?


Background: Bandits


• Theorem [Gittins/1979]: The optimal policy decouples and solves a bandit problem for each arm independently

– Significantly reduces the dimension of the problem space

– Gives a minimum regret bound of O(log T)

– But, the optimal functions gi(si, fi) are hard to compute

– Need approximate methods…


Background: Bandits

Bandit Policy

1. Assign priority to each arm

2. “Pull” arm with max priority, and observe reward

3. Update priorities

Priority 1

Priority 2

Priority 3

Allocation

Estimation


Background: Bandits

• One common policy is UCB1 [Auer/2002]

Number of successes

Number of failures

Total number of observations

Number of observations of

arm i

Observed payoff

Factor representing uncertainty


Background: Bandits

• As total observations T becomes large:

– Observed payoff tends asymptotically towards the true payoff probability

– The system never completely “converges” to one best arm; only the rate of exploration tends to zero

Observed payoff



Background: Bandits

• Sub-optimal arms are pulled O(log T) times

• Hence, UCB1 has O(log T) regret

• This is the lowest possible regret

Observed payoff








– Background




Background: BanditsW

ebp

age

1

Bandit “arms”

We

bpa

ge 2

We

bpa

ge 3

= ads

~106 ads

~109 pages


Background: Bandits

Ads

Web

page

s

Content Match = A matrix

• Each row is a bandit

• Each cell has an unknown CTR

One bandit

Unknown CTR


Background: Bandits

Why not simply apply a bandit policy directly to our problem?

• Convergence is too slow ~109 bandits, with ~106 arms per bandit

• Additional structure is available, that can help Taxonomies


Taxonomies for dimensionality reduction

Root

Apparel Computers Travel

• Already exist

• Actively maintained

• Existing classifiers to map pages and ads to taxonomy nodes

Page/Ad

A bandit policy that uses this structure can be faster


Outline

Multi-level Bandit Policy for Content Match

• Experiments

• Summary


Multi-level Policy

Ads

Webpages

… …

……

……

classes

classes

Consider only two levels


Multi-level Policy

ApparelCompu-

ters Travel

… …

……

……

Consider only two levels

Tra

vel

Co

mp

u-

ters

Ap

pare

l

Ad parent classes

Ad child classes

Block

One bandit


Multi-level Policy

ApparelCompu-

ters Travel

… …

……

……

Key idea: CTRs in a block are homogeneous

Ad parent classes

Block

One bandit

Tra

vel

Co

mp

u-

ters

Ap

pare

l Ad child classes


Multi-level Policy

• CTRs in a block are homogeneous

– Used in allocation (picking ad for each new page)

– Used in estimation (updating priorities after each observation)


Multi-level Policy


Used in allocation (picking ad for each new page)

– Used in estimation (updating priorities after each observation)

Research © 2008 Yahoo!C

A C T

AT

Multi-level Policy (Allocation)

?

Page classifier

• Classify webpage page class, parent page class

• Run bandit on ad parent classes pick one ad parent class


A C T

AT


• Classify webpage page class, parent page class

• Run bandit on ad parent classes pick one ad parent class

• Run bandit among cells pick one ad class

• In general, continue from root to leaf final ad

?

Page classifier

ad


A C T

AT

ad


Bandits at higher levels

• use aggregated information

• have fewer bandit arms Quickly figure out the best ad parent class

Page classifier


Multi-level Policy


Used in allocation (picking ad for each new page)

Used in estimation (updating priorities after each observation)


Multi-level Policy (Estimation)


– Observations from one cell also give information about others in the block

– How can we model this dependence?



• Shrinkage Model

Scell | CTRcell ~ Bin (Ncell, CTRcell)

CTRcell ~ Beta (Paramsblock)

# clicks in cell# impressions

in cell

All cells in a block come from the same distribution



• Intuitively, this leads to shrinkage of cell CTRs towards block CTRs

E[CTR] = α.Priorblock + (1-α).Scell/Ncell

Estimated CTR

Beta prior (“block CTR”)

Observed CTR


Experiments

Root

20 nodes

221 nodes…

~7000 leaves

Taxonomy structure

We use these 2 levels

Depth 0

Depth 7

Depth 1

Depth 2


Experiments

• Data collected over a 1 day period

• Collected from only one server, under some other ad-matching rules (not our bandit)

• ~229M impressions

• CTR values have been linearly transformed for purposes of confidentiality


Experiments (Multi-level Policy)

Multi-level gives much higher #clicks

Number of pulls

Clic

ks


Experiments (Multi-level Policy)

Multi-level gives much better Mean-Squared Error it has learnt more from its explorations

Mea

n-S

quar

ed E

rror

Number of pulls


Experiments (Shrinkage)

Number of pulls Number of pullsMea

n-S

quar

ed E

rror

Clic

ks without shrinkage

with shrinkage

Shrinkage improved Mean-Squared Error, but no gain in #clicks


Summary

• Taxonomies exist for many datasets

• They can be used for

– Dimensionality Reduction

– Multi-level bandit policy higher #clicks

– Better estimation via shrinkage models better MSE







– Background




Challenges and Open Problems

• Bandit policies typically assume stationarity

• But, sudden changes are the norm in the online advertising world:

– Ads may be suddenly removed when they run out of budget

– New ads are constantly added to the system

– The total number of ads is huge, and full exploration may be too costly

– Mortal multi-armed bandits [NIPS/2008]


Mortal Multi-armed Bandits

• Traditional bandit policies like UCB1 spend a large fraction of their initial pulls on exploration

– Hard-earned knowledge may be lost due to finite arm lifetimes

• Method 1 (Sampling):

– Pick a random sample from the set of available arms

– Run UCB1 on sample, until some fraction of arms in the sample are lost

– Pro: Quicker convergence, more exploitation

– Con: Best arm in the sample may be worse than best arm overall

– Pick sample size to control this tradeoff



• Traditional bandit policies like UCB1 spend a large fraction of their initial pulls on exploration

– Hard-earned knowledge may be lost due to finite arm lifetimes

• Method 2 (Payoff threshold):

– New bandit policy: If the observed payoff of any arm is higher than a threshold, pull it till it expires

– Pro: Good arms, once found, are exploited quickly

– Con: While exploiting good arms, the best arm may be starving and may expire without being found

– Pick threshold to control this tradeoff



• Challenges:

– Selecting the critical sample size or threshold correctly, for arbitrary payoff distributions

– What if even the payoff distribution is unknown?


Challenges and Open Problems

• Mortal multi-armed bandits

• What if the bandit policy has some information about the budget?

– The bandit policy can control which arms expire, and when

– “Handling Advertisements of Unknown Quality in Search Advertising” by Pandey+/NIPS/2006

• Combining budgets with extra knowledge of ad CTRs

– E.g., Using an ad taxonomy

• Using a bandit scheme to infer/correct an ad taxonomy


Conclusions


Conclusions

• We provided an introduction to Online Advertising

– Discussed the eco-system and various actors involved

– Discussed different flavors of online advertising• Sponsored Search, Content Match, Display Advertising


Conclusions

Online Advertising

Revenue Models

Advertising Setting

Misc.

CPM CPC CPA


Sponsored Search

Ad exchanges


Conclusions

• Outlined associated statistical challenges

– Sponsored search, Content Match, Display

• We believe the following to be a technical roadmap

Offline Modeling Online ModelsTime series

Explore/Exploit

Multi-armed bandits

Regression, collaborative filtering, mixture of experts

Multi-resolution models

Selection bias Slate correlation

Noisy labels


Conclusions

• Offline Modeling– By far the best studied so far

– Not a careful study of selection bias, slate correlations, noisy labels. Good opportunity here

– More emphasis on matrix structure, goal is to estimate interactions

• Explore/Exploit– Some work using multi-armed bandits; long way to go

• Time series model to capture temporal aspects– Little work

• Holistic approach that combines all components in a principled way

Documents

Research © 2008 Yahoo! Statistical Challenges in Online Advertising Deepak Agarwal Deepayan Chakrabarti (Yahoo! Research)