Panos Ipeirotis New York University Opinion Mining using Econometrics A Case Study on Reputation Systems Joint work with Anindya Ghose and Arun Sundararajan

Panos IpeirotisPanos Ipeirotis

New York UniversityNew York University

Opinion Mining using Econometrics Opinion Mining using Econometrics A Case Study on Reputation SystemsA Case Study on Reputation Systems

Joint work with Anindya Ghose Joint work with Anindya Ghose and Arun Sundararajanand Arun Sundararajan

Comparative Shopping in e-MarketplacesComparative Shopping in e-Marketplaces

Are Customers Irrational?Are Customers Irrational?

Are Customers Irrational?Are Customers Irrational?

$11.04

$18.28

-$0.61

-$9.00

-$11.40

-$1.04

Price Premiums

Price premiums @ Amazon Price premiums @ Amazon

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

-100 -75 -50 -25 0 25 50 75 100

Price Premium

Nu

mb

er

of

Tra

ns

ac

tio

ns

Irra

tiona

l (?)

Average price premiums @ AmazonAverage price premiums @ Amazon

0

200

400

600

800

1000

1200

-100 -75 -50 -25 0 25 50 75 100

Average Price Premium

Nu

mb

er

of

Tra

ns

ac

tio

ns

Irra

tiona

l (?)

Why not buying the cheapest?Why not buying the cheapest?

You buy more than a product

Customers do not pay only for the product

Customers also pay for a set of fulfillment characteristics

Delivery

Packaging

Responsiveness

…

Reputation Matters!

Reputation SystemsReputation Systems

Facilitate electronic commerce

Integral part of online marketplaces

Provide information about unobserved fulfillment characteristics (most of which we take for granted in traditional commerce)

Reputation in ecommerce is complex

Different buyers value different fulfillment characteristics

Sellers have varying abilities on these characteristics

Example of a reputation profileExample of a reputation profile

Reputation profiles: ObservationsReputation profiles: Observations

Reputation profile capture more than “averages”

Well beyond “average score” and “lifetime”

Rich textual content: information about a seller on a variety of dimensions (or fulfillment characteristics).

How the seller’s performance (potentially on each of these characteristics) has evolved over time

Buyer-seller networks

Reputation in ecommerce is complex

Different buyers value different fulfillment characteristics

Sellers have varying abilities on these characteristics

Previous work studied only effect of “average score” and “lifetime”

Our research agendaOur research agenda

What are the dimensions of online reputation?

What characteristics comprise the important parts of a seller’s overall reputation? (politeness? packaging? delivery?)

How do these dimensions affect pricing power?

Does a better reputation enable a seller to charge a higher price?

Which dimensions affect this pricing power most significantly?

Average numerical ratings?

Number of prior successful transactions?

Assessments of ability on specific fulfillment characteristics?

Do competitors with better reputations limit a seller’s pricing power?

Can prior reputation predict marketplace outcomes?

Given a set of sellers, their reputations, and their prices, can one predict which seller will successfully make the sale?

DataData

Overview

Panel of 280 software products sold by Amazon.com

Data on all “secondary” market transactions

Amazon Web services facilitate capturing transactions

Complete reputation profile for all sellers who completed one or more transactions during this period

Summary

280 products X 180 days

1,078 sellers, of which 122 transacted

12,232 transactions

107,922 “observations” (seller-competitor pairs)

Data: TransactionsData: Transactions

Sales of (mostly new) software


Capturing transactions and “price premiums”


Seller ListingItem Price

When item is sold, listing disappears



While listing appears, item is still available

time

1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10



While listing appears, item is still available

time

1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10

Item still not sold on 1/7



When item is sold, listing disappears

time

1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10

Item sold on 1/9

Data: Variables of InterestData: Variables of Interest

Regular Price Premium

Difference in the price charged by a seller and the listed price of a competing seller at the time the transaction occurred

(Seller Price – Competitor Price)

Calculated for each seller-competitor pair, for each transaction

Each transaction therefore generates N observations, where N is the number of competing sellers


Difference in the price charged by a seller and the average price of all competing sellers at the time the transaction occurred

(Seller Price – Avg. (Competitor Price) )

Calculated for each transaction

Each transaction generates 1 observation

Price premiums @ Amazon Price premiums @ Amazon

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

-100 -75 -50 -25 0 25 50 75 100

Price Premium

Nu

mb

er

of

Tra

ns

ac

tio

ns

Average price premiums @ AmazonAverage price premiums @ Amazon

0

200

400

600

800

1000

1200

-100 -75 -50 -25 0 25 50 75 100


Nu

mb

er

of

Tra

ns

ac

tio

ns

The dimensions of reputationThe dimensions of reputation

How reputation affects price premiums?

Decomposing reputationDecomposing reputation

Is reputation just a scalar metric?

Previous studies assumed a “monolithic” reputation.

We break down reputation in individual components

Sellers characterized by a set of n fulfillment characteristics

What are these characteristics (valued by consumers?)

We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)

We scan the textual feedback to discover these dimensions

seller lifeseller ranking

Data: Reputation ProfilesData: Reputation Profiles

Decomposing and scoring reputationDecomposing and scoring reputation

Decomposing and scoring reputation

We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”)

The sellers are rated on these dimensions by buyers using modifiers (adjectives or adverbs), not numerical scores

“Fast shipping!”

“Great packaging”

“Awesome unresponsiveness”

“Unbelievable delays”

“Unbelievable price”

Dimensions from text: ExampleDimensions from text: Example

ij

Parsing the feedback

P1: I was impressed by the speedy delivery! Great Service!

P2: The item arrived in awful packaging, and the delivery was slow

…Identified modifier-dimension pairs

P1: “speedy – delivery”, “great – service”

P2: “awful – packaging”, “slow – delivery”

…Reducing textual feedback to a n X p matrix

Dimensions: 1-delivery, 2-packaging, 3-service

ij

11 12 13" ", " ", " "speedy NULL great

21 22 23" ", " ", " "slow awful NULL Postings

Decomposing and scoring reputationDecomposing and scoring reputation

Scoring reputation

“Fast shipping!”

“Great packaging”

“Awesome unresponsiveness”

“Unbelievable delays”

“Unbelievable price”

How can we find out the meaning of these adjectives?


We assume that each modifier assigns a “score” to each dimension

:score associated with appearing as the modifier for the k-th dimension

ri: weight of posting that appears on the i-th position (weight down old posts)

wi: weight assigned to the i-th dimension

Thus, the overall (text) reputation score Π(i) is:

11 1 1

1 2

1

( ,1) ... ( , )

( ) , ,...,

( ,1) ... ( , )

i in

pi ip pn n

a a n w

i r r r

a a n w

( , )a k

scores forfirst dimension

scores forn-th dimension

,

( ) ( ( , )) ( , )i j ji j

i w a i R i Sum of ri weights in which

j modifies dimension i

estimated coefficients

scores forfirst posting


Scoring the dimensions

Use price premiums as “true” reputation score

Use regression to assess scores (coefficients) for each dimension-modifier pair

Regressions

Control for all variables that affect price premiums

Control for all numeric scores of reputation

Examine effect of text: E.g., seller with “fast delivery” has premium $10 over seller with “slow delivery”

,

( ) ( ( , )) ( , )i j ji j

i w a i R i estimated coefficients

Some indicative dollar valuesSome indicative dollar values

Positive

Negative

Natural method for extraction of sentiment strength and polarity

ResultsResults

Some dimensions that matter

Delivery and contract fulfillment (extent and speed)

Product quality and appropriate description

Packaging

Customer service

Price (!)

Responsiveness/Communication (speed and quality)

Overall feeling (transaction)

ResultsResults

Further evidence

Classifier (aka choice model) that predicts sale given set of sellers

Binary decision between seller and competitor

Naïve Bayes and Decision Trees (SVM’s forthcoming)

Only prices and characteristics: 53%

+ numerical reputation, lifetime: 74%

+ encoded textual information: 89%

Other applicationsOther applications

Summarize and query reputation data

Give me all merchants that deliver fast

SELECT merchant FROM reputation

WHERE delivery > ‘fast’

Summarize reputation of seller XYZ Inc.

Delivery: 3.8/5

Responsiveness: 4.8/5

Packaging: 4.9/5

Pricing reputation

Given the competition, merchant XYZ can charge $20 more and still make the sale (confidence: 83%)

SummarySummary

Key contributions

New technique that automatically scores “sentiment” based on economic data

Validation by multiple methods (estimating an econometric model, building classifiers)

New evidence of the extent to which interdisciplinary research can be fun and distracting

Broader contribution

Economic data is abundant and there is rich literature on how to handle such data

Economic data can be used for training for MANY applications

Moving aheadMoving ahead

Extensions of current work

Dimensionality reduction, grouping dimensions topics that might correspond more closely to the “true” dimensions of reputation

Latent Dirichlet Allocation, (probabilistic) Latent Semantic Analysis, Non-negative Matrix Factorization, Tensors

Identifying weights for dimensions, using normalized scores

“Correct” game theoretic model of market competition

Exploiting network structure

Exploring connection with the “trustrank” literature

Network position as an additional dimension of seller reputation

Buyers as seller/category specific “authorities”

Thank you!Thank you!

http://economining.stern.nyu.edu

Prior studies of reputation Prior studies of reputation

Positive feedback significant, negative not

Ba and Pavlou (2002) for CD’s, software, electronics; Bajari and Hortacsu (2003) for collectible coins?

Negative feedback significant, positive not

Lee et al. (2000) for computer equipment, Reiley et al. (2000) for collectible coins

Nature of price: winning online auction bid (usually eBay)Measure of reputation: average numerical score, # of transactions

Both positive and negative feedback significant

Dewan and Hsu (2004) for rare stamps, Melnik and Alm (2002) for gold coins, Houser and Wooders (2005) for Pentium chips

Documents

Panos Ipeirotis New York University Opinion Mining using Econometrics A Case Study on Reputation Systems Joint work with Anindya Ghose and Arun Sundararajan