Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
Ad Serving at Spotify ScaleA journey of incremental full stack overhaul
Kinshuk Mishra, Director of [email protected]@_kinshukmishra
What I do● Founded ads engineering team at Spotify in 2011
● Build all things ads engineering - team & software
● Major focus areas :
○ Ad delivery (Backend and Web)
○ Multi-platform native ads (Client Platform)
○ Ad performance (ML and Data)
In essence it is pretty simple
Client
User Profile database
Ad Server
Campaign Management Portal
Billing/Reporting
Ad campaign database
Data CollectionSystem
Spotify Ads infrastructure in 2011
EdgeServiceDesktop
LogDelivery HDFS
User Profile
Batch
Basic Ad Server
CampaignManagement
Billing/Reporting
Spotify Ads infrastructure in 2017
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
Multi-platform clients
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
Data collection
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
Intelligence
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
Ad Delivery
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
Demand fulfillment
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
Architecture overhaul is hard
● While keeping the business running
● While innovating on new products
● When you should have done it yesterday
Future needs●
● Growth in scale
● Emergence of new client platforms
● Cheap cloud computing
● New products to meet business objectives
● Technical debt
Original ad server designEdge Service
Routerhash(userid)
Ad server ring with partitions
Ad server instance
Memcache
Memcache
Memcache
Memcache
Campaign DB
User DBDesktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
The ad server transition
EdgeService
LogDelivery HDFS
User Profile
Batch
Smart Ad Server
CampaignManagement
Billing/Reporting
Ad Server Proxy
(routing) Basic Ad Server
Gradual transition from basic to smart ad serving
Desktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
After the ad server transition
Proxy Service
LogDelivery HDFS
User Profile
Batch
CampaignManagement
Billing/Reporting
Smart Ad Server
Desktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
Division of responsibilities
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
Ad decisioning
Ad fetch orchestration
Client context
Ad Trigger & Render
Before After
Remember division of responsibilities?
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
Ad decisioning
Ad fetch orchestration
Client context
Ad Trigger & Render
BAD GOOD
Multiplatform Client design
iOS
Proxy Service
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
ModelingSelf-Serve Service
Creative Generation Payments
Billing/Reporting
Event Stream Historical
ETL1 ETL2 ETL3
UserEntity1(attribute1, attribute2) UserEntity1(attribute1, attribute3) UserEntity1(attribute1, attribute3’)