MongoDB @

Viacom1

(or how to support a few million kids sliming their favorite celebrities all at once.)

Who Are We?Michael MakunasDirector - Multiplatform Engineering@mmakunas

Ramesh NuthalapatiArchitect - Multiplatform Engineering@nuthalapati

2

Who Are We?

Viacom Media NetworksViacom creates original entertainment content for every audience, on every screen around the world. The company's brands strive for creative excellence and cultural impact across programming, motion pictures and digital experiences that live online, on mobile, and via apps.

3

Today We Are Talking About...

Funnel

Viacom’s internal high volume polling, voting, and data collection service.

4

…and How MongoDB just works.

5

• Launched in 2011 on 2.0 • Currently on 2.4

Use CasesAward Show Voting

• MTV Video Music Awards (VMAs)• European Movie Awards (EMAs)• Kids Choice Awards (KCAs)• CMT Music Awards

Polling• MTV March Musical Madness• Spike Bellator On Air Match Voting

Contests• MTV Ultimate Fan Experience

6

What is “High Volume” to Viacom?

• From big to little, continuous “horizontal” support across 100+ global sites

• Voting, polling, and data collection occurs 24/7• No “maintenance windows”• Sudden spikes to 15K+ votes per second from on-air

“Throws” during prime time viewing: “Go to the site and Vote Now!”

• Storage of 100s of millions of votes for analysis after events

7

A Very Important Question...

8

Who will survive the Human Slime Car Wash!?!?!?

9

Statistics (Pushing The Limit)

2015 Kids Choice Awards

• 531 Million votes (a new record!)• 7 Million votes during the show

10

Some DefinitionsFunnel

• Our internal name for our systemSite

• A brand and/or site specific namespace for collecting data

Tent-pole• A major programming (entertainment) event

Collection• An individual poll, voting event, sweepstakes, etc. • NOT a MongoDB Collection

Entry• A single vote or poll entry

Processor• Performs an action on a entry (validate it, persists it, etc.)

11

History• Why did we build Funnel?• Why did we choose Mongo?

12

Why Did We Build It?• Previous systems unmanageable • One system for data collection

• Contests, sweepstakes, small polls• Built in-house with ATG Dynamo and MySQL• Per-site deployment model meant multiple installations

per brand• Flexible schema == MySql “xml blobs”

• Separate system for high volume polling• Comercial• Lack of customization• Per-brand installation

13

Why Did We Choose Mongo?

• Flexible schema document style database provides the quick-to-market support needed by 100+ sites, brands, and countries (All who want things “just a little bit different”)

• Proven scalability• JSON in and JSON out

14

Architectural Goals• High write throughput over read throughput and

consistency• Handle up to 10,000 1KB writes per second. Read

throughput is secondary• High Availability• Replace dozens of separate installations and services • Wide adoption across internal brands

15

Server Layout

16

Primary

Secondary

Secondary

Replica Set 1 (day-to-day use)

Replica Set 2 (tent-pole events)

Primary

Secondary

Secondary

Server Layout

• High availability• Primary, Secondary configuration with automatic

failover• Primary stores all reads and writes, secondary stays

in-sync with Primary• Monitoring with MongoDB Management Service (MMS)• Email Alerts from MMS for high volume op inserts

17

More on MMS

18

KCA 2015 MMS Graph

Funnel UI

Schema For a Typical Collection

"_id" : "kca-live-vote-2015-event","_class" :

"com.mtvi.funnel.domain.model.collection.Collection","site" : "kca.nick.com","name" : "kca-live-vote-2015-event","description" : "KCA Live Event","owner" : "nuthalar","start" : ISODate("2015-03-02T05:00:00Z"),"end" : ISODate("2015-05-10T04:00:00Z"),"createdDate" : ISODate("2015-03-03T17:34:55.113Z"),"lastUpdateDate" : ISODate("2015-03-26T14:40:15.665Z"),"tags" : ["live-show"],"processorChain" : [...],“questions” : [...] -- Questions has Options and

Validators

20

Processors• Each entry received by Funnel will be handled by an ordered list of

processors (the ProcessorChain) As each processor in the list is visited,

the processor will decide whether to continue to the next one or to

terminate further processing

• A Processor can raise an error, and its detail message will be

configurable and localizable. When an error is raised, it can signal the

system to take a few different actions depending on the severity of the

error condition:

Log the error but continue to processing

Bubble up the error to the client and cease further processing

Ignore the error, cease further processing, and return an OK

response to the client

21

Processors (UI)

22

Questions: Email

23

Questions: Email{ "name":"email", "displayText":"email", "persistable":true, "source":"ENTITY_BODY", "validators":[

{ "enabled":true,"failureMessage":"Please enter a valid email address to proceed.","contentType":"EmailAddressValidator", "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.EmailAddressValidator" }, { "enabled":true, "failureMessage":"This field is required. Please enter a response to proceed.", "contentType":"RequiredValidator", "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.RequiredValidator" } ], "seqNum":NumberLong(0), "tallyEnabled":false, "isMandatory":false, "contentType":"EmailQuestion"}

24

Questions: Single Choice

25

Questions: Single Choice

26

{ "multipleAnswers":false, "options":[ { "optionValue":"andrew-garfield-male-action-star", "displayText":"andrew-garfield-male-action-star", "customOption":false },

{ "optionValue":"channing-tatum-male-action-star","displayText":"channing-tatum-male-action-star","customOption":false},{ "optionValue":"chris-evans-male-action-star","displayText":"chris-evans-male-action-star","customOption":false }, { "optionValue":"chris-pratt-male-action-star", "displayText":"chris-pratt-male-action-star", "customOption":false },], "writeInOptionEnabled":false, "name":"favorite-male-action-star", "displayText":"favorite-male-action-star", "persistable":true, "source":"ENTITY_BODY",

"validators":[ { "enabled":false, "failureMessage":"This field is required. Please enter a response to proceed.", "contentType":"RequiredValidator", "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.RequiredValidator" }, { "enabled":true, "failureMessage":"Please select a valid choice to proceed.", "contentType":"ChoiceValidator", "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.ChoiceValidator" }, { "enabled":true, "failureMessage":"Please fill in the 'Other' value to proceed.", "contentType":"WriteInRequiredValidator", "_class":"com.mtvi.funnel.domain.model.collection.questions.validators.WriteInRequiredValidator" } ], "seqNum":NumberLong(0), "tallyEnabled":true, "isMandatory":false, "sm4MappingEnabled":false, "sm4Mapping":"", "exodusMapping":"", "contentType":"SingleChoiceQuestion", "_class":"com.mtvi.funnel.domain.model.collection.questions.QuestionWithOptions"}

An Entry would look like…

27

{"_id" : ObjectId("53c698c1ebd268ff9b76034d"),"answers" : {

"question" : "option-3","source" : "mobile","True-Client-IP" : "166.77.6.8"

},"createdDate" : ISODate("2014-07-

16T15:22:41.593Z")}

Entry Trend Graph

28

A Tally would look like…

29

{"_id" :

ObjectId("53c698c51180cc552e79c12d"),"option-1" : 1249596,"option-2" : 341404,"option-3" : 466646,"questionName" : "question","timeWindowType" : "ALL_TIME"

}

OptimizationsWe didn’t get it perfect the first time:

• Correct Indices - Index on created Date of Entry Collection. EnsureIndex is called during collection creation and update

• Batch update of vote tallies - Update the count per second instead of per Entry

• Hourly Stats – Pre-caluculating hourly totals saved resources during live events to get stats for past 24 hours, 7 days and 30 days

30

SPAM Prevention

• De-duplicator processor: De-duplicates X number of entries for a defined period (eg. 30 Entries per minute). Internal to Funnel application

• VERISIGN DDoS Protection service

• Akamai WAF (part of Kona Web Application Firewall)

• CAPTCHA

• Authentication (FLUX - Viacom Internal System)

31

Before & After

Previous Systems• Decentralized• Multiple Installations• Outdated App Server• Rigid Schema requiring

rigid change control process

32

Funnel• Service Oriented• Performant• Flexible Schema; easily

extended

Our Track Record• 100+ Sites• 15K Collections• 10+ Billion Votes to date

33

Where else MongoDB is Used @ Viacom

• Used in production since v 1.6• Content Management System • Viacom Identity Platform (Auth & User Profiles)

34

Wait…

35

Who survived the Human Slime Carwash!?!?!?

36

Questions ?

37

Thank You!

38

• Michael Makunas -- @mmakunas• Ramesh Nuthalapati -- @nuthalapati