Big Data : Bits of History, Words of Advice

Big Data : Bits of History, Words of Advice

Venu Vasudevan

GLSEC Big Data Meetup

Big Data : Bits of History, Words of Advice

Big Data PastBig

Fast

intelligent mediaIoT

satellites

Big Data : Behavioral

Big Data

- The ‘V’ view of Big Data challenges - Number of V’s up for debate

Big Data : Architectural

untidy data

firehose

clean analytics

fast & good

slower & much better

Lambda architecture

Lake architecture

Stream architecture

Technical

Technical

This Talk

Behavioral View

Technology Solution

Stack

‘Middleware’ (benefit of hindsight)

some more some

governance culture (gap)

data economics

owne

rshi

p

foo

d fi

ght

s

data eco

nom

ics

3 data pointsBig

Fast

intelligent mediaIoT

satellites

Iridium

• mobile routers (10K mph), fixed people

• no repeated patterns

• satellites N-S movement

• earth E-W movement

• regular topology, irregular exceptions

• solar flares

• military satellite presence

Fast Data Problem

• cellular frequency allocation (graph coloring problem)

• frequent fast recalculations (fast routers + semi-fast earth)

• transmit-no transmit (solar flares, military satellite presence)

• moving ‘seam’

seam

irreg

ular

ities

Fast Data Problem

• cellular frequency allocation (graph coloring problem)

• frequent fast recalculations (fast routers + semi-fast earth)

• transmit-no transmit (solar flares, military satellite presence)

• moving ‘seam’

• + ‘France’

seam

irreg

ular

ities broadcast

= +$$$

broadcast = -$$$ (lawsuit)

Fast Data Problem

• quest for (OO)DB technology to address ‘France’ as make-or-break use case

• query expressive power

• complex constraint satisfaction

• query handling throughput

• 3-4 month benchmarking effort

seam

broadcast = +$$$

broadcast = -$$$ (lawsuit)

Fast Data Problem

• quest for (OO)DB technology to address ‘France’

• query expressive power

• query handling throughput

• 3-4 month benchmarking effort

• France solved ‘out-of-band’ (legally)

seam

broadcast = +$$$

broadcast = -$$$ (lawsuit)don’t overfit your architecture to

an extreme requirement

unless it’s from an extreme (paying) user

Big Data Problem

• systems management

• manage 66 ‘nodes’

• nodes moving at 10K mph

• ‘seam’ moving of 20K mph

• sounds harder than trivial, but not too hard

‘Pre’ Lambda Solution

• Dumb edge | smart core approach

• 15K events/sec/satellite

• 1M events/sec

• Fast & Approximate - FMEA: ’compiled’ lookup table for failure modes

• Slow & Precise - Model-based reasoning on satellite models

untidy satellite firehose

(1M events/sec)

actionable insights

‘Pre’ Lambda architecture

Model-Based Reasoning

FMEA

‘Pre’ Lambda Solution

• Dumb edge | smart core approach

• 15K events/sec/satellite

• Fast & Approximate - FMEA: ’compiled’ lookup table for failure modes

• Slow & Precise - Model-based reasoning on satellite models

• Simple, straightforward & wrong.

untidy satellite firehose

(1M events/sec)

actionable insights

‘Pre’ Lambda architecture

Model-Based Reasoning

real-time expert system

FMEA

Yet, an architecture that is

‘rinsed and repeated’

over the years

why does dumb edge smart cloud endure?

• edges are expensive ($2B)

• when edges go wrong (break/blow up /collide) , they make headlines

$

$$$$$

why dumb edge smart cloud

• edges are expensive ($2B)

• when edges go wrong (break/blow up /collide) and make headlines

• nobody messes with an ‘edge’ once it works

• clouds don’t make for good news headlines

$ T-0

$$$$$ T-30 yrs

why dumb edge smart cloud

• edges are expensive ($2B)

• when edges go wrong (break/blow up /collide) and make news headlines

• nobody messes with an ‘edge’ once it works

• thus, implementing an end-to-end architecture causes culture clashes

over my dead body

iterate & refine

an almost repeat (Industrial IoT)

• edges are messy & domain specific

• creating them means dealing with culture clashes

• but .. an ounce of edge is worth a pound of cloud

$$$$$ T-30 yrs

$ T-0

Things to consider• Problem statement. What’s your ‘France’?

• colorful sub-problem. strategy overfit.

• Architecture. small fixes to IT/OT gap can go a long way to a simpler problem

• Technology Choices. best practices & the risk of ‘rewardless risk’

• right - make average programmers productive with new tech

• frequent - turn great programmers into average

Big Data to Deep Metadata

streaming video(TV) ~ 1 petabyte/day

second

minute

hour

day/week

epochal

detect & replace ads

Create Playlists by Player,

Play, Sentiment

Identify minor characters with rabid fan following

rejuvenate old content

derive new

content

‘chapterize’ by Player,

Play, Sentiment

Platform Triage Challenge new Product, new market

• one core technology, many markets

• platform triaging challenge. what drives the platform?

• highest (but uncertain) $ potential?

• ‘extreme’ requirement?

• sparsest competition?

• use case outlier is your biggest customer

deep metadata

technology

SaaS data

platform

Advertising

Search

Video concept

maps

ad replacement use case• speed

• few days (on-demand content)

• few seconds (real-time rebroadcast with new ads)

• precision

• low - best effort, for low cost international content for niche audiences

• high - frame level for expensive content. e.g. Sports/$10M/episode programming

• errors

• 90% accuracy - ok for long tail content

• ‘five nines’ for premium content

precision accu

racy

spee

d

ad replacement opportunity space

largest customer

occam’s razor works (again)

• build to simplicity

• loose coupling between data engg & equipment engg

• modularize complexity

• ‘differentiate your product’ changes

• ‘necessary evil’ changes

data-only approach

+1st party integration (dynamically configure

ad splicers)

3rd party knobs (dynamically refresh CDN)

Architecture

but, what if ..

• Data is untidy

• Interpretation is subjective/cultural

• Automation is aspirational but quixotic

human-powered analytics

• some analytics tasks are too ‘slippery’ for machines

• data hard to characterize

• uneven video quality of ‘old’ archives

• untidy

• insights are subjective

human-powered analytics

• some analytics tasks are too ‘slippery’ for machines

• need for human augmentation

• humans generate ‘training’ sets to bootstrap m/c learning

• humans completely take over some tasks

machines vs humans

• crowdsourcing & human-powered computing

• has been the ‘next big thing’ for a while

• checkered history:

• uneven output

• fraud

• uneven throughput

Machines Humans

fast slow

brittle malleable

objective subjective

clear nuanced

machines vs humans

• much of that has changed

• Amazon Mech Turk

• 500K active users

• the ‘human machine’ can return substantial jobs in under 30 mins

• quantifiable as a machine for many media tasks - latency, quality, error rate, thruput

Hybrid Architecture

Things to consider• Beware ‘France’ in other forms:

• customer with loudest voice & ‘holy grail’ hairball

• Dealing with data quality & variability

• crowdsourcing has come a long way as credible ‘engine’

• If big data the answer, what is the question? (have strong opinion held weakly)

• decision rationalization

• process automation

• human ‘power tool’ (e.g. compelling visualization) vs imperfect automation

startup data jiu-jitsu

• How to create a data-driven strategy before the data shows up?

• rationalize future SaaS revenue models

• justify product decisions in a data-driven manner

need data for product

need product for data

startup data jiu-jitsu

• How to create a data-driven strategy before the data shows up?

• how ‘intelligent’ can lighting control be with 50-100K users?

• how do people use dimmers (continuous or quantized) — UX implications

data set dilemma

• standard sources (e.g. Kaggle & UCI) insufficient

• few ‘physical world’ datasets

• expensive to collect

• may be specialized (vendor-specific)

• dataset proxies for IoT actuation may not work

• energy utilization != switch usage

big data, small start

• physical world data likely to be smaller (1-10 homes, few months)

• setup costs limit size of public datasets

• e.g. UMass Smart* light switch dataset

big data, small start

• consider data ‘augmentation’

• standard practice in AI (deep learning) - horizontally flipping, random crops …

• under-used in data space

• may need some thought on perturbation models for your domain

real

synthesized

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

In short ..

• big data success - equal parts tech & non-tech

• solving right problem, not just problem right

• revisit problem, and what success means

@venuv62 venu.vasudevan@nextio.co