From Screens to Context

Tim Park

Microsoft

tpark@microsoft.com

@timpark

Telemetry Ingestion

Car Client

type: engine

ts: 05/03/2014 08:00:24

rpms: 23.23

fuel: 2.2

type: location

ts: 05/03/2014 08:00:23

latitude: -33.92253

longitude: 18.4239

type: location

ts: 05/03/2014 08:00:26

latitude: 38.2

longitude: 2.2

type: location

ts: 05/03/2014 08:00:25

latitude: 38.2

longitude: 2.2

Inge

stio

n W

eb S

ervi

ce

MQTTu

C(1

28

KB

, 80

MH

z)

3G

Avalanche!

122,000 cars in their fleet.

35 interesting data points a second during operation

45 minutes a day per car across their fleet.

94,500 data points per car per day.

11.58 billion data points per day (133,984 per second).

4.2 trillion data points per year.

Make sure the telemetry you collect has business impact.

Even then, there is still a ton of data coming your way.

Telemetry Ingestion

MQTT AMQP

Authentication and Authorization

Ingestion Service

REST

Device Registry

Data Pipeline

Cars Browsers, MobileServers

Data Pipeline

Data Pipeline

Storm

Table Storage

Azure Machine Learning

Driver Risk Scoring

Destination Prediction

Raw TelemetryClassified Driving Segments, Location PredictionsSummarized / Cleansed / Derivative Data

Application API

Driver Risk Application Destination Prediction Application

Hadoop

messagestream

Blob

Storage

Event Hub

type: location

ts: 05/03/2014 08:00:23

latitude: -33.92253

longitude: 18.4212

type: location

ts: 05/03/2014 08:00:25

latitude: -33.92273

longitude: 18.4239

type: location

ts: 05/03/2014 08:30:01

latitude: -33.92269

longitude: 18.4239

type: location

ts: 05/03/2014 08:00:24

latitude: -34.2213

longitude: 18.9124

session: 132

total_time: 23m12s

total_distance: 18.19km

start: -33.92253, 18.4212

finish: -33.92273, 18.4239

session: 133

…

Streaming Data Pipelinetype: location

ts: 05/03/2014 08:00:23

latitude: -33.92253

longitude: 18.4212

type: location

ts: 05/03/2014 08:00:25

latitude: -33.92273

longitude: 18.4239

….

Storm Data Pipeline Topology

Location filtering

Raw

Location

Stream

Session Identification

Cleansed

Location

Stream

Door Ajar

Ignition

Driving

Sessions

Apache Storm Infrastructure

spout

spout

bolt

bolt

Machine Learning

split

dataset

training data test data

train model

score model

algorithm

Supervised Machine Learning

Deployment

Logical Deployment View

Frontdoor Service (API Surface Consolidation)

Ingestion Service Device Registry Service Consumption Service

REST Auth QueryProvisioning

Devices Applications

WebsocketsRESTMQTT MQTT

Node.js service boundaries

Deployment Sizing

We send up roughly 2 messages a second during operation of the vehicle.

37% peak concurrent usage of car fleet is typical.

Design goal of monitoring car fleet of 30,000 vehicles.

Deployment designed to be capable of 60% usage of fleet.

Need to be able to support 60k messages/sec load from car fleet.

Each ingestion instance can handle roughly 1000 messages/second

60 ingestion instances

20 frontdoor instances

10 consumption instances

5 device registry instances

App A

bins / lib

Guest

OS

Hypervisor

Host OS

Server Hardware

App A’

bins / lib

Guest

OS

App B

bins / lib

Guest

OS

App A

bins / lib

Host OS

Server Hardware

App A’ App B

bins / lib Docker

VMs vs. ContainersVM

Container

CoreOSStripped down Linux distro

Optimized to run containers

Autoupdating

Systemd / Fleetctl

etcd

“Warehouse scale computing”

Thinking in cores and containers instead of VMs

Deis Deployment Internals

I C

F I

CoreOS

I I

I C

CoreOS

I F

I I

CoreOS

I I

R I

CoreOS

I I

F I

CoreOS

I I

R I

CoreOS

R I

F I

CoreOS

I I

C I

CoreOS

F I

I I

CoreOS

R I

I C

CoreOS

Deis Request Router

Summary

Context is the next user interface

Collecting context is very high scale by its nature

Need to use new approaches to store this data.

Need to use new approaches to process this data.

Need to use new approaches to learn from this data.

nitrogen.io

7

Thanks!

Tim Park

@timpark