1

Logging and machine data at Scale

James Governor @monkchips co-founder.

2

“The lumber industry sells what used to be waste — sawdust, chips, and shredded wood — for a pretty profit. Today you’ll find these by-products in synthetic fireplace logs, concrete, ice strengtheners, mulch, particle board, fuel, livestock and pet bedding, winter road traction, weed killing and more”

– Jason Fried, CEO Basecamp

3

4

5

6

7

8

9

Zero Users Care What The System Health is

All Users Care What Their Experience Is

Nines Don’t Matter if Users Aren’t Happy

Charity Majors, Honeycomb.io

10

Observability bridges tracing, logging, and monitoring

Logs are no longer about history

Real time, streams, and adhoc queries

11

https://simplicable.com/new/machine-data

12

Use Cases

ComplianceDevOps, IT Service management/root cause analysisecommerce, digital marketing optimisationFraud detectionIoT and integration with data from physical assetsHealthcareReal time security

13

Always Compliance

Anti-money laundering

PCI

GDPR is coming – fines potentially 4% of global turnover

14

Splunk

Dubai Airport instruments literally everything– Even hand washing in the bathroom from electronic taps and soap

Rackspace ingests 3TB of data every day– Security monitoring and management

Tactical Assault Light Operator Suit (TALOS)– Monitoring including vital signs

15

ELK as a disruptive entrant

Started as text search, now as a default for many orgs

Sprint – 3bn events per day– Digital Transformation 4.0 Project– real-time data from Retail Management and Store Ops– 200 dashboards represent events from logs, databases, emails,

syslogs, test messages, and internal and vendor application APIs.

Uber, Facebook, Netflix

16

17

18

Edge Filtering

IoT

Sampling and polling

19

Tracing at Facebook

2014 Mystery Machine took 2 hours to compute a model from 1.3M traces

2017 Canopy generates and processes 1.3 billion traces per day

(spanning end-user devices, web servers, and backend services, backs 129 performance datasets from high-level end-to-end metrics to specific custom use cases)

Source https://blog.acolyer.org/2017/11/22/canopy-an-end-to-end-performance-tracing-and-analysis-system/

20

Roll your own kind of sucks

21

Using machine data for business process optimization

Data isn’t on the balance sheet

But data as product is

Data optimized services and customer experiences

Data culture

Observability is the new hotness

22

Cloud Native Tools

Grafana – visualization graph and dashboards,

Prometheus – monitoring and alerting, time series

Fluentd – data collector and log aggregation

Graphite – time series and graphs