Cloud Tech III: Actionable Metrics

Actionable MetricsEnabling Decision-Making in Netflix’s Decentralized

Environment

Cloud Tech IIIOctober 6, 2012Roy Rapoport

@royrapoport, rsr@netflix.com

Thursday, October 18, 12

• Been in tech for about 20 years

• Systems engineering, networking, software development, QA, release management

• Time at Netflix: 1195 days (3y:3m:1w)

• (Current) job at Netflix: Make things better (Security Monkey, Python Platform, Central Alert Gateway, Breaking Stuff.. )

Metrics Humor

% of instances with even public IP addresses

Technology Overview

Technology Overview• SoA, REST, Mostly Java

• Simple overall architecture:

Culture Overview

• Freedom and Responsibility

Culture Overview

• Distributed Operations

Culture Overview

• Distributed Operations

• Get out of the way of Developers

The Metric Lifecycle

•Send

•Look

•Send

•Look

•Alert

Systems

• Flexible

• Scalable

• Self-Service

TelemetryFlexible, Scalable, Self-Service

import netflix.metrics[...] self.nm = netflix.metrics.Metrics("core_cag")[...]def api(self): self.nm.nfCounter("api") [...] self.nm.nfCounter(“application_%s” % application)[...]

VisualizationFlexible, Scalable, Self-Service

AlertingFlexible, Scalable, Self-Service

• Static vs Dynamic Thresholds

• Compare to history

For Example ...

What the ...

Last 3 hours’ core_tools.core_cag_api

For Example ...Visualization (Continued)

Last 4 days’ core_tools.core_cag_api

even more questions!

Last 10 days’ core_tools.core_cag_api

What caused the spike?

Show alert volume per application

Someone had a rough few days...

Don’t Like Surprises...{ "alerts": [ { "applyTo": "cluster", "condition": { "minPercent": 90.0, "noise" : .2, "maxPercent": 25.0, "type": "DoubleExponential" }, "metricName": "core_cag_api", "severity": "major" } ], "clusters": [ "core_tools" ]}

Threshold Tuning

• An Abbreviated History ...

Threshold Tuning(in the beginning)

Some priests offer their prayers to alien creatures best left forgotten. This ill-advised worship twists their minds in odd ways. Overlords find these warped men useful due to the unnatural powers they can channel. The dark priests most favored by their strange gods have powerful protections, and defeating one of them is sure to bring down a terrible curse upon the victor.

- http://www.descentinthedark.com/_d_/dark_priests.php

• Systems owned by IT

• Want an alert? Submit a ticket

• Want to tune an alert? Submit a ticket

Threshold Tuning(It gets better)

• You get to configure your own threshold

• Freedom!

• Also, you have to configure your own thresholds

Threshold Tuning(Are we there yet?)

• Play with historical data

• Huge difference

• Still falls short

Threshold Tuning(Yeah, that’s the ticket)

• Computers can be good at this

If Time Allows ...

Events vs Metrics

• Irregular Interval

Events vs Metrics

• Point in time

Events vs Metrics

• Point in time

• Lack magnitude

Why Build It?

• Change management

• Vs Change control

Why Build It?

• What Changed?

Why Build It?

• What Changed?

• Better Alerting

Chronos

• Rapidly Prototyped

Chronos

• Adapters and reporters

Chronos

• Easy querying

Chronos

• Easy querying

• Alarming

• Something happened

Chronos

• Easy querying

• Alarming

• ... X times in Y minutes

Chronos

• Easy querying

• Alarming

• ... X times in Y minutes

• Something didn’t happen

Chronos

• Easy querying

• Alarming

• Medium volume

Chronos

• Easy querying

• Alarming

• Medium volume

• Recursive

End Result

• Massive decrease in change control tickets

End Result

• Not talking about SOX or PCI

End Result

• Better visibility into changes

End Result

• Decreased TTR

End Result

• Decreased TTR

• Especially for bad code deployments

End Result

• Decreased TTR

• Especially for bad code deployments

• You should do this

I Didn’t Mention

• End-to-end testing and alerting

• External availability and performance

• Open Connect

• Jobs

Questions?

Cloud Tech III: Actionable Metrics

Technology

Corporate Metrics Tech Doc

Sieve: Actionable Insights from Monitored Metrics in ... · Sieve: Actionable Insights from Monitored Metrics in Microservices Jörg Thalheim1, Antonio Rodrigues2, Istemi Ekin Akkus3,

LSPE Presentation: Actionable Metrics at Netflix

Actionable Agile Metrics for Predictability › download › Chapter-13-Pull-Policies.pdf · Chapter13-PullPolicies 4 Inthesmallairports,unfortunately,thereisstillusuallyonlyone upfront,credential-checkingagenttoservebothoftheselines.The

Actionable SaaS Metrics: Churn Rate

Actionable Software Metrics: An Industrial Perspectivesmartinez/wp-content/papercite-data/pdf/ram20… · contribute any actionable information [25]. Essentially, a metric that can

Advanced Metrics in Social Media MMC6936 · generation metrics, search data (SEO metrics), social media optimization, social listening using actionable tools, social media ROI, and

Analysing Your Marketing Mix Through Actionable Metrics Within A Fair Attribution Model

ACTIONABLE CONTINUOUS DELIVERY METRICS · 2020-05-19 · ACTIONABLE CONTINUOUS DELIVERY METRICS Suzanne Prince, Head of Product, ThoughtWorks Products. Head of Product for ThoughtWorks

SharePoint 2013 Usage Analytics and Making Metrics Actionable

Sieve: Actionable Insights from Monitored Metrics in Distributed Systems · 2018-05-15 · Sieve: Actionable Insights from Monitored Metrics in Distributed Systems Jörg Thalheim1∗,

Actionable Metrics at Production Scale - LSPE Meetup June 27, 2012

The new Cross-Channel Navigator: Actionable Channel Metrics in Oncology

From Scrum to Flow using Actionable Agile Metrics

Outsourcing Achieving High Performance through Application ......Metrics and performance management Measuring application development and maintenance on a consistent and actionable

Tech Talk: Up Your Agile Game with Predictability Metrics

Actionable Agile Metrics for Predictabilitysamples.leanpub.com/actionableagilemetrics-sample.pdfTABLEOFCONTENTS Preface PARTONE-FLOWFORPREDICTABILITY Chapter1-Flow,FlowMetrics,andPredictability

VMworld 2013: Create a Key Metrics-based Actionable Roadmap to Deliver IT as a Service

Achieving “Actionable Logistics” · 2017-05-19 · Achieving “Actionable Logistics ... • Clear outcomeClear outcome--related performance metricsrelated performance metrics

6 actionable google analytics metrics you can use to improve your marketing