Building A System That Never Stops [FutureStack16 NYC]

BUILDING A SYSTEM THAT NEVER STOPS

Nic Benders Chief Architect, New Relic

BUILDING A SYSTEM THAT NEVER STOPSGROWING

BUILDING A SYSTEM THAT NEVER STOPSLEARNING

BUILDING A SYSTEM THAT NEVER STOPSCHANGING

©2008-16 New Relic, Inc. All rights reserved.

@nicbenders

http://nicbenders.com/presentations/scaling-a-system/

IN 1 MINUTE

requestsacceptsover 21M

storesover

analyticevents200M

aggregatesover 800M metrics

50Bqueriesover

datapoints

▪

differentservices

containsover 250

maintained/built by 25+ engineering

teams

▪

4.3 morethan

SSDstorage

PETABYTES


8 LESSONS FROM OUR DATA

PIPELINE


Collector

New Relic App Agents

Confidential ©2008-15 New Relic, Inc. All rights reserved.

▪ Double-click to edit


Collector



Collector


Browser R.U.M.Beacon

New Relic Browser Agents


Collector




Mobile Proxy

New Relic Mobile Agents




New Relic Mobile Agents

Zzz

Collector Browser R.U.M.BeaconMobile Proxy



NOTHING Lasts Forever

Lesson 1


Lesson 2

Create a culture where

CHANGE IS OK




EXPERIMENTS Don’t always work

Lesson 3


Lesson 4

SYNCHRONOUS is going to be a problem


Kafka

Kafka

Browser Data Consumer

Browser Data Router



Collector


Kafka

Kafka



Collector

Zzz

Browser Data Router




Lesson 5

Master the

ROLLOUT


RolloutTechniques

Kafka

Load Balancer Cluster

Kafka


Browser Data Router

Browser Data

Consumer V2


%- %+


Collector

Old New


Lesson 6

It’s the

OPERATIONSstupid


Monitoring Tooling Operations Runbooks



Lesson 7

Use the rightWORKLOAD

DISTRIBUTION


Workload Distribution

DDDDD

CB

B

BB

A

AA

AA

C BD A B DDD

Active Management Random

C B BBD DAA D C

B BDB C

AA DD

C

CA B


DDDDD

CB

B

BB

A

AA

AA

C BD AB DDD

C B BBD DAA D C

B BDB C

AA DD

C

CA B

Active Management

Random

Use Both!


Lesson 8

Technology enables

CULTURE

Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166

Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166

��

35





@nicbenders

http://nicbenders.com/presentations/scaling-a-system/