Lessons Learned from New Relic's DevOps Journey

Lessons Learned from New Relic’s DevOps Journey

2

Housekeeping

Confidential ©2008-15 New Relic, Inc. All rights reserved.  

Today’s webinar is being recorded. A link to the

recording will be emailed to all registrants shortly after.

Please be sure to ask any questions via the “questions panel”

There will be a brief survey after the webinar for any additional questions or

feedback.

©2008-15 New Relic, Inc. All rights reserved.  

Devops IS NOT:▪ A job or team title▪ A particular process▪ An end to itself▪ A technology or set of tools▪ About getting to "devops"

Devops IS:▪ A methodology▪ A mindset▪ An organizational attitude▪ About getting to "yes"▪ A way of organizing teams

What even is a devops?

5

©2008-15 New Relic, Inc. All rights reserved.  

● Devops IS:o A word for a collection of org chart and cultural changes that

address the unique demands of building large consumer software projects today.

What even is a devops?

6

©2008-15 New Relic, Inc. All rights reserved.  

Developers: seek change to ship releases and achieve product goals

Natural Conflict

Operations: seeks stability to improve uptime and sleep

7

©2008-15 New Relic, Inc. All rights reserved.  

Natural Conflict

Operations, Test, and Support: Seek stability and risk reduction

Product, Business, and Developers: Seek growth and change

8

©2008-15 New Relic, Inc. All rights reserved.   9

©2008-15 New Relic, Inc. All rights reserved.   10

©2008-15 New Relic, Inc. All rights reserved.   11

©2008-15 New Relic, Inc. All rights reserved.  

Stage 1: The Beginning (20-100 engineers)

Pre-history:▪ Traditional IT Operations group primarily concerned with servers,

telephones, and networking.▪ Operations was on call for everything, but didn’t do deployments

or get too concerned with the needs of the applications.

Site Engineering:▪ No more phones or offices, just focused on application delivery and

reliability.▪ If we were “devops”, it was because we were programmers doing

operations.

12

Set the right pattern early

Infrastructure is a Force Multiplier

©2008-15 New Relic, Inc. All rights reserved.  

Stage 2: Growth (100-300 engineers)

Steady increase in scale and tempo.▪ More frequent deployments to reduce engineer time cost (if it’s

hard, do it more often)▪ Teams placed on call for their own services, with easy escalation

back into Site Engineering

15

Oncall IS Operations Ownership

©2008-15 New Relic, Inc. All rights reserved.  

Protect Your Sleep!

Track your incidents and ruthlessly squelch noisehttps://github.com/newrelic/insights-about-pagerduty

Manage your incidents with a process everyone can follow:

http://www.ocadotechnology.com/our-blog/articles/how-we-set-up-chatops-within-slack

Leaven your Product Teams

©2008-15 New Relic, Inc. All rights reserved.  

Distributed Expertise

Not just more data and services, too many more▪ Too many to have in one Site Engineering group ▪ Matrix-managed a team of SREs into Forward SRE positions within

product groups

19

©2008-15 New Relic, Inc. All rights reserved.  

Stage 3: Shuffling the Deck (500+ engineers)

Site Reliability Engineer is now a peer title to Software Engineer▪ Teams can hire and allocate SRE positions as they like▪ SREs naturally bunch up in operations-focused groups, but many

teams have them▪ SRE alignment driven by interest group activities rather than

reporting

20

Total Ownership with Contract Points

©2008-15 New Relic, Inc. All rights reserved.  

Biological Complexity

Each team should be a small, full-ownership group whose operational engagement extends as far down the stack as necessary to make their product work. It tries to capture the energy and rapid-response stance of a startup team.

22

©2008-15 New Relic, Inc. All rights reserved.  

Q&A

Please use the questions panel in your webinar client.

©2008-15 New Relic, Inc. All rights reserved.  

Continue the conversation!

Visit our Online Technical Community at

https://discuss.newrelic.com/c/nerdvana/devops-chat

For more information, see our DevOps hub:

https://newrelic.com/devops