Facilitating Continuous Delivery at Scale

Dan Woods

/danveloper

danielpwoods@gmail.com

Senior Software Engineer: Delivery Engineering

Learning Ratpack

What is this talk about?

• Continuous Delivery is tremendously important for unprecedented scale

!• The “Rube Goldberg” machine of a microservice

architecture needs to keep moving !• The culture needs to be amenable to continuous delivery !• Understand that continuous delivery is an organic

concept within an organization

Culture

Freedom and Responsibility!!• One of the greatest aspects of the culture !• Means that if something isn’t working for you, you are

empowered to change it !• We can’t think about things in traditional Ops (or even

DevOps) terms

Culture

Supporting Teams!!• Need to really understand what the various problem

domains are throughout the org !• Try to design for the overlap in use-cases !• From a tooling standpoint, we need to have versatile

infrastructure that can adapt

Facilitation

Facilitating Continuous Delivery!!• Can’t be locked in to a single path or way of doing things !• Need to be able to pivot as quickly as the product teams !• Recent emergence of new technology stacks has made

this ever-apparent

Facilitation

Integrating with Netflix!!• Netflix has a large insight infrastructure, built on JVM

technologies !• Solutions for the various problem domains might not fit

into that structure !• Need to have an answer for getting these apps

integrated into the platform

Facilitation

Platform Sidecar!!• Decouples the application from the platform !• Provides a standard RESTful API for utilizing the platform !• Comes with some reduced insight, but also some

valuable functionality !• Very easy internally to build this into an application’s

deployable artifact

Facilitation

Platform Sidecar!!!!!!

https://github.com/netflix/prana!!

!!!!

Facilitation

This is about enabling the “Freedom and Responsibility” culture by having answers that are not

“no”.

Facilitation

How Does This Fit Into Continuous Delivery?!!• Continuous Delivery is the full show — “code to cloud” !• Having decoupled tooling available means that we can

enable teams to move quickly !• They can be more productive, push code more rapidly !• Less time spent on “integration”, more time spent

developing products

Facilitation

Designing Toward Extensibility!!• No one-size fits all for Continuous Delivery !• Designing workflows based around configuration !• Version all the things! !• Have “best practices”, not shoehorns

Facilitation

Being flexible and driving Continuous Delivery through configuration means that teams can iterate more

rapidly.

Facilitation

Designing for Availability and Durability!!• Infrastructure supporting CD pipelines needs to be highly

available !• Need to be durable to outages — if we’re having an

outage, a production team certainly is as well !

• Design for failure and hope for the best; continuous delivery is not limited to a zone or region, need recovery built-in

Facilitation

Designing for Availability and Durability!!• Getting into the realm now of self-healing systems !• Some teams at Netflix already have this capability !• We’re designing the tooling to be flexible enough to

facilitate most teams

Facilitation

Designing for Availability and Durability!!• Active/Active deployment model !• Non-core functionality can be gracefully degraded out of

the stack !• Ensure that continuous delivery tooling is always

available; this is always the first stop in the event of an outage

Facilitation

Designing for Availability and Durability!!• Netflix Continuous Delivery tooling of the future is a

distributed platform !• Making use of service discovery makes use more resilient

than ever !• For availability purposes, we can leverage Eureka for

load balancing and routing to send requests to the right region/zone/service combination

Facilitation

Build Extensibility - Immutable Infrastructure!!• The “Golden Nugget” of deployable artifact !• Has been advocated since forever, often ruled out !• Netflix has created extensive tooling around designing

and building immutable infrastructure !

• This is one of the only requirements to run in the Netflix cloud (it is the only way to get there)

Facilitation

Immutable Infrastructure - Resources:!!

http://nebula-plugins.github.io/!!

https://github.com/netflix/aminator!

Facilitation

Other Extensible Aspects!!• Extensibility in the build system: Gradle/Nebula !• Extensibility in the configuration model: Archaius !• Extensibility in the data store: Cassandra - Astyanax;

Redis - Dynomite !• Extensibility in the deployment systems: “designing”

continuous delivery pipelines !

Facilitation

Documentation as a First Class Citizen!!• Often the first thing sacrificed when moving quickly !• Is of paramount importance in a distributed platform !• Need to make it appealing for developers to write

documentation !• Need to make it easy for consumers to find/parse your

documentation !

Facilitation

Documentation as a First Class Citizen!!• Build the documentation with the code (literally make it a

module of the project) !• We’ve landed on using Asciidoctor for documenting all of

our processes and APIs !• Generate a manual to HTML, which can be served

directly inline with the service (keeping the documentation as close to the code as possible)

!• Avoid auto-generated docs!! !

Facilitation

Rules for Good Documentation!!• Be sure to include the context for what you’re doing !• Speak at a layman’s level !• Always approach documentation as though the reader

has no knowledge of what you’re doing !• Remember that terminology may need to be described

as well !

Facilitation

DONT DO THIS

Facilitation

DONT DO THIS

Facilitation

YES! DO THIS!

Facilitation

We’re Hiring.