Building for DevOps Success

www.devopsguys.comPhone: 0800 368 7378 | e-mail: [email protected]

2016

BUILDING FOR

DEVOPS SUCCESS

2www.devopsguys.com

Phone: 0800 368 7378 | e-mail: [email protected]@TheDevMgr

ABOUT ME

• 20 years of web-scale engineering experience

• Built e-commerce systems for Dell, Pizza Hut, KFC and more…• Dell generated $10 million/day.

• The rest….$150 million in revenue every year.

• Co-founder of the DevOpsGuys• http://www.devopsguys.com

• @thedevmgr

http://www.devopsguys.com/

3www.devopsguys.com


DEVOPSGUYS

ACCELERATE & TRANSFORM

THE WAY ORGANISATIONS

DELIVER SOFTWARE


@TheDevMgr


SUCCESS!

(BUT FIRST COMES FAILURE)


@TheDevMgr

KNIGHT CAPITAL GROUP

WALL


@TheDevMgr

American global financial services company

market making

electronic execution

institutional sales

trading

In 2012…the largest trader in US equities


@TheDevMgr

3.3 billion trades

$21 billion

A day!


@TheDevMgr

On July 31, 2012 Knight

had approximately $365

million in cash.


@TheDevMgr

Life is good!


@TheDevMgr

NYSE announces plans to launch

a new Retail Liquidity Program

Enable retail customers to

get price improvement for

their stock orders


@TheDevMgr

Receives large orders from the trading platform

and breaks them up into multiple smaller

orders to find a buyer/seller match for the

volume of shares.

The larger the parent order, the more child

orders would be generated.

Photo Credit: Torben Bjørn Hansen via Compfight cc

http://www.flickr.com/photos/21067003@N07/3120819790/

http://compfight.com/

https://creativecommons.org/licenses/by/2.0/


@TheDevMgr

27 July 2016

“Power Peg”

Knight stopped using this

functionality in 2003

but the code was neither

deleted nor deactivated.

Photo Credit: Charles Crosbie via Compfight cc



@TheDevMgr

Code update repurposed

an old flag that was used

to activate the “Power

Peg” functionality.

The code was thoroughly

tested and proven to

work correctly and

reliably.


@TheDevMgr

What could possibly go wrong?


@TheDevMgr

Knight manually deploy

updates to a limited

number of servers per

day

8 servers are “updated”

Photo Credit: jovike via Compfight cc



https://creativecommons.org/licenses/by-nc/2.0/


@TheDevMgr

If there is an SEC filing about your

deployment something may have gone

terribly wrong!

18www.devopsguys.com


THE SEC FILING

During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second

technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from

the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.

-- SEC Filing | Release No. 70694 | October 16, 2013http://www.sec.gov/litigation/admin/2013/34-70694.pdf



Missed one!


@TheDevMgr

August 1st, the markets open

Knight begin processing orders

7 servers begin processing orders correctly

one does not!

Photo Credit: harry harris via Compfight cc



https://creativecommons.org/licenses/by-nc-sa/2.0/


@TheDevMgr

“Power Peg” is back from the dead!

Photo Credit: San Diego Shooter via Compfight cc



https://creativecommons.org/licenses/by-nc-nd/2.0/


@TheDevMgr

Count the shares bought/sold against a

parent order as child orders were executed

Send the child orders for execution

Stop once the parent share count has been

fulfilled

Photo Credit: marfis75 via Compfight cc



https://creativecommons.org/licenses/by-sa/2.0/


@TheDevMgr

27 July 2016

Cumulative tracking

functionality moved to an

earlier stage in the code

execution

And removed the count

tracking from “Power

Peg”Photo Credit: Vox Efx via Compfight cc





@TheDevMgr

LOOP


@TheDevMgrPhoto Credit: JD Hancock via Compfight cc





@TheDevMgr

Imagine what would happen if you had a

system capable of sending automated,

high-speed orders into the market

without any tracking to see if enough

orders had been executed


@TheDevMgr

27 July 2016

Photo Credit: Wooly Matt via Compfight cc

OH s**T!





@TheDevMgrPhoto Credit: fPat via Compfight cc

OH s**T!





@TheDevMgr


@TheDevMgr

9:30 AM – Markets Open

9:31 AM – Traders realise something is wrong

9:32 AM – Why hasn’t it stopped?

Photo Credit: somewheregladlybeyond via Compfight cc





@TheDevMgr

No kill switch

Trading 8 million shares / minute

+50% of the normal trading volume

stocks +10%

Photo Credit: Marshall Astor - Food Fetishist via Compfight cc



https://creativecommons.org/licenses/by-sa/2.0/


@TheDevMgr

No documented procedures

Start uninstalling the new code

Removed the working code

Amplified the issue

Photo Credit: Jason Michael via Compfight cc

Left the broken code





@TheDevMgr

45 minutes of trading

212 orders

millions of child orders


@TheDevMgr

4 million transactions against 154 stocks

397 million shares


@TheDevMgr

$460 million loss in 45-minutes

Photo Credit: zachstern via Compfight cc





@TheDevMgr

Knight only has $365 million in cash


@TheDevMgr

In 45-minutes Knight went from being the

largest trader in US equities and a major

market maker in the NYSE and NASDAQ

Photo Credit: MattysFlicks via Compfight cc





@TheDevMgr

NADA….

Bankrupt


WHAT’S THIS

DEVOPS THING?


@TheDevMgr

PEOPLE, PROCESS AND THE RIGHT TOOLS WORKING TOGETHER TO MAKE YOUR

SOFTWARE DELIVERY LIFECYCLE FASTER AND MORE PREDICTABLE.


@TheDevMgr


@TheDevMgr

OPERABILITYScalability

Deployability

Resilience

Monitoring Alerting

Disaster Recovery

Supportability

Maintainability


@TheDevMgr

CONTINUOUS DELIVERY + OPERABILITY = DEVOPS



KEY DEVOPS PATTERNS & PRACTICES

Developer self-service

Continuousmonitoring

Desired state management

Monitor everything

Integrated toolchains

Infrastructure as code

Collective Ownership

Continuous Improvement

Collaborative culture

Continuous experimentation

Full stack engineers

Autonomous teams

Integrated teams

Joint meetings

Job rotation

Servant leadership

Continuous integration

Continuous testing

Continuous delivery

Minimum viable product

Small batches

Automated Builds

Automated testing

Release automation

Canary rollouts

Instrument everything

Feature flags

Version everything

Test-driven development

Technical debt

Test everything

Minimum viable process

Test-drivendeployment

Common metrics

Optimise flow

Culture

Process

people

technology


8 DEVOPS LESSONS


#1

IDENTIFY THE GOAL


@TheDevMgr


@TheDevMgr

+


@TheDevMgr

Company

Department

Team

Individual


@TheDevMgr

• Sprint

• 2 weeks6 18

• Month

• 2 sprints

Aspirational

(60%)

• Quarter

• 3 months

• Annual

• 12 months

1 2 3Vision


@TheDevMgr

• Sprint

• 2 weeks6 18

• Month

• 2 sprints

Aspirational

(60%)

• Quarter

• 3 months

• Annual

• 12 months

1 2 3

Positive

(70%)

VisionObjectives


@TheDevMgr

• Sprint

• 2 weeks6 18

• Month

• 2 sprints

Aspirational

(60%)

• Quarter

• 3 months

• Annual

• 12 months

1 2 3

Positive

(70%)

Considered

(80%)

VisionObjectivesPlan


@TheDevMgr

• Sprint

• 2 weeks6 18

• Month

• 2 sprints

Aspirational

(60%)

• Quarter

• 3 months

• Annual

• 12 months

1 2 3

Positive

(70%)

Considered

(80%)

Confident

(95%)

VisionPlanTasks Objectives


@TheDevMgr

“SUCCESS IS NOT CHECKING A BOX.

SUCCESS IS HAVING AN IMPACT.

IF YOU COMPLETE ALL TASKS AND

NOTHING EVER GETS BETTER, THAT'S NOT

SUCCESS.”

Christina Wodtke, OKR Coach

https://twitter.com/cwodtke


#2

BE AGILE


@TheDevMgr

Scrum

Lean

Kanban

XP

DSDM

SAFe



SCRUM IS POPULAR

Image source: State of Agile Survey, 2015



Daily Stand-Ups

Retrospectives

Burndowns

VelocityIteration Planning

Story Mapping

Planning Poker



Value people and how they communicate

Measure progress through working software

Collaborate with your customers

Respond to change



THE MANIFESTO

• Individuals and interactions over processes and tools

•Working software over comprehensive documentation

•Customer collaboration over contract negotiation

•Responding to change over following a plan



BEING AGILE

Image source: State of Agile Survey, 2015


@TheDevMgr

LEARN TO INSPECT & ADAPT


#3

ALIGN TO VALUE


@TheDevMgr


@TheDevMgr

Organization

Roles

Teams

Cadence

Taxonomy

Plan

Practices

Alignment

Autonomy

“Let’s try to give our teams three things….

Autonomy, Mastery, Purpose”


@TheDevMgr

AlignmentEvery team and business tracks scenarios and features consistently.

AutonomyEvery team chooses how to manage stories and/or tasks


@TheDevMgr

OPTIMISE YOUR ORGANISATION FOR PRODUCT DELIVERY


#4

DELIVER

CONTINUOUSLY


@TheDevMgr

TIME


@TheDevMgr

Continuous DeliveryFrequent Release Events

WaterfallRare Release Events

Smoother EffortLess Risk

Effort PeaksHigh Risk

Time Time

Ch

ange

Ch

ange


@TheDevMgr


@TheDevMgr

AUTOMATION IS IMPORTANT

Comprehensive, fast and reliable test and deployment automation

Trunk-based development & continuous integration

Application code and app and system configuration all in

version control

Together, the factors on the left model continuous delivery which

leads to…

Lower levels of deployment pain

Higher levels of IT performance (higher throughput & stability)

Lower change fail rates

Higher levels of org performance(productivity, market share,

profitability)


@TheDevMgr

https://xebialabs.com/periodic-table-of-devops-tools/


@TheDevMgr

15,000 engineers, working on 4,000+ projects

All code is checked into one source tree

(billions of files!)

5,500 code commits/day

75 million test cases/run daily


@TheDevMgr

DEPLOYMENT PAIN CAN TELL YOU A LOT ABOUT YOUR IT PERFORMANCE


#5

MEASURE WHAT’S

IMPORTANT



WHERE DO I START?

• Start at the highest level

• Is measurement happening?

• Are metrics a thing?

• Ask around!


@TheDevMgr

START WITH YOUR IMPORTANT THINGS TO YOUR TEAM



IDENTIFY A GOAL

• Is there existing data?

• Who are the key people?

• What things are important to support this goal?


@TheDevMgr

METRICS PROVIDE OPPORTUNITY FOR COMMUNICATION AND ALIGNMENT



IDENTIFY A GOAL: SHIP EVERY DAY

• Is there existing data?

• Who are the key people?






• Test Maturity • Unit tests

• Component tests

• Integration tests

• Acceptance tests

• Performance tests

• Security tests




0

1

2

3

4

5

6

Unit Integration Acceptance Performance

Test Maturity

Product A Product B Product C


@TheDevMgr

WORKING SOFTWARE IS THE PRIMARY MEASURE OF PROGRESS.


#6

ACT ON FEEDBACK


@TheDevMgr

THE DELIVERY TEAM MUST RECEIVE FEEDBACK AND ACT ON IT


@TheDevMgr


@TheDevMgr

INVOLVE EVERYBODY IN THE FEEDBACK PROCESS


@TheDevMgr


@TheDevMgr

BROADCAST THE INFORMATION


@TheDevMgr


@TheDevMgr

FEED THE WASTE SNAKE


@TheDevMgr


@TheDevMgr

FEEDBACK MUST BE ACTED UPON


@TheDevMgr


@TheDevMgr

HOW MANY TIMES PER DAY IS THE ANDON CORD PULLED IN A TYPICAL DAY AT A TOYOTA MANUFACTURING

PLANT?


@TheDevMgr

3,500 TIMES PER DAY


@TheDevMgr

“SHIT HITS THE FAN, BUT THAT DOES NOT MEAN YOU SHOULDN’T USE THE SAME

DISCIPLINE.”


#7

BUILD TRUST


@TheDevMgr

HIGHEST PREDICTOR OF PERFORMANCE

The Study of Information Flow: A Personal Journey; Westrum, Ron

https://www.researchgate.net/profile/Ron_Westrum2/publication/261186680_The_study_of_information_flow_A_personal_journey/links/546117bc0cf2c1a63bff7d29.pdf



HOW WELL DO YOU IDENTIFY WITH?

• On my team, information is actively sought.

• On my team, failures are learning opportunities, and messengers of them are not punished.

• On my team, responsibilities are shared.

• On my team, cross-functional collaboration is encouraged and rewarded.

• On my team, failure causes enquiry.

• On my team, new ideas are welcomed.


#8

OPTIMISE FLOW


@TheDevMgr

EVERYTHING YOU DO SHOULD OPTIMISE FLOW


@TheDevMgr

ANY OPTIMISATION NOT MADE AT THE CONSTRAINT IS AN ILLUSION


@TheDevMgr

THROUGHPUT IS PROFITABLE, EFFICIENCY IS NOT


@TheDevMgr

WHAT IS YOUR LEAD TIME FOR CHANGES?


@TheDevMgr

HOW LONG DOES IT TAKE TO GO FROM CODE COMMITTED TO CODE SUCCESSFULLY

RUNNING IN PRODUCTION?



DEPLOYMENT LEAD TIME PREDICTS

• Ability for Dev and Ops to share a “common source of truth”

• Effectiveness of our automated testing in the deployment pipeline

• Ability to quickly deploy into production without causing chaos and disruption

• Ability to detect and correct problems through monitoring

• Ability for Dev and Ops to work together in a way that is “win / win”

• How quickly developers can get feedback on their work • Testing, deploying, production outcomes, customer outcomes


@TheDevMgr

Thousands of teams +

Micro services Architecture +

Continuous Delivery

= 50 MILLION DEPLOYMENTS/YEAR

136K DEPLOYS/DAY



FLOW OPTIMISATION

• Split into 2 teams

• “Boss” has a stopwatch (on a phone)

• Record first chip to “boss”, record last chip to “boss”

• Developers job is to “flip the chip” (black dot up or down)

• Stack chips in piles of 10

• Complete the batch before passing to next person

SPLIT INTO 2 TEAMS

Team A

Run # Time First Time Last

1 45 45

2 30 38

3 4 21

4 4 29

5 8 25

Team B

Run # Time First Time Last

1 55 55

2 58 58

3 55 55

4 51 51

5 1.01 1.01


@TheDevMgr

DEPLOY SMALLER CHANGES MORE FREQUENTLY



8 LESSONS

1. #IdentifyTheGoal

2. #BeAgile

3. #AlignToValue

4. #DeliverContinuously

5. #MeasureWhatsImportant

6. #ActOnFeedback

7. #BuildTrust

8. #OptimiseForFlow


THANK YOU

Technology

Building for DevOps Success