DevOps at its CoreAnn Marie Fred

IBM July 15, 2015

About Me

Duke BS CS 1999

UNC MS CS 2005

Husband and 2 daughters

About MeIBMer since 1998

pervasive & mobile

web applications & portals

IT systems management

cloud computing

websites

About MeDevOps practitioner and evangelist since 2011

5 very different DevOps teams:

SmartCloud Continuous Delivery dev/test

SmartCloud Orchestrator dev/test

IBM Cloud Orchestrator on Cloud dev/test

Service Engage dev/test/ops

Marketplace dev/test/ops > manager/squad lead

Part 1: DevOps at its Core

DevOps: Goals

Happy Developers

Happy IT Operators

Happy Business Stakeholders

Happy Consumers

Happy DevsWork-Life Balance

Fun

Passion

Uptime

Quality

Velocity

Business Value

Sharing Knowledge & Experience

Broad & Deep Skills

Clear Communications

Continuous Process Improvement

Happy OpsWork-Life Balance

Fun

Passion

Uptime

Quality

(maybe not Velocity)

Business Value

Sharing Knowledge & Experience

Broad & Deep Skills

Clear Communications

Continuous Process Improvement

Happy BusinessWork-Life Balance

Fun

Passion

Uptime

Quality

Agility (+)

Velocity

Business Value

Sharing Knowledge & Experience

Broad & Deep Skills

Clear Communications

Continuous Process Improvement

Happy ConsumersFun

Passion

Uptime

Quality

Agility

Velocity

Business Value

Clear Communications

DevOps: Core ValuesWork-Life Balance

Fun

Passion

Clear Communications

Sharing Knowledge & Experience

Broad & Deep Skills

DevOps: Core ValuesUptime

Quality

Agility

Velocity

Business Value

Continuous Process Improvement

Warm Fuzzies: Work-Life Balance, Fun, and Passion

Attract & Retain Talent

Hire for the Warm Fuzzies.

Assess Management on the Warm Fuzzies.

Consciously make these happen.

The other DevOps Core Values help immensely.

Clear Communications: Required

Talking

Chat Services

Wikis and Blogs

Forums

Bug Reporting & Tracking

Agile Planning & Tracking

Clear Communications: Extras

Co-location

Social Media

Conferences

User Groups and Meetups

Email and Newsletters

Specific Agile Methodologies (Scrum, Kanban, etc.)

Sharing Knowledge & Experience, Broad & Deep

Skills: RequiredSee: Clear Communications

Document as you build: how and why

Informal Training

Internet Resources

Code Reviews

Sharing Knowledge & Experience, Broad & Deep

Skills: ExtrasWork on things you don’t know how to do.

Pair Programming

Mob Programming

Cross Pollination

Open Source Model

Formal Training

Books

Uptime: Required

See: Quality

Monitoring

Graceful Fail-over

Sharing the Pager

Post-Mortems

Continuous Delivery

Continuous Delivery

Automated Deployment into Production

Deployment Automation

Infrastructure as Code

Frequent Deployments (ideally, several per day)

Small, Incremental Changes

Zero Downtime

Continuous Delivery

Automated Tests in Production or At Least Staging

Including NFRs: performance, security, etc.

Staged Rollouts

Dark Launches

Rolling Deployments

Green/Blue Deployments, etc.

Experiments

A/B Testing, etc.

Uptime: Extras

See: Sharing Knowledge & Experiece, Broad & Deep Skills

Graceful degradation

Dedicated Support Team

Quality: RequiredSee: Uptime

Bug Tracking

Source Control

Code Reviews

Testing NFRs: security, performance, etc.

Continuous Integration

Continuous Testing

Manual Testing

Continuous Integration

On Every Check In of Source Code:

Build Automatically

Test the Source Code Automatically

Static Code Analysis

Linters for Code Style

API Tests

Anything that doesn’t require a running system

Package Build Artifacts for Deployment

Store Build Artifacts in a Library

Continuous TestingRetrieve Build Artifacts from Library

Deploy Automatically in the Lab

As close to production as possible (Docker, VM images, Chef)

Automatically Test Running Systems

Report Test Results Quickly

Voting and Non-Voting Tests

Failed tests make the build fail

If the build fails, fix it before moving on

Quality: Extras

See: Sharing Knowledge & Experience, Broad & Deep Skills

Dedicated Test Team

Continuous Delivery

AgilitySee: Clear Communication

Agile Planning and Tracking

Frequent Releases

Continuous Delivery Helps

Short Feedback Loops

Autonomous, Empowered Teams

Quality and Agility"Agile has not only improved our ability to plan effectively but it also has allowed us to spend more time working on user stories by improving the quality of the code we deliver." - Richard Gebhardt, Technical Team Lead for Service Engage.

VelocitySlow at first, while building the frameworks for quality and uptime.

Faster in the long run.

Low Bug Count

Low Technical Debt

Frequent Releases

Agility and Velocity

"By having daily scrum meetings, everyone knows who is working on what, resources quickly get directed to problem areas and impediments are addressed quickly and efficiently." - Chuck Brant, Service Engage Release Manager.

Business ValueSee: Uptime, Quality, Agility, Velocity

The Right Stakeholders

Feedback from Consumers

User Centered Design

Data Driven Decision Making

Evaluate teams on business value.

Monitor business value.

Agility and Business Value

"As a product line manager, working on one-week sprints gives me the flexibility to change priorities as things change. I can still get the big-ticket items delivered - but I can do so without getting locked into an unyielding plan and can react rapidly to shifting business priorities." - Pete Marshall, Product Manager, IBM Service Engage/Cloud Foundation Services.

Continuous Process Improvement

See: Sharing Knowledge & Experience

Blameless Post Mortems

Sprint Retrospectives

Part 2: How We DevOps

Inspired by Spotify, Informed by

Scrum & KanbanSquads

Guilds

Most people have more than one role

Not exactly Spotify…

Business: Managers

First-Line Managers handle:

Staffing

HR issues

Money

Compliance

Red tape & paperwork

Removing impediments

Career planning

Coaching

Planning: Squad Leads

These are the stakeholders.

Loosely defined milestones and long term plans from upper management.

Define and prioritize the Story backlog.

“As a [who], I want to [what].”

Description, Business Value, Acceptance Criteria

Run weekly Story Scrub with the squad to ensure that stories are well defined.

Manage inter-squad dependencies.

Review what the squad delivers to make sure it meets the acceptance criteria.

Get customer feedback.

Executing: Technical Leads

They own the implementation of the stories.

Which high-priority stories are ready to be worked on?

Story points: planning poker by the team

Story breakdown into tasks, with the team

Final choice of stories for the sprint, with the team

Adding tasks to the Kanban board

They also help coordinate work between squads, and define component boundaries and interfaces.

Executing: Scrum Masters

Scrum Masters (rotating position) keep the process moving.

Run daily stand-up meetings

Reproduce defects, add valid defects to the Kanban board to be worked on

Investigate deployment errors, get them fixed

Run weekly group code review or learning activity

Run weekly retrospective

Present weekly playback

Dev/Test/Review: Squad Members

Implement the tasks on the Kanban board.

Defects have higher priority than new tasks.

Pick up a new top-priority task when you’re ready. Your code is under review, and you’ve reviewed others’ code.

Prefer tasks you don’t know how to implement.

Tasks are often implemented with help.

Free time to work on what you like, once the tasks for the sprint are implemented.

Review/Deploy/Operate: Committers

Strict code review process, based on the OpenStack model, using Gerrit.

Excellent code reviewers can earn a “+2” or “committer” designation.

Code changes must be reviewed/approved by at least two people other than the originator, one of whom must have a “+2” designation.

“+2s” handle the final merge to master, which kicks off the green/blue deployment to production.

“+2’s” are the ones on call for critical problems.

ALL squad members help fix problems, not just “+2s”.

Switching to Github pull request model now.

TestIndividual squads own their own unit tests and functional tests.

Code coverage reports required, aim for 100%. Low coverage numbers are a risk to SVT/IVT.

SVT/IVT team writes automated tests for things like:

End to end, cross-component flows

Multi browser, multi geography, etc.

Performance, scalability, security, etc.

Automated tests can fail the build / prevent a deploy.

SVT/IVT also does manual and free-form testing.

The same team sets up frameworks for running experiments (A/B tests, etc.)

OperationsCombination of DevOps squads and dedicated Ops teams

PaaS (Bluemix, Heroku): Fewer low-level operational concerns

Docker containers: Portable across deployment platforms

Monitoring & Alerting (New Relic, Pingdom, PagerDuty)

Logs (ELK stack, Logentries, custom tools)

Site performance testing (Sitespeed, Webpagetest, Yahoo, Google PageSpeed…)

Security testing (AppScan, etc.)

ResourcesMe: amfred@us.ibm.com, Twitter @DukeAMO

DevOps Days conferences

Code as Craft, Etsy’s Engineering Blog

Spotify Squad Model

Blogs from Netflix, Facebook, IBM

Books from O’Reilly Media