Agile 2013 Talk: How DevOps Changes Everything

How DevOps changed everything

Karthik Gaekwad

@iteration1 #AgileDevOpsWednesday, August 7, 13

Howdy!• I’m Karthik Gaekwad

• Senior Web Engineer

• Mentor Graphics Embedded

DevOps and the CIA

• From Austin, TX

• We have great BBQ, chill people and brutal summers!

#AgileDevOps@iteration1 Agile 2013Wednesday, August 7, 13

My Background

• National Instruments (NI)

• Ecommerce Team (Agile)

• R&D Cloud Team (DevOps)

• Mentor Graphics Embedded

• Cloud Team (DevOps)

• Organizer: CloudAustin, DevOpsDays Austin, TheAgileAdmin


http://cloudaustin.org

http://cloudaustin.org

http://devopsdays.org/events/2013-austin/




http://theagileadmin.com/

http://theagileadmin.com/

Audience Survey

DevOps and the CIA#AgileDevOps@iteration1 Agile 2013Wednesday, August 7, 13

The most important DevOps things I’ve learned over the last 4 years

Let’s not waste time!


First, we have to talk about agile...



There was a waterfallRequirements

Design

Implement, Test

Maintain

Deploy


RolesRequirements

Design

Implement

Maintain

Deploy

Product Manager, Business Analyst

Business Analyst, Web Designer, Architect

Programmer Analyst, Web Developer

Programmer Analyst, System Administrator

System Administrator


S#AgileDevOps@iteration1 Agile 2013

Waterfall..

Wednesday, August 7, 13

DevOps and the CIA#AgileDevOps@iteration1 Agile 2013

“But I told you this already”

Communication



“That is not my job!”

Accountability



“That is not a priority”

Time to deliver



**About the time I started...


New Product: Partslist

Collaborate and tag parts to build large systems


Transitioned to SCRUM2 Week Iterations

Targeted Feature setsDemo at end of iteration


The Product Team > Traditional Organization


ResultsWorking together, building things that were

wanted


Results

Between Business and IT


Results

Better products being delivered, at a faster pace.

Methodology expanded to all other web teams...


Results

I love my job! #ForReal


Agile: Retrospective

• Scrum brought team together.

• Building products on time, which were actually needed.

• Superior collaboration between business and IT.

• Organization shift -> Scrum


Ops conversations

“Agile is cool! but it’s hard for ops to be

agile because it takes months to buy and provision servers.”



Ops conversations

Deployer Dude

“All day everyday”


But in the community...


But in the community...


“We’re making cloud products now”

“Go build us cloud products

now”Sincerely,

Management



“Cloud is AWESOME!”



“Cloud is OSSM”

-Dave Nielson

-NIST



“Cloud is AWESOME OSSM”

On-demand

Self-service

Scalable

Measurable


“Give me API or give me death”

@littleidea



“Agile DevOps

Cloud Team”


DevOps

“A cultural and professional movement”

Adam Jacobs


http://www.youtube.com/watch?v=Fx8OBeNmaWw

http://www.youtube.com/watch?v=Fx8OBeNmaWw

DevOps

“System administrators participating in an agile development process alongside developers and using many

of the same agile techniques for their systems work.”

The Agile Admin Blog


http://theagileadmin.com/what-is-devops/

http://theagileadmin.com/what-is-devops/

Alison ChaikenEmbedded Linux, Thought Leader

“Get rid of the devs; get rid of the ops; let’s

just find people who can solve problems”


CULTURE

MEASUREMENT

AUTOMATION

SHARING

Pillars of DevOps

John Willis

Damon Edwards

#AgileDevOps@iteration1 Agile 2013

By:


DevOps 101Ideas and Stories

Free!! No charge!!



“Agile DevOps

Cloud Team”


Buzzword friendly!


We had a goal:

“Build a cloud authentication prototype”


CULT

URE

Our 1st sprint(s) sucked!


CULT

URE

But we suffered from “Wall of confusionitis”

Devs: “REST Services!”

Ops: “System Stability!”


CULT

URE

Not seeing eye to eye#Frustrating #Angry


CULT

URE


CULT

URE

Really, this is “Platform Architecture”, but I’m too lazy to redo the image


CULT

URE

Devs and ops understand good architecture. #GoodPlaceToStart


CULT

URE

Think platform first, then app later. #PaaS #REST


• Everyone understood architecture

• Helped everyone speak same language

• Reduced friction

• Called it “The System Model”

AUTO

MATIO

N


1st win: Architecture


1st win: Architecture

• “Model Driven Automation”

• Championed by ops

• Everyone needs to understand the model

• Figure out how to deploy the model

AUTO

MATIO

N


1st win: Architecture• Programmable Infrastructure Environment

(PIE)

• Based on version controlled architecture diagram

• Provisioned cloud boxes

• Ran deploys

• Set standard naming for boxes

• On demand deploy for apps

AUTO

MATIO

N


Outcomes

• Ops learns Java!

• Ops learns version control!

• Developers get comfortable with production environments!

• ...Now we’re speaking many common languages!

AUTO

MATIO

N


No Chef? No Puppet?

AUTO

MATIO

NAre y’all crazy???


AUTO

MATIO

NEverybody be cool

• We believe the cleanest abstraction layer is starting with the entire system you’re trying to model

• In 2009, chef and puppet were younger

• We were scared of Ruby

• We needed windows cloud support

• Plus dawg, cloud was a wild wild world then!


AUTO

MATIO

N

Please automate!!


AUTO

MATIO

N

Do I pick Chef or Puppet??

“whatever makes you happy”


SHARIN

G

For chef/puppet, you’ll need to code.

#NotJust4Devs #NotScary


SHARIN

G

Stuck? Just ask your DevOps dev guy!

#NotJust4Devs


SHARIN

G

Don’t hide.Share the knowledge


SHARIN

G App configs


SHARIN

G App configs

Easy Change Management

Let your automation control these kind of files


Tools PhilosophyDEVO

PS

• Needed to work for cloud

• Facilitate collaboration

• Charts and graphs

• Everyone wants to use it


Tools PhilosophyM

EASU

RE

• Monitoring = Cloudkick

• Monitor servers

• Monitor code (custom synthetic checks)

We use #datadog now


Tools PhilosophyM

EASU

RE

Teach your devs how to monitor their code, and they will love the tool

Example: #NewRelic


Tools PhilosophyM

EASU

RE

If it measures business metrics, management will love you too!

#statsd #dogstatsd

HINT: Management pays you. Make them happy


Tools PhilosophyM

EASU

RE

WASTE OF TIME:

SSH’ing into a box to read log files


Tools PhilosophyM

EASU

RE

Get a logging solution.

Plethora of choices!

#Logstash #Graylog #Splunk #Sumologic

Which one? CloudAustin July Meetup


http://cloudaust.in/2013/06/26/july-meetup-log-like-you-mean-it/

http://cloudaust.in/2013/06/26/july-meetup-log-like-you-mean-it/

SHARIN

G

You’ll need more toolsFind them or write them!

You’re devops now!

Tools Philosophy


Tools PhilosophyM

EASU

RE

DevOps victory:

“People” use tools that solve their core problems


DevOps 201MOAR Ideas and StoriesFor ops who can code

and dev not scared of linux


CULT

URE

Is different for every team.

What works for your team, works for your team


CULT

URE

This worked for us...#AlwaysBeInclusive


CULT

URE

Stuff Breaks

Like...all the time


CULT

URE


CULT

URE

#DontDoThis#AgileDevOps@iteration1 Agile 2013


CULT

URE

Be Cool. Fix the issue at handIf you panic, everyone else does too.


CULT

URE Post Mortem

Do a Post mortem


CULT

URE Post Mortem

It’s okay to admit mistakes

You’re not Batman...


CULT

URE Post Mortem

Why x5Add more tests

Make sure it doesn’t happen again


CULT

URE RELEASE OFTEN

Release with 4 changes is less risky than

Release with 40,000 changes


DEVO

PS Testing


DEVO

PS Test Everything

Test the codeTest the infrastructureIt’s all the same now

#infrastructureAsCode


DEVO

PS Free Time?

Write more tests!


DEVO

PS

CONTINUOUSLY


DEVO

PS CONTINUOUSLY


DEVO

PS

Manual ChangesTo

Production


DEVO

PS Manual Changes

Don’t do this.

Do this:1. Make manual changes in dev

2. Add to your chef/puppet script3. Release to test/prod


DevOps 301MOAR MOAR Tips

For when your team grows


Security in the loop

Add security audits as a part of your sprint.

We do an audit periodically


@Gauntlt @RuggedDevOps



Hire an infosec pro to your devops team



No budget?

Try a cross functional working group


Expanding Teams?

• Rotate times for meetings (status/planning)

• Scrum standups were recorded and dropbox(ed)

• “OK to call” philosophy

• Campfire chat (saves history)

• Ownership of new features


Expanding Teams?

• No secret: Face to face is always better

• Prioritize travels for visits to teammates


Everyone asking for DevOps?

The embedded DevOps model works*


Everyone asking for DevOps?

Make sure your team culture is strong enough so you don’t lose person

permanently :)


Start Small...

Don’t say “WE’RE 100% DEVOPS TODAY”


DevOps 401MOAR MOAR MOAR Ideas

For next time. I don’t have any. Tweet #agiledevops for your best ideas!


STILL STUCK?Just ask!

We’re devops, and this is real(even though we can’t define it)

Twitter: #devopsamahttp://devopsweekly.com/

Go to a devopsdayshttps://groups.google.com/forum/#!forum/devops


http://devopsweekly.com/

http://devopsweekly.com/

https://groups.google.com/forum/#!forum/devops

https://groups.google.com/forum/#!forum/devops

DevOps: Retrospective• DevOps and CAMS

bring team together.

• Building stable products on time, which were actually needed.

• Superior collaboration between devs and ops

• Organization shift -> DevOps


DevOps: Retrospective• DevOps and CAMS

bring team together.

• Building stable products on time, which were actually needed.

• Superior collaboration between devs and ops

• Organization shift -> DevOps


• Scrum brought team together.

• Building products on time, which were actually needed.

• Superior collaboration between business and IT.

• Organization shift -> Scrum


Thank you!

Let’s work together, and solve the problems that our business wants us

to!

#DevOps


Technology

Agile 2013 Talk: How DevOps Changes Everything