But when I attended AppSec USA a couple of months ago ...public.dhe.ibm.com/software/dw/security/se-devops/... · DevOps has been a big ... and it's getting a lot of coverage on developerWorks

Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder

POWERS: Hi, I'm Calvin Powers, I'm the managing editor

for security at IBM developerWorks. DevOps has been a big

movement in the IT industry, and it's getting a lot of

coverage on developerWorks especially in our Agile

Development Zone. And we've got a very active Enterprise

DevOps Blog which you really ought to check out if you get a

chance.

But when I attended AppSec USA a couple of months ago --

back in the fall -- I had the pleasure of meeting and

hearing Gene Kim speak at that event, and he impressed me

with his ability to connect IT security with the DevOps

movement. And so, I asked him to expand on that a little

bit and he has just published a series of articles on

developerWorks about his views on DevOps.

As you know, Gene Kim was the founder and CTO for Tripwire

for 13 years, and it's that 13 years of experience that led

him to writing his most recent book, which is called The

Phoenix Project -- let me hold it up here.

[LAUGHTER]

The Phoenix Project. And you should run out and buy it.

And so I thought we would bring Gene Kim on to have a bit of

a discussion with us. And to help me out and help put

together a bit of a roundtable, I asked two of IBM's DevOps

subject matter experts to join us on the call today.

-1-


First, we have Ann Marie Fred, she's a software developer in

Tivoli and is an active contributor to our Enterprise DevOps

Blog. Ann Marie, welcome to the roundtable and tell us a

little bit about yourself.

FRED: Hi, nice to meet everybody. I'm Ann Marie

Fred, and yes, I've been working for IBM for about 15 years

now and I've been in the cloud area for the last three

years. Most recently, I've been working on two projects:

one was SmartCloud Continuous Delivery, which is our

continuous delivery/DevOps based offering; and also, I'm

working on some code for SmartCloud provisioning.

POWERS: All right. And we also have with us today, we

have Michael Elder. He is a senior technical staff member,

he's in our Rational brand. And he leads the product

offering for IBM SmartCloud Continuous Delivery, which helps

our enterprise customers begin adopting DevOps practices.

Michael, welcome to the roundtable, and tell us a little bit

about yourself.

ELDER: Sure. Hi, everyone. Again, my name's Michael.

I'm responsible for the SmartCloud offering from Rational

and Tivoli, which is a joint project. And specifically, our

goal was to enable various enterprise customers to adopt

small practices for DevOps that we thought they could

-2-


contain.

You know, the book talks a lot about some of the cultural

challenges and the adoption hindrances that kind of get in

the way, so our goal is to come up with a couple of things

that we thought would be sort of simple, achievable

incremental steps that we could do.

So, I work with Ann Marie as well as several other

developers both in Rational and Tivoli in this space, and we

integrate with the various cloud platforms from IBM to

support continuous delivery directly from the developer

process.

POWERS: All right. So let's bring Gene in. Gene,

welcome to the roundtable. Tell us a little bit about what

motivated you to write this book. I'm going to hold it up

again, The Phoenix Project, a novel about IT DevOps and

helping your business win. Give us a little bit of history

about this book, Gene.

KIM: You know, I started studying a group of

organizations that back in 1999 we used to call geniuses of

people with great kung fu. [For the other] people that

acted differently, talked differently, but most importantly

they had profoundly different operational results than your

typical organization.

-3-


And so over the years, working with the Institute of

Internal Auditors and working with the Software Engineering

Institute, they helped me kind of coin a better name for

these groups. And we call them now the high-performing IT

organizations that have the best feature flow rates, the

best operational stability, the best security and the best

posture compliance.

And you know, what we found is that these organizations

think so differently, act so differently and get results so

profoundly better than your typical IT organization. And

this book was really meant to sort of show how we understand

what non high-performing IT organizations look like and it's

almost all the same.

But when you have cultural warfare going on between

development, IT operations and information security, you're

almost doomed to failure -- that bad outcomes are almost

preordained.

POWERS: Fair enough.

KIM: And in the first part of the book, yes, it's

like some of the feedback we've gotten through the book is

like, wow, you've just described either the organization I'm

in or the organization I was in prior. And I think that was

because we carefully constructed that novel to show that

-4-


yes, these things are exactly the outcomes that happen when

you don't have dev, ops and security working together.


KIM: Yes, that was the first half of the book. And

the other half is really meant to show kind of what are

those breakthroughs that you can get once you get those

different functional areas working together. And what are

the patterns you can put into place that actually are not

mammoth projects but are small things you can do differently

that create far better outcomes.

And so much of that is now being folded into the DevOps, I

would say "body of knowledge" and now we're trying to show

not only what the value of doing that is but how you do

that.

POWERS: Now, you used a word there that doesn't usually

get used when we're talking about IT security. You used the

word "novel," and it's really interesting to me that the

book is...it's an educational book but it is framed and

portrayed and written as a novel with a narrative and plot

and characters and everything rather than a textbook about

DevOps. Why did you and the other authors choose to do

that?

KIM: Well, I would say there were two reasons. One

-5-


is that the book was very closely modeled after a book that

we studied for over 10 years, which is called The Goal by

Dr. Eliyahu Goldratt. It was written in the 1980s, and it

was about...it was a novel about a plant manager who had to

fix his cost and due date issues in 90 days otherwise

they're going to shut the plant down.

You know, this book probably most profoundly affected my

professional career, and even though I had never worked in a

plant, certainly never managed a plant, for me it's just...I

couldn't help but walk away thinking there were some

important lessons here that we wanted to take away.

The second thing that I think sort of validated our desire

to do this is that we found that storytelling is the most

effective mode of communication -- that storytelling

actually bypassing all the rational parts of the brain. And

if you want to get those neurons firing, you know, it's

actually through stories that you know, the human brain is

almost sort of designed to sort of understand and be

receptive to.

And so, our goal was to show that we understand the problem

and there is a better way and to take the reader along a

journey to make sure that they feel like these problems that

we're describing are relevant to them. And I'd love to ask

Ann Marie and Michael, you know, hopefully the book was able

-6-


to sort of trigger some of those sort of sympathetic

reactions?

FRED: Yes, I liked the novel format. I found that it

was a fast read, and I think it's because I was really

engaged in it and just kind of letting it flow. Yes, and to

me it just sort of pointed out a lot of areas where I want

to learn more myself.

ELDER: I would definitely echo those comments. I

think that when you read through it and you see the

frustrating parts, you can really relate to it and that

makes it more real, right? It's not just a storyline, a

fictional line; it's something that you can kind of say, I

knew that guy in a prior role, or I know that person now and

how they impact the project in either positive or negative

ways. So that was...it was kind of neat to sort of see the

life of what we do made into a format that was entertaining

and dramatic as opposed to technical and boring, so.

KIM: In fact, I mean, as we were writing it, I mean,

I'm sure there were times we were writing a person that we

were [LAUGHTER] like that person named Brent, right? The

person who, no outage can be fixed without Brent, no major

project can be done without Brent, because Brent is always

in the way. Right? I mean, I think most of us have...will

be familiar with that person and sometimes have even been

-7-


that person.

POWERS: ...in particular the fact that there were these

moments where you fixed the problem and then he had no idea

how he actually did it.

KIM: I know those people that, they pull it out, and

then after that moment of clarity it vaporizes back into the

nothingness.

POWERS: Right.

KIM: I love that phrase, like every time that

happens Brent gets a little smarter and the organization

gets a little dumber.

FRED: Yes, actually we had an animated discussion

about that at lunch a couple of days ago about what's the

best way to deal with the Brents in your organization and

you know, one person was arguing, well, you can't slow them

down, you have to let them keep doing what they're doing.

And other people were saying, no, you have to make sure

everything that they do is documented. So that really spoke

to me as well.

KIM: And by the way, the irony is you have to do

both, right? I mean, in order for us to get the most out of

-8-


Brent and keep Brent focused on the highest leverage, work

that only Brent can do, we have to make sure that Brent is

not doing things that he shouldn't be doing, right? Like

punching the hole in the boat and then having to fix it the

next morning.

So I just think it's kind of one of those grand ironies that

in order for Brent...and we say this with love and

compassion, right? I mean, because some of this is even,

all of us have probably been Brent, is that in order for the

organization to succeed and Brent to be happy, we need to

surround him with the right processes and standardized work

instructions so that Brent doesn't waste his time with

things that Brent shouldn't be doing.

POWERS: Let's get into the...let's get into the common

[cool] mysticism part of the book. Gene, tell us about the

three ways of DevOps.

KIM: Well, so one of our goals was...actually, one

of the complaints that I think is actually valid is that

DevOps [INAUDIBLE] actually say what DevOps is. And so, one

of the things that we want to do is actually codify the

principles that you could derive all the DevOps patterns

from.

And so, in the book there's this character, this Yoda-like

-9-


character, this Mr. Mioggi like character named Eric who

helps coach the protagonist in the book. And you know, so

he speaks in these kind of platitudes but I think they're

very important principles that really show kind of what the

underlaying motive for DevOps is.

The first way is all about understanding the flow of work as

you go from left to right, from development to IT

operations. And the question is, why dev and IT ops? And

it's because that's what's between the business and the

customer. Right? And so making sure that we see the flow

of work, the work should only go in one direction: forwards,

never backwards.

The second way is the reciprocal: how do you get the right

feedback loops created from IT operations into development

so that the goal of any process improvement methodology and

philosophy is to shorten and amplify feedback loops. So how

do we take the key learnings that we learned kind of at the

sharp end of the spear in IT operations and get those

embedded into development so that we can prevent those

things from happening again? Or if that can't happen,

certainly we can detect and correct for it next time it

happens.

And the third way is all about creating a culture of

continual experimentation, continual risk taking and

-10-


understanding that repetition is a prerequisite to mastery.

And one of the things that I wish I had read and learned 10

years ago is that whether we're talking about Splitz

training or special forces in the military, or learning a

musical instrument, you know, repetition matters.

And so, it's better to practice a musical instrument 15

minutes a day than it to practice three hours once a week.

And so, too, there's certain things that as information

security professionals, as IT operations, we need to be

practicing all the time, whether it's deployments, recovery

work, disaster recovery, you know, penetration testing, you

know, these can't be done just once a year because that

doesn't actually materially change how we do work. So,

repetition creates habits; habits create changed outcomes.

And so, it's from those kind of a very obscure

mystical-sounding principles that Bill, the protagonist, is

actually able to form his own breakthroughs and you know,

create the DevOps patterns in his own organization without

anyone actually having to tell him exactly what to do. It

wasn't too obtuse, you know, for the readers.

POWERS: Well, let's ask. Michael, do you see any

evidence of those three ways in the approach you guys take

on the SmartCloud offerings?

-11-


ELDER: Definitely. The biggest one for us is focusing

on the first and second items, right, providing a way that

you can do a continuous deployment from the development line

all the way into at least a testing QA environment. And

then trying to amplify the feedback and the quality

verification, make sure that what was delivered was not

causing regressions, was actually meeting the future

specifications.

You know, the story line in the book where many rapid fixes

delivered on top of each other back to back in very

compressed, crunched, unrealistic timelines actually creates

more chaos instead of reducing chaos.

And that's I think the biggest aspect of trying to amplify

the feedback loop in a way that's controlled, so that when

you do get a positive result, the next thing you do is a bit

more positive, a bit more stable, a bit more improved. And

I think that, to me, is the biggest item. It's exactly the

kind of thing we focus on in the product deliverables that

we produce as well.

FRED: Yes, and I would say, you know, we went through

this a few years ago with adopting agile practices, and now

we're kind of in the middle of pushing the DevOps principles

through our organization as well. I would say it's kind of

a daily battle -- like it's very easy for people to slide

-12-


back into the old way of doing things, so...

POWERS: Right.

FRED: ...you really have to be disciplined and

constantly pushing, like this is the right way to do things,

we're doing this for a reason. You know, and these are the

things that are actually going to save you time.

And it's also, it's kind of a trial and error process. Like

some things you think will save time end up just being extra

paperwork; some paperwork you drop because you thought you

didn't need it but then you find out that everything falls

apart.

So I think it's kind of a constant learning experience. But

you know, we also have to learn from each other. There are

other people who have done this before, so we're not

starting from zero.


FRED: It's really a discipline, yes.

KIM: One of the things I just love about continuous

integration and continuous delivery is that...continuous

deployment, is this notion that you have a deployment

pipeline and when things go wrong you stop the deployment

pipeline, right? And the ideal, right, no new features

-13-


until we're in a deployable state again.

And in a previous lifetime, I remember being in an

organization, or I had a friend who was in an organization

where, you know, he/she had, you know, a broken build

system, right, where, you know, we...and if you don't have

continual builds, then you can't do continuous testing. If

you can't do continuous testing, then you can't do

continuous integration.

And this creates this downward spiral where if integrations

and merges become painful, you do it less frequently and if

we do it less frequently, that means it takes more time.

It's just this horrible downward spiral.

And I think the whole sort of value system around continuous

deployment and continuous integration is so important for

DevOps. I mean, I would say it is an actual prerequisite to

get these kind of fast feature flow and stability that we

want out of DevOps.

POWERS: You know, that's a great segue into our next

topic, because in your book there is explicit references to

lean manufacturing and the theory of constraints. And I

can't help but notice, you know, part of DevOps is

operations, and so I think it's a natural question to ask,

how much are things like lean manufacturing and theory of

-14-


constraints and just basic fundamental operations and

research a foundation for the DevOps movement?

KIM: Oh, I think the DevOps movement has been very

influenced by Deming and Goldratt and the lean folks. And I

think, in my mind, right, what I'm hoping is that the novel

and the book pushes forward one more increment, is the

notion that IT operations has more in common with the plant

floor than most people would expect.

And I think everything from code commit, you know, down to

in production, you know, that is not artisan work, that

actually is operations and there's a lot more recurring work

that happens, and that actually has a lot more in common

with like a bill of materials and a bill of resources and

routings. Right?

And so the whole notion that we can sort of create these

repeatable pipelines into production and ideally automate as

much as possible is something that is completely out of the

lean manufacturing playbook.

And you know, just I think one of the things that we try to

do in the book is try to make those mappings very explicit.

And I think, I'm just delighted beyond words that there's

already this affinity between lean manufacturing and DevOps,

and I'm hoping that this book will push it one more step

-15-


forward.

POWERS: It seems like there's probably a lot of people

that might gristle at the notion that they're working on an

assembly line in IT. Not that I'm arguing with you, but I

just...I just wonder if you get much pushback on that point.

KIM: Oh, of course. In fact, Bill, the protagonist

in the novel, right, he says, we use our brains, not our

hands, right?

POWERS: Right, right.

KIM: [LAUGHTER] You know? You know, our work is not

repetitive. And, yet, if you take a look at sort of the

cadence of most operations and most deployments, deployment

work, right, there is actually a lot of recurring work that

happens in every project and every deployment.

ELDER: So, on this particular topic, I do think

there's a lot of correlations between how we develop

software and deliver software and how that process works.

For me, the perspective I've always taken is that you look

at when the assembly line became popular with folks like

Henry Ford, they started manufacturing parts that were more

interchangeable, more standardized, as opposed to these

special artisan pieces that only Bob knows how to create and

only if you have the rest of Bob's machine will actually

-16-


[fit] together, that actually having a standard part that

you can kind of replace and plug in.

And I think that notion for both software delivery in terms

of the business features and you know, you make the point

late in the book around driving automation to make the

pipeline repeatable, each of those things -- both the

software layer and the automation layer -- become cogs in

that larger machine.

And if you can standardize around them and provide that

process for how you create them, how you improve them, I

think ultimately that by having a better support within the

machine itself, the machine...the overall machine becomes

more elegant, more stable, because each of its individual

pieces are more elegant and more stable. So, I tend to

agree with the facts that there are a lot of correlations

with software delivery and manufacturing in general.

FRED: Yes, it's interesting. I was just working on a

presentation from...with Professor Ron Dattero from Missouri

State University about the Toyota production system and how

that's related to DevOps.

And when I first saw the topic, I thought he was insane. I

was like, what are you talking about? But then as I was

reading through the presentation, I was like, oh, you know,

-17-


they're talking about built-in quality, the importance of

people and teamwork, reducing waste, you know, don't let

your builds pile up behind the test team, delivering things

just in time. You know? It was amazing to me how it really

does sort of tie in to everything that we do.

KIM: And I think one of the things that the lean

folks do better than anyone -- even better than the theory

constraints in the Goldratt school -- is saying that the

highest aspirational goal, right, of a plant is single-piece

flow. Right? So that means no inventory, you have a

continual pipeline of work, you know, almost like an

assembly line, no...that means no worker process, that means

no wait time.

And so I think kind of the whole notion of like when you

look at Amazon doing a thousand deploys a day, doing

deployments on demand, I mean, I think that is an ultimate

embodiment of what a lean practitioner calls single-piece

flow. So I totally agree with you, and I think that's one

of the neatest things I learned in my indoctrination into

the lean world.

POWERS: Okay. Let's move on. And Michael, in some

offline discussions, Michael had raised this topic about

getting buy-in from the management hierarchy. You want to

launch us in on that one, Michael?

-18-


ELDER: Sure. So, as we have talked to larger

enterprises, one of the constraints around the story line in

the book is that you have a company that while it has larger

revenue stream they still are primarily focused on one large

project. You have many roles that kind of sit together in

the same space; I'll talk about the geos later. But the

fact that they're able to achieve buy-in because they really

have this sort of top-down failure, right? We had to have

the entire train run into the side of the mountain, crash,

burn, flames, everything, before you got some sense of, we

need to do something differently.

And I thought that it was an accurate model that you get

top-down buy-in and then you have to establish trust among

peers. You know, there's the "off-site meeting," as it was

called in the book, where everyone kind of comes together

and develops some sense of vulnerability and trust among

each other. To me, that is probably the largest cultural

adjustments you have to make.

POWERS: In the South we call those "Come to Jesus

meetings."

ELDER: Very much so.

Ultimately, though, I'm curious, Gene, what your perspective

is, whether you believe that you always have to have that

-19-


plane crashing into the side of the mountain effect before

you can have meaningful change in the culture; or, if you've

seen other places where you can make those changes without

crashing the stock price into the ground.

KIM: You know, in fact, you've...Michael, you just

sort of shaped what I would call the moral crusade of why we

wrote the book, because there's a formative moment where

while I was with Tripwire I was working with a gentleman

named Eric Passmore; he's the CTO of AOL.

And you know, I've told this story before in front of Eric,

and you know, for the IT operations people, he was the SVP

of global engineering at the time. You know, he had 1,300

developers working for him. And among the IT operations

team, he was "that Eric." Right?

He was the person that sort of guaranteed that operations

would never get what they need, right?

Until, I remember this, we did this off site, and there was

this moment where he said, oh, my gosh, the reason why we

couldn't ship a certain feature was because IT operations

couldn't upgrade the Linux kernel from 2-4 to 2-6 and get

multithreading support.

And he said, oh, that was as much of a reason for a code

-20-


freeze, right, as anything development related. And he

realized, wow, at that...almost like with a snap of a

finger, he became IT operations best friend. I mean, he

became "THAT Eric," right, who demolished roadblocks and

then became one of the staunchest supporters of the things

that IT operations needed.

And one of the things that just blew me away was that, you

know, one of the things that he helped champion was just

changing who was doing the packaging and moved it from

operations to development. And by doing that, we took the

deployment time for the AOL.com homepage from like six hours

to like 45 minutes.

And that had such a huge impact on me, because that wasn't a

huge mega project, it wasn't this huge cultural

transformation; it was just swapping who was doing what,

right, and it had this incredible difference in outcome.

And so that had a huge, that was one of the big aha! moments

for me, was that, you know, wow, it doesn't take the plane

running into the ground, right, into the side of the

mountain. It just takes a real understanding of a shared

goal and shared outcome that's larger than development tests

or operations.

And so one of the profound hopes that we have in the book is

-21-


that by showing the patterns, these undesirable effects,

right, that people will recognize themselves and their peers

in the book and say, hey, we don't have to crash into the

mountain in order for us to believe that this is pertinent

to us...

That, you know, we can actually have some healthy

discussions around this and then start, you know, pulling

the...pulling back on the stick and gaining altitude long

before the business is actually jeopardized. Did that make

sense, by the way?

ELDER: It does, it does. I think there's an

interesting point, though, that there is still some failure

point that you have to observe to really realize the value

of that closer collaboration, right? Or the gentleman you

described, it was one feature that might have benefitted the

customers, but it wasn't the stock price crashing per se,

but there was this catalyst effect that said, that's why we

need to cooperate more effectively.

POWERS: He had to have an aha! moment.

ELDER: ...more so than the value that it might derive.

POWERS: Interesting.

KIM: And can I just point out one thing about what's

so interesting to me is that so much of the way we behave is

-22-


embedded in sort of the way our organizations are rewarded,

right? Typically development is to be rewarded on fast

feature delivery, right? You know, more features delivered,

the better.

And usually that comes at the expense of quality of features

and non-functional requirements. And that means make

changes as fast as we can, it doesn't matter what the

quality is, while operations is motivated by up-time

availability, and that usually means make no changes ever,

right, over my dead body, right? And so...which can be

quickly arranged, in my experience.

And so, because those measurements are so embodied in kind

of how dev and ops are managed, you know, almost everybody

will feel the effects of that tension, right, of like this

chronic conflict that comes from one organization making as

many changes as they can and the other organization being as

resistant to changes as they can be, right?

So, you know, that's why I say those kind of horror stories

are almost preordained, just because of the way the dev and

ops people are managed.

ELDER: I completely agree with that. I think that we

see that fundamental challenge over and over again, and

personally I tend to think that as long as there's a divide

-23-


between the director and VP level of these orgs, you're

always going to have that inner conflict.

I mean, one of the things in the book near the end, maybe

not essentially a spoiler, but there's this notion that you

sort of have to centralize a role around operations that has

impact on both development and operations as that single

unit that's rewarded based not on what they do individually

but how they compete effectively against their competitors

in the marketplace and more so how they please users, right,

as a team.

FRED: And I mean, we found that on our own team we

had to sort of embed some operations people within our

development team in order to really learn from them, you

know, how it is that they get their jobs done. And also, we

had to go put some developers on the operations team. And

boy, most of them go kicking and screaming, actually,

especially when they realize that they have to own pagers.

Nobody really wants to do that.

But it's a very valuable learning experience, and we're

trying to do more of that so they get that experience. And

then they'll go back and they'll say, okay, here's what we

did wrong in development, here are the things we need to

fix. And then they sort of become evangelists for that

transformation, you know?

-24-


KIM: Yes, provided they don't go native, right?

[LAUGHTER]

POWERS: Well, let's talk about the line employees a

little bit, the guys that sit in front of the screens in the

data center all day and the developers who write code all

day. How is this going to, how is this DevOps approach

affecting them? I mean, aren't they sort of at ground zero

for all of the chaos?

KIM: Yes, most certainly. In fact, in the book, you

know, the person we created to embody that person is Brent,

right? Brent is...and it's interesting, when I was looking

at the Amazon reviews, and which I'm just delighted that

there's like 83 of them right now, but one of the things

that sort of actually make me think that we, you know, got

something, we didn't glorify him enough, in fact, some

people say that Brent is a villain.

It's like no, no, no, Brent is just trying to do what Brent

needs to do, right? And so I think in the ideal, especially

for IT operations, the people who are doing the majority of

the work, you know, the line contributors, the effects that

they will feel is that they're spending more time doing

creative work, improving the input environment, improving

the deployment mechanism and doing less time firefighting.

-25-


Right? I mean, I think one of the most spiritually damaging

things we can actually do to people, to other people and

ourselves as human beings, is, you know, put them in a

system where they feel trapped, in a system that preordains

failure.

So when you're in IT operation and you have to live with the

downstream consequences of decisions made upstream of you

year after year after year that cause carnage and mayhem in

the production environment, you know, that make you work on

weekends and make every deployment, you know, sick for

weeks, for year after year, I mean, and you know, bring back

problems to the family.

I think these are the people whose lives will be most

improved by things like DevOps, because now we can actually

embed them into development, just like Ann Marie said, and

actually help change the outcomes and change the mindsets

and change how we make decisions.

And think on the development side, you know, it means that

we have to share the custodianship of the production

environment and the health of the deployment pipeline. And

I think that is great, because when we work on those things,

that's what actually allows us to speed up the tempo of

development and spend less time on rework and firefighting

-26-


even in development.

POWERS: Very good. Now, one of our colleagues who

couldn't join us expressed to me in some e-mail that, you

know, just one tender criticism that maybe the book was a

little bit too operations centric and maybe quite didn't

give development teams their fair shake sometimes. I think,

Michael, you had some thoughts along those lines as well.

Can you kind of voice that concern for us?

ELDER: Sure. So, I think that the book makes a great

dramatic effect around the challenges of a development team

with zero discipline. But I think that if you look at some

organizations that have already adopted agile development

practices, there's already a feedback loop with continuous

integration, maybe even hopefully automated testing, static

analysis, et cetera, but that feedback loop hasn't yet been

extended into the operations process.

And so for some organizations, like the one described in the

book, you have to bring process and agility and feedback

loops even to the development process within itself and then

kind of carry it forward, and others where you have some

sense of feedback loop there but the handoffs are still very

manual, very error prone, right, there's no way that you're

automating the complete deployment into the system, it seems

like that's really an organization that still needs to

-27-


extend it but where perhaps development is not the root of

all evil in the universe.

So I guess the question that we had was whether or not you

really feel from the vantage point that development tends to

be that cause because it is so feature driven, so, you know,

sort of a whiplash of the business side, right, bringing in

this capability of maybe yesterday we've got to go do it, or

whether that was just sort of a dramatic effect to make a

point.

KIM: You know, it's funny, well, I'll make two

observations. One is in my experience, and by the way, I

was a former developer. My first job was actually a QA

engineer for Sun Microsystems for the File System test

suite, so I have a lot of compassion for development.

It's been my experience that when there have been agile

processes in place and when there have been even continuous

integration, one of the things they don't do is actually

test in an environment that resembles production.

And so, they go do all this work, you know, to put

continuous integration practices in place, and yet they're

missing sort of the foundational piece that allows them to

actually, well, in some ways, and the worst case, right,

they're testing just a code and they never have the

-28-


environment, you know, in the test plan.

And I think in my experience is that's at least 80 percent

of the organization. And so the notion wasn't to vilify

anybody, especially not development, because of course, some

of my best friends are developers.

[LAUGHTER]

It was just to show how much better it can be, right, if we

can just have development and operations share custodianship

around both the code and the environment. I guess what I

thought you were going to go with is like, boy, the kind of

comment that I've gotten most is like, wow, you guys are

sure hard on the information security person, right?

He...they sort of came off as, I don't want to say like a

buffoon, right, but I mean, he was a person who was

ostracized from the organization, marginalized. And Calvin,

when you and I were at AppSec USA, right, I mean, I think

there were a lot of chuckles in the room as kind of people

would recognize, you know, people that we knew, you know,

information security practitioners who everyone else hides

from because, you know, you invite this person to the

meeting, they're guaranteed to slow things down.

POWERS: That's right. We're the ones that are always

shaking our fists saying no, no, no, yes.

-29-


KIM: Right. We were on this moral crusade, right,

that [should tell you] why we can't do things.

POWERS: That's right, which is exactly why we need to

get more risk based so that we can slide into this mode of

thinking, this continuous delivery mode of thinking. Well,

the way...the way I think that affects us is we need to

understand the risks and take them where necessary and

integrate our security operations into that continuous

delivery.

KIM: Right. And one of my biggest fears is that

information security will be the people saying, over my dead

body, right? We will never move to fast-flow features or

DevOps practices, you know, while I'm here. Right? Which

might be a career-jeopardizing move.

But I'm hoping that the book will actually show how

information security actually has some of the most to gain,

you know, by DevOps practices because we can now find and

fix issues faster. And you know, the fact that we have

short cycle times means that we can actually sort of

integrate more work, you know, we can actually inject more

work into the system and not only produce stability but also

improve security as well.

-30-


FRED: Well, yes, I like the fact that you pointed out

that by using infrastructure as code and putting that into

source control now you have sort of an audit trail of what

exactly has been done to the system, right?

So that's a contribution from the development teams to

operations, hey, look at this, you know, now you have a

perfect way of tracking exactly what changed, who did it,

and also systems...you know, you can have systems where you

just reapply your deployment over and over again so if

somebody sneaks a change in it will just get wiped out in a

few minutes anyway. So I think it's nice to show that the

continuous delivery can actually improve your security and

audit posture also.

POWERS: That's interesting.

KIM: Here, here. Yes.

ELDER: And, in fact, I like the point that was made

when, I think it was when they were talking about deploying

more quickly using a cloud infrastructure and Brent made the

comment about, I've already got my own local tools that kind

of help me, right, I think a big part of what Ann Marie is

talking about, being able to pool some of that content that

Brent's created and help it benefit the integration at large

helps decouple Brent from being on the critical path of

everything, because you codify his tribal knowledge into a

-31-


format that's more consumable for everyone else.

KIM: Absolutely. Well, and by the way, just, I know

this is...it seems some what germane, yeah, and so our goal

is not to vilify anybody except for one person, the VP of

marketing, right?

[LAUGHTER]

KIM: It was actually, we agonized about that for

months, is, you know, is she really a villain or is she also

sort of trapped in behaviors that are a side effect of the

way she's managed.

And essentially what we found is like, you know, one of the

parts of storytelling, and, you know, especially when you're

sort of trying to tell the hero's journey, you actually do

need a villain. And so, you know, after much sort of

agonizing, we said all right, you know, we've got to have a

villain, and the villain is her, so...

ELDER: Maybe that's your strategy, though, right, a

common enemy unites, you get dev and ops together to hate

one of the other lines of businesses on the critical paths

to delivering value to customers.

POWERS: Oh, Lord, have mercy. All right. On that fine

note, I think we're going to have to wrap this up. I want

-32-


to say a thanks to my SMEs. Ann Marie, thank you so much

for joining us. Any closing remarks or plugs you want to

get in?

FRED: Oh, no. Just thank you very much for the book.

And yes, I'm personally going to go learn more about lean

myself, so.

POWERS: Terrific. You can all read her on the

Enterprise DevOps Blog on developerWorks, we'll have a link

on the video. And Michael, thanks to you also, and thanks

for your excellent questions and contributions. Any...?

ELDER: I will, if I can, take a quick opportunity,

take a look at what we have on jazz.net around SmartCloud

Continuous Delivery. You know, as Ann Marie pointed out, we

do have the DevOps blog as well. We very much love to

engage our community. And so, take a look, see what's out

there, and let us know what we can do to improve it or make

it better.

POWERS: Okay. And once again, that's jazz.net

SmartCloud Delivery, was that right?

ELDER: SmartCloud Continuous Delivery, that's correct.

KIM: I mean, I'll even amplify that. That's great

for developers and IT operations. I think continuous

-33-


integration, continuous deployment practices are some of the

most important prerequisite skills to get sort of the

outcomes that we want when we talk about DevOps.

POWERS: All right. And once again, the book is The

Phoenix Project: A Novel about IT, DevOps and Helping Your

Business Win. And I would be remiss if I didn't point out

that it's co-authored also with Kevin Behr and George

Spafford, the one and only George Spafford. So, kudos on

that. Gene, thanks for joining us. Any closing remarks or

plugs you want to get in?

KIM: Oh, no. Thank you so much. And if...I'll send

together for show notes some resources about some of the

principles and underpinning theories that went into the book

as well as certainly a link to the set of articles that we

did for...that we worked on together, Calvin.

[END OF SEGMENT]

-34-

Documents

But when I attended AppSec USA a couple of months ago ...public.dhe.ibm.com/software/dw/security/se-devops/... · DevOps has been a big ... and it's getting a lot of coverage on developerWorks