EvolveStaying relevant at cloud speed

Mark MydlandDirector of QAApp Lifecycle Tools

What is normal?

What’s Normal?

Tools of yesteryear

The only thing constant is change

Change has been good

Information at our fingertips

Entertainment on demand

Companies change… or Die

Change is not an option it is a guarantee

Change has been good

Information at our fingertips

Entertainment on demand

Companies change… or Die

Change is normalChange has been good

Information at our fingertips

Entertainment on demand

Companies change… or Die

The Evolution of IT

The IT Department

The Consumer

The Developer

The IT Department

The 90s: Cost Center

IT considered a cost center

Hub of IT

Technical decisions made by IT

The 00s: Key Infrastructure

IT deployments driven by business need

Technical direction driven by management

Still a cost center

Now: Morphing IT

Carefully planned IT investments

Customer driven

Potential profit center

The Consumer

The 90s: PC Generation

Windows and the PC are standard business tools

The 00s: The Internet Generation

Consumers embrace the Internet as a fundamental part of their lives

Now: The Mobile Generation

Mobile devices become the tool for Internet access

Who is the Consumer?

The customer buying from us

The employees working for us

How they consume IT has changed their expectations

Developer evolution

Questions?

Our Journey to Cloud Cadencethoughts to share from Microsoft Developer Division’s agile transformation

Mark MydlandDirector of QAApp Lifecycle Tools

Continuous

Devices

Services

Connected

When you deliver, test and validate exactly what the customer specified, what happens?

Three Waves of Learning to VS 2012First wave addressed technical debt

Second wave amplified flow of value

Now it’s time to focus on cycle time

In parallel, we start to delivercontinuous services in parallel to on-premise product

Size of DevDiv Code Base

738 Active “devs” last 7 days

14,000 Builds per month

1,615,627 Work items

23,681,882 Source code files

6.5

Stats as of October 2012 from Divisional TFS Server

Terabytes of data

8047 Managed Assemblies and Native Binaries17,000,000 Code Assets (namespaces, classes, methods...)

Creates conflicting priorities among stakeholders

Should force a discussion

of single backlog

At divisional level, reconciled largely through resourcing

DevDiv: Servant of Three Masters

2005 – Starting Culture

Conway’s Law

The best and brightestAutonomy, job rotation, promotion

The currency of loveHeadcount

Dysfunctional TribalismDon’t ask, don’t tell

Schedule chicken

Metrics are for others

Our tribe is better

Our customers are different

Waste

Easy credit

No interest charge for debt

VS 2005 Bug Step @Beta 1Deferral common(see top line)

Teams carry undone work (both functional and ‘ility)

Endgame hard to predict

Background: Debt Crisis

3-4 Years to Build It Service It (10 years)

M1…M3: Develop the code

err…M3.1…M3.3 Recode

Beta1: Integrate and pray

Beta2: Test like hellRC (release candidate) 0..n: Final buildsRTM: Ship it!

QFEs and hot fixes

Service packs

M0: Plan and cost the release

Business Model: One-time License

Product Cycle Pre 2005

The First Wave of Learning: VS 2008First wave addressed technical debt

Second wave amplified flow of value

Now it’s time to focus on cycle time

In parallel, we start to delivercontinuous services in parallel to on-premise product

Focus of MQ (Milestone for Quality)

Get CleanUnderstand our Debt

Test AutomationProduct Code BugsTest Case BugsTest Infrastructure Bugs

Put our Debt on the Table

Cost our Debt

Eliminate the Debt

Stay CleanMeasure our Quality

Quickly & ConsistentlyAnytime, Anywhere

Quality earlier in the cycle

Make life easier

Product Backlog End-to-End Scenarios, Value Props, Experiences, Features

Scenario Scenario

Value Proposition

Value Proposition

Value Proposition

Value Proposition

Experience Experience Experience Experience

Feature

Feature

Feature

Feature

Feature

Feature

Feature

In your business, do you need to…?

Epic: What if you could…? Would you buy or upgrade?

Theme: Let me show you how to…? Would you use it that way?

User story: As a…, I can…

Small BatchSizes MinimizeWork in Process

Sprint

GROOMING

SHIPPABLEIncrement

DailySTANDUP

ACCEPTANCETesting

PRODUCT BACKLOG

Scrum Team (Feature Crew) Delivers ≥1 FeaturesTeam responsible for producing a feature

Work together as a team to hit Feature Complete

Small Interdisciplinary Team (6±3)

Engineering Principles

Drive quality upstream/Don’t defer workIncrease efficiency by doing things right the first time, Bug tail not filled with hidden work

Automate testingAutomated testing at multiple levels; Tests run anywhere

Common definition of what done meansConsistent definition of quality across division enables meaningful metrics for progress

Keep main branch customer readyConsistent delivery of high quality CTPs. Product not tattooed with part finished code

Ready to ship when the time is rightFeature Branches isolate features. RI when quality gates are met; FB PU Staging Main

2005 Debt vs. 2008 Debt15x Debt Reduction!

Minimal Deferral

2x Schedule Improvement

High Predictability

Huge Customer Sat Rise

VS 2005 Beta 1Product Bugs only

VS 2008 Beta 1ALL bug debt

Changes to Pre- and Post- GameBuild it (2 years) Service It (10 years)

Planning: Start and groom product backlog

MQ: Improve our practices, get ready for the next versionM0: What are our goals for this release? What customer value do we deliver?M1…M3 5-week sprints: Develop and test the code, drive to doneCTP (customer technical preview): Targeted customer release to collect feedback on mainline scenarios

Beta1: First broad customer visibility; collect feedback

Beta2: Validate recent changes with customers; collect feedback

Quarterly: Powertools and Feature Packs for current release

Business Transformation from Packaged Product to SubscriptionRC (release candidate) 0..n: Final builds

RTM: Ship it!

The Second Wave of Learning: VS 2010First wave addressed technical debt

Second wave amplified flow of value

Now it’s time to focus on cycle time

In parallel, we start to delivercontinuous services in parallel to on-premise product

Having removed debt, we could focus on flow of value

Focus on the flow

Aspirational Lean Scenarios

Production mismatchesWaiting for build setupUI regressionsPerformance regressionsMissed requirements or changesNo reproPlanning black box

Late surprisesBuild breaksParallel development painBewildering adminButterfly effects or legacy fearCode & fix

What if you could experience No More…

How well are we estimating hours?

And burning down the

tasks?

Which impediments are still open?

Are we removing impediments in

time?

How quickly are implementing requirements?

Burndown Dashboard

Information burners make all work and risk visible

VS 2010 Recognition

GARTNER: Magic Quadrant for Application Life Cycle Management 5 June 2012

VOKE: Test Market Mover ArrayJuly 2010 Evaluating VS2010

Community votes for best product

IDC: IT PPM Market LandscapeDecember 2010 Evaluating VS2010

Industry analysts recognize Microsoft leadership

The Third Wave of Learning: VS 2012First wave addressed technical debt

Second wave amplified flow of value

Now it’s time to focus on cycle time

In parallel, the we start to delivercontinuous services in parallel to on-premise product

Shortening cycle timeNo (nearly) ‘end game’Late feature requests can’t be handledIntegrations cause fear -- late integrations cause pileups in main

Can’t afford a step in the wrong directionAlignment of features to strategy not always clearChanges are difficult above the feature crew level

Conway’s lawProduct level walkthroughs too time consumingTesting needs to reflect customer not organization flow

Engineering Principles Redux

Drive quality upstream/Don’t defer workIncrease efficiency by doing things right the first time, Bug tail not filled with hidden work

Automate testingAutomated testing at multiple levels; Tests run anywhere

Common definition of what done meansConsistent definition of quality across division enables meaningful metrics for progress

Keep main branch customer readyConsistent delivery of high quality CTPs. Product not tattooed with part finished code

Ready to ship when the time is rightFeature Branches isolate features. RI when quality gates are met; FB PU Staging Main

Defense in depthTests run everywhere in rolling builds with ‘ility tests; Devs write most of regression suite

Find bugs, fix bugsHigh-value exploratory testing; Test features not stories; Fix bugs immediately

Work in small batchesFinish 2-3 stories / team / week; Push for one piece flow

Minimize work in progressUse Story branches; Integrate done, not complete; Keep main ship ready

Reacting to late changes

Drowning in regression testing

VS 2012 ship

1,685 test cases run nightly

7 vendors running automation

99% pass rate 14 test investigations per day

VSU Basics tests pass at 100% daily

VS 2012 start

14,000+ test cases run nightly

23 Vendors running automation full time

33 Product bugs found per year

80% pass rate 2,800 test investigations

Reacting to late changes

Eliminate the ‘end game’

VS 2012 ship

Continuous test pass running

All tests run every night

99% pass rate maintained

Rolling builds run in all active branches

VS 2010

2 week test passes at the end of each milestone

P2 and P3 tests run only during the pass

<50% initial pass rate

315 person weeks (TLM Redmond) 3 won’t fix bugs found

Configuration tested this run

Portfolio management using MMFs and Kanban

Scenario with links to storyboard, value definition, and OGF

Minimal Marketable Features (MMF) with links to storyboards, value definition and acceptance tests

Scenario definition

Clearly express the customer promise

Scenario storyboards

What we will do to deliver on the customer promise

Scenario measurement

Are we building: the right product? the product right? customer value?

Scenario measurement

Are we building: the right product? the product right? customer value?

The Third Wave of Learning: VS 2012First wave addressed technical debt

Second wave amplified flow of value

Now it’s time to focus on cycle time

In parallel, the we start to delivercontinuous services in parallel to on-premise product

PRODUCT BACKLOG

OPS BACKLOG

Sprint Monitor

Wave 3: Continuous Feedback

Priorities

Code & Tests

User Experience

Live Site

Testing

Ideas

Quickly storyboard ideas

Now make it continuous as a request from a PBI

Capture results in the product backlog

Priorities

PRODUCT BACKLOG

OPS BACKLOG

Sprint Monitor

Priorities

Code & Tests

User Experience

Live Site

Testing

Ideas

Work, planned and not, visible

Iteration backlog on a task board

Code & Tests

PRODUCT BACKLOG

OPS BACKLOG

Sprint Monitor

Priorities

Code & Tests

User Experience

Live Site

Testing

Ideas

Continuous Feedback while Coding

CodingFeedback

Loop

Test a little

Code a little

Testing

PRODUCT BACKLOG

OPS BACKLOG

Sprint Monitor

Priorities

Code & Tests

User Experience

Live Site

Testing

Ideas

Explore and Map

http://procl.files.wordpress.com/2011/06/map-of-north-america-1806.jpg

Explore and Map

http://procl.files.wordpress.com/2011/06/map-of-north-america-1806.jpg

Live Site

PRODUCT BACKLOG

OPS BACKLOG

Sprint Monitor

Priorities

Code & Tests

User Experience

Live Site

Testing

Ideas

Escalatefrom production

Real users matter

User voice

2 Metrics: Continuous Flow Of ValueDevelopIdea to working software

OperateIncident to resolution

PRODUCT BACKLOG

Sprint

CYCLE TIME

MEAN TIME TO REPAIR

OPS BACKLOG

Monitor

Reduction of Waste

Trustworthy Transparency

Increased Flow of Value

Shortened Cycle Time

Continuous Feedback

Continuous Business

Improvement

Our Journey:

The virtuous improvement cycle

http://erikasaca.files.wordpress.com/2011/03/ruby-red-slippers2.jpg

There’s no place like production

Questions?

Where to start?

Build a backlogPrioritize work 1..nRuthlessly increase the granularity

Plan an iterationShorter is betterCommit to a concrete goal

Define done, doneStandards to checkin, complete a story, and finish an iteration

Hold a daily standupNot a status meetingCommunicate your plan for the day

You get what you inspect, not what you expect

Make the work visibleAttend the sprint review, put hands on the product, hold the team accountableInstill engineering disciplineUnit testing, continuous integration, code reviews, and rolling builds are a mustA failing test is worse than no test

Focus on business valueMeasure your progress against customer satisfactionAlign crews to strategic initiatives

Emphasize small batchesWithout doubt the most challenging thing in moving to an agile cycleUse Kanban with agile practices to see the bottlenecks

Questions?