Beyond Scrum of Scrums

Copyright © 2011 Constant Contact Inc.

Beyond a Scrum of Scrums: Scaling Up Agile with Kanban

Gil IrizarryProgram ManagerConstant Contact

June 2011

2Copyright © 2011 Constant Contact, Inc.

Agenda

• A bit about me and Constant Contact

• Scrum and Kanban at the team-level

• Scaling up to an engineering org


My background

• Over 20 years software development and management experience

• Over 5 years in an agile software development environment

• Most recently working with Kanban, prior to that Scrum and RUP

• CSM and PMP certifications, Kanban coaching training with David Anderson

• BS from Cornell, ALM from Harvard, certificate in Management from MIT Sloan


My contact info

• [email protected]

• [email protected]

• Contact me if you would like updated slides


Background on Constant Contact

• SaaS company offering on-line e-mail marketing, event marketing and surveys. Recent enhancements extend the services to the social media space

• >$200MM gross revenue per year

• >800 employees

• >450K paying customers

• Engineering and Operations total about 150 people

• First Scrum team formed in 2006


The Goal

• We want to release more features more frequently

• Why?

• React quicker to changing market conditions

• Get new features to users more quickly

• Frequent releases are smaller releases: our customers often cannot absorb a large set of changes


The Challenge

• 13 software development teams

• 5 development and testing server clusters

• 2 production server clusters in 2 data centers (soon to be 3)

• A code base that was first created around 2004 and built up over time

• And all utilizing one operations team


What do we have?

• A mess of inter-dependencies

• Image from

http://viewyonder.com/2009/07/12/data-center-consolidation-road-to-unified-computing-service-analysis/ - Data Center Spaghetti. This problem exists in many other parts of the enterprise!

http://viewyonder.com/2009/07/12/data-center-consolidation-road-to-unified-computing-service-analysis/

http://viewyonder.com/2009/07/12/data-center-consolidation-road-to-unified-computing-service-analysis/


So, how do we address this?

• Scrum of Scrums!

• Um, wait…


A little review: how things used to be


This is appropriate if you are building an…

“Originally planned to enter service in May 2008, the [Dreamliner] project has suffered from repeated delays and is now more than three years behind schedule.” -- Wikipedia


Quick Review of Scrum

• Fixed iterations

• Daily stand-ups

• What did you do yesterday, what did you do today, any impediments

• Retrospectives

• Burn-down chart

• Board with To Do, In Progress and Done states


Scrum in a slide

Sprint: 4 weeks

3 Meetings

3 Documents

3 Roles

PO

1

SM

3

Product Backlog

1

Sprint BL

2

Burn-down

3

Product

Stakeholders

On-going Collaboration

Sprint Planning

Sprint Planning

1

Daily Scrum

2

SprintReview

SprintReview

3

Daily

TEAM

2


Scrum of Scrums (of Scrums (of Scrums…))

• How to integrate teams in Scrum?

• Hold a Scrum of Scrums

• What happens if you have too many teams?

• Hold a Scrum of Scrums of Scrums, and so on…

• Add a MetaScrum or two


Foundational Principles of Kanban

• Start with what you do now

• Agree to pursue incremental, evolutionary change

• Respect the current process, roles, responsibilities & titles

From: http://agilemanagement.net/index.php/Blog/the_principles_of_the_kanban_method (David Anderson)

http://agilemanagement.net/index.php/Blog/the_principles_of_the_kanban_method

http://agilemanagement.net/index.php/Blog/the_principles_of_the_kanban_method


5 Core Properties of Kanban

• Visualize the workflow

• Team board states are a reflection of the value stream

• Limit WIP

• Manage Flow

• Implied that flow should be continuous

• Make Process Policies Explicit

• Improve Collaboratively (using models & the scientific method)


Sample Kanban Board


Sample CFD

11/9

/201

0

11/1

5/20

10

11/2

1/20

10

11/2

7/20

10

12/3

/201

0

12/9

/201

0

12/1

5/20

10

12/2

1/20

10

12/2

7/20

10

1/2/

2011

1/8/

2011

1/14

/201

1

1/20

/201

1

1/26

/201

1

2/1/

2011

2/7/

2011

2/13

/201

1

2/19

/201

1

2/25

/201

1

3/3/

2011

3/9/

2011

3/15

/201

1

3/21

/201

1

3/27

/201

1

4/2/

2011

4/8/

2011

4/14

/201

1

4/20

/201

1

4/26

/201

10

10

20

30

40

50

60

Makeover Project - Cumulative Flow Diagram

User StoryMockupsReady-DoneIn DevelopmentDev DoneIn TestingComplete


Compare with Burndown chart

19-J

ul

21-J

ul

23-J

ul

27-J

ul

29-J

ul

2-Aug

4-Aug

6-Aug

10-A

ug

12-A

ug

16-A

ug

18-A

ug

20-A

ug

24-A

ug

26-A

ug

30-A

ug

1-Sep

3-Sep

7-Sep

9-Sep

13-S

ep

15-S

ep

17-S

ep

21-S

ep

23-S

ep

27-S

ep

29-S

ep1-

Oct0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00Social Project - Burndown to date

Social Media Tasks

Linear (Social Media Tasks)

Tas

ks R

emai

ning


Scrum and Kanban: Let’s combine them!(But how? And why?)

http://en.wikipedia.org/wiki/Reese's_Peanut_Butter_Cups#Marketing_and_advertising




Transistioning from Scrum to Kanban

• We did not change how teams were currently working

• We modeled existing hand-offs within the team, i.e. each team’s kanban board reflected that teams style of work


The Dilemma

Remember that teams want (and often need) to work continuously, but releases are discrete.

Multiple teams working together create interdependencies

How to resolve this?


Transforming Scrum (and Scrum of Scrums)

• At the enterprise-level, team(s) management becomes a problem of dependency management

• Planning, coordination and stand-ups at the departmental and organizational level need to be executed with this mind


What we have done at CTCT

• At the organization-level:

• Prioritized project list

• All At Once Planning

• Classes of Service

• (Near) Continuous integration and (mostly) automated regression testing

• Dependency and deliverable review

• Release train


What we have done at CTCT

• At the team-level:

• Continuous delivery

• Better metrics and metrics-driven estimation


Prioritized Project List

• Product Management, Engineering and Operations leads meet every two weeks to prioritize all development projects

• This becomes the organizational backlog, which drives each team’s backlog


All At Once Planning


All At Once Planning

• Prior to the start of an iteration, teams use the prioritized project list to plan their upcoming work.

• Planning involves the identification of deliverables and dependencies.

• Dependencies are discussed with dependent teams.

• A meeting is held (all at once planning mtg) in which all development teams present their dependencies to each other and to the operations team.


All At Once Planning, cont’d

• At the end of the meeting, each team has their planned deliverables and incoming dependencies.

• If they haven’t already, they determine their capacity and, based on the priorities, commit to a set of work.

• This means that a team may have capacity to do work, but may not get to it in a release if that work pushes the operation team beyond its capacity.

• Once the iteration starts, we will have each team’s set of commitments.


Classes of Service

• Inevitably, the operations team can often become a bottleneck for development.

• We attempt to manage this through classes of service. We borrow a concept from Kanban that says similar projects are grouped into classes and each class is assigned an allocation.

• For example, we may decide that 20% of ops time should be spent on infrastructure improvements, and 80% spent on servicing development


(Near) Continuous integration and (mostly)automated regression testing

• It is important that builds, deployments and testing are as automated and continuous as possible. Tools we use:

• Hudson for continuous integration

• Puppet for automated deployments

• Maven for managed builds

• Selenium for automated testing

• svn for source control and versioning

• We still have a lot of work to do in this area


Dependency and deliverable review

• We have a daily meeting, about 20 minutes, long in which we review each project, not each team.

• Project leads review the deliverables and dependencies to which they have committed and say if they are on track or not.


Dependency and deliverable review, cont’d

• We found the teams could still miss a deliverable even if they had no impediments. Deliverable tracking provides a better view of the state of the iteration

• However, we still call the review meeting “Scrum of Scrums” because the name stuck!


Release Train

• Releases are iterative but development is continuous


Release Train, cont’d

• Scrum aims to make development iterative but this causes problems:

• How do you handle all the testing at the end of the sprint? What if defects are found a day before the sprint ends?

• If a deliverable misses a release (the train), it simply waits to capture the next one.

• To be reiterate (and be explicit): we don’t penalize a team if a deliverable is not done at the end of a release and misses the release train


Release Train, cont’d

• To make this work:

• Releases must be short

• Must have those tools for automated deployments

• Marketing and support must be flexible enough to react to last-minute product changes


Kanban at the team-level

• Teams plan continuously

• Teams test continuously

• It’s OK if a team finds a defect on the last day of the release.

• It’s OK if a team starts work for the next release in the current release

• Development and testing can flow more smoothly, because we do not want the end of an iteration to feel like this:


Kanban at the team-level


Metrics

• We gather for each team:

• Cycle time on items after grouping them by size:

• Completion time for small, medium and large

• Spread of cycle times

• Work items completed

• Open defects in production, to give a gross measure of technical debt


Metrics guide planning and estimation

• Over time, we expect that the spread of cycle times for a given item size goes down.

• So, over time, an estimate of completion time for items of a given size should become more accurate.

• We have eliminated planning poker. Work items are just sized as smalls or mediums and the average cycle times for those sizes from the last release become the estimate for the upcoming release.

• Large items are broken down in smaller items


2010 R7 2010 R8 2011 R1 2011 R2 2011 R3 2011 R40

5

10

15

20

25

30

35Average Cycle Times for work items

Average of Cycle Time (small - 1 Story Point)

Average of Cycle Time (medium - 3 Story Points)

Average of Cycle Time (large - 5 Story Points)


How did we do?

• Releases used to take 12 weeks. Now they take 5 weeks.

• More importantly, testing used to take 6 weeks. Now it takes 1 week. Testing used to be 50% of the release cycle, but now is just 20%.

• We have a better picture of our release at any given moment

• A year ago, CTCT was not in the social media space. Now our service has ties to Facebook, Twitter and LinkedIn. We are becoming a leader in the social media marketing space.


How did we do?


Time for Questions

Business

Beyond Scrum of Scrums