A story of reducing technical debt

Founder + Digital Craftsman

Markus Kobler

Avoiding Technical Bankruptcy

Overview

The technical debt build up

Five Techniques that Worked

1) Getting a Canary

2) Rejecting Scrum for Kanban

3) Planning Poker

4) Automating Deployments with ‘Munge’

5) Sharing Ownership with Fez & Pirate Hats!

Dealing with Collapse : A Success Story

Taglab did a bit of everything web related...

Strategy

Transactional Sites

Design

Email(Even built/hosted our own campaign system!?!)

BannersMicro-Sites

Brochure-ware

Social Media

Hosting

Ad Campaigns

24/7/365 Support J2EE Web Apps

Messaging Middleware

Workflow Systems

Notable Clients Included

Betfair

Times Online

Starbucks

The Economist

Disney Channel

Turner

Global Switch

Channel Five

NBC UniversalSkandia

STA Travel

Yell Online

The Spectator

British Gas

Avis

One Water

Liverpool Victoria

My First three years at Taglab had been exciting

- Six fold growth

- Wide range of big and small projects

- Rounded skillset from architectural & development to system administration

However after the initial exciting growth the company started to hit a celling

- Common in small business, 80% fail in their first 5 years*

- J2EE was loosing its perceived competitive advantage

*The E-Myth 2001

Over time technical debt started to build up

Its around this time Taglab asked me to rejoin the company as CTO...

Supporting 40-50 live production web apps

- 100+ Environments in total (UAT, Integration, etc)

- Key required 24/7/365 SLA’s

Hosted Apps had 45+ external integration points

- RESTful & SOAP API’s

- But many XML(sometimes)/RPC

Designed by Architects with Capital A...

...but built by developers (with a lowercase d)

Pain stemmed from multiple points

Pain stemmed from multiple points

Legacy and modern Codebase

- 7+ year old code, Singleton galore with poor test coverage

- Some code even needing decompiling from bytecode...

- To modern lightweight clean GWT, Hibernate, Spring

Hosting Mix

- Mix of our own rack in Docklands to clients hardware and the cloud

- Although our rack provided many benefits some hardware dated back 5-7 years

Projects were increasingly Death Marches

Death March TypesH

appi

ness

Chances of Success

Kamikaze Mission Impossible

UglySuicideTeam are sacrificial

lambs at the slaughter for successful delivery

Team dreams of fame, glory and riches ‘if ’

they succeed

Projects and Team are doomed.Only alternative is to be fired

Projects are doomed, but everyone agrees it will be

glorious failure!

Death March - Edward Yourdon

At its core this is a story aboutH

appi

ness

Chances of Success

Positively and creatively changing

peoples perspective to improve

delivery

Problem 1 : Notification Hell

Nagios monitoring was already in place

- but needed updating

Were getting 4-5 alerts a day!

- Regular Database/Hardware failures

- Deployment, Code Quality problems

- Subtle networking issues (ISP or self inflicted)

You dreaded your phone

Make things worse, we needed more checks!

- CI still needed + much better test coverage

Getting a Canary (or Nabaztag)

Flickr: julianbleecker

Flickr: krissatmontreal

Could differentiate personal from work txt messages

Problems started feeling a little less ‘catastrophic’

The office knew when was not right time ask for a quote on ‘one more copy change...’

Over time major alerts dropped to 1 every few weeks

Getting a Canary (or Nabaztag)

Problem 2 : Juggling Delivery + Support

We where hungry for work, so new projects varied wildly in size and complexity

- Some were multi year builds

- Others ‘OMG we need it done by the weekend’

Support’s unpredictability impacted new build work

- More common in-practice with teams impacted by 2nd/3rd line support issues

Mixing both functions can be efficient and satisfying

'Known Knowns' - requirements captured in the specification

'Known Unknowns' - requirements flagged as ‘assumptions’

'Unknown Knowns' - requirements the client had not yet told you about

'Unknowns Unknowns' - requirements neither you or client knew about

Successful Delivery is about Accepting UncertaintyFlickr : ooocha

Sounds like Agile Right?

Many competitors and clients were touting SCRUM

“Don’t you know the difference between the Chicken and Pig!”

But clients still found it hard to sign off budgets

“Need to know what is being delivered before I signing off costs!”

Work/Budget would often not fit neatly into an iteration(s)

How would unpredictable support be incorporated?

- Having a ‘batman’ would have been wasteful or not enough

We needed something less rigid

Rejecting Scrum for Kanban

To-Do Doing Done

Task TaskTask Task

Task

Task

TaskTask

Task

Task

TaskTask

Task

So we started simply

Rejecting Scrum for Kanban

To-Do Doing

Task TaskTask

Task

Task

Task

Task

Task

Task

Task

Task

Task

Task

Added columns that made sense to us

UAT CAT Live

Task

TaskTask

TaskTask

Rejecting Scrum for Kanban

Backlog In Progress

Task Task TaskTask

Task

Task

TaskTask

Task

Task

Task

Task

Task

Added columns that matched our internal processes

UAT CAT Live

Task

Task

Task

Task

Task

Planned DevInternal

Test

Task

Task

Task

Task

Task

Task

TaskTask

Rejecting Scrum for Kanban

Backlog In Progress

Task Task

Task

Task

Task TaskTask

Task

Task

Task

Task

Added swim lanes

UAT CAT Live

Task

Task

Task

Task

Planned DevInternal

Test

Task

Task

Task

Task

Task

TaskTask

Majo

r Proj

ect 1

Majo

r Proj

ect 2

Other P

rojec

ts

Intern

al

TaskTask

Task

Task

Rejecting Scrum for Kanban

Backlog In Progress

Task Task

Task

Task

Task TaskTask

Task

Task

Task

Task

Introduced Work In Progress (WIP) limits and SLA’s

UAT CAT Live

Task

Task

Task

Task

Planned Dev InternalTest

Task

Task

Task

Task

Task

TaskTask

Majo

r Proj

ect 1

Majo

r Proj

ect 2

Other P

rojec

ts

Intern

al

TaskTask

Task

Task

5 Day SLA

15 4 12

IssueImprov-ement

Improv-ement

Rejecting Scrum for Kanban

Backlog In Progress

Task Task

Task

Task

Task TaskTask

Task

Task

Task

Task

Visually distinguish types of tasks

UAT CAT Live

Task

Task

Task

Task

Planned Dev InternalTest

Task

Task

Task

Task

Task

TaskTask

Majo

r Proj

ect 1

Majo

r Proj

ect 2

Other P

rojec

ts

Intern

al

TaskTask

Task

Task

5 Day SLA

15 4 12

Improv-ement

Improv-ement

Issue

Issue

Rejecting Scrum for Kanban

Work is continuously pulled through system

Visually provides a Micro & Macro view of work

Support issues easily absorbed

WIP and SLA’s provided key indicators of quality and resource planning

Problem 3 : Estimation + Tracking Velocity

Flickr : kozumel

Planning Poker

Cards going up in value

User Story discussed

Once happy, everyone estimates in private

Reveal at same time

Hi & Lo estimates discuss differences

Repeat estimate again if no consensus

Flickr: lejoe

Shared ownership of projects problem solving

Could start analytically looking back at projects past progress

Enabled plotting projects into future based on previous tasks/estimates

As a PM you can often get team to develop simplest solution and feel good about it!

Planning Poker

Problem 4 : Deployment

Deployment should be the last problem developers need to worry about

- Needed to re-instil confidence back in the team

Had to work well with Designers and Developers

Needed to adapt to different environments

- Our Rack, Client’s hardware, The cloud

Reducing time to deploy would reduce our costs

- Worst cases deployments took 5 hours!

Designers Needed Something Simple

Flickr : thunderchild5

Developers Demanded More Rigour

Flickr : shutupyourface

Automating Deployments with ‘Munge’

Mix of automation and strict project conventions

- Minor changes could be released in seconds

- Major releases, tested and deployed in 5-20mins (depending on changes)

Clean build environment and version controlled released meant predictable builds and rollbacks

Used live environment metrics to determine deployed hosts

Problem 5 : Shared Ownership of Prod Issues

Whole team needed to share ownership of dealing with production issues

Most problems only needed one person looking at it to begin with

Ideally ownership should be visual to rest of team

So we got some hats of responsibility

Flickr : doctorow

So we got some hats of responsibility

Flickr : c_r_i_s

So we got some hats of responsibility

Flickr : striatic

So we got some hats of responsibility

Flickr : blackcountrymuseums

So we got some hats of responsibility

Finally a Success Story

Flickr : dybarber

Finally a success story

2009, Setanta Sports was on verge of collapse

- A key client dependant on their sports coverage

- Unsure if someone would step in with finance

- Or someone else would buy the rights

Particularly complex CRM/Sales intergration

- Estimated >5 weeks worth of effort

- Failure to deliver by immovable deadline was to be very damaging for all parties involved

Sign off received 3 days before changes were required

Success Story

Work broken into 5 interleaved dev streams

Hit first key 3 day deadline

- 4 major releases followed over next 7 days

- More followed in the subsequent weeks/months

Sales in the days that followed dwarfed anything that came before with 0 downtime

One of the most satisfying projects have been lucky enough to be involved in

All thanks to hard prior work put in by the team

Final Thoughts

Our biggest challenge was turning bleak situation into positive opportunities

- Some times you need a creative solution

- Continuously take educated technical risks

Software delivery is an art-form

- but backed by scientific approaches

Embrace change and unpredictability

- but always keep a clear guiding vision

An informative workspace is key

- Although is largely about ‘Psychological Manipulation’

Credits

Death March (Yourdon Press Series) - http://www.amazon.co.uk/Death-March-Yourdon-Press-Edward/dp/013143635X/ref=sr_1_1?ie=UTF8&qid=1290856412&sr=8-1

Nabaztag - julianbleecker - http://www.flickr.com/photos/julianbleecker/156303245/

Nabaztag - krissatmontreal - http://www.flickr.com/photos/krissatmontreal/3027612184/

061108-F-5586B-137 - ooocha - http://www.flickr.com/photos/ooocha/3051551020/

Poker Phase - kozumel - http://www.flickr.com/photos/kozumel/4063207100/

Planning poker! I've a straight flush! - lejoe - http://www.flickr.com/photos/lejoe/4553607341/

The Big Easy - thunderchild5 - http://www.flickr.com/photos/thunderchild5/2397161370/

Factory - Robotic Arms - shutupyourface - http://www.flickr.com/photos/shutupyourface/105779625/

Me in Ada's pirate hat, monorail queue, Walt Disney World, Orlando, Florida - doctorow - http://www.flickr.com/photos/doctorow/2137686/

Fez / 87.365 - c_r_i_s - http://www.flickr.com/photos/c_r_i_s/2852508138/

past tense - striatic - http://www.flickr.com/photos/striatic/2418853122/

1940s flying helmet , LC1995_15_1 - blackcountrymuseums - http://www.flickr.com/photos/blackcountrymuseums/4970602229/

Fremantle Bridge Collapse, 22 July 1926 - dybarber - http://www.flickr.com/photos/dybarber/3461384691/

Thank you.

distinctive.co

@markuskobler