© 2015-2016 DeLormeMay 3, 2023 1 © 2015-2016 DeLorme

Agile Software Development at DeLorme

Mark D. Grover, CSM, Ph.D.inReach Web Team Scrum MasterPresented to: Agile Maine, Dec 8 2015

© 2015-2016 DeLormeMay 3, 2023 2

DeLorme: What We Do

Founded in 1976Current products:

inReach communicatorDigital data productsPaper AtlasesGPS devices and companion desktop products (consumer and GIS)

© 2015-2016 DeLormeMay 3, 2023 3

Software Teams

Web, Services, and Manufacturing Automation Team (today's subject)Mobile Firmware/Apps TeamLegacy Support Efforts including GISIT, DBAs, plus distributed QA

© 2015-2016 DeLormeMay 3, 2023 4

In Days of Yore

Activist integrators/managersLarge projects, distant from customer feedbackSudden change orders to developersUnpredictable and slow deliveriesSilosPersonal projectsEvery day a surprise

© 2015-2016 DeLormeMay 3, 2023 5

Recent Development Focus: inReach

Satellite 2-way text communicatorGlobal coverage24-hour worldwide SOSRuggedModel evolution"Home base" online supportOver 50,000 active unitsEnterprise support

© 2015-2016 DeLormeMay 3, 2023 6

Good Pre-Agile ImprovementsSource Control (c. 1997)Code-sharing discipline: multiple product versions from common core librariesDev/Test/Prod versions of code and products (c. 2007)Restrictions on access to each levelAutomated TestingCode Reviews for everything, everyoneRelease/Deployment versioningContinuous IntegrationBug reporting systemsCo-located teamsStand-ups

© 2015-2016 DeLormeMay 3, 2023 7

New project — New Team

inReach Web, late 2011Message Gateway"App" tier: device/user profiles, message analysis/history, Explore site, SOS managementFinance, Sales, Marketing, Billing, Administration support tierOther: APIs, Manufacturing support, Sync app, standalone gateway, data delivery and cuttingC#, JavaScript, SQL

© 2015-2016 DeLormeMay 3, 2023 8

Introducing Agile

January, 2013Enter a championEstablishment of roles (PO, SM, team)Buy-in from managementInternal training, discussion, acceptanceStakeholder and management involvementNew tools: Greenhopper/Jira, Git, Stash, TeamCity. Later: NuGet, Octopus, NewRelic

© 2015-2016 DeLormeMay 3, 2023 9

FundamentalsAgile ManifestoFirst model: Classic ScrumProcess Principles:

Team, as a whole, makes the rules.Rules can be changed.

Team:Personnel level has varied but only slightly

6-8 devs, 2.5 QA, 1+ PO/UI, .5 SMOrganized around existing talentCo-located here, but with third-party partners worldwide

© 2015-2016 DeLormeMay 3, 2023 10

Working Agreement1. Embrace the Manifesto and the Principles.2. Work as a team – involve others in decision-making.

Include stakeholders.3. Use the inReachWeb Web Team distribution list – share

your questions and knowledge.4. Our daily scrum meets at 915 AM at the Couch of

Learning. When working remotely, submit a report via email to the team if you will be absent.

5. Meetings: Be on time. End on time.6. Don't break the build.

© 2015-2016 DeLormeMay 3, 2023 11

7. Update the Backlog before the daily scrum.8. Don't leave for the day until your promoted code has

built.9. Don't take on un-estimated features or improvements

(or at least get those present in the office area to generate a quick estimate when it can't be avoided).

10.Raise critical issues to the team and fix them in the current sprint whenever possible.

11.Clearly document all test steps.12.Keep this working agreement simple and short.

Working Agreement II

© 2015-2016 DeLormeMay 3, 2023 12

ScrumStarted with classic ScrumGen 18 becomes Sprint 18Three week sprintsPublic deployment with each sprintVariations introduced quickly:

Tickets generated by QA from regression get zero story points, as well as emergencies/hotfixesMid-sprint hotfixes are occasionally required, charged to current sprintTicket swapping in/out for equivalent story pointsIntroduced "Sweep Week"

© 2015-2016 DeLormeMay 3, 2023 13

Sweep Week

Development 1 (two weeks)

Test/Deploy 1 (one week)

Development 2

Test/Deploy 2

© 2015-2016 DeLormeMay 3, 2023 14

Results

Regular deliveriesOn-time, on-targetReasonably few hotfixesAdded 24-hr support: Introduction of Engineer-on-Call (24/7 for one week at a time)

© 2015-2016 DeLormeMay 3, 2023 15

Evolution Begins

Sept 2013: Loss of original PO/CheerleaderIntroduction of Dual POs

one to focus on strategic partners, upper management, and financeone to focus on consumer/professional customers, Customer Care concerns and UI design

With growing product popularity and attention…

Three-week deliveries inadequateEveryone wants more responsiveness, including team

© 2015-2016 DeLormeMay 3, 2023 16

Sprint 36 (May 2014)

Introduced the one-week overlapping sprint, with mid-week deploymentsTeam initially had a tendency to under-commit, and then became more aggressive by adding "swapping rule"Result: Many tickets get postponed and swapped out for more important items

Most important gets doneSome items get postponed

© 2015-2016 DeLormeMay 3, 2023 17

Sprint 48 (August 2014)

Identified problem: Team distractions, requests, emergencies that require technical intervention and researchTechnical requests sent to the team, Engineer-On-Call triagesFeature requests to the POsBug reports to JIRA if possibleAPI questions directed to forum

© 2015-2016 DeLormeMay 3, 2023 18

Sprint 57 (Nov 14) – Evolution cont’d

Began experiment with epics. Dedicate most of team to one long-term project.

Eventually, down to one or two staff on a long-term epic.Introduced pull-oriented (Kanban-like) approach.

Changes from "sprint-oriented" to "ticket-oriented"No specific commitmentsPO can/add remove new items, marking highest prioritiesTeam members take new items as they become freeAt cutoff time: no regrets

© 2015-2016 DeLormeMay 3, 2023 19

6/10/2015

2015Day 1 2 3 4 5 6 7 8 9 10 11 12

PO/SM Ticket Review6/10/2015

Gen 85 Deployment to Test

Estimation Meeting6/11/2015

Gen 84/85 Demo6/17/2015

Gen 85 Production Testing6/17/2015

Gen 85 Retro and Planning6/8/2015

Gen 85 QA Assessment of Resources6/12/2015

PO/SM Ticket Review6/12/2015

Gen 85 Cutoff6/12/2015

Gen 85 QA Regression Testing6/15/2015

Gen 85 Deployment to Production

6/17/2015

Gen 85 Closed

6/19/2015

6/8/2015 6/12/2015Gen 85 Development

6/15/2015 6/19/2015Gen 86 Development

Life of a Sprint

© 2015-2016 DeLormeMay 3, 2023 20

Visibility

Google Analytics

What’s Up

NodePing

Loggly

PagerDuty

Automated TestsNewRelic

Octopus

TeamCity

© 2015-2016 DeLormeMay 3, 2023 21

Scrum Master Philosophy(Half-time developer)Keep everything in motion.Watch for stuck processes.Anticipate. Keep the chaos at bay.Be vocal about inefficiencies.Do what others don't want to do but needs to get done.Don't make decisions: See that they get made.Don’t micromanage – you are not even a manager.Spread awareness; fill in communication gaps; keep everything visible.Don't interfere with a good thing.Don't expect glory.

© 2015-2016 DeLormeMay 3, 2023 22

Low wallsAll team members in line-of-sightRoom for 16Includes dev, QA, SM, PO/UIMix of traditional and stand-up desksConference areas and rooms available

New Co-location Plan: June, 2015

© 2015-2016 DeLormeMay 3, 2023 23

MetricsAs of June, 201585 generations, good records for 64 sprintsCompleted ticket counts:

Highest: 67 (six-day sprint)Lowest: 12 (four-day sprint)Note: counts include any "free" regression tickets

Story Points (normalized to one week)Average: 88High: 228Low: 16Standard Deviation: 40 (!)How useful are these numbers to the PO?

Tickets for the project8700+ created since September, 2011~300 open~6000 completed/fixed (69%)~700 duplicate, unreproducible, incomplete~900 indefinitely suspended/deferred~800 rejected (won't fix)

© 2015-2016 DeLormeMay 3, 2023 24

How does a development team handle day-to-day operations issues?

Engineer On CallIT On Call

Train/empower:Product OwnerCustomer CareTechnical Sales SupportTier 2 Support?Maintenance Team?

What About “Ops”?

© 2015-2016 DeLormeMay 3, 2023 25

Challenges and TemptationsDistractions and interruptionsUnrealistic stakeholder expectationsAdding more work after sprint cutoffNon-agile contracts; custom workMaking tickets too bigNot letting the team know what's going onNot asking the team for helpWorking before PO approvalManagement pressure on the PO (“remote control”)Team wants to decide “what”; POs want to decide “how”; SM wants bothWorking without a Kanban (ticket)Rushing step to “completion” (code, review, test, deploy)Shiny Object SyndromeEverything can be newer and better (infinite refactoring syndrome)

© 2015-2016 DeLormeMay 3, 2023 26

Glorious FutureSystem Traffic and Usage Multiplying Each YearHow to Grow?

Team Size vs. Customer BaseThroughput, Reliability, Cloud, AgentsSpecial project teamsShould teams align with architecture?Operations vs. R&D

© 2015-2016 DeLormeMay 3, 2023 27

Conclusion: Successfully Agile

Co-location is KeyTeams are self-correctingVisibility breeds assent/approvalA responsive attitude is inspiringEmpowered teams are awesome