2013-10-14
1
©2004 Bedarra Research Labs. All rights reserved.
Agile.Next – Accelerating Product Agility Lean Value Driven Product Development
Dave Thomas
Bedarra Research Labs, YOW! Conferences,
Carleton University, Queensland University of Technology, Lodestone Foundation, Depth First
Outline
The Demand for Greater Agility
Products are from Venus - Projects are from Mars
Business Agility – Streamline Your Organization
Technical Agility - Cutting Code Faster
Discussion
© 2012 Bedarra Research Labs. All rights reserved.
2013-10-14
2
©2010 Bedarra Research Labs. All rights reserved.
Benefits of Agile Development
Improved Predictability! § Simplified work flow § Short iterations
§ Collaborative estimation § Simplified visual progress tracking
Improved Quality! § Improved requirements communication
§ Integrated teams § Test driven development § Continuous integration
A little Improved Productivity § Reduced meetings § Reduced rework
§ Improved Response to change § Improved Communication
§ Reduced Stress
©2012 Bedarra Research Labs. All rights reserved.
“We’ve given you Objects, Agile, Tools, Hardware, New Offices, Mentors, Great Compensation because we believe you need these to be your best and have great confidence in your abilities to meet our competitive challenges. Here is what here is what the business needs from you. 1. We need to be much faster in selecting and executing high value projects. This means much better estimates of $ and time to value based on our capacity. 2. We need to turn up the dials on productivity while maintaining quality and features.” …. THIS IS YOUR MISSION…THIS TAPE WILL SELF DESTRUCT ….
CEO – All Hands Product Challenge “As long as we keeping do software the same way as everyone else
were have no competitive advantage.”
2013-10-14
3
Better, Faster, and Cheaper!!!
Typical Implications! 1. Cut out any complicated
functionality even if it is important
2. Rush it through with missing functionality and too many defects
3. Deploy a Prototype, followed by a support nightmare
4. Over staff (in source or outsource deliver a mess late)
©2012 Bedarra Research Labs. All rights reserved.
Cycle Time!
Software Value Chain
Better, Faster, Cheaper – A New Road?
We must adapt and improve our way of building software to meet the challenge.
1. Advance – use alternative techniques to communicate, design, estimate, build, test and deploy.
2. Refactor our organization to streamline and enable more concurrency and reduce cycle time without reducing quality. We need leverage what works and not be constrained by current best practices. If it is slow it has to go!
3. Explore and Experiment – we need to envision alternatives and evaluate them quickly before betting too much on any approach. We need to fail vast to maximize ROI and time.
4. Focus on Value – Target resources and innovations to where they will make a difference.
© 2012 Bedarra Research Labs. All rights reserved.
2013-10-14
4
Change is easy, as long as someone else is doing it
A Natural-Born ‘Leader’ Emerges...apparently self
organization needed his leadership
“Who him? ... Oh, he’s the Product Owner you’ve been hired to replace…he failed to
feed the developers on demand
March Sprint 1 2 3 4, Sprint! …..
©2010 Bedarra Research Labs. All rights reserved.
Leadership Matters!
Has integrity/stands by values, Leads by example, Executes with transparency, Has the wisdom of experience , Articulates the
vision, Says when they are wrong, Judges fairly, promotes trust, Delegates authority, Coaches versus directs, Promotes
constructive diversity, Makes timely consistent decisions, Sense of humour, Positively motivates and educates, Always tries to
find the best wrong answer, Has time to listen
©2011 Bedarra Research Labs. All rights reserved.
“Agile Leadership is Fragile” Nigel Dalton
Sustained Leadership is still the Largest Challenge
2013-10-14
5
• Respect isn’t optional
• Colored Hats ; What’s your color; Who stole my cheese; Psychology of Computer Programming
• Working with difficult people
• How different people Learn and Change
• Respect the Organizational API – Lead with Humility
• Dilbert invisible to our Dilbert?
• Embrace competent strong opponents
• Diversity Imperative – Human, Geographic and Technical
© 2012 Bedarra Research Labs. All rights reserved.
Software Sociology Considered Essential
Specialists are Essential, Special People are Not!
Specialist • Expert in one or more technical or business areas • Deep experience in one or more technical or business areas • Generalist understanding of other areas
Special Person • I can’t be with the team because …. • I don’t use the same wrong tools, process because … • That is now how we do it in UX, DB, OPS, … • I only do x, y or z because I’m special
© 2012 Bedarra Research Labs. All rights reserved.
2013-10-14
6
Collective Ownership
Collective Ownership (more than Pairs)
4 eyes on the code • shares knowledge • reduces risk • reduces stress • prevents defects Achieved by pairing, reviews, inspections...
©2010 Bedarra Research Labs. All rights reserved.
Executive
Technical Director
Team Leader
Individual Contributors…
Individual Contributor…
Management Ladder
Lean Software Organization Technical Ladders, Playing Coaches and Communities
Distinguished Engineer
Principal Engineer
Outstanding Contributor
Technical Ladder
©2011 Bedarra Research Labs. All rights reserved.
Management Architects
Leads
Customer Product
Mgr. Tools
Process Infrastructure
Platforms Coaches
Release Deployment
Support Test
Driven Development
Products
Learning COPs
Align Compensation with Work Products and Goals
2013-10-14
7
Lean – Ouch! Thinking?
©2010 Bedarra Research Labs. All rights reserved.
Lean is a top down process to improve the software value chain by reducing waste, improving flow and continuous improvement
Agile is a bottom up team centered process for improving development predictability and quality.
Agility is the ability of the business to respond to rapidly changing demands.
©2010 Bedarra Research Labs. All rights reserved.
Lean Thinking – The Zen of Agile
2013-10-14
8
Lack of common vocabulary, rhythm and tools
Too many meetings – 1/meetings is a productivity metric
Lack of transparency
…
©2010 Bedarra Research Labs. All rights reserved.
Lean: Examples of Software Waste
Lack of understanding (requirements, architecture, design)
Lack of training (requirements, architecture, design, development, test)
Defects (requirements, architecture, design, programs, tests)
Unmanaged supplier risks
Unnecessary fire drills (feature requests disguised as defects)
Excess repair vs. timely replacement
Manual testing
Unnecessary process artifacts
Big Up Front (Analysis, Architecture, Design,
Too many any dependencies
Gold Plating – Requirements, Code, Tests, Tools/Frameworks
Lean – KISS (Keep It Simple Stupid) Staff your team with people who have a track recorder for timely
quality delivery
Fix price your consultants and make sure the delivery with acceptance tests and reward with fast acceptance payment
Reward Tangible Requirements with Timely Quality Deliver
Business ensures the value is as stated by deliverables into their performance plan
Avoid dependencies on anything especially infrastructure, reorgs, space, expertise, tasks forces ...
Reduce meetings to increase available work effort and make the ones
Be a wildly optimistic downside planner, prepare for everything major to fail, second guess your own plan
©2004 Bedarra Research Labs. All rights reserved.
2013-10-14
9
Projects are from Mars - Products are from Venues
© 2012 Bedarra Research Labs. All rights reserved.
Projects are about apps and resource; Products are about Features IT sees feature teams == project teams; Products require component and feature teams; Projects App is all we need, reuse is optional and unlikely; Product Architecture and Reuse is Essential Project focus is my App ; Products all about Interfaces, Dependencies Project apps can evolve : Products need Design to Last Projects resources pooled ;Products own their $, backlogs, teams … Projects delivered incrementally ; Products need a schedule and deliverables for major functionality
IT Project versus Product Architectures
Component Break Down Structure (Software Bill Of Materials)
Application Layer
Service Layer
Platform Layer HW/SW Encapsulation, Drivers, Protocols
Means subsystem
Means component, framework or library
App Tier SOC Tier Data Tier
My Project
My
Pro
ject
My Project
2013-10-14
10
Lean and Agile Values and Principles
Product Development Activities At-A-Glance
Envisioning
Product Owner Team Common Work Practices
Definition Development DevOps
Prototypes/Models
Requirements Backlog
Risk Backlog
Team… Team…
GUI Guidelines
Architecture
Product Backlog
Release Backlogs
Team…
Team…
Team…
Team…
Potentially Shippable Product
Team Release Backlog
Shippable Code Increments
Sprint Release Backlogs
CI&T
CI&T
©2006-2007 Bedarra Research Labs and Object Mentor
Visualize Flow – Put It On The Wall You Can Only Improve what You Can Measure
Artifact Status Reporting Status of artifacts such as personas, problems, use scenarios, stories
Subjective Reporting Qualitative data entered by teams at retrospectives
CI Environment Reporting Status of stories written, stories completed, tests written, tests passed and tests failed
Query & Reporting
Cumulative Flow
Retrospective Reports
Unit and Story Tests
©2011 Bedarra Research Labs. All rights reserved.
2013-10-14
11
Velocity and Burndown So XP 2000!
© 2012 Bedarra Research Labs. All rights reserved.
Cumulative Flow Diagram
Complete backlogs ensure that all work is visible, not just new feature work
Use Complete Backlogs
Development Work Items A list of the work items that need to be completed by the Scrum team during the Sprint that contribute to the Product being created. Examples include: -- Writing code and tests -- Writing stories -- Writing acceptance criteria and tests -- High-level software design -- Wireframes - Design and Code Reviews -- Software or GUI prototyping -- Feature planning and estimating -- Release engineering tasks -- Beta test plans -- Documentation plans -- Localization plans
Backlog
Non Development Work Items
A list of any work items that impacts the productivity of the team outside development such as: -- Training courses -- Task forces -- Performance reviews and evaluations -- Answering questions from other teams -- Vacations and holidays
2013-10-14
12
Portfolio Management – Managing to Capacity Program Feature Team
P1 F1 Blue
F2 Blue
F3 Red
F4 Red
F5 Red
F6 Red
P2 F7 Yellow
F8 Green
F9 Green
F10 Purple
F11 Purple
P3 F12 White
F13 White
F14 White
F15 White
Component F16 Orange
F17 Orange
F18 Orange
Company Backlog
Program Backlogs
Team Backlogs
Program
Feature
Epic
Story
Task
MRI Mechanical Control
Table Movement
Horizontal Movement
Medical Imaging
Position w/ Joy Sticks
It’s All in Backlogs! Evidence for Action Lumpy Stories – sizes, estimates, clarity
Story Rework Due To Changing Requirements
Uncompleted Stories
Story Rework Due to Integration Problems
Missing Important Functionality
Done but not usable, poor performance
Done doesn’t mean acceptance tested?
More stories keep getting added to the backlog
Lack of business Value/Tangibility
Blocked on Another Team
Blocked waiting on Product Owner
All resources 100+% allocated
©2009 Bedarra Research Labs. All rights reserved.
2013-10-14
13
Coo! Is that Scrum or Kanban Wall? No It is Our Wall!
© 2012 Bedarra Research Labs. All rights reserved.
My really cool feature …
What is on the back of the card?
Tangible Requirements
• easily expressed by Business
• easily understood by IT
• acceptance easily understood by Business
• acceptance easily understood by IT
• acceptance criteria => acceptance test (AT)
©2004 Bedarra Research Labs. All rights reserved.
Story Acceptance Criteria
2013-10-14
14
Tangible Requirements • easily expressed by Business
• easily understood by IT
• acceptance easily understood by Business
• acceptance easily understood by IT
• acceptance criteria => acceptance test (AT)
©2004 Bedarra Research Labs. All rights reserved.
Story Competitive Delta
Analysis Customer Field
Studies & Interviews
Technology Evaluations
Market & Product Analysis Brainstorming
& Visioning
QFD House of Quality
Prototyping Acceptance Criteria
Story Acceptance Criteria
Purpose of Envisioning
Envisioning develops a clear product vision /roadmap to deliver the right product to the
right market using the right technology through consultation with users and
choosers.
1. Determine the technical feasibility § Can we build it?
2. Determine the product differentiation § Can we build it better than the next guy?
3. Explore functional variations § Can we make it useful and usable?
4. Explore form variations § Can we make it useful and usable?
5. Develop a product vision and roadmap § Does everyone on the team have the same shared vision? § How do we partition it so that we can build it reasonably quickly? § How are we realistically going to roll it out?
2013-10-14
15
10% of overall effort
Envisioning
Competitive Delta Analysis
Practices Brainstorming & visioning Competitive analysis (SWOT) Delta analysis QFD Customer studies Hardware, platform & component evals Prototyping/modeling
Deliverables Requirements backlog Risk backlog Analysis & Verification Reports Prototypes/Models Look-and-Feel Guidelines
Requirements Backlog
Risk Backlog
Customer Field Studies & Interviews
Technology Evaluations
Prototypes & Models
GUI Guidelines
Market & Product Analysis Brainstorming
& Visioning
QFD House of Quality
Prototyping Deliverables Acceptance Criteria
Envisioning develops a clear product vision /roadmap to deliver the right product to the right market using the right technology through consultation with users
and choosers.
Low-Fidelity Envisioning Prototype Example Low-fidelity prototype
§ Initially rough and then later refined drawings § Interactive branching allowed walkthrough § User model, task model, task flows § 3 structure and navigation alternatives § 2 visual form alternatives
Concept iterations § 6 iterations (expanding from 8 to 48 screens) § 3 sprints
§ 3 internal / 2 external customer sessions
Detail iterations § 3 iterations (148 screens) § 8 sprints § 3 internal / 1 external customer sessions
Investment § Less than 2% of overall effort
2013-10-14
16
AIL Feature Planning Flow
Development Envisioning Definition
35 stories
25 stories
20 stories
80-100 ids
60-80 days
55-60 ids
5 stories ? ids
4 stories ? ids
5 stories ? ids
20 stories
15 stories
15 stories
? ids
? ids
? ids
15 Key Stories Architecture Diagram GUI Prototype Customer Feedback
80 Clear User Stories 14 Unclear User Stories Feature Breakdown Component Breakdown Dependency Chart Release Backlogs
6 Weeks 2 Weeks
80 User Stories Initially 63 Additional stories after second round of envisioning
45 Weeks
35 stories
25 stories
20 stories
+ 23 later on
+15 later on
+ 25 later on
2 Weeks 2 Weeks
1st Iteration
2nd Iteration
Artefacts
Feature Containing Red, Green & Gold Epics
4 4 5
Story Points Yes but Not End of Estimate Story
© 2012 Bedarra Research Labs. All rights reserved.
Estimates – Do Them Fast and Often
Collectively Owned By Team
§ 2 or 3 point estimates to include uncertainty
§ All items in backlog
§ AIL uses Ideal Days
§ Planning Poker (wide band Delphi)
1 Point
2 point
3 point
Low, Medium, High
Relative Points
Ideal Days
2013-10-14
17
Part* Based Estimates – Managing Risk for Large Projects
Activity Examples Worst Best Expected Likely Java Code (classes, methods)
Interface
Pricing Table (table size)
Process (steps)
Product/Data Definition (Tables/Fields)
Rules (Decision Table (conditions/rules)
Forms (screens, widgets)
Printed Forms (page types, widgets)
* Also called Activity
Part and Story Estimates – Managing Risk for Large Projects
Story Estimates
Part Estimates
Risk Window
2013-10-14
18
Essential Architecture == Interfaces (APIs) The essence of the product and skeleton for evolution
• APIs versioned in the code base
• Should ALWAYS be able to create the architecture pictures from the code
• Allow the architecture to emerge during envisioning through key classes and interfaces
• architect is a role not a job • ability to effectively communicate • competent architects understand the technology/team and business
(NOT post technical) • playing coaches competent smell and read code
• Try Extreme Design
©2004 Bedarra Research Labs. All rights reserved.
Showing Progress Through Architecture Create a working ‘thin slice’ through the architecture
Also called tracer bullets, essential use cases
Allows you to test functionality and show progress to stakeholders
Example: ‘S1, S45, S22 and S18, we can show a simple search…’
Database
Component
Component
Server
Client
Platform Layer Service Layer Application Layer
Database
S1
S22
S45
S18
2013-10-14
19
Feature and Component Team Coordination .
Component C
Component B
Component A
Feature Epic
Component Epics
Component C
Feature A Priority 1
Feature B Priority 2
Feature C Priority 3
Component A Component B
Feature A Work
Feature C Work
Feature B Work
Feature A&B Work
Feature C Work
Feature A Work
Faster, Better, Cheaper
How can we reduce the software cycle time?
©2009 Bedarra Research Labs. All rights reserved.
Needs Solution
Development
2013-10-14
20
Answer - Ship Less Code!
© 2012 Bedarra Research Labs. All rights reserved.
KLOCS (1000 lines of code) Kill! Dependencies Strangle OO Frameworks inject dependencies Data Driven Always More Flexible Than Code
Use Less Objects and Less Code !
Object Refactoring harder than Changing Data and Functions
Many Apps have few if any domain objects!
© 2012 Bedarra Research Labs. All rights reserved.
Table Driven Programming
Rules Decision Table Calculation Spreadsheet Data Validation Domain and Range Table Mapping Lookup Table Flow Data, Work Flow, Message Events, Matches State Table Process, Reports Input-Output Table Acceptance Criteria BDD Domain Models Entity-Attribute Dictionary
Enable Customer Self Service Atom feeds versus APIs Query versus custom Expose Scriptable Services Self Described Data
2013-10-14
21
Table Driven Programming A picture is 1000 words, a table 200 and a diagram 50
Advantages
• Easily understood by Business, BA, Dev and QA
• Easy to create, refactor and extend using Excel
• Modularity through structured tables
• Consistency /Completeness Checking
• Easy to version and Diff
• Efficient Automated Data Driven implementation
• Data Driven means changes can be “hot deployed” to a running application
Applications Insurance, Banking, Taxation, Healthcare, ATC, Real-time...
©2009 Bedarra Research Labs. All rights reserved.
Reduce Integration Time and $$$
• Put ATOM/RSS feeds on our legacy/partner systems – journal files, events …
• REST and JSONify your services
• Use ODBC as a simple interface to complex server systems
• Use a simple MashUp tool to deliver a integrated application view
©2004 Bedarra Research Labs. All rights reserved.
2013-10-14
22
Script to Save Time and $$$ • More and more applications are disposable at least in part –
make it work and get it out there, scale it later if you need to ( e.g Facebook …)
• Use Ruby (IronRuby, JRuby) , Python(Jython, IronPython) , PHP, Groovy, JavaScript, Clojure… to get development productivity
• Leverage cloud services (map reduce, cloud DB)
• Leverage core internal and external services via REST/ODBC
©2004 Bedarra Research Labs. All rights reserved.
We all know that testing costs a lot and takes time, especially when working in a changing complex environment. Lets not bother!
What it takes 1. Modular micro service architecture § instance deployment and tear down § loosely data coupling
2. Simple Functionality Deployment
3. Stringent Monitoring for Deviant Behavior
4. Applications – Telecom, Finance, Web, Sensors …
© 2012 Bedarra Research Labs. All rights reserved.
Code-Deploy-Monitor – Testing Considered Harmful!
2013-10-14
23
Enabling Loose Coupling
• All APIs are value based and where possible stateless • Isolation of services in separate processes/machines
• Simple Pipes and Filters when possible
• RSS/ATOM feeds from events/updates/logs
• Query interface such as ODBC
• Occasionally Disconnected – replication and sync; event source..
• Simple efficient implementations using co-routines..
• Orchestration/Composition using Scripting Process • Service Bus, Messaging.. Node.js, Erlang, Actors
• FP thinking encourages value orientation and composition
©2003 Bedarra Research Labs. All rights reserved.
©2003 Bedarra Research Labs. All rights reserved.
Service Architecture and Design
2013-10-14
24
Improved Testing Value
• Unit Tests << Component Tests << Acceptance Tests
• BDD – Software By Example … Programming By Example
• Look at Randomized Testing aka Quick Check
• Introduce Design Rule Checking • Lint, JSHint, Naked Strings • Type Inference • Pluggable Types
© 2012 Bedarra Research Labs. All rights reserved.
“Testing shows the
presence not absence of
bugs”*
Let the Hardware Do The Work! $49,000 buys a computer 1 TB RAM with 40 TB disk and 32 cores!
$200 buys 1000 4 GB cpus on Amazon for 1 hour!
• Automated Build and Test
• All interesting data is in memory!
• Inexpensive Data Conversion/Translation
• Data Compression and Encryption is “cheaper” on multi-core
• Data Base is an oxymoron
• Speed and Memory enable Simpler Algorithms
• Enable End User Computing
©2009 Bedarra Research Labs. All rights reserved.
2013-10-14
25
Summary
1. Focus on continuous delivery of value
2. Maximize Flow
3. Leadership and Skills Matter
4. Favor targeted high value change over systemic change
5. Build Products not Projects
6. Respect the Individual and Organizational APIs
7. Just Enough Design and Architecture
8. Features and Components both essential
9. Ensure every feature has an associated acceptance criteria
10. Acceptance Tests >> API Test >> Unit Test
11. Automate everything
12. Use the right tool/practice for the right job
© 2012 Bedarra Research Labs. All rights reserved.
Successful Software Development is about a Winning Culture
Software is a team sport, and like all team sports practice, constructive peer feedback, and coaching are essential.
Winning teams need to implicitly know the moves of each player, as well as the movements of the team as whole.
The ultimate expression of process is a culture where building software is more like playing jazz. People Just Do It!