Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Business Value ofAgile Methods
Benefits of Testing Early & OftenDr. David F. Rico, PMP, CSEP, ACP, CSM, SAFe
Twitter: @dr_david_f_ricoWebsite: http://www.davidfrico.com
LinkedIn: http://www.linkedin.com/in/davidfricoAgile Capabilities: http://davidfrico.com/rico-capability-agile.pdf
Agile Resources: http://www.davidfrico.com/daves-agile-resources.htmAgile Cheat Sheet: http://davidfrico.com/key-agile-theories-ideas-and-principles.pdf
Author Background Gov’t contractor with 32+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe
2
Career systems & software engineering methodologist Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000NASA, USAF, Navy, Army, DISA, & DARPA projects Published seven books & numerous journal articles Intn’l keynote speaker, 130 talks to 12,000+ people Specializes in metrics, models, & cost engineeringCloud Computing, SOA, Web Services, FOSS, etc. Adjunct at five Washington, DC-area universities
What is Agility? A-gil-i-ty (ә-'ji-lә-tē) Property consisting of quickness,
lightness, and ease of movement; To be very nimble The ability to create and respond to change in order to
profit in a turbulent global business environment The ability to quickly reprioritize use of resources when
requirements, technology, and knowledge shift A very fast response to sudden market changes and
emerging threats by intensive customer interaction Use of evolutionary, incremental, and iterative delivery
to converge on an optimal customer solution Maximizing BUSINESS VALUE with right sized, just-
enough, and just-in-time processes and documentationHighsmith, J. A. (2002). Agile software development ecosystems. Boston, MA: Addison-Wesley.
3
What are Agile Methods?
4
People-centric way to create innovative solutions Product-centric alternative to documents/process Market-centric model to maximize business value
Agile Manifesto. (2001). Manifesto for agile software development. Retrieved September 3, 2008, from http://www.agilemanifesto.orgRico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.Rico, D. F. (2012). Agile conceptual model. Retrieved February 6, 2012, from http://davidfrico.com/agile-concept-model-1.pdf
Customer Collaboration
Working Systems & Software
Individuals & Interactions
Responding to Change
valuedmore than
valuedmore than
valuedmore than
valuedmore than
Contracts
Documentation
Processes
Project Plans
Frequent comm. Close proximity Regular meetings
Multiple comm. channels Frequent feedback Relationship strength
Leadership Boundaries Empowerment
Competence Structure Manageability/Motivation
Clear objectives Small/feasible scope Acceptance criteria
Timeboxed iterations Valid operational results Regular cadence/intervals
Org. flexibility Mgt. flexibility Process flexibility
System flexibility Technology flexibility Infrastructure flexibility
Contract compliance Contract deliverables Contract change orders
Lifecycle compliance Process Maturity Level Regulatory compliance
Document deliveries Document comments Document compliance
Cost Compliance Scope Compliance Schedule Compliance
Courage
NetworkComputer
Operating SystemMiddlewareApplications
APIsGUI
How Agile Works Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success
5Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.
Agile Traditional1 2 3 Faster
Early ROI
Lower Costs
Fewer Defects
Manageable Risk
Better Performance
Smaller Attack Surface
Late
No Value
Cost Overruns
Very Poor Quality
Uncontrollable Risk
Slowest Performance
More Security Incidents Seven Wastes1. Rework2. Motion3. Waiting4. Inventory5. Transportation6. Overprocessing7. Overproduction
MINIMIZES MAXIMIZES
JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes
Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success
What is Agile Testing? Traditional testing is a late, manual process Agile testing is an early and automated process Goal to deliver early & often and V&V components
6Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txtCrispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
AGILE TESTING- Early Incremental Testing -
TRADITIONAL TESTING- Late Big Bang Integration Testing -
Test Criteria Accompany StoriesAutomated Tests Written FirstUnits Coded-Tested One at TimeCode is Frequently Checked InCode Automatically RetrievedCode Automatically CompiledTests Automatically Executed Instant Feedback & Test Reports
Test Criteria Written After FactManual Tests Written Much LaterUnits Coded Late All at One TimeCode Checked In Late in ProjectCode Manually Submitted to TestCode Manually Compiled & BuiltTests Manually Executed LateLate Project Feedback & Reports
Code Automatically DeployedLate Defects Freeze Projects
BASIC—Test Driven Development Term coined by Kent Beck in 2003 Consists of writing all tests before design Ensures all components are verified and validated
7Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
ADVANCED—Continuous Integration Term coined by Martin Fowler in 1998 Process of automated build/regression testing Evaluates impact of changes against entire system
8Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
ALL DEVELOPERS RUN PRIVATE BUILDS
DEVELOPERS COMMIT CODE TO VERSION CONTROL
INTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY
100% OF SYSTEM TESTS MUST PASS FOR EVERY BUILD
A SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD
FIXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY
REPORTS AUTOMATICALLY GENERATED & REVIEWED
Thousands of TestsContinuously Executed
No More Late BigBang Integration
User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur
9Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
BuildIntegration
Server
VersionControlServer
BuildScripts
UsesWatches
BuildStatus
ProvidesDeveloper A
Developer B
Developer C
CommitsChanges
CommitsChanges
CommitsChanges
Builds
Database
Analysis
Testing
Reporting
Documentation
Deployment
Early, Automated, Fast,Efficient, & Repeatable
Constant ReadinessState & CM Control
Lean, Waste Free, Low WIP,No Deadlocked Test Queues
Rapidly & SuccessfullyDev. Complex Systems
Agile Testing Process“Continuous Integration”
Agile Testing Practices Agile testing consists of seven broad practices Automated build, database, inspection, tests, etc. Include reporting, documentation, deployment, etc.
10
Practice
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Description
Frequently assembling products and services to ensure delivery readiness
Frequently generating/analyzing database schemas, queries, and forms
Frequently performing automated static analysis of product/service quality
Frequently performing automated dynamic product and service evaluation
Frequently generating automated status reports/messages for all stakeholders
Frequently performing automated technical/customer document generation
Frequently performing automated delivery of products/services to end users
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
Agile methods are based on traditional measures Story points, velocity, and burndown basic metrics Experts use Agile EVM, test, ROI & portfolio metrics
11Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.
AGILE METRICS1. Agile CODE Metrics2. Agile PROJECT Metrics3. Agile TRACKING Metrics4. Agile TESTING Metrics5. Agile VALUE Metrics6. Agile HEALTH Metrics7. Agile PORTFOLIO Metrics
1. Agile CODE Metrics Code Size Code Complexity Object Oriented Code Coverage Code Defects Relational Design
2. Agile PROJECT Metrics Software Size Software Productivity Software Effort Software Quality Software Schedule Software Success
3. Agile TRACKING Metrics Story Points Sprint Burndown Release Burndown Velocity Feature Progress Agile Earned Value
4. Agile TESTING Metrics Test Coverage Test Automation Integration Builds Running Tested Features DevOps Automation Deployment Frequency
7. Agile PORTFOLIO Metrics Portfolio Kanban Epic Progress Portfolio Radar Release Train Radar Lean Portfolio Metrics Enterprise Scorecard
6. Agile HEALTH Metrics Teamwork Quality Collaboration Quality Agile Process Maturity Agile Adoption Rate Degree of Agility Product Flexibility
5. Agile VALUE Metrics Total Lifecycle Costs Total Lifecycle Benefits Benefit to Cost Ratio Return on Investment Net Present Value Real Options Analysis
Agile Metrics Taxonomy
12
METRIC DESCRIPTION
TEST COVERAGE Percent or degree to which software source code is tested
TEST AUTOMATION Ratio or degree to which software tests are automated
INTEGRATION BUILDS Frequency of automated software builds and integrations
RUNNING TESTED FEATURES Number of completed and tested features or user stories
DEVOPS AUTOMATION Ratio or degree to which deployments are automated
DEPLOYMENT FREQUENCY Frequency of automated software deployments or deliveries
Software test automation emerged during the 1970s Reached their height in personal computer (PC) era Most are FOSS and used by successful agile teams
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Agile Testing Metrics—Definitions
Agile Testing Metrics—Example
13Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Agile Testing Workflow
14
Traditional vs. Agile Cumulative Flow
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Traditional Cumulative Flow Agile Cumulative Flow
Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP & lead times
Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
Agile Testing Costs & Benefits
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.
Most agile testing tools are “free” open source Build server costs no more than a commodity PC 10x more efficient/effective than traditional testing
15
Agile Testing Economics Traditional testing finds a defect in about 10 hours Manual code inspections find a defect in 1 hour Agile testing finds a defect every 6 minutes
16Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
Agile Cost of Quality (CoQ) Agile testing is 10x better than code inspections Agile testing is 100x better than traditional testing Agile testing is done earlier “and” 1,000x more often
17Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
Continuous Integration Fewer integrations leave in higher bug counts Frequent, early integrations eliminate most defects Goal is to have as many early integrations as possible
18Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.
Number ofIntegrations
Less Defects•More Integrations•Early IntegrationsMore Defects
•Few Integrations•Late Integrations
Agile Testing Anti-Patterns Agile teams don’t often use TDD, CI, CD & DevOps Implement independent test teams after Sprints done Sprint Waterfalling, Scrummerfalling, & Wagile result
19Heusser, M. (2015). 12 years of agile testing: What do we know now. Proceedings of the Agile Gathering, Grand Rapids, Michigan, USA.
Incorrect• Phased Testing• Separate Teams• Delayed Testing
Correct• Integrated Testing• Integrated Teams• Continuous Testing
Scaling Agile Testing Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with more attention
20Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
MICRO ADJUSTMENTS- Focused Impact Tuning-
MACRO ADJUSTMENTS- Wide Impact Tuning-
Add More CPUs & MemoryParallelize System BuildsReplace 3rd Party Test LibrariesReduce or Remove Test TimeoutsSelect Different TestsRefactor Code & ComponentsTune Network & SoftwareTune Database & Middleware
In-Memory CompilationParallelize Test RunsPre-Install Test LibrariesRemove Process RandomnessUse Faster Code & Test Tools Incremental vs. Big Bang TestsParallelize Build & InstallTune & Optimize Build Process
Enterprise Continuous Delivery Created by Jez Humble of ThoughtWorks in 2011 Includes CM, build, testing, integration, release, etc. Goal is one-touch automation of deployment pipeline
21Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.
CoQ
• 80% MS Tst• 8/10 No Val• $24B in 90s• Rep by CD• Not Add MLK
22
Hewlett-Packard is a major user of CI, CD, & DevOps 400 engineers developed 10 million LOC in 4 years Major gains in testing, deployment, & innovation
Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.
TYPE METRIC MANUAL DEVOPS MAJOR GAINS
CYCLE TIME
IMPROVEMENTS
Build Time 40 Hours 3 Hours 13 x
No. Builds 1-2 per Day 10-15 per Day 8 x
Feedback 1 per Day 100 per Day 100 x
Regression Testing 240 Hours 24 Hours 10 x
DEVELOPMENT
COST EFFORT
DISTRIBUTION
Integration 10% 2% 5 x
Planning 20% 5% 4 x
Porting 25% 15% 2 x
Support 25% 5% 5 x
Testing 15% 5% 3 x
Innovation 5% 40% 8 x
Cont. Delivery (Hewlett-Packard)
Assembla went from 2 to 45 releases every month 15K Google developers run 120 million tests per day 30K+ Amazon developers deliver 8,600 releases a day
23Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.
62x FasterU.S. DoD
IT Project
3,645x FasterU.S. DoD
IT Project
Continuous Delivery (with DEVOPS)
Enables enterprises to be flexible but disciplined Allows enterprises to distribute project work teams Ensures distributed project teams are collaborating
24
Agile Tools“Across the Life Cycle”
Project Management
RequirementsDOORSRequisite ProSLATE
DesignRhapsodyTelelogic System ArchitectRational System Architect
CodingEclipseVisual StudioSun Studio
TestingJUnitNUnitXunitCPPUnit
GtestFitFitnesseSelenium
Quality AssuranceCheckStylePMDEMMAJdependCoberturaGcov
Configuration MgtSubversion (SVN)Concurrent Versions Sys.ClearCase
Build AutomationAntNAntMavenMake
Continuous Integ.Cruise ControlHudsonBuildBot
CollaborationWebExSkypeMeetMeWimba
WikiMediaWikiTracWikiPhpWiki
DocumentationNDocJavadocDoxygeniText
Version OneRallyScrum WorksVSTS
Agile TeamAgile EnterpriseScope ManagerStory Studio
XP Plan ItIterateXP TrackerAgilo
XP CGIXP WebXplannerIce Scrum
Project CardsTarget ProcessXtreme PlannerTeam System
CommunityEnterpriseMingleHansoft
There are literally hundreds of agile testing tools There are tools for building, testing, and deployment Integration tools monitor repositories and initiate tests
25
Agile Tools“In-Depth Test Automation”
Smart, J. (2009). Automated deployment with maven and friends: Going the whole nine yards. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
Agile Testing Usage Statistics Agile test use is low in spite of its age, i.e., 15 years Many do not understand its utter simplicity and power Failure to use agile testing undermines project success
26Kim, D. (2013). The state of scrum: Benchmarks and guidelines. Indianapolis, IN: Scrum Alliance.
Agile PracticesRetrospectives
Refactoring
Done Definition
Test Tools
Test Driven Dev.
CM Tools
Simplicity
Pair Programming
Technical Debt
Agile Testing 13%
Continuous Integrations
Weekly
Daily
2-3 TimesPer Day
Never
2-3Times
PerIteration
Common Barriers to Agile Testing Industry very slow in adopting agile testing model Cost, difficulty, and territorialism are common issues Developers must take initiative for disciplined testing
27
Technical BarriersOrganizational BarriersDevelopers don’t want to test
· Infrequently committing code· Committing broken code· Failing to immediately fix builds· Not writing automated tests· Not ensuring 100% of tests pass· Not running private builds· Resorting to traditional testing
Resistance to change· Fear of investment costs· Fear of learning new skills· Test group territorialism· Organizational policy conflicts· Overhead of maintaining CI· Complexity and scaling· Not developing a quality culture
··
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project
28Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.
What’s the Bottom Line?“Agile Testing Done Early & Often”
Agile TestingTraditional TestingDramatically reduces risks
· Automates manual processes· Instant verification & validation· High project visibility· Greater confidence and morale· Incremental business value· 24x7 deployability to users· Highly quality and reliability
Late defect discovery· Low quality software· Poor project visibility· Lack of deployability· Late big-bang integration· Testing is a bottleneck· Poor customer satisfaction· Outright project failure
··
Dave’s Professional Capabilities
29
SoftwareQuality
Mgt.
TechnicalProject
Mgt.
SoftwareDevelopment
Methods
OrganizationChange
SystemsEngineering
CostEstimating
GovernmentContracting
GovernmentAcquisitions
LeanKanban
Big Data,Cloud, NoSQL
WorkflowAutomation
Metrics,Models, & SPC
SixSigma
BPR, IDEF0,& DoDAF
DoD 5000,TRA, & SRA
PSP, TSP, &Code Reviews
CMMI &ISO 9001
InnovationManagement
Statistics, CFA,EFA, & SEM
ResearchMethods
EvolutionaryDesign
Valuation — Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.
Lean-Agile — Scrum, SAFe, Continuous Integration & Delivery, DevOps, etc.
STRENGTHS – Data Mining Gathering & Reporting Performance Data Strategic Planning Executive & Manage-ment Briefs Brownbags & Webinars White Papers Tiger-Teams Short-Fuse Tasking Audits & Reviews Etc.
● Action-oriented. Do first (talk about it later).● Data-mining/analysis. Collect facts (then report findings).● Simplification. Communicating complex ideas (in simple terms).● Git-r-done. Prefer short, high-priority tasks (vs. long bureaucratic projects).● Team player. Consensus-oriented collaboration (vs. top-down autocratic control).
PMP, CSEP,ACP, CSM,
& SAFE
32 YEARSIN IT
INDUSTRY
Books on ROI of SW Methods Guides to software methods for business leaders Communicates the business value of IT approaches Rosetta stones to unlocking ROI of software methods
http://davidfrico.com/agile-book.htm (Description) http://davidfrico.com/roi-book.htm (Description)
30