Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Development model Improvements Process Concurrency Challenges Summary
Levers for changing software deliveryv1.2.0
Michiel Kalkman
Development model Improvements Process Concurrency Challenges Summary
System development scaling
Figure 1: Solo efforts can redefine space asthey go along
Figure 2: Team efforts need a sharedunderstanding of space
Image sources: wikimedia.org a, b
Development model Improvements Process Concurrency Challenges Summary
Our reference phases
Concept Inception Construction Release Production Disposal
Feedback
IdeationVisionFeasibility
DesignFundingResourcing
AnalysisDevelopmentTesting
TestingAcceptanceDeployment
OperationsSupportAssessment
Data conversionUser migrationRetirement
Figure 3: Reference Lifecycle - Typical fixed-resource project
Development model Improvements Process Concurrency Challenges Summary
General Observations
Phases are ordered sequentiallyWith feedback loops to register defects, learnings
Phases have distinct functionsTransitions between phases require a context switch or handover
Work in earlier phases affects later phasesLater phases affect earlier phases via feedback
Development model Improvements Process Concurrency Challenges Summary
General Goals
We want to,
Minimise feedback volume, rate and processing timeMinimise the cost of phase transitionsMinimise the volume of non-feature work passing through the cycleMaximise the speed of work passing through the cycleMaximise the rate of work passing through the cycle
Development model Improvements Process Concurrency Challenges Summary
Project Phases
Delivery
Construction Release Production
Feedback
Figure 4: The DevOps Cycle
Development model Improvements Process Concurrency Challenges Summary
Identified issue example
ScenarioAnalytics show us that an increasing number of customer journeys end inFAILED or ABORTED states. Root cause analysis has determined that themajority of these cases are caused by valid, but less common, sequences ofinteractions that are not handled correctly, are missing code to handle themor are using obsoleted code that cannot execute.
ActionRaise test coverage requirements at unit and integration levelBlock a release unless it meets the coverage requirements
Development model Improvements Process Concurrency Challenges Summary
Add Quality Gates
Delivery
Construction ReleaseQuality Gate ProductionQuality Gate
Feedback
Figure 5: Add quality gates
BenefitsLower feedback volume and rate
CostRaise the cost of phase transitions
Development model Improvements Process Concurrency Challenges Summary
Quality Gate examples
Unit test coverageLoad or soak tests of Release before ProductionAutomated static or dynamic analysis of code policies. Example,
Security patternsMissing codeCyclomatic complexity
Development model Improvements Process Concurrency Challenges Summary
Identified issue exampleScenario
Our support team is unable to keep up with the backlog of issues, whichkeeps growing along with the mean time to issue resolution. Analysis thatmost of the time spent by the support team goes to collecting event logsfrom different systems based on timestamp information and other attributesto reconstruct transactions and individually engaging engineers to providecontext.
ActionAll log events will be accessible from a single systemAll log events will be structuredAll systems will implement correlation identifier generation and propagation forservice transactionsCorrelation identifiers will be added to log events
Development model Improvements Process Concurrency Challenges Summary
Increase Defect Traceability
Delivery
Construction Release Production
Feedback
Traceability TraceabilityTraceability
Figure 6: Tracing
BenefitsLower feedback processing time (support and incident handling)Faster remediation (lower lead time)
Development model Improvements Process Concurrency Challenges Summary
Traceability examples
Centralised logging with propagated correlation IDsDistributed tracing implementationUser activity measurement connected to backend tracingSelf-service dashboarding and alert managementChange visibilityBusiness metric generation and reporting
Development model Improvements Process Concurrency Challenges Summary
Option : Improve Transition Throughput
Delivery
Construction Release Production
Feedback
Figure 7: Concurrent streams of work
CostIncreases phase transition cost (handover, interface, gates)
BenefitsIncrease the volume of work
Development model Improvements Process Concurrency Challenges Summary
Option : Improve Phase Output
Delivery
Construction Release Production
Feedback
Figure 8: Higher quality output
BenefitsLower feedback volumeLower the volume of non-feature work
Development model Improvements Process Concurrency Challenges Summary
Combined model
Delivery
Construction ReleaseQuality Gate ProductionQuality Gate
Feedback
Traceability TraceabilityTraceability
Figure 9: Combined improvements
Development model Improvements Process Concurrency Challenges Summary
Cost of defects
1
10
100
Preven�on
Correc�on
Failure
Figure 10: 1-10-100 Rule
Development model Improvements Process Concurrency Challenges Summary
Example : Cost scenario of feature rollout
Outcome Value CostSuccess Positive Planning time
+ Execution time
Rollback Zero Cost of success+ Recovery time+ Opportunity cost
Failure Negative Cost of rollback+ Emergency time+ Lost function cost+ Reputation cost
Development model Improvements Process Concurrency Challenges Summary
Early detection
Defects found early are less costly to remediate
If we want to lower the cost of defect remediation, there are two key drivers in thisobservation,
1 Early detection of defects2 Fast remediation of detected defects
Development model Improvements Process Concurrency Challenges Summary
Right Shifted
Design Implemention Test
Figure 11: Testing happens last
Development model Improvements Process Concurrency Challenges Summary
Right Shifted - Step details
Feature Test
Design Implemention Design Implementation Execution
Figure 12: Examination of testing steps - these also contain design and implementation
Development model Improvements Process Concurrency Challenges Summary
Right Shifted - Step contexts
Feature Test
Design Implemention Design Implementation Execution
Business contextVariable time
Technical contextVariable time
Fixed timeCan be automated
Business contextVariable time
Technical contextVariable time
Figure 13: Steps with context
Development model Improvements Process Concurrency Challenges Summary
Context mapping
Feature Test
Design Implemention Design Implementation Execution
Business Context Technical Context
Figure 14: Right Shifted - late duplication of contexts
Development model Improvements Process Concurrency Challenges Summary
Flow independent use cases
Business Context Technical ContextValidation Context
Design FeatureDesign QA Criteria Implement FeatureImplement TestsExecution
Tester DeveloperAnalyst
Figure 15: Feature development actors, cases, contexts
Development model Improvements Process Concurrency Challenges Summary
Ideal development
Construction Phase
Quality Gate
Business Context Technical Context
Feature Design Feature Implementation
Test Design Test Implementation Execution
Figure 16: Left Shifted - early alignment and no late duplication
Development model Improvements Process Concurrency Challenges Summary
Minimise external dependencies
Feature team
Load testing team
Release team
Infra team
Monitoring team
Security team
Feature team
Feature team
Feature team
Figure 17: One team per non-featurecapability
DevDev
Tester
SecurityPlatform
DevDev
Tester
SecurityPlatform
DevDev
Tester
SecurityPlatform
Figure 18: Capabilities embedded in teams
Development model Improvements Process Concurrency Challenges Summary
Goals for concurrency
Reduce idle time, maximise productive timeReduce delivery risk
Break one large project into small featuresFail fast - detect problems early
Avoid single points of blockage/stallingConcurrent, isolated streams of work
Capture value early with fast exposure of features to production
Development model Improvements Process Concurrency Challenges Summary
Independent Feature Development
Work Stream 1 - Feature A
Work Stream 2 - Feature B
Itera�on 1
Figure 19: Testing V-Model
Development model Improvements Process Concurrency Challenges Summary
Independent Feature Development
Requirements,
Up-front planningCoordination activitiesContinuous monitoring and adjustmentDedicated investment in process management
Development model Improvements Process Concurrency Challenges Summary
Multiple Development Streams
Test execu�on
Test defini�on
Defect raised
Feature delivered
Stre
ams
of w
ork
Time
Figure 20: Parallelised, Left-Shifted Delivery
Development model Improvements Process Concurrency Challenges Summary
Technical Debt
"Ship now,deal with it later"
"What is layering?"
"Now we know howwe should have done it"
"We don't have �mefor design"
Figure 21: Martin Fowler’s quadrants of tech debt acquisition
Development model Improvements Process Concurrency Challenges Summary
Technical Debt Types
1 Eventually paid off2 Sustainable3 Compounding
Development model Improvements Process Concurrency Challenges Summary
Technical Debt
CausesTime-to-market drive - Delivery pressure to meet value pointTeam ramp up, learning curve - Immature engineering practicesProgressive learnings - Mismatch between technology and applicationProduct repositioning - Mismatch between platform and application
EffectsOpportunity costCompound interest - debt load increases over timeDevelopment overhead - Any change is more complex than necessaryTesting overhead - Complexity doesn’t increase linearlyStill has to be maintained - Any maintenance is more complex than necessary
Industry average 15 to 50 errors per 1000 delivered LoC(Code Complete, Microsoft Press)
Development model Improvements Process Concurrency Challenges Summary
Technical Debt
Key PointsTech debt is taken on for a reason - usually related to creating valueTech debt itself is not good, but having it can be a good signalTech debt itself is not costly, the cost is in carrying the debt
ActionsProvide a decision making framework for taking on debt
And paying it off (or not)Make debt visibleTrack the cost of carrying debt
Development model Improvements Process Concurrency Challenges Summary
Pressure of re-work and technical debt
Figure 22: Cumulative flow of re-work andtechnical debt pressures
Re-work is lost opportunityInterest on technical debtcompoundsTrack both new debt and its interestpressureTrack re-work and cycle times
Development model Improvements Process Concurrency Challenges Summary
Pre-project feedback
Pre-Project Project
Post-Project
Concept Inception
Feedback
Construction Release Production
Feedback
Disposal
Figure 23: Concept feedback
No well-defined feedback mechanismsNo clear outcomesPoliticsMove more decisions to Construction phase pre-project
Development model Improvements Process Concurrency Challenges Summary
Non-feature budget expenditure
Process changes require significant expenditure of resources. Find secondary benefitsthat address pain points or have a financial impact to lower the cost.
Improving Traceability for Defects can also be used to lower to cost of SupportAutomation and parallelisation can be used to scale DevelopmentIf a form of compliance is on the roadmap, implementing automation for AccessControl, Change Management and Control Monitoring will reduce that cost
Development model Improvements Process Concurrency Challenges Summary
ROI of Manual vs Automated Testing
Figure 24: By stage
There is always an upfrontinvestment with automationUnderstand where the crossover withmanual testing isMake this visible to stakeholdersKeep track of and report on thevalue of automation as the projectprogresses
Development model Improvements Process Concurrency Challenges Summary
Value of work by stage
Figure 25: By stage
Get completed features toproduction as soon as possible, noexcusesUse feature flags to gate activationif necessary
Development model Improvements Process Concurrency Challenges Summary
Shift Left = Process change + automation
What HowDefine test requirements early Engage test team at each step
Make testing everyone’s responsibilityExecute tests as soon as possible Gate progress at each step on testingRapidly act on test defects Implement consistent traceability, reportingMinimise test execution barriers Automate tests, accessible to allMinimise test execution time Automate tests, make test runs cheapHave test execution consistency Automate tests, make test runs predictable
Development model Improvements Process Concurrency Challenges Summary
Summary
Figure 26: Effect of changes across concerns
Development model Improvements Process Concurrency Challenges Summary
Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)
You are free to:
Share — copy and redistribute the material in any medium or formatAdapt — remix, transform, and build upon the material
The licensor cannot revoke these freedoms as long as you follow the license terms.
Under the following terms:
Attribution — You must give appropriate credit, provide a link to the license, andindicate if changes were made. You may do so in any reasonable manner, but notin any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes.
https://creativecommons.org/licenses/by-nc/4.0/