Upload
jose-luis-soria
View
605
Download
0
Embed Size (px)
DESCRIPTION
Slides from my presentation about release pipelines at Codemotion (Madrid - November 2014)
Citation preview
MADRID · NOV 21-22 · 2014
Designing a Release Pipeline
Jose Luis Soria
Continuous Improvement Manager at Ria Financial
@jlsoriat
MADRID · NOV 21-22 · 2014
What is a
Release Pipeline?
MADRID · NOV 21-22 · 2014
What is a Release Pipeline?
Automated manifestation of
your delivery process.
Feedback mechanism.
Detection of unfit release
candidates.
Pull system.
Useful for CD, or any other
delivery model.
MADRID · NOV 21-22 · 2014
Pipeline design considerations
Emergent design. No BDUF.
Start early.
Start simple and evolve with the system.
Begin with the most valuable assets.
Address the bottlenecks.
MADRID · NOV 21-22 · 2014
#1 Define
Components
MADRID · NOV 21-22 · 2014
MADRID · NOV 21-22 · 2014
What is a (software) component? (*)
A set of artifacts (binaries, dynamic code,
configuration files, other supporting files) that
can be deployed and verified together without
affecting other areas of the application.
(*) in a release pipeline
MADRID · NOV 21-22 · 2014
Choosing components
Deploy and test the smallest independent
entity.
Rely on the architecture:Logical / physical.
Layers / tenants.
See the whole.
MADRID · NOV 21-22 · 2014
What do we need to know?
For each component:Meaningful name.
Description.
Priority / order (when to address it?)
Source (most likely, version control.)
Target (where it gets deployed.)
Pre-requisites.
Dependencies.
Configuration tokens.
MADRID · NOV 21-22 · 2014
Configuration tokens
1. Make a list of environment-dependent
information.
2. Tokenize it in the configuration.
3. Gather the values for all the environments.
MADRID · NOV 21-22 · 2014
Component sheet
MADRID · NOV 21-22 · 2014
Activity: defining components
Our application consists on:
A web site built on top this technology stack:
MVC framework.
Client-side logic (HTML5, JavaScript.)
Entity model mapped to the DB using a ORM.
Data model residing in a DB.
MADRID · NOV 21-22 · 2014
Activity: defining components
Our application consists on:
A web site built on top this technology stack:
MVC framework.
Client-side logic (HTML5, JavaScript.)
Entity model mapped to the DB using a ORM.
Data model residing in a DB.
Key business logic resides on a web services layer.
We also maintain a mobile client for two platforms.
MADRID · NOV 21-22 · 2014
Activity: defining components
Our application consists on:A web site built on top this technology stack:
MVC framework.
Client-side logic (HTML5, JavaScript.)
Entity model mapped to the DB using a ORM.
Data model residing in a DB.
Key business logic resides on a web services layer.
We also maintain a mobile client for two platforms.
For some operations we make calls to third-party web
services.
MADRID · NOV 21-22 · 2014
Example: component sheet
MADRID · NOV 21-22 · 2014
Implementation notes
It should be possible to independently build, deploy and
test anything defined as a component.
You should decide how dependencies will be made
available:
Source Control.
Artifact repositories.
Deployed (third-party) artifacts.
Etc.
MADRID · NOV 21-22 · 2014
Tooling
Version control systemsGit, Mercurial, SVN, TFS…
Artifact repositories
Artifactory, Nexus, Frog, NuGet…
MADRID · NOV 21-22 · 2014
#2 Identify
Sub-pipelines
MADRID · NOV 21-22 · 2014
Single pipeline
A single pipeline servicing all the components
and teams.
May be able to detect which component has
changed and operate only on that one.
MADRID · NOV 21-22 · 2014
One pipeline per component
Each component has its own pipeline.
Different pipelines may have different designs.
Individual pipelines may fan-in to a system pipeline.
More flexible but more complex.
MADRID · NOV 21-22 · 2014
One pipeline per team
Each team has its own pipeline.
Different pipelines may have different designs.
Individual pipelines may fan-in to an integration pipeline.
MADRID · NOV 21-22 · 2014
Mixed approach
Different teams building different components.
Keep it simple!
MADRID · NOV 21-22 · 2014
Implementation notes
It is easier if you use a tool that allows to define
sub-pipelines, fan-in, fan-out, etc.
MADRID · NOV 21-22 · 2014
#3 Define
Stages & Orchestration
MADRID · NOV 21-22 · 2014
MADRID · NOV 21-22 · 2014
What is a stage?
A set of steps or activities that are performed on a
release candidate.
It lets any release candidate advance towards
production, or discards it.
When a release candidate passes through a stage,
our confidence on it is increased.
It is a source for feedback.
Frequently taken for environments (but they’re not
the same)
MADRID · NOV 21-22 · 2014
What is orchestration?
It is the way we arrange the stages so release
candidates flow through them, in their way to
production.
MADRID · NOV 21-22 · 2014
Tips for stages & orchestration (I)
Feedback is the key. Arrange stages and
orchestration based on the feedback we need.
Stages are filters. The orchestration should be
arranged to stop the pipeline if a stage fails.
Stages can contain both manual and
automated steps.
MADRID · NOV 21-22 · 2014
Tips for stages & orchestration (II)
Stages can be manually or automatically
triggered (think about approvals.)
Automate as much as possible. Including the
approvals.
MADRID · NOV 21-22 · 2014
Tips for stages & orchestration (III)
Grow your pipeline wide, not long
http://bit.ly/1jsNGP5
Build only once.
Use environment-agnostic binaries.
Version everything.
MADRID · NOV 21-22 · 2014
What do we need to know?
For each stage:∘ Meaningful name.
∘ Clear goal.
∘ Does it need a manual approval to be triggered?
∘ Does it need a manual verification when it has finished?
∘ Sources.
∘ Flow (orchestration.)
MADRID · NOV 21-22 · 2014
Pipeline-level orchestration
(examples)
Commit Manual testing Release
Minimum pipeline
Fully automated
Partially automated, or manual
Legend:
MADRID · NOV 21-22 · 2014
Pipeline-level orchestration
(examples)
Commit Acceptance testing Release
Exploratory testing
Capacity testing Security testingUser Acceptance
testing
Complex pipelineFully automated
Partially automated, or manual
Legend:
MADRID · NOV 21-22 · 2014
Stage-level
orchestration
(example) Code is builtOnce and Only Once:
In the first stage.
Subsequent stagesare run in parallel
if configured that way.
Pipeline gets triggered:When a developer does a check-in,
or manually
A new instance of the pipeline is
created
Get next stage, relate it to the
pipeline instance, prepare
parameters, notify for monitoring
Automatically-triggered stage?
Trigger stageWait for the user to
trigger the stage
Gather stage results and notify them for
monitoring
Succeded?
Stop the pipeline instance
Any stages left?
NOYES
NO
NO
YES
YES
MADRID · NOV 21-22 · 2014
About sources
Version control.
Artifact repositories.
Environment libraries.
MADRID · NOV 21-22 · 2014
Which stages do I need?
Think about the kind of feedback you need.
Think about what should stop a release
candidate to get to production.
Create a Value Stream Map.
MADRID · NOV 21-22 · 2014
Value stream map (example)
Assessment Approval Planning
Capacity tests
ReleaseAcceptance
tests
CodeSpecification
2 days 1 dayValue-added time
Waittime
Development: cycle time ~ ?
3 days 3 days ?
? ?
?
4 days 1 day 2 days 2 weeks ? ?
Delivery: lead time ~ ?
Exploratory tests
?
UAT
?
?
MADRID · NOV 21-22 · 2014
Prevalent stages: the Commit stage
Eliminate early release candidates that are unfit for
production.
Close to (or the same as) a CI build.
Quick validations: build, unit testing, static analysis, etc.
Packaging.
For Continuous Delivery, it runs on each commit (no
branches – feature toggles.)
For other models, decide when it gets triggered (for
example, on each merge to trunk.)
MADRID · NOV 21-22 · 2014
Prevalent stages: the Commit stage
http://bit.ly/1jsSkwA
MADRID · NOV 21-22 · 2014
Prevalent stages: the Automated
Acceptance Test stage
http://bit.ly/1jsSkwA
MADRID · NOV 21-22 · 2014
Prevalent stages: the Manual
Test stage
http://bit.ly/1jsSkwA
MADRID · NOV 21-22 · 2014
Prevalent stages:
non-Functional Testing stages
http://bit.ly/1jsSkwA
MADRID · NOV 21-22 · 2014
Activity: defining stages &
orchestration
“We have a basic suite of automated
acceptance tests that we plan to grow along
with the system.”
“The team does (manual) functional testing.”
MADRID · NOV 21-22 · 2014
Activity: defining stages &
orchestration
“We have a basic suite of automated
acceptance tests that we plan to grow along
with the system.”
“The team does (manual) functional testing.”
“We need to support 2,000 concurrent users.”
MADRID · NOV 21-22 · 2014
Implementation notes
Choose a tool that allows to easily model and
visualize the flow.
Choose a tool that supports what you need for
orchestration:∘ Approvals.
∘ Validations.
∘ Parallelization.
∘ Alerts.
∘ Etc.
MADRID · NOV 21-22 · 2014
#4 Define
Environments
MADRID · NOV 21-22 · 2014
What is an environment?
A set of servers, devices or any other
resources we need in order to run and validate
a release candidate in its way to production.
MADRID · NOV 21-22 · 2014
Tips for defining environments
Prepare for deployment automation.
Lock down environments. Restrict access.
Different stages could target the same environment if
needed.
Prepare for auto-provision.
Make environments disposable. Don’t turn them into
bottlenecks.
Environments may not be tied to stages. It should be easy
to point any stage to any environment.
MADRID · NOV 21-22 · 2014
Activity: defining environments
“We have a basic suite of automated
acceptance tests that we plan to grow along
with the system.”
“The team does (manual) functional testing.”
“We need to support 2,000 concurrent users.”
MADRID · NOV 21-22 · 2014
Implementation notes
Use virtualization.
Use cloud-based environments.
Use tools for managing templates,
configuration, auto-provision, etc.
MADRID · NOV 21-22 · 2014
Tooling
Phisical machines
Virtualization tools
VmWare, VirtualBox, Hyper-V, etc.
Containers
Docker
Cloud
Amazon, Google, Azure…
Environment definition
Vagrant, PowerShell DSC…
MADRID · NOV 21-22 · 2014#5 Define Steps
MADRID · NOV 21-22 · 2014
What is a step?
Any activity that is done in the context of a
stage, that allows us to get feedback and prove
the fitness of the release candidate.
Examples:∘ Deploy a component.
∘ Run automated tests.
∘ Run manual tests.
∘ Update metrics.
∘ Alert the user of some event.
∘ Etc.
MADRID · NOV 21-22 · 2014
Tips for defining steps (I)
Consider:
The goal of the stage.
The kind of feedback you need.
Sources.
Targets (environments.)
Build and package only in the Commit stage.
MADRID · NOV 21-22 · 2014
Tips for defining steps (II)
Consider:
Most times, deployment is present, but not
always.
(Automated) Smoke Testing should follow
any deployment.
Think about both automated and manual
steps.
MADRID · NOV 21-22 · 2014
Activity: defining steps
“We want to filter out anything producing static
analysis warnings.”
“We want to try exploratory testing.”
“We may use the same environment for load
testing and security testing.”
MADRID · NOV 21-22 · 2014
#6 Define
Automation & Tooling
MADRID · NOV 21-22 · 2014
Tips for step automation (I)
Automate everything.
Automate everywhere (for all the
environments.)
Preference for automation:
Fully automated steps.
Manually triggered automatic steps.
Manual steps.
MADRID · NOV 21-22 · 2014
Tips for step automation (II)
Build only once.
Version everything.
This includes the automations.
Have environment lockdown in mind.
MADRID · NOV 21-22 · 2014
Deployment automation
considerations
Deploy the same way to every environment.
The target environment should be a (implicit)
parameter for the automations.
Set up (tool-agnostic) one-click deployments.
Treat configuration tokens as parameters for
the automations.
Prepare for rollbacks.
MADRID · NOV 21-22 · 2014
Database deployment
considerations
Database deployment is not the same as
database development.
Decide about the deployment strategy:
Schema & Data compare.
Delta scripts (better for Continuous
Delivery.)
ORM tools (schema update, migrations, etc.)
MADRID · NOV 21-22 · 2014
Test automation considerations
Q2 tests are not necessarily run through the UI.
Smoke tests may be run through the UI.
Frequently, non-functional testing can be
automated.
Leave environments and data in a known state.
A few things can’t be automated (UAT & Q3
testing.)
MADRID · NOV 21-22 · 2014
What do we need to know
For each step to be automated:
Automation tool or technology.
Execution model.
Parameters (at least you’ll have the
configuration tokens.)
Source / target.
MADRID · NOV 21-22 · 2014
About execution models
Native OS tool.
Agent.
Remote execution.
MADRID · NOV 21-22 · 2014
Activity: defining automation &
tooling
“Production environment is in an isolated
network”
“Operations people won’t allow us to install
anything there”
MADRID · NOV 21-22 · 2014
Tooling
Environment provision
Puppet, Chef, Ansible
Virtualization tools (VM templates, etc.)
App deployment
Scripting (Unix shell, PowerShell, etc.)
Puppet, Chef, Ansible, VS Release Management
DB Deployment
RedGate, DBDeploy, etc.
Testing
Testing frameworks
UI automation frameworks
Non-functional testing tools
MADRID · NOV 21-22 · 2014
#7 Define Execution model,
monitoring & metrics
MADRID · NOV 21-22 · 2014
Continuous delivery flow model
Pipeline instances are created on each commit.
Any commit is a release candidate.
One-piece continuous flow model.
There is no way back. Any error makes the
release candidate to be discarded.
Fixes are treated as new release candidates.
They are run through the entire pipeline from
the beginning.
MADRID · NOV 21-22 · 2014
Other flow models
Pipeline instances are created as needed. A
release candidate might comprise several
commits.
Decide on the batch size. Larger batches may
be cheaper but limit feedback.
Errors might be fixed in the context of the stage
where they arise.
MADRID · NOV 21-22 · 2014
Monitoring the pipeline
Transparency.
Rely on a proper tool.
Set up alerts for key events.
Use a Cumulative Flow Diagram (CFD.)
Gather metrics and act on them.
MADRID · NOV 21-22 · 2014
Primary metrics
Cycle time.
Mean Time Between Failures (MTBF.)
Mean Time To Recover (MTTR.)
MADRID · NOV 21-22 · 2014
Secondary metrics
Test coverage.
Duplication of code.
Coupling.
Compilation warnings.
Code churn.
Build frequency
Etc.
MADRID · NOV 21-22 · 2014
Tooling
Out of the box metrics in the orchestration tool.
Ad-hoc reporting.
Sonar, FxCop, Ndepend, FindBugs…
MADRID · NOV 21-22 · 2014
#8 Plan for future enhancements
MADRID · NOV 21-22 · 2014
Examples – DevOps culture
Improve branching model if needed.
Debugging optimization, symbol servers.
Canary releases.
Blue/green deployments.
A/B Testing.
MADRID · NOV 21-22 · 2014
Examples – DevOps culture
Preventive profiling.
Telemetry, analytics, Application Performance
Monitoring (APM.)
Proactive resiliency enablement (Simian army
http://nflx.it/SPeTGj)
MADRID · NOV 21-22 · 2014
Tooling
Application profilers
APMNew Relic
App Insights
Raygun
Compuware APM
MADRID · NOV 21-22 · 2014
Artifact and metadata repositories
Binaries repository
TFS Build Drops folder
Reporting system (receives data from all stages) TFS Reporting – tracks Cycle
Time, MTTR & MTBF Custom TFS Reporting on Work Items
Commit stage TFS Build Definition, customized from
Default template
CodeGet dependencies (package
restore)Build
Commit testsContinuous integration
Code analysisBasic functional tests (manual)
Version artifactsName the pipeline instance
(set the build number)Update the artifact repository
Update the Symbol Server
Acceptance test stage TFS Build Definition, customized
from Lab template.MTM for Test Case
management
Change configuration (automated)
Deployment (automated)MSBuild for WP
Powershell for AzureAcceptance tests
(automated)MSTest, CodedUI (?)
Release stage TFS Build Definition, customized from Lab
template
Branch or merge to Release branch (for archiving)
Deployment (automated)MSBuild for WP
Powershell for AzureSmoke testing (automated)
CodedUI (?)Error detection and recovery
(resiliency enablement)SCOM-TFS Integration
Intellitrace in ProductionPreemptive AnalyticsCapacity test stage TFS Build
Definition, customized from Lab template
Deployment (automated)MSBuild for WP
Powershell for AzureSmoke testing (automated)
CodedUI (?)Performance & load testing
(automated)Visual Studio perf & load
tests
Environments (automatic provision, locked down so only automated deployments are allowed) Using Lab Manager SCVMM Environments
Development (isolated)
Local deploymentto WP and Azure emulators
Test Production
Iteration 6 Pipeline: further improvements
Manualtrigger
Automatictrigger
Version Control (code and configuration) TFS Version Control
Main branch (w/ feature
toggles)
Release branch (for archiving)
Staging
Deployment & testingtriggered by Test Agent
Pipeline monitoring
Custom solution
UAT stage
UAT (manual)Acceptance criteria from
TFS requirementsTFS Feedback tool
Manualtrigger
Deployment triggeredby Test Agent
Using deploymentfrom Capacity test stage
Deployment & testingtriggered by Test Agent
Security test stage
Security tests (manual)WACA
Any additional tools?
Automatictrigger
Using deploymentfrom Capacity test stage
Exploratory test stage TFS Build Definition, customized
from Lab template
Deployment (automated)MSBuild for WP
Powershell for AzureExploratory testing (manual)
MTM
Manualtrigger
ExploratoryTest
Deployment & testingtriggered by Test Agent
Artifact repository
NuGet Server
Symbol Server
TFS Symbol Server
MADRID · NOV 21-22 · 2014
#9 Inspect & Adapt
MADRID · NOV 21-22 · 2014
Continuously improve the pipeline
Component or architectural changes.
New skills in the team.
New resources, tools, environments.
Reserve time, and make the team accountable
for improvement.
MADRID · NOV 21-22 · 2014
Summary
1. Define components.
2. Identify sub-pipelines.
3. Define Stages & Orchestration.
4. Define Environments.
5. Define Steps.
6. Define Automation & Tooling.
7. Define execution model, monitoring and metrics.
8. Plan for future enhancements.
9. Inspect & adapt.
MADRID · NOV 21-22 · 2014
Thanks!
Jose Luis SoriaContinuous Improvement Manager at Ria Financial
[email protected] - @jlsoriat
http://geeks.ms/blogs/jlsoria
Slides: http://www.slideshare.net/jlsoria
http://aka.ms/releasepipeline