Testing in Continuous DeliveryAistė Stikliūtė, Agile turas 2015

AgendaWhat is Continuous Delivery and why it’s (slowly) going mainstream

Examples

Challenge - testing• Problems• Solutions

What is Continuous Delivery

C.D.

Build

Deploy

Test

Release

Do you do Continuous Delivery?

How often do you / can you deploy to Prod?

What if you’ve got to do a big change?

Who decides when to deploy / release?

Some examples

… examples

The testing challengeMonths → Weeks → Days → Hours → Minutes

Have the cake and eat it too?

● do thorough regression testingand do it often?

● add changes fastand keep up with automated

tests?

● ensure good qualitywithout (much) manual testing?

Cycle time: idea production

Design Develop Test Merge to trunk

Deploy to test

environmentTest Deploy to

stage Test

Deploy to production

Cycle time to production: testingManual tests

before merge

Automated tests after

mergeStaging tests

Analyse resultsUpdate testsBugfix

Clarify open questionsBugfix

Analyse resultsBugfix

Solve the testing challengePatterns and ideas

[obvious] agile testing

Testers involved in team:grooming, planning, development

“Show me”testing before it’s finished, input for automated tests

When it’s finished, tester must be ready to test

[obvious] developer testing

Develop Test

developer responsibility

You are responsible for your code• Testing• Deploying• Supporting

• Defined code ownership• Better automated tests• Monitoring. Faster corrections• Faster & more successful deployments

branching patterns

Centralized Finds and fixes problems ASAP

Distributed Develop every change in its own branch and test

Temporary branches

Effective, combines centralized and distributed

Matrix of services

Use centralized CI for a massively scalable system

test automation

Fast tests Reliable tests

test automation

Test optimization

Well-thought infrastructure

Run in parallel

Test categories

test automation

Test coverage

Test environment

Keep tests green “Pass” ratio

testing in production

A/B testing

feature toggling

monitoring

feature toggling

● Needs to be implemented● Reduces need for lab-testing, but it’s still needed● Needs to be tested itself● Reduces risk● Enables A/B testing

on off staff user_list opt-in 0-100%

A/B testing

monitoring

Errors Performance User behavior

User community /

support

automatic notifications

The future