Performance Testing in Continuous Delivery Pipelines

Andrey PokhilkoChief Scientist, BlazeMeter

Importance of CI and CD

● Machine time costs nothing, human time priceless

● De-facto winning practice

● Most advanced teams go with CD

● A lot of teams are still in process of adopting it

Agenda

1. Things we put into CI

2. Best practice (for now)

3. Challenges of testing in CI

4. Jenkins for Performance Testing

Things we put into CI

Triggering and Preparations

● When to do the job

● VCS checkout + dependencies checkout

● Building project (compiling etc.)

● Put resulting packages into repos

Quality Control

1. Static code analysis

2. Unit tests

3. Functional tests

4. Performance testsneeds deployment

Deployment

● To QA environment (for further manual tests)

● To staging environment

● To production environment

The Best Practice

VCS Checkout + Dependencies

Build Unit Tests

Functional Tests

Perfromance Tests

Deployment

Filter Filter Filter Filter

One Way Road (Driverless)

VCS-Driven Pipeline

● Natural evolution of CI systems

● Branching and pull requests

● Jenkins 2.0 pipelines

● Taurus Tool as part of this approach

Why testing is challenging in CI

Challenge #1: Test Environment

● Applications are complex

● Lots of dependencies

● Third-party systems

Databases

Microservices

Third-parties

Challenge #2: Time Consuming

● Preparations

● A lot of functional tests

● Performance tests are naturally long

Challenge #3: Debugging CI Jobs

● Evolving test complexity

● Debugging and troubleshooting

● Build history is a value

Challenge #4: Results Analysis

● Rich reporting needed

● Jenkins UI has its limits

● Automated decision making for CD

● Collaboration for non-tech people

wut???

Overcoming Challenges

Overcome: Test Environment

● Containers help (you have to use containers :)

● You don’t need to be as realistic

≈

Overcome: Time Consuming

● Prepare what you can upfront - in the night

● Reuse what you can

● Short tests can reveal a lot of things

● Don’t try to put spike and endurance tests into CI

● Parallelize tests, Jenkins 2.0 or Taurus helps

Overcome: Debugging and Improving

● Taurus Tool, the handheld piece of CI

Overcome: Debugging and Improving

Load Test Job

Local Debugging

Overcome: Reporting & Result Consumption

● Purpose-built services with integrations

● Overview & Status in Jenkins

Overcome: Decision Making

● Deploy to staging is great win

● Have pass/fail criteria, including APM info

● It’s ok to have deploy as separate job

(for UAT, to avoid disruptions)

Jenkins for Performance Testing

Why Use Jenkins for Testing?

● Multi-step process

● Long process

● Repetitive process

Start Simple

Load Test 1

Load Test 2

Load Test 3

Center of Excellence

Developer

Operations

QA

Maintain Jobs

Become Continuous

● Periodic jobs have many applications

● Have valid failure criteria (thresholds)

● Use non-blocking downstream jobs

Pursue Continuous Delivery

FastFeedbackDeveloper /

DevOps

Build + xUnitFunctional

Tests

Load Tests Deploy Staging

SlowerFeedback

Some Useful Plugins

Plot Plugin

Performance Publisher Plugin

BlazeMeter Plugin

Taurus Tool

● Why not plugin, but command-line tool

● Universal plug for tests inside Jenkins

● Taurus designed to work with other Jenkins plugins

gettaurus.org

Final Summary

1. We have to test inside CI

2. There are ways to start it simple

3. There are tools & plugins to help

4. Jenkins 2.0 pipelines FTW!

Thank you!Any questions?