Performance and stability testing \w Gatling

With Gatling

Performance & Stability Testing

Development Testing

@MeDmitry Vrublevsky, Software developer @

/in/dmitryvrublevsky

Roadmap

- Mission overview

- Gatling introduction

- Some usage example

Mission

Given: Graph database (Neo4j)

Challenge: execute query as FAST as we can

Neo4j

- Written in Java

- Native graph storage

- Extendable

So?

- Default API is too slow

- Implement custom extension

- Control all the things

1. Developed

2. Deployed

3. ???

4. Success!

Performance

Specification

Do “X” operations

in “Y” seconds

Specification

Do “10000” query operations

in “10” seconds

POST /extension/query Body: “MATCH (root)-[:HAS*]->(child)”

Response: { // JSON data }

Client

LB

DB1 DB2 DB3

- our extension

Problems?

Client

LB

DB1 DB2 DB3

- our extension- possible problem

Gatling

http://gatling.io

Load testing framework

Free & Open source

High performance

Friendly DSL

Nice html reports

Coding ?

Bundle

1. Download bundle 2. Extract 3. Ready

http://gatling.io/#/download

Optional

- Maven integration - SBT plugin - Gradle plugin

package com.neueda.exampleimport io.gatling.core.Predef._import io.gatling.http.Predef._import scala.concurrent.duration._class Example extends Simulation { val httpConf = http.baseURL("http://www.example.com") val example = scenario("Example") .exec( http("get_example_index") .get("/") .check(status.is(200)) ) setUp( example.inject(rampUsers(10) over (10 seconds)) ).protocols(httpConf) }

package com.neueda.exampleimport io.gatling.core.Predef._import io.gatling.http.Predef._import scala.concurrent.duration._

class Example extends Simulation {}

val httpConf = http.baseURL("http://www.example.com")

val example = scenario("Example")

http("get_example_index") .get("/") .check(status.is(200))

setUp().protocols(httpConf)

example.inject( rampUsers(10) over (10 seconds) )

package com.neueda.exampleimport io.gatling.core.Predef._import io.gatling.http.Predef._import scala.concurrent.duration._class Example extends Simulation { val httpConf = http.baseURL("http://www.example.com") val example = scenario("Example") .exec( http("get_example_index") .get("/") .check(status.is(200)) ) setUp( example.inject(rampUsers(10) over (10 seconds)) ).protocols(httpConf) }

http://gatling.io/docs/2.1.4/cheat-sheet.html

================================================================================2015-04-07 23:02:54 5s elapsed---- Example -------------------------------------------------------------------[##################################### ] 50% waiting: 5 / running: 0 / done:5---- Requests ------------------------------------------------------------------> Global (OK=5 KO=0 )> get_example_index (OK=5 KO=0 )================================================================================

================================================================================---- Global Information --------------------------------------------------------> request count 10 (OK=10 KO=0 )> min response time 255 (OK=255 KO=- )> max response time 302 (OK=302 KO=- )> mean response time 274 (OK=274 KO=- )> std deviation 13 (OK=13 KO=- )> response time 95th percentile 293 (OK=293 KO=- )> response time 99th percentile 300 (OK=300 KO=- )> mean requests/sec 1.08 (OK=1.08 KO=- )---- Response Time Distribution ------------------------------------------------> t < 800 ms 10 (100%)> 800 ms < t < 1200 ms 0 ( 0%)> t > 1200 ms 0 ( 0%)> failed 0 ( 0%)================================================================================

================================================================================2015-04-07 23:02:54 5s elapsed---- Example -------------------------------------------------------------------[##################################### ] 50% waiting: 5 / running: 0 / done:5---- Requests ------------------------------------------------------------------> Global (OK=5 KO=0 )> get_example_index (OK=5 KO=0 )================================================================================

================================================================================---- Global Information --------------------------------------------------------> request count 10 (OK=10 KO=0 )> min response time 255 (OK=255 KO=- )> max response time 302 (OK=302 KO=- )> mean response time 274 (OK=274 KO=- )> std deviation 13 (OK=13 KO=- )> response time 95th percentile 293 (OK=293 KO=- )> response time 99th percentile 300 (OK=300 KO=- )> mean requests/sec 1.08 (OK=1.08 KO=- )---- Response Time Distribution ------------------------------------------------> t < 800 ms 10 (100%)> 800 ms < t < 1200 ms 0 ( 0%)> t > 1200 ms 0 ( 0%)> failed 0 ( 0%)================================================================================

Reports \o/

class Test extends Simulation { val httpConf = http.baseURL("http://neo-database:7474") val test = scenario("GetGraph") .exec( http("execute_query") .post("/extension/query/execute") .body(StringBody("MATCH (root)-[:HAS*]->(child)")) .check(status.is(200)) .check(jsonPath("$.results").count.is(100)) ) setUp( test.inject( rampUsersPerSec(0) to (1000) during (60 seconds) ) ).protocols(httpConf) }

http("execute_query") .post( "/extension/query/execute" ) .body(StringBody( "MATCH (root)-[:HAS*]->(child)" )) .check( status.is(200) ) .check( jsonPath("$.results").count.is(100) )

1

2

3

4

rampUsersPerSec(0) to (1000) during (60 seconds)

Req

uest

s / S

econ

d

0

150

300

450

600

Seconds

0 10 20 30 40 50 60

When request/second goes up

And some threshold reached

Then we receive Timeout error

Everything is ok?

Maybe we can do higher load?

How to spot the problem?

Drop all

non-essential

parts

Client

LB

DB1 DB2 DB3

- our extension

0

250

500

750

1000

0 10 20 30 40 50 60

Client - Server communication

problems

• Incorrect server setup

• Unconfigured networking

• Low max open connection limit

Learned

- Performance testing should be done - Test results should be interpreted

properly - Test results should be verified in

different environments

We need to test different queries

Feeders!

val feeder = Array( Map("query" -> "..."), Map("query" -> "..."), Map("query" -> "...") )

.queue

.random

.circular

val test = scenario(“GetGraph") .feed(feeder) .exec(http("execute_query") .post("/extension/query/execute") .body(StringBody("${query}")) .check(status.is(200)) .check(jsonPath("$.results").count.is(100)) )

CSV JSON JDBC

Sitemap Redis

Custom

http://gatling.io/docs/2.1.4/session/feeder.html

csv("data.csv").random

Our own feeder

- Custom feeder - Lazy loads data - Performant

Dynamic user load?

http://gatling.io/docs/2.1.4/general/simulation_setup.html

test.inject( nothingFor(5 seconds),

atOnceUsers(10),

rampUsersPerSec(10) to (100) during (20 seconds),

constantUsersPerSec(100) during (40 seconds),

rampUsersPerSec(100) to (500) during (20 second),

constantUsersPerSec(100) during (60 seconds))

Checks

http://gatling.io/docs/2.1.4/http/http_check.html

- status - currentLocation - header - regex - xpath - jsonPath

regex("Invalid Page title").notExists

status.is(200)

jsonPath("$.posts").exists

regex("""<div class="article">""").count.is(10)

Realtime monitoring

http://gatling.io/docs/2.1.4/realtime_monitoring/index.html

Execution progress visual feedback

Graphite (InfluxDB) +

Grafana

Recorder

http://gatling.io/docs/2.1.4/http/recorder.html

HTTP

http://gatling.io/docs/2.1.4/http/recorder.html

- HTTP Protocol - SSL - WebSocket - SSE

Jenkins integration

https://github.com/jenkinsci/gatling-plugin

- Any IDE with Scala support is OK

- Intellij IDEA

- Eclipse (Scala IDE)

- Netbeans

Our use cases

GET /api/node/1 POST /api/node UPDATE /api/node

Basic API tests

- Generate background load - Make stability tests

- Cluster communications - Load balancer setup - Spikes

Load generator

- Simulate significant write load - Check how database reacts

Data import

Easy Fun

Performant