T9 Session 10/27/2016 1:30:00 PM

Build Your Open Source Performance Testing Platform in the Cloud

Presented by:

Gopal Brugalette

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073 888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.starcanada.techwell.com/

Gopal Brugalette Gopal Brugalette is a senior applied architect in performance engineering at upscale fashion retailer Nordstrom where he works on a variety of e-commerce omni-channel initiatives. Gopal's responsibilities include preparing the Nordstrom.com site for peak events, developing engineering frameworks, and expanding performance engineering activities into the development cycle and production. He has presented at numerous industry events and been featured in online magazines. Previously Gopal was at IBM and Washington Mutual Bank. Before he got into IT, he was a researcher at the Center for Experimental Nuclear Physics. In his spare time Gopal enjoys developing his permaculture farm and woodworking.

10/17/2016

1

Building an Open‐Source Load Testing Framework in the Cloud

Gopal Brugalette

Principal Engineer

A busy day @ Concur158,000 trips booked

357,000 expense reports1 million mobile logins

580,000 mobile receipts uploaded

32,000 clients, 100 countries2

10/17/2016

2

Gopal BrugalettePrincipal Engineer, Performance

3

Introduction – so you want to build a framework

Building the Framework

Requirements

Management

Design

Build

4

10/17/2016

3

Why build a framework?

Cost 

Capabilities

ControlControl 

Challenge5

The stakeholders 

•WhoWho

•What

•Why•Why

•When6

10/17/2016

4

What will it do

CI Pipeline Integration

Spike Load and 

Performance

Endurance or Soak

StressProduction Full‐Scale

7

Who are your users

D•Devs

•SDETS

•Perf EngineersPerf Engineers 

•Automation

8

10/17/2016

5

B k d

Who will build it

•Background

•Skills

•WhyWhy

•Team Dynamics

9

Leadership

Enterprise Direction

Success MeasurementsSuccess Measurements

Value

Budget 10

10/17/2016

6

Manage the Effort

Find the TimeProject 

MethodologySustain

11

Components of the framework

Scripting LoadTest AUT

APIs

UI

Orchestration

Scripting& Exec

LoadGeneration

Metrics

Test Artifacts

AUT

AnalyticsReporting

12

10/17/2016

7

Designing the framework

Hand it off

Re‐usable 

Structured inputs and outputs

Modularity

More than sum of parts13

Deliver Value and Iterate

R1 

• Basic load and graphs

• Pipeline integration

R2 

• Dynamic load

• Deeper analytics

R3

• Alerting

• Reporting

R4

• Auto‐scripting

• More alerting

14

10/17/2016

8

D i T l S l ti

Development Principles

Design

•Architecture

•12 factor app

Tool Selection

•Integration

•Capabilities

•Containers

•Cloud, on‐Prem

•Community Support

15

16

10/17/2016

9

Working in AWS

Networking &Concepts

• Automate It! 

• SLA’s

• Cost &

AWS Services

• EC2, Lambda

•Dynamo, S3

• Kinesis SQS

Networking & Security

• ELB

• VPC

• Security• Cost &Capacity

• Kinesis, SQS

• Cloudwatch, Elastic Search

• Security Groups

• IAM Roles

17

Scripting & Execution Engine 

Tool Consideration

l ll•Jmeter,Gatling,Artillery...

Scripting Integration

18

• Applications under Test

• Language & capabilities

•Code based or UI

•Automation support

•Results Output

•AWS Load Generation

18

10/17/2016

10

Load generation

Resources

dynamic or static

• EC2

• Lambda

Considerations

•Capacity and Spin Up

•Network Access

•On‐prem •Cost

•Management Effort

• Test Types19

API’s

Hide Details Standardize Integrate

Expand Tool Features

ExtensibleLambda in 

AWS

20

10/17/2016

11

Orchestration

Workflow/Pipeline Tools

Test Assets

Spin upp

• Jenkins

• Go

• Rundeck

API’s

• Lambda SMS DynamoDB

p p

Run Test

Analyze Results

21

• Lambda ‐ SMS, DynamoDB

• Server‐based logic

• Gateway

Custom Scripting & Logic

21

Report

Spin down

Data Store

A li i &Options

• Elastic search

• Influx

• Splunk

• APM Tools

• Dynatrace

Data Collection

• Agent/forwarders

• Log Stash/Beats

• Telegraf

• Direct writes

• API's

Extracting Data

• API

• Query Language

• Raw or Analyzed

• Size & Capacity Limits

Application & Server Metrics

• CloudWatch

• APM

• Logs and Plugins

• topbeats

Dynatrace

• New Relic

API s

• Post

• Streaming

• Visualization Support

22

10/17/2016

12

AnalyticsPerformance Data

• Response Time

• Errors

Application & Server

• Stack traces

• Resource Utilization

Information

• Trending

• SLA, Pass/Fail

• Throughput • Processing Time • Anomalies

• Distribution

Custom Built‐in Plug‐Ins & Packages

Implementation

• R

• Python

• Scala

• Julia

• Time Series

• ELK, InfluxDB, Cloudwatch

• APM

• Dynatrace, New Relic

g g

• Watcher, Elastalert

• Kapacitor

• TimeLion

• R, Python Packages23

Reporting

Visualization CI Integration AlertsVisualization

• ES/Kibana

• Grafana

• Chronograf

• Tableau

CI Integration

• Pass/fail

• Stop the build

Alerts

• Email

• Pager

• Slack

• Watcher• Tableau

• Shiny

• APM tools

• Watcher

• Elastalert

• Kapacitor

24

10/17/2016

13

Scaling

Testing the framework

g

Resiliency

Pilot teams

Analytics

Alerting & Reporting25

Culture of Who has

Encourage Adoption

Culture of Reuse?

Who has Responsibility? 

Market It Train Them

26

10/17/2016

14

Vid

Training

•Videos•Wiki

•Lunch & Learn•1‐1•Office Hours

27

Support your framework

Process CommunicationProcess

•Ticketing System

•SLA’s

Communication

•Email

•Slack, IM

•Pager support

•Follow the sun

•Office‐hours

•Wiki 

28

10/17/2016

15

Example Frameworks

Scripting LoadTest AUT

APIs

UI

Orchestration

Scripting& Exec

LoadGeneration

Metrics

Test Artifacts

AUT

AnalyticsReporting

29

A Simple Framework

Gi

Jenkins

Git

JenkinsJmeterJmeter

Static Servers

30

10/17/2016

16

Analytics and Reporting

Jenkins Performance Plugin

31

A Basic Framework

Gi

Jenkins

Git

JenkinsJmeterJmeter

ES

Kibana

Logstash/Beats

Static EC2 Servers

32

10/17/2016

17

Analytics and Reporting

33

Jenkins, Jmeter to Influx

34

10/17/2016

18

Influx/Grafana Dashboard

35

Dynamic Scaling

Dynamic JmeterMachines

Jenkins

JenkinsJmeterJmeter

ES

Kibana

Machines

Logic

Dynamic Creation

Spin Up MachinesDistribute Scripts, Data

36

10/17/2016

19

Dynamic Scaling II

Jenkins

λ

ES

Logic

Call λ λλλλ

λλ

λ

λ

Kinesis

S3or

Jmeter,Gatling,Artillery (Java)

Distribute Scripts, Data

λ

37

API’s

API’s

ES

KibanaAPIGW

λ

λλλλλ λ

λλ

λλλλ λ

λ

38

GWGatlingJmeter

API’s

10/17/2016

20

Custom Analytics

R

Tableau

39

ES

Jenkins

Questions?

Reach me @

gopal.brugalette@concur.comOr LinkedIn

40