Load Testing With JMeter

Larry DavidsonOctober 21, 2008

Load Testing with JMeter

What is Load Testing?

2

3

App

User User User User

Normal Use – Few Concurrent

4

App

User User User UserUser User User User User User User User User User User User User User User User User

App

Spike Use – Many Concurrent

User User User User User User User User User User User User User User User User User

5

AppApp

Spike Use – Many Concurrent

User User User User User User User User User User User User User User User User User

Load Testing

Key goalsCan app support expected concurrent users?At what load does the app break?

StepsIdentify primary user pathsIdentify expected concurrent users

Both now and down the roadSet up virtual users to hit the appRun the testAnalyze the results

Load Testing Approaches

Open Source ToolsJMeterabhttp_loadThe GrinderSiege

Commercial ToolsLoadRunner (around $200,000 to own!)WebLoadThird-party $ervices

7

Why We Chose JMeter for Internal Testing

Easy to install and use Free!Java — most platformsGUI vs. command lineJust download and run!

Feature-rich Post formsRecord from browserLoad test data from filesAdd logic, variables & functionsRun one test from multiple machinesTest many protocols, not just HTTP

8

Installing & Running JMeter

1. Download from http://jakarta.apache.org/jmeter 2. Unzip (path with no spaces preferred)3. Go to "bin" directory4. Run appropriate file

Windows:  Double-click jmeter.bat Mac:          Double-click ApacheJMeter.jarUnix/Mac:  Run ./jmeter from command line 9

Looks complicated ... but it's not!

10

Using JMeter's GUI

Hierarchy based "Test Plan" is root element

Add test elements as childrenRight-click and choose "Add > ..."Give items meaningful names

"WorkBench" is for non-test helpersHTTP Proxy recording

Modify properties in right panelProperties saved after leaving entry fields

11

What We're Trying to Do

Simulate usersDefined numbers of concurrent usersDefined period of timeVarious user types (normal, registered, admin, etc.)

Simulate web requestsSpecific web pagesForm submissionsAlternate or random paths

Analyze reportsResponse times acceptable?Handles expected load?What load causes failure?

12

JMeter's Nomenclature

User                   Thread

Request             Sampler

Report                Listener

13

Let's run a Load Test already!

14

Setting Up Your First Load Test

Simulate usersAdd > Thread Groups

Number of Threads (concurrent users) = 10Rampup Time (seconds until users active) = 1Loop Count (how many repetitions) = 1

Simulate web requestsAdd > Sampler > HTTP Request

Server Name or IP = www.yahoo.comPath = /

Analyze reportsAdd > Listener > Aggregate GraphAdd > Listener > View Results Tree 

15

Running Your First Load Test

Choose "Run > Start" (ctrl-R)Click on the "Aggregate Graph"

Data fills in as test runsClick "Display Graph" for chart

Watch upper right squareGreen = running, Gray = done47/100 means 47 of 100 threads are running

To run again with clean data... Choose "Run > Clear All" (ctrl-E)Choose "Run > Start" (ctrl-R)

16

Running!47 threads (“users”)

Verify Details with View Results Tree

Details for each request, good for troubleshootingSampler Result

Thread number and detailsResponse timesServer response codes and headers

RequestRequest sent to web server

Response DataServer response body (HTML, etc.)View as text or rendered HTML, etc. 

17

Let's get more advanced!

18

A Test with Three User Types

Users can do various things on a siteGood to simulate several common tasks at same time

For this demo, we’ll test 3 users1. Yahoo Viewer2. Google Viewer3. CNN Viewer

To start, add 3 thread groups under Test PlanSet to run all concurrently (this is default)Add HTTP Request to each, with appropriate URLs

Add Listener under Test Plan and RunNotice that all 3 URLs are requested at once

19

Logic Controllers

Group ItemsSimple

Choose One Per Loop or Thread GroupInterleave, Random, Switch*

 LoopingLoop, ForEach*, While*

Logical*If

Time ControlThroughput, Runtime 

20

Randomly Selecting From 3 URLs

Let’s modify the 3 user types Test PlanMove the 3 requests under one Thread Group

1. Yahoo2. Google3. CNN

Choose all 3, right click, choose “Insert Parent...”Choose “Controller > Random Controller”

Add Listener under Test Plan and RunYou’ll see that only one random request is chosen

21

Add an If Controller

Let’s modify the existing Test PlanAdd “Controller > If Controller” under Thread GroupCopy the 3 requests

1. Yahoo2. Google3. CNN

Paste under the “If Controller”Delete the “Random Controller”

22

Let's make a recording!

23

Recording your load test

Add Proxy Server to WorkBench“Non-test Elements > HTTP Proxy Server”Choose a "Target Controller" for recorded data Click "Start"

Set browser to use Proxy ServerEdit browser's Network Settings

Usually under Preferences > Advanced > Network/ProxyProxy = localhost; Port = 8080

Browse and recordClick "Stop", then edit recorded requests

24

Recording tips and tricks

Limit what's recordedEnter "text/html" in Content-Type filter (Include)Enter *.gif, *.css, etc. in URL Patterns to In/Exclude 

Use Grouping if recording multiple pagesAdd HTTP Request Defaults under Controller

Fill in domainNow matching recorded domains will be blank

Record time between page requestsAdd "Constant Timer" under Proxy Enter "${T}" in delay field to record actual time

25

Recording forms

JMeter records all submitted fieldsEasier than examining source code or Firebug!

JMeter also records browser headersSome pages (e.g., Google) require valid browsers

Remember to accommodate cookies if needed!add “Config Element > HTTP Cookie Manager”

Edit field values after they're recordedOR ... why not simulate varying user data!

26

Let's add variables!

27

JMeter Variables

Define global variablesTest Plan (root element)Config Element > User Defined Variables

Define virtual user variablesPreprocessors > User ParametersConfig Element > CSV Data Set Config

Reference as ${variableName}Can be used in nearly any propertyScope determined by location in hierarchy

28

Variables from a CSV file

Create a CSV file with the data you want to useNo label row; save in same folder as .jmx file

Add "Config Element > CSV Data Set Config"Add under the Thread Group that will use itSet Filename = filename.csvSet Variable names = username,password (e.g.)

Set variables in HTTP Request "Send Params"Name = username, Value = ${username}Name = password, Value = ${password}

Will read in data with each new thread, or with loops

29

Debugging

Add Debug SamplerSet JMeter variables as "True", others "False"

Add View Results Tree ListenerRunClick View Results TreeClick Debug Sampler in Results Tree

Click "Response data" tab to view variablesLooped user vars will show with numbers

30

JMeter Functions

Referenced as ${__functionName(properties)}Like variables, can use in JMeter properties

Use Options > Function Helper DialogChoose a functionEnter parameter valuesClick "Generate", then copy/paste function

Helpful functions${__javaScript(expression to evaluate)}${__threadNum()}${__Random(min, max, varname)}${__regexFunction(regEx, template, match, ...)}

31

So what about reports?

32

Reading Reports

Most times are reported in millisecondsThroughput

Requests per second

90% LineValue below which 90% of request times fall

Standard DeviationA measure of certainty / uncertaintyMeasures variation from average

LatencyDelay between request and response

33

Response Time Goals to Aim For

0.1 second — perceived instant1.0 second — uninterrupted flow10 seconds — user loses interest

Going further...

35

Higher loads with multiple "slave" computers

Install JMeter on other computersFind the IP addresses of the other computersStart JMeter server on all computers

Windows: Double-click [jmeter]/bin/jmeter-server.batMac command line: cd [jmeter]; ./jmeter-server

Add IP addresses to master computer's JMeterOpen [jmeter]/bin/jmeter.properties in a text editorAdd IPs to remote_hosts=127.0.0.1,192.168.1.241

Choose "Run > Remote Start All"

36

Limitations of JMeter

Can't run embedded Javascript, Flash, etc.But ... use recording to capture those requests

Limited by your computer's network connectionYou may have to work with IT to optimize

1 computer limited to about 300 concurrent usersBut... use multiple slave computers to handle more

 Reports are a bit lackingBut... save results and import into Excel, etc.

37

Reference

http://jakarta.apache.org/jmeter

38