View
493
Download
2
Category
Preview:
Citation preview
Load Testingwith Jason Lotito
“Guaranteed to be the best talk on load testing with PHP @ MidwestPHP 2016 during that time
slot in that room with me as the speaker"
Jason Lotito
• DevOps Lead Architect
• MeetMe.com
• @jasonlotito
• jasonlotito.com
• jlotito@meetme.com
Load TestingNot Unit Testing, Integration Testing, Acceptance Testings, manual Testing, or Standardized Testing
Load Testing
• Putting demand on a system
• Determine behavior under load
• Determine maximum operating capacity
• Discover Bottlenecks
• Determining steps to increase operating capacity
• Optimize for costs
Why RedLine13?
• Simple, up and running in little time
• Freemium, free level you only pay AWS
• Scriptable (PHP, Node.js)
• AWS for Infinite Scalability™
• I know Rich, one of the founders
RedLine13 has a bunch of features you can read about on the website. I’m really here to talk to you about the cool stuff I do with RedLine13.
In our example, we’ll use sleep to simulate performing actions such as
making HTTP requests to an API
<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }
CustomTest.php
This is where your code goes
<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }
CustomTest.php
<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }
CustomTest.php
<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }
CustomTest.php
<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }
CustomTest.php
<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }
CustomTest.php
<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }
CustomTest.php
Start Time
Start/End Time Difference
<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }
CustomTest.php
URL/String
<?php/** * @return bool */function isRedlineVerbose(){ return (int) getenv( 'REDLINE_VERBOSE' ) === 1; } /** * @param int $timestamp * @param int $elapsedTime */function recordPageTime( $timestamp, $elapsedTime ) { isRedlineVerbose() && output( "Record Page Time: timestamp $timestamp; elapsed time: $elapsedTime" ); } /** * @param string $url
redline.php
<?php/** * @return bool */function isRedlineVerbose(){ return (int) getenv( 'REDLINE_VERBOSE' ) === 1; } /** * @param int $timestamp * @param int $elapsedTime */function recordPageTime( $timestamp, $elapsedTime ) { isRedlineVerbose() && output( "Record Page Time: timestamp $timestamp; elapsed time: $elapsedTime" ); } /** * @param string $url
redline.php
<?php/** * @return bool */function isRedlineVerbose(){ return (int) getenv( 'REDLINE_VERBOSE' ) === 1; } /** * @param int $timestamp * @param int $elapsedTime */function recordPageTime( $timestamp, $elapsedTime ) { isRedlineVerbose() && output( "Record Page Time: timestamp $timestamp; elapsed time: $elapsedTime" ); } /** * @param string $url * @param int $timestamp * @param int $elapsedTime */function recordURLPageLoad( $url, $timestamp, $elapsedTime ) { output( "$url -> $elapsedTime seconds" ); }
redline.php
* @param int $timestamp * @param int $elapsedTime */function recordPageTime( $timestamp, $elapsedTime ) { isRedlineVerbose() && output( "Record Page Time: timestamp $timestamp; elapsed time: $elapsedTime" ); } /** * @param string $url * @param int $timestamp * @param int $elapsedTime */function recordURLPageLoad( $url, $timestamp, $elapsedTime ) { output( "$url -> $elapsedTime seconds" ); } /** * @param int $kilobytes */function recordDownloadSize( $kilobytes ) { isRedlineVerbose() && output( sprintf( "Download Size: %dkb", $kilobytes ) ); } /**
redline.php
* @param string $url * @param int $timestamp * @param int $elapsedTime */function recordURLPageLoad( $url, $timestamp, $elapsedTime ) { output( "$url -> $elapsedTime seconds" ); } /** * @param int $kilobytes */function recordDownloadSize( $kilobytes ) { isRedlineVerbose() && output( sprintf( "Download Size: %dkb", $kilobytes ) ); } /** * @param string $error */function recordError( $error ) { output( "Error: $error" ); } /** * @param string $msg * @return bool
redline.php
/** * @param int $kilobytes */function recordDownloadSize( $kilobytes ) { isRedlineVerbose() && output( sprintf( "Download Size: %dkb", $kilobytes ) ); } /** * @param string $error */function recordError( $error ) { output( "Error: $error" ); } /** * @param string $msg * @return bool */function output( $msg ) { echo sprintf( "[%s] %s]\n", getmypid(), $msg ); return true; }
redline.php
$kilobytes ) ); } /** * @param string $error */function recordError( $error ) { output( "Error: $error" ); } /** * @param string $msg * @return bool */function output( $msg ) { echo sprintf( "[%s] %s]\n", getmypid(), $msg ); return true; }
redline.php
<?phprequire_once('redline.php'); require_once('CustomTest.php'); $customTest = new CustomTest(1, mt_rand(1000000,9999999)); $customTest->startTest();
run.php
<?phprequire_once('redline.php'); require_once('CustomTest.php'); $customTest = new CustomTest(1, mt_rand(1000000,9999999)); $customTest->startTest();
run.php
<?phprequire_once('redline.php'); require_once('CustomTest.php'); $customTest = new CustomTest(1, mt_rand(1000000,9999999)); $customTest->startTest();
run.php
<?phprequire_once('redline.php'); require_once('CustomTest.php'); $customTest = new CustomTest(1, mt_rand(1000000,9999999)); $customTest->startTest();
run.php
Instance Number
<?phprequire_once('redline.php'); require_once('CustomTest.php'); $customTest = new CustomTest(1, mt_rand(1000000,9999999)); $customTest->startTest();
run.php
User Number
Random Number
You can use error reporting to simulate real time logging
Errors show up in real time, logs show up after the test is completed. Use error reports to let you
know if you should stop your load test.
Sample App: Chat Service
• 1-on-1 Chat
• Group Chat
• Browse Chat Rooms
• Search for Chat Room Topics
1-on-1
Group30-40%
60-70%30-40%
60-70% 60-70%30-40%
Time
Start at the same time
Increase group chat %
Could mean just more group chat or both more group and less 1-on-1
Revert to normal ratio
20:00 40:000:00
1-on-1
Group30-40%
60-70%30-40%
60-70% 60-70%30-40%
Time
Could be done in your script or in RedLine13
1-on-1
Group350
350+300350
350+300 350350
Time
300 test sleeps for 20 minutes
After 20 minutes, it stops.
Things You Need
• Production-level systems to test against
• Monitoring
• Automated systems to deploy changes
• Dedicated people
• A time to test (load testing day!)
Planning the Load Test
• Start at what you expect normal traffic to be
• 500rps
• Start a second test that is just the burst traffic
• 900rps
• Ramps up over 20 mins
• Lasts 40 mins at peak
• Ramps down over 20 mins
Learning
• Is proper monitoring setup?
• Is a scaling plan in place? Can you scale?
• What happens to the app under load?
• What slows down first?
• Are things alerting as expected?
As Load Tester, you are responsible for…
• Ensuring those involved with the system under test are aware and available
• Putting the system under test under load
• Completely destroying the system under load
• Providing a clear and concise write up of exactly what occurred
• Take charge! It’s fun!
Reporting Results: Short Version
• Short version: During an hour starting with 0rps to 1400 rps in the first 10 minutes....
• ...when starting with 5 instances and scaling to 11 instances, the response times were: 50% 23ms, 75% 54ms, 95% 303ms, and 99% 1069ms.
• ...when starting with 11 instances, the response times were: 50% 16ms, 75% 24ms, 95% 45ms, and 99% 59ms.
Reporting Results: Detailed Version
• Provide more information
• Results
• Story driven
• Have data supporting results prepared in case
• Account for changes such as auto-scaling
• With auto-scaling: 99% at 1069ms
• Without auto-scaling: 99% at 59ms
Testing elasticsearch on c3.4xlarge
Detailed Reporting
Also included in this report was a link to the actual GitHub repository. Make sure you are keeping your load tests in version control!
Things to Keep In Mind
• Understand expected usage
• X% of users using the app while
• Y% are chatting with one another
• Users are logging in
• Creating accounts
• Backend systems
• Determine what’s important
Things to Keep In Mind
• User input
• Random filters
• Weighted filters
• Cached results are expected
• Client constraints
Things to Keep In Mind
• User flow through service
• Try to understand how users use the app
• Script should try to mimic
Things to Keep In Mind
• Be careful about testing a single system
• System will have logging
• System will have backend services
• You’d be surprised what can cause failure
• A load test helps you learn before it’s in production
Things to Keep In Mind
• User interaction
• MeetMe is social, so we’ve load tested chatting
• 1 test per 2 users, both chatting with one another
Things to Keep In Mind
• Have developers available
• Better, have developers with you when load testing
Things to Keep In Mind
• Find a problem, Fix it, Reroll app, Rerun test
• FFRR, or F2R2
• I just made that up
• Don’t use it.
Things to Keep In Mind
• Start testing from your laptop
• Seriously, my MacBook Air could bring down Erlang services
• Database indexes are a thing
• While running a load test, you can run a single client form your laptop
Things to Keep In Mind
• Someone should be testing the app/service as well
• Response times are only a number
• What does 50ms vs 300ms response times feel to the user
• What impact does 2x/3x/4x load have
• When auto-scaling, how does the client handle
Things to Keep In Mind
• Review how the client is using the API
• Review how the API developer expects the client to use the API
• Model after what the client is doing
• Call out differences early
Thanks!
• @jasonlotito
• jasonlotito.com
• jlotito@meetme.com
• https://joind.in/talk/80910
Recommended