Oh Crap, My Code is Slow - Madison PHP 2016

Madison PHP Conference, September-October 2016 1

Oh Crap, My Code is SlowChris Tankersley@dragonmantankMadison PHP Conference, September-October 2016


1.79 MHz 8-bit Processor128K RAM

640x192 max resolution64 color palette

RS-232 Serial PortCartridge Bay

2 Joystick Ports

Disk Extended Color Basic 2.1


520 Mhz Apple S1512MB RAM

390x312 resolution (~303 ppi density)16 million colors

WatchOS


32 Cores512GB RAM

1-10Gbps NICs10 Terabyte FusionIO Cards



“premature optimization is the root of all evil.”

Donald Knuth, “Structured Programming With Go To Statements”

Madison PHP Conference, September-October 2016 7www.phpbench.com


It Doesn’t Matter


We Are the 3%


Who is at fault?• 3rd Party Connections• I/O Performance• Database





4 Hours• Cache Ad Campaign Data• Cache Analytics Data• Run Numbers

• Sync Products


4 Hours• Cache Ad Campaign Data• Cache Analytics Data• Run Numbers

• Sync Products


The Problems


Running Numbers was heavy• Primary server would spike under load• Secondary servers would get out of sync and go into “rollback”


5:20am 5:25am 5:30am 5:35am 5:40am 5:45am0

500

1000

1500

2000

2500

3000Replication Lag

Lag in Seconds


5:20am 5:25am 5:30am 5:35am 5:40am 5:45am0

1000

2000

3000

4000

5000

6000Replication Lag

Lag in ms


Product Sync was Slow• Just took hours to run



Product Sync


Just Start Logging• Add DEBUG log messages with timestamps• Where is it slow?


Seldaek/monologuse Monolog\Logger;use Monolog\Handler\StreamHandler;

// create a log channel$log = new Logger(‘job_debug');$log->pushHandler(new StreamHandler('path/to/your.log', Logger::DEBUG));

// add records to the log$log->debug(date(‘Y-m-d H:i:s’) . ‘ – Contacted API’);// Do our business logic$log->debug(date(‘Y-m-d H:i:s’) . ‘ – Finished with page’);

CulpritsMadison PHP Conference, September-October 2016 25


What did we find?• All calls to Product API had to be full sets, couldn’t subset• Calls to Product API were slow, but not horrid• Generating and inserting the Products were slow due to business logic• Blocked Operations:

• Getting next page from API• Processing products


Our Workflow// Original WorkflowGet Page X from API

For Each Product:Extract Data from XMLTransmogrify the Data into a Product ObjectSave Object to DBIf No Next Page:BreakElse:Page++Continue


Solution – Out of Band Processing// Original WorkflowGet Page X from API



Solution – Out of Band Processing// Job 1 - Cache Product API CallsGet Page X…X+10 from API

Cache XML to DatabaseIf No Next Page:

BreakElse:

Page++Continue

Call Job 2Respawn Job


Solution – Out of Band Processing// Job 2 – Insert ProductsGet Page X…X+10 from DB



Run Totals



Background• Was originally PHP• Turned into a MongoDB Script because it was too slow


5:20am 5:25am 5:30am 5:35am 5:40am 5:45am0

500

1000

1500

2000

2500

3000Replication Lag

Lag in ms



What did we find?


Check the Server Metricshttps://aws.amazon.com/blogs/aws/new-cloudwatch-metrics-for-amazon-ebs-volumes/

SuspectMadison PHP Conference, September-October 2016 38


Our Solution – Throw Hardware At It


Our Solution – Throw Hardware At It• Increased IOPs on the SSD’s• Larger Instances on AWS


Our Solution – Move out of MongoDB• Rewrite the script back into PHP• Run in our worker system


The Result


The New Bug

It took 5 hours to run

SuspectMadison PHP Conference, September-October 2016 44




What did we find?


Code Profiling


xhprof• Low-cost dynamic analysis for PHP• PHP Extension• Store results in a DB• Has a pretty good GUI• https://www.digitalocean.com/community/tutorials/how-to-set-up-x

hprof-and-xhgui-for-profiling-php-applications-on-ubuntu-14-04

https://www.digitalocean.com/community/tutorials/how-to-set-up-xhprof-and-xhgui-for-profiling-php-applications-on-ubuntu-14-04

https://www.digitalocean.com/community/tutorials/how-to-set-up-xhprof-and-xhgui-for-profiling-php-applications-on-ubuntu-14-04


Pretty Graphs


Useful Metrics


What we find?• Hydrating objects was expensive• We were doing deep hydration, resulting in extra DB and hydration

calls• We had authentication checking happening in a loop, due to bad

logging code


The result?

It brought it down to around 3.5 hours


Valgrind• General programming tool for checking memory debugging, memory

leaks, and code profiling• Supported by xdebug• KCacheGrind/QCacheGrind to view output


Enable it in xdebugzend_extension=/usr/lib/php/20151012/xdebug.soxdebug.profiler_enable=1xdebug.profiler_output_dir=/var/www/tests/xdebug


Function Calls and Code Flow


What we find?• We were looping a lot• We were looping big loops inside small loops• We were looping through a lot of the same data multiple times


The Result – Reduce the Looping

Runtime was reduced to 30 minutes


Tips for Slow Code• Use Monolog to add Debugging messages• Use xhprof to profile “live” code• Use xdebug and Valgrind to get deeper profiling


Thank You!• https://github.com/dragonmantank• Author of “Docker for Developers”

• https://leanpub.com/dockerfordevs

• http://ctankersley.com• [email protected]• @dragonmantank

https://github.com/dragonmantank

https://leanpub.com/dockerfordevs

http://ctankersley.com/

mailto:[email protected]


Credits• Slide 13 – Andrei.D40 – Stacks of Books, Flickr• Slide 34 – Upper Snake River Valley Historical Society – 3339 loggin,

Flickr

Technology

Oh Crap, My Code is Slow - Madison PHP 2016