RightScale Conference Santa Clara 2011 - How do you tune application performance in the cloud? What if you want elasticity? Well talk through our experiences scaling and performance tuning the RightScale Cloud Management Platform in the cloud. Plus well share tips for sizing, auto- scaling, monitoring, and troubleshooting large-scale cloud deployments.
1. 1Optimizing Your Cloud Applicationsin RightScaleRafael H. SaavedraVP of Engineering, RightScaleWatch the video of this presentation
2. 2#Agenda Introduction 3-tier application architecture Vertical & horizontal scaling RightScale monitoring and cluster graphs New Relic RPM Support for optimizing DB performance Load testingReal Cloud Experience. Shared. 3. 3#Cloud computing characteristics Multi-tenancy Shared resource pooling Geo-distribution and ubiquitous network access Service oriented Dynamic resource provisioning Self-organizing Utility based pricingReal Cloud Experience. Shared. 4. 4#Cloud computing advantages No upfront investment Lowering operating costs Highly scalable Easy access Reduces business risk and maintenance costs Enables process automationReal Cloud Experience. Shared. 5. 5#3-tier application architecture Load balancers An array of application servers Master-slaveReal Cloud Experience. Shared. 6. 6Optimizing Your Cloud Applications in RightScaleVertical & Horizontal Scaling 7. 7#Cloud performance optimization Instance size (vertical scaling) Instance autoscaling (horizontal scaling) Server arrays RightScale support for performance optimization ServerTemplates are configured to capture performance data Collectd RightScripts Hardware & OS monitoring data Specialized plugins MySQL, HAProxy, Apache, NgInx, IIS, etc Monitoring graphs: individual, cluster, stacked, heat maps Alerts & escalations New Relic RPMReal Cloud Experience. Shared. 8. 8#Scaling up spectrum of instance sizes Compute units vs memory128.0m2.4xlarge 64.0m2.2xlarge 32.0 cc1/cg1.4xlarge m2.xlarge c1.xlargeMemory (GB) 16.0m1.xlarge Scalablem1.large Applications8.0High Performance4.0 Computingm1.small2.0c1.medium1.0 Test & Dev t1.micro0.50.05.010.0 15.0 20.0 25.0 30.0 35.0 40.0Compute UnitsReal Cloud Experience. Shared. 9. 9#Server arrays provide horizontal scalingReal Cloud Experience. Shared. 10. 10#Server arrays provide horizontal scaling The array scales up or down based on performance votes Tags allow scaling on an arbitrary decision set Decision threshold controls reaction time Sleep time allows new resources to have an impact Scaling can be time dependent Detailed setup instructions: http://bit.ly/c1oLr2 Fast response to changes in load conditions using alerts Allocation of servers to availability zones based on weights Deployment-based so configuration is consistent Arrays can be pre-scaled to support anticipated demandReal Cloud Experience. Shared. 11. 11 Optimizing Your Cloud Applications in RightScaleMonitoring & Cluster Graphs with RightScale 12. 12#Server monitoring graphsReal Cloud Experience. Shared. 13. 13#Cluster monitoring Individual graphs Good for a dozen servers Displays all standard graphs with full detail Stacked graphs Displays the contribution of many servers to a total Great to see the sum and variability of activity in a cluster Difficult to make out individual servers Examples: requests/sec, cpu busy cycles, I/O bytes/sec Heat maps Displays a bar for each server Great to see uneven distribution across servers Great to quickly spot performance problems across many servers Difficult to read absolute values or see the total cluster activityReal Cloud Experience. Shared. 14. 14#Cluster monitoring architecture Architecture Monitoring front-end serverspull data from storage servers Up to 100 servers on one graph(to be increased) monitoring monitoring storagefront-end serversservers your serversReal Cloud Experience. Shared. 15. 15#Cluster monitoring Current cluster monitoring: one graph per serverReal Cloud Experience. Shared. 16. 16#Stacked graphs Each color band shows contribution of one server Servers are stacked on top of one anotherReal Cloud Experience. Shared. 17. 17#Heat maps Each horizontal strip shows one server The color shows how hot the server is runningReal Cloud Experience. Shared. 18. 18#Heat map with 100 serversReal Cloud Experience. Shared. 19. 19#Stacked graph of the same 100 serversReal Cloud Experience. Shared. 20. 20 Optimizing Your Cloud Applications in RightScaleApplication Performance Analyticswith New Relic 21. 21#New Relic RPM Real-Time App Performance Analytics Supports Ruby, PHP, Java & .Net SQL & NoSQL performance Web transaction tracing Performance notifications Availability monitoring Scalability analysisReal Cloud Experience. Shared. 22. 22#New Relic RPM Direct access from RightScale dashboardReal Cloud Experience. Shared. 23. 23#New Relic RPM Historical statistics over a period of timeReal Cloud Experience. Shared. 24. 24#New Relic RPM Distribution of the most time consuming requestsReal Cloud Experience. Shared. 25. 25#New Relic RPM Statistics about response times from different countriesReal Cloud Experience. Shared. 26. 26#New Relic RPM Detailed response times by browserReal Cloud Experience. Shared. 27. 27#New Relic RPM 2 Examples An expensive query The N+1 query problemReal Cloud Experience. Shared. 28. 28 Optimizing Your Cloud Applications in RightScaleOptimizing Database Performance 29. 29#Optimizing DB performance RightScale MySQL ServerTemplates Configuration files tailored to instance size innodb_buffer_pool_size key_buffer_size thread_size sort_buffer_size The never ending task of identifying current bottlenecks Disk seeks Performance of disk operations Scale up when working set cannot fit in memory avoid active swapping Constant monitoring of performance graphs, logs and query Schema considerationsReal Cloud Experience. Shared. 30. 30#Schema considerations Lookups need to be indexed Sorting requires an index Joins need to be done on indices Become slower as tables grow Compounded indices should be used consistently Do not abuse indices Each index requires a disk write Compact tables if they become fragmented Deleted rows do not remove the corresponding index entriesReal Cloud Experience. Shared. 31. 31#Monitoring DB performance Standard collectd statistics User vs wait time (disk operations) Performance of disk operations Scale up when working set cannot fit in memory MySQL collectd plugin Monitor INSERT, SELECT, UPDATE operations The breakdown of read operations can indicate missing indices Monitoring /var/log/mysqlslow.log file Identify slow queries Use MySQL EXPLAIN command to identify query planReal Cloud Experience. Shared. 32. 32#MySQL Collectd Plugin Uses MySQL SHOW STATUS command to collect statistics A large set of counters that are divided into 10 categories Connections IO Requests Select Rates Read Rates Key Rates Commands Rates Query Cache Tables Memory Misc.Real Cloud Experience. Shared. 33. 33#MySQL Collectd Plugin Uses MySQL SHOW STATUS command to collect statisticsReal Cloud Experience. Shared. 34. 34#Mysqlslow.log & explain commandReal Cloud Experience. Shared. 35. 35#MySQL performance depends on locality Wait time should be minimum when working set fits in memory Performance degrades once wait time is significant wait time insignificant user time dominatesReal Cloud Experience. Shared. 36. 36#MySQL reads graphs Read-random-next represents a table scan Read-next represents an index scanReal Cloud Experience. Shared. 37. 37Optimizing Your Cloud Applications in RightScaleLoad Testing 38. 38#Load testing using httperf RightScale provides ServerTemplates in the marketplace https://my.rightscale.com/library/server_templates/Httperf-Load-Tester/24714 Tutorial on httperf setup and configuration http://support.rightscale.com/03-Tutorials/02-AWS/E2E_Examples/E2E_Gaming_Deployment/Adding_Httperf_Load_TesterReal Cloud Experience. Shared.