Upload
mavenwire
View
4.184
Download
4
Tags:
Embed Size (px)
Citation preview
OTM Performance Review and Benchmarking
OTM SIG User Conference ‘08
AgendaPerformance Analysis MethodologiesOur Holistic Bottom-Up ApproachBenchmarkingOther OptionsQ&A
MethodologiesThe “Top-Down” Approach
Great for determining the root-cause of specific performance issues.
The “Bottom-Up” ApproachGreat for doing a site review and identifying issues and bottlenecks (current and potential).
The Holistic ApproachEnsure that all components (technical, functional and external) are taken into consideration.
MavenWire’s Holistic Bottom-Up ApproachOur Approach
Hardware / PlatformsOperating SystemJava TuningApplication Server TuningOTM Thread TuningOTM Diag ServletsAgentsRatesItinerariesLogsBenchmarking
Hardware / PlatformsCPU and Hardware Platform Matters!CPU Speed – Not a Good Indicator of Performance
Other factors (cores, memory bandwidth, on-chip cache) necessitate benchmarking
OTM Requires both high multi-threading and high single-thread performance
Lots of cores and high per-core performance
Performance of Current PlatformsLinux / x86-64Windows / x86-64 (note: memory limitations)SolarisHP-UX / PA-RISC
Note: HP-UX / Itanium currently unknown
AIX / POWER
Operating System / StatsReview system performance under production load for the previous 2 weeksUtilize System Tools to Monitor
sar / kSartop / prstat / topas / etc
Utilize Tools to TrendNagios / Munin / etc
Oracle DBThe following have improved DB performance
Ensure you’re updating DB stats regularlyPatched to 10.2.0.3Partitioning is enabledCURSOR_SHARING from EXACT to SIMILAROPTIMIZER_MODE = CHOOSESTATISTICS_LEVEL = ALLIn some cases the following helped
OPTIMIZER_FEATURES_ENABLE = 9.2.0
Otherwise – Tune it like a normal Oracle DBStandard DBA skill set and best practicesUtilize tools like the Oracle DB StatspackIncrease memory (PGA / SGA / etc) allocationPin frequently used packages/procedures to memoryDecrease storage IO Wait (more spindles, etc)Separate out indexes, tablespaces, logs
Java TuningOTM is HIGHLY Dependent on JVM PerformanceJRockit Performs considerably faster than other JVMsMany Current-Generation JVMs Self-Tune (including JRockit)Platform Specific ParametersAllocate as much Memory as Possible
2-3GB depending on platformFor both Web and App server
Monitor Garbage CollectionMost frequent JVM performance issue
OTM v6.0 will utilize a 64-bit JVMNOT a silver bullet!
Application Server TuningShouldn’t need to tune Apache or TomcatWebLogic vs. OAS
OTM has been running on WL since 1999May no longer be an issue with BEA acquisition
WebLogic TuningUtilize the WebLogic Console
http://otmapp.mavenwire.com:7001/console
Number of Execute Threads$OTM_HOME/weblogic/config/gc3domain/config.xml.templateExecuteThreadsShould be 70-90, depending on load
Percentage of Socket Reader Threads$OTM_HOME/weblogic/config/gc3domain/config.xml.templateThreadPoolPercentSocketReadersMay have to increase to 75
DB Connection PoolNow Tuned within the OTM Application
OTM TuningEnsure You’re Running the Latest RUThread Tuning
Take your time – verify resultsTune iteratively – avoid contentionIf you don’t know what a thread pool does – ask firstPay attention to Queue Size and Wait Time
Temporarily Tune Threads via the EventDiagServlet
Careful about killing threads!
Permanently Set Threads via the glog.properties file
OTM Thread Tuning
No Bottleneck? Check the MediatorDiagServlet
OTM Diagnostic ServletsOTM Event Diag Servlet
http://otmweb.mavenwire.com/GC3/glog.webserver.event.EventDiagServlet
OTM Topic Queue Assignments Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.event.TopicQueueAssignmentsServlet
OTM Mediator Diag Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.mediator.MediatorDiagServlet
OTM Object Lock Diag Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.synch.object.ObjectLockDiagServlet
OTM Connection Pool Diag Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.datasource.CPDiagServlet
OTM Bean Cache Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.beancache.BeanCacheServlet
Agents
Try to model agents so only one agent fires for any given eventReview your saved conditions for optimal performanceLeverage other functionality than Agents –Auto Assignment Rules, Contact Notifications
Rates
Ensure that rate offerings that are expired are also inactiveIf you have rate offerings active ensure that rate records are associated to them.Avoid redundancy in your rate structure. If you rate offering can only handle once piece of equipment there is no need to also define it at the rate record level.
Itineraries
The more itineraries valid for a given move the longer OTM will take to plan. Test using the order planning action “Show Routing Options”Simply your itineraries where ever possible
Logs
To much logging turned on will impact performance.Turn logging on as needed to troubleshootLog features that can impact performance
SQLRatingEngineRatingEngineDetailRatingEngineDebugPersistence
MavenWire Benchmark SuiteSuite of Benchmarks compiled to allow comparative hardware / platform testingDoes not require OTM installation, decreasing setup complexity and testing timeFreely available to the OTM CommunityAll Benchmarks utilized are Open Source Software (OSS), allowing for free use and modification as necessaryFull Details, including download, installation, runtime and comparison data available at:
http://www.otmfaq.com/forums/blogs/chrisplough/
Benchmarking - VolanoMarkVolanoMark
Java-based benchmark that simulates high transactional and multi-threaded loadReflects the performance of the following OTM activities
Web UI, Agents, Integration, General Workflow, General OTM Activities (not including optimization and planning based)
Higher numbers are better
Benchmarking - DaCapoDaCapo
Java-based benchmark that simulates highly computational, algorithmic, single-threaded processingReflects the performance of the following OTM activities
Optimization and Planning / Bulk Planning
Lower numbers are better
Benchmarking – Soap StoneSoap Stone
Java-based benchmark that tests data throughput between servers and replicates application protocols, such as HTTP, RMI and RAW.Reflects the throughput and protocols utilized between the various OTM Tiers
Browser / Web: HTTPWeb / App: RMIApp / DB: RAW
Higher numbers are better
Benchmarking – HammeroraHammerora
Benchmark based on the TPC-C and TPC-H benchmarks.Reflects the performance and scalability of the DB TierLower numbers are better
Other OptionsWeb Tier
Load BalancingSSL AcceleratorWAN / Web App Accelerator
App TierOTM SCA
DB TierOracle RAC
Online ResourcesPerformance
kSarhttp://ksar.atomique.net/
Nagioshttp://www.nagios.org/
Muninhttp://munin.projects.linpro.no/
BenchmarkingFull Replication Details
http://www.otmfaq.com/forums/blogs/chrisplough/
VolanoMarkhttp://www.volano.com/benchmarks.html
DaCapohttp://dacapobench.org/
Soap Stonehttp://soap-stone.sourceforge.net/
Hammerorahttp://hammerora.sourceforge.net/
Q & A and Discussion
Questions?