SOA-36: Tuning and Scalability for Your Sonic ™ Enterprise Messaging Andreas Gies Principal Architect

SOA-36: Tuning and Scalability for Your Sonic™ Enterprise Messaging

Andreas GiesPrincipal Architect

© 2008 Progress Software Corporation2 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging

D I S C L A I M E R

Setting Performance Expectations

System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment.

Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach.

D I S C L A I M E R


Agenda

Overview – Typical performance projects Discovery – Defining performance goals Benchmark – Testing performance Tuning – Improving results


Performance Engineering Terms

“System Under Test”

“Test Harness”

“Load” = “Sessions” * “Delivery Rate”

“Latency” = ReceiveTime – SendTime“Test Components”

“ExternalComponents”

“Platform” “System Metric”

R

V

“Variable” =client param,app param,system param

V

V

Tip: Limit scope to those test components thatare critical to performance and under your control


Performance project documents

Proposal• Goals and Scope• Architecture• Scenario definition

Project plan• Details on architecture and scenarios• Test cases• Schedule & logistics

Performance report• Executive summary• Test results• Conclusions


Common Pitfalls

Testing functionality, not performance No model for scalability Tuning decisions not applicable to production Insufficient time to test and tune Lack of Performance Engineering skills Inflexibility in exploring high-performance

architecture options

Rule of Thumb: Schedule a two hour discovery call to clarify customer’s understanding of performance issues.


The Performance Engineering Project

The Project is Goal Driven

Analyze

Test

Tune

For each iteration1. performance vs goal2. identify likeliest area for gain

Startup tasks1. success criteria2. resources & skills3. tool selection

Tip: Schedule daily meetings to share results andreprioritize test plan.


Performance Project Skills

SOLUTION

IntegrationExpert

TestingExpert

RequirementsExpert

Cos

t/Ben

efit

Load/Distribution

Design Options


Tools for a Messaging Benchmark

Platform

Component

System Under Test

TestHarness

TestConfigurator

TestAnalyzer


Agenda



Performance Project Specification

Rule of Thumb: Focus on message loads over 10 msg/sec,and process loads over 10,000 per hour.

Project spec outline: Project scope Goals and procedures Architecture Business scenarios Test cases

Business view• Value• Relevance• Risk

ESB goals• Throughput• Latency• Scalability

Architecture• Messaging• ESB Process• SOA Services


Specification: Platform Architecture

Back OfficeField DMZ

Platform architecture spec For each Machine:

• CPU number, type and speed• Disk / Memory

Network topology & specs


Specification: SOA architecture

PartnerField HQF

ina

nc

e

SF

A

CRM

ESBESB

Tra

ckin

g

Po

S

Partner ESB

SOA architecture spec Brokers and service containers Geographical distribution Scaling model


Specification: Data Architecture

Data Schemas 1…n Data Schemas 2…m

?

Tip: Transform tools vary in efficiency:XSLT – slowest (but most standard)DataXtend® SI much faster and most flexibleCustom service – fastest, but less flexibleData architecture spec

Key transform events Schema complexity Opportunities for caching Hi-load databases


Business Scenario patterns

Async Process

Realtime Feed

Request/Reply

Tip: The easiest way to manage decoupled SOA processes isthe Sonic ESB Itinerary.


Scenario Specification

Scenario spec details: Business process description QoS, availability, dependencies Goals and expected benefits

validate CreditCard Event

make a paymentEvent

Midas BillingSystem

Make Payment with Credit Card

make_payment (Progress Adapter)

CC Validate

Payementech(Mocked Object)

Make_payement API

Client Apps

is Billing System Up? Queue

yes

no

Credit Card SyntaxValidation

1

2

3

4

5

7-n

6

8

9

10

XML validation

7

Tip: The User’s language,The User’s diagram,

The User’s needs


Specifying Test Cases

Test case spec details: Scenario being simulated Load characteristics Basic ESB process model Services to deploy or simulate Variables manipulated Data measured

Inputs• Message• Protocol• QoS

ESB process• Service impl• Distribution• Params

Results• Throughput• Latency• Validation

Tip: Avoid application details; focus on performance.


Test Variables

Tip: External JMS client variables are easily managedwith the Test Harness.

Sonic ESB Service config Endpoints / Connections Process design XML operations

SonicMQ®

Connection / session Quality of Service Interaction mode Message def

Load Client sessions Messages / second Message size

Application Interaction pattern Service overhead Database interaction


Agenda



Configuring the ESB

Upload services• prototype services

• simulation services

Upload processes• test cases

• simulated external services

Other resources• external web services

• databases


Provisioning Brokers and Containers

Message brokers CAA Cluster / DRA ESB Containers & Connections

Tip: For each client connection scenario define a namedJNDI Connection Factory and document its characteristics.


Client Session Configuration

Rule of Thumb: Create one connection for every 10 to 20 sessions

Test HarnessQoS

Msg genreports

ESB

Reply


Configuring Client Quality of Service (QoS)

HTTP and Web service clients JMS Client

• Send mode• Ack mode• Flow control

ESB service JMS options Broker disk overhead

Tip: Best compromise is at least once QoS based on CAA, Persistent, Async disk and DUPS_OK.


Generating Message Content

StatusRequest

priority=2

org=1

Cust Info

gen random int

gen sample xml

Addr svc lookup

transform rule

Tip: Collect sample messages early on, to ensure thatrouting rules and lookups run as expected.


Simulating Clients with Test Harness

System Under Test

TestHarness

Connection

Broker

JNDI

MsgPool

Reply

Request

MessageGeneration


Running Performance Test Iterations

Logistics• multi-client• test intervals• Warm-up / cool-down

Data collection Repeatability

efficiency

relevance

accuracy

Rule of Thumb: Start with 10-20 intervals of 30 secs; if steady state is soonreached, reduce to 10 intervals of 10 seconds, plus warmup.


The Zen of Benchmarks

Spiral Up Childish curiosity Focus on goals Ensure repeatability Deal with now Big picture Use every resource Prioritize


Agenda



Performance Tuning with Sonic ESB®

Platform factors Top Ten Tuning Tips Tuning options

• Broker

• ESB


ESB System Usage: CPU

Sources of CPU cost:• Application code

• XML Parsing

• CPU cost of i/o– Network sockets– Log/Data disks

• Web service protocol– Web Service security

• Security– Authorization– Message or channel encryption


ESB System Usage: Disk

Sources of Disk overhead:• Database services

• Large File / Batch services

• Message recovery log

• Message data store

• Flow to disk overflow

Tip: To estimate best-case message log performance, usethe DiskPerf utility bundled with Test Harness.


ESB System Usage: Network

Sources of Network overhead:• Client requests• ESB Service invocations• ESB Web service callouts• CAA broker replication messages• Admin messages

Network performance limit:• Network bandwidth:

– Network Interface Card (NIC)– Switches, routers and firewalls

• Network load

Tip: To estimate best-case network performance, usethe DiskPerf utility bundled with Test Harness.


Tip #1: Increase sender and listener threadsto make service & client scale

Increase # Listeners for key entry endpoints Add more Service/Process instances Warning: Intra-Container ignores endpoint

• split scalable services into separate containers• turn intra-container messaging off• note: sub-Process is always intra-container

Container1

ServiceX

…

Container2

ServiceX

… …

Tip: Stop increasing Listeners when CPU usagenears maximum acceptable.


Tip #2: Implement optimal QoSto balance speed versus precision

ESBQoS

MQ

SettingMessage Loss Events

Duplicate

Msg Events

N/A DiscardableBuffer overflow,

Any crashMissed Ack, Crash of UoW

Best EffortNonPersistent,

DupsOK ack

Broker crash,

Client crashMissed Ack, Crash of UoW

At Least OncePersistent,

DupsOK ackNever

Missed Ack, Crash of UoW

Exactly Once Transacted Never Never

Tip: With CAA configured, Best Effort service is equivalentto At Least Once, with substantially lower overhead.

(Based on CAA brokers and fault-tolerant connections)


Tip #3: Re-use JMS objectsto reduce setup costs

Objects with client and broker footprint:• Connection• Session• Sender/Receiver/Publisher/Subscriber• Temporary destination

Tuning strategies:• Reuse JMS objects in client code• Share each Connection across sessions• Share Sessions across Producers and Consumers

– but not across JVM Threads• For low-load topics/queues

– Use Anonymous Producer – Use wildcard or multi-topic subscription

Rule of Thumb: Up to 500 queues per Broker and10,000 topics per broker.


Tip #4: Use intra-container service callsto avoid broker hops

BrokerDispatch Outbox

Marshall Msg

Send Msg

…

…Dispatch Outbox Unmarshall Msg

Marshall Msg Call onMessage

Send Msg …

Receive Msg

Unmarshall Msg

Call onMessage

Instantiate Proc

…

Inter-Container Messaging

Intra-Container Messaging

Tip: If you save service resources in sonicfs, the ESBcontainer will dynamically load and cache them.


Tip #5: Use NonPersistentReplicated modeto reduce disk overhead

Normal broker mechanisms require disk sync• Contributes to latency across the board

• Interferes with batching of packets

• Limited bandwidth

Disabling disk sync eliminates this overhead• Send mode NonPersistentReplicated

• Optional broker params to disable entirely

• WARNING: Log-based recovery will lose recent messages

• BUT: CAA failover will not

Rule of Thumb: Network overhead for Replication channel isabout ½ the Persistent Msg load of the broker.


Tip #6: Use XCBR instead of CBRto eliminate Javascript overhead

CBR rules implemented via Javascript• Dynamic change with complex rules

• Very high overhead for runtime engine

XCBR rules extract data fields for comparison• Only simple comparisons supported

• No script engine overhead

• Use message property data key for best effect

Rule of Thumb: Invocation of the Rhino JavaScript engine requiresabout 100 milliseconds CPU time on a typical platform.


Tip #7: Use message batchingto accelerate message streams

Message transfer overhead is generally fixed Hidden ack messages amenable to tuning:

• AsyncNonPersistent mode decouples ack latency• Transaction Commit allows 1 ack per N messages• DupsOK ack mode allows ‘lazy’ ack from consumer• Pre-Fetch Count allows batched transmit to consumer

ESB Design option: send one multi-part message instead of N individual messages

XML transforms and other services handle multi-record data efficiently

Producer Broker Consumer

Msg

Msg

…Ack

Msg

Msg

Ack

…


Tip #8: Minimize XML/SOAP operationsto avoid parsing overhead

Bypass SOAP and Web services processing• Use HTTP Direct Basic instead of SOAP or WS• Risk of invalid XML if source is unreliable

Combine multiple XML parsing steps into one• Save target XPath results as Message props• Also relevant for BPEL correlation ID’s

InputMessage

XMLTransform

CustomJAXB

InputMessage

propY=9propX=1

InputMessage

JAXB ObjMsg part

XCBR

JAXBService


Tip #9: Use high-speed encryptionto reduce security overhead

Default SSL encryption uses old RCA stack• At least 2X slower than more modern options

Change to any JSSE compliant stack:• Modify client DSSL_PROVIDER_CLASS to:

progress.message.net.ssl.jsse.jsseSSLImpl

• Change broker SSL provider from RSA to JSSE

Use more efficient cipher suites:• RSA_With_Null_MD5 is the smallest and fastest

Reduce broker memory overhead by deleting any unused ciphers


Tip #10: Use stream API'sto improve large message performance

SonicMQ Recoverable File Channels• Uses JMS layer to manage large file transfer• Queue-based initiation of transfer• High-speed JMS pipeline for blocks of data• Recovery continues at point interrupted

Sonic ESB open-source Large Message Service• Provides dynamic provisioning• Interacts with ESB processes

SonicStream API (version 7.5 or later)• Topic-based, pipeline into Java Stream api• No recovery


Broker Tuning Parameters

Core Resources:• JVM heap size• JVM thread, stack limits• DRA, HTTP and

Transaction threads• TCP settings

Message storage:• Queue size and save

threshold• Pub/sub buffers• Message log and store

Message management• Encryption• Flow control and flow-

to-disk• Dead message queue

management

Connections• Mapping to NIC’s• Timeouts• Load balancing

Rule of Thumb: For non-trivial queues, multiply defaultsettings by 10 to 100.


ESB Tuning Options

Load balancing and scalability of services:• Number of distributed service instances

• Number of service listener threads

Container Java VM heap size Intra-Container messaging Endpoint and connection parameters

• Same principles as JMS client

Tip: Start with small Java heap and only increase -Xms size if it improves performance.


Progress Software Developers Network:• http://www.psdn.com

• Search: “performance”, “sonic test harness”, “tuning”

SonicMQ Performance Tuning Guide Benchmarking Enterprise Messaging Systems white

paper Sonic Test Harness documentation Progress Professional Services

• Developer training courses

• Sonic Performance Analysis package

Other Performance Engineering Resources

No “Magic Bullet”, but plenty of places you can go for info:


Questions?


Thank You


Documents

SOA-36: Tuning and Scalability for Your Sonic ™ Enterprise Messaging Andreas Gies Principal Architect