Ricardo TorreProduct Strategist – BizTalk360
BizTalk Performance Deepdive
Understand architectural, design, and development patterns to improve BizTalk Server performance
Answer BizTalk performance related questions
Session Objectives and Takeaways
And a few other things
Development Optimizations
Optimize what?
Agenda
How to performance test, collect and analyse?
Architecture Optimizations
Optimize what?
Look at your architecture and identify optimization targets• BizTalk Applications• BizTalk Server• SQL Server• IIS Server• Windows Servers
How should I start?
Develop your applications with performance in mindDefine your testing strategy, tools and goalsCollect your baselineIdentify your current bottleneckChange somethingTestAnalyse
Use Visual Studio Load testing
Analyse during testing
Don’t be happy until you find a physical bottleneck
Development Optimizations
Adhere to BizTalk way of doing thingsDon’t work against it
Pipeline components
• Stream-based approach to pipeline components• Take advantage of BizTalk’s Streams
(Microsoft.BizTalk.Stream.dll)• VirtualStream and Event Streams• XPathMutatorStream• NamespaceTranslatorStream and XmlTranslatorStream
• Use PassThruReceive and PassThruTransmit pipelines whenever possible
• Promote items to the message context only if you need them for:• Message Routing (Orchestrations, Send Ports)• Demotion of message context properties (Send Ports)
• Instrument your source code to make your components simple to debug
• Avoid ordered delivery and transaction support whenever possible
Orchestrations
Tune the orchestration dehydration settings per the BizTalk Operations / Performance GuidesEliminate persistence points when possibleKeep orchestration state as small as possibleEliminate unnecessary context properties and distinguished fields
Use BizTalk Server 2013
XslTransform vs XslCompiledTransformA separate .NET class is generated for each transformation map.In most cases, the XslCompiledTransform class significantly outperforms the XslTransform class in terms of performance.Caveat with XslCompiledTransform: because the XSLT is compiled to MSIL, the first time the transform is run there is a performance hit, but subsequent executions are much faster.XslCompiledTransform class is the best choice in a "Load once, Cache, and Transform many times" scenario as the initial cost for map-compilation is highly compensated by the fact that subsequent calls are much faster.
BizTalk map (XslTransform) Test Case
1. One-Way FILE Receive Location receives a new CalculatorRequest xml document from the IN folder2. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb)3. The inbound request starts a new instance of the DefaultStaticLoop orchestration4. The orchestration executes a loop (1000 iterations) in which it uses a Transform Shape to apply a map to the inbound CalculatorRequest message (80KB)
XslCompiledTransform Test Case
1. One-Way FILE Receive Location receives a new CalculatorRequest xml document from the IN folder2. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb) 3. The inbound request starts a new instance of the CustomDynamicLoop orchestration4. The orchestration executes a loop (1000 iterations) in which it uses the XslCompiledTransformHelper static class to apply a map to the inbound CalculatorRequest message (80KB)
Test Results
BizTalk map (XslTransform) Test Case• Total Elapsed Time = ~57.3 seconds • Average Elapsed Time/Transformation = ~57 ms
XslCompiledTransform Test Case • Total Elapsed Time = ~3.6 seconds• Average Elapsed Time/Transformation = ~3.6 ms
Architecture Optimizations
Work with SME for each product in the stack
Optimize your architecture:• BizTalk Configuration Best practices• SQL Server/Disk Subsystem• IIS
Take the easy wins first
Apply all the BizTalk best practices
You will find the biggest performance improvements by removing logical bottlenecks
Be happy when you are CPU, IO or Memory bound
SQL Servers
The most common problem with SQL Server instances with BizTalk is PFS Contention
Don’t be afraid of using Multiple Message Boxes in a single SQL Instance
Invest on a good SAN
Multiple files and filegroups for the BizTalk MsgBoxDb
Primary (default file group)
Misc Data Misc IndexesPredicate
DataPredicate Indexes
Message Data
Message Indexes
For more information:BizTalk Server MessageBox Database Filegroups SQL Script
‘text in row’ in the MessageBox
Causes text, ntext, and image fields to be stored “in-row” instead of on a separate page in the tableMight be beneficial to enable this on the following tables in the MessageBox:• Parts• Spool• DynamicStateInfo_HostName
EXEC sp_tableoption N'Parts', 'text in row', '4000'
IIS Optimizations
Use Webgarden for IIS scalability
You can also tune the internal IIS engine for very specific
“UK Retailer” Compare Message Flow
“UK Retailer” Compare Hardware
2 BizTalk Servers: • 0.5 Business Transactions / second• 6.3 seconds avg. latency (5 sec wait time)
4 BizTalk Servers: • 0.69 Business Transactions / second• 11.2 seconds avg. latency
BUT…
“UK Retailer” Compare Results
Throughput will not increase, but we are not CPU or I/O bound… Why?
When a new row is inserted into an index, SQL Server will use the following algorithm to execute the modification: 1.Record a log entry that row has been modified. 2.Traverse the B-tree to locate the correct page to hold the new record. 3.“Latch” (lock) the page with PAGELATCH_EX, preventing others from modifying it. 4.Add the row to the page and, if needed, mark the page as dirty. 5.“Unlatch” the page
Eventually this can lead to “latch convoys” in highly concurrent workloads
Background – what is latch
Session ID Latch Type Database
Schema Object Name Index Name
Wait Type
241PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 109
273PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 63
348PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 63
177PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 62
302PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 62
311PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 62
108PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 47
110PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 47
122PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 47
166PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 47
234PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 47
300PAGELATCH_EX
BizTalkMsgBoxDb dbo
TrackingData_0_1
PK_TrackingData_0_1 47
294PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 46
107PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 31
111PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 31
121PAGELATCH_EX
BizTalkMsgBoxDb dbo Spool IX_Spool 31
Latch Convoy On Spool table
Contention on Small Number of Pages in Heavily Accessed Tables(Select, Update, Delete, Insert all running concurrently on multiple threads)
High rate of Insert/Select/Update/Delete access patterns against very small tables We have observed latch contention in MessageBox schema on 16 Cores and aboveThis problem is now being reported by production deployments, waits 100+msImpact: Latch waits > 50-100ms+Observed waits on internal SQL Structure• ACCESS_METHODS_HOBT_VIRTUAL_ROOT
WorkaroundMinimize load on MessageBox • App design to avoid persistence points• Use inline sends if necessary
Multi-MessageBox may alleviate this problem somewhat
BizTalk Workload Characteristics
Resources
Visual Studio Load TestingOrchestration ProfilerBizTalk Performance Optimization GuideBizTalk Benchmark Wizard
Q & A
Recommended