BFS-US SSDU Research Best Strategies to avoid worst performance failures in Investment Banking Sujatha Badri Narayanan Karthik Authoor Venkata Infosys

BFS-US SSDU ResearchBFS-US SSDU Research

Best Strategies to avoid worst performance failures in Investment Banking

Sujatha Badri NarayananKarthik Authoor Venkata

Infosys Limited (NASDAQ: INFY)


Abstract

• Investment banking functionalities are grouped into front office, middle office and back office.

• Performance testing specifically related to trading applications within investment banking plays a vital role.

• Era has shifted from traditional manual trading in stock exchanges like NASDAQ, Globex to electronic trading.

• People trade millions of shares in micro seconds with various exchanges these days using electronic trading.

2


Why this Paper?“Every large system that works started as a small system that worked.” – Anonymous

“Fast, good, cheap: pick any two.” – Anonymous

“Lots of methodologies have come and gone, paradigms have changed but the requirements are always the same; Make it good and make it fast.” – Anonymous

In Electronic trading, complex heterogeneous systems are spread over diverse technologies with various platforms ranging from legacy mainframe to today’s web based technologies are used.

Today’s systems and interfaces are built with their own protocols both at application layer and network layer.

Even minor performance improvement in these systems will yield mammoth benefits leading to savings in millions of dollars.

In this whitepaper we will attempt to describe some of the good performance testing strategies that can be followed to immunize ourselves from disasters in the Investment banking domain specifically in the trading business.

3


Few Performance experiences as seen by User

4

Sl.No User experience Scenarios in the Trading Business

1 Session login failed.2 Market related data reaching the end user late due to network traffic rendering the info

useless3 Market data graphs failed to pull the real time data from the server

4 Viewing the detailed transaction statements on the web application giving distorted or half information due to delay in response from server

5 CMP(Current Market price) dashboard showing old market price for the stocks as the real time market update is slow

6 Placing a trade order from Stock trading web application during market mayhem (lot of trading activities happening at the same time). Server timed out.

7 Placed a trade order from web application during heavy traffic. Trade got executed but the confirmation never came back leaving the transaction in dilemma stage

8 Making a critical payment from bank web app to a third party using secure payment network. The money got debited from my account but due to heavy traffic on the third party server, the transaction failed.

9 Downloading “My” bank statement on the desktop which failed10 Making a mobile payment from network switches from 3G to Wireless and thus failing the

transaction


Proposed Strategies for better user experiences

Following are the key strategies proposed, that might help to provide solutions to some of the

performance issues that have been experienced by users as listed in the previous slide:

1) Performance Testing from an Architect’s View – Performance test driven

architecture

2) Infrastructure Optimization like Server, Database, Network etc -

3) Optimal combination and usage of tools based on technology used – Fiddler,

Web Load, VSTS, Jmeter, LR, Wily Introscope, Dynatrace, Shunra, Strobe, Xcap

etc.

4) Component based testing – Stressing the system at component level

5) Seamless Failover Testing – Time taken for switch over during disaster5


Strategy 5:

Seamless Failover Testing

Strategy 2:Infrastructure Optimization

Stra

tegy 4

:

Com

pon

ent b

ased

testing

Performance Strategy layers – An End-to-End View

6

Strategy 1 : Architect’s View

Strategy 3 : Optimal combination and usage of tools based on technology

Load Balanc

er

Web

Serv

er

Web

Serv

er

Ap

p S

erv

er

Ap

p S

erv

er

MF

DB


Strategy 1: Performance Testing from an Architect’s View

7

Life Cycle Phases Performance Strategy Activities

Requirements gathering / analysis / validation

System understanding NFR gathering Understanding NFR & validating them Modeling

o NFRo Workload

Architecture Analysis & Design Scalability assessment of the architecture Validation of the design from a Performance

stand point Capacity and sizing planning of the

infrastructure Performance Predictive Modeling Designing the performance Test strategy


Strategy 1: Performance Testing from an Architect’s View.. ContdFollowing are the performance test strategy activities (as mentioned against each

lifecycle phases in the previous slide) that can be leveraged

1) Performance Testing from an Architect’s View – Performance test driven

architecture

2) Infrastructure Optimization like Server, Database, Network etc -

3) Optimal combination and usage of tools based on technology used – Fiddler,

WebLoad, VSTS, Jmeter, LR, Wily Introscope, Dynatrace, Shunra, Xcap etc.

4) Component based testing – Stressing the system at component level

5) Seamless Failover Testing – Time taken for switch over during disaster

8


Strategy 2: Infrastructure Optimization

Why Infrastructure Optimization ?• Major component in an organization’s CAPEX• Level of optimization is a key goal for many organizations• Superior user experience which is the core business should also not get

affected

Most Commonly used servers in Investment Banking

Web Server : Tomcat, IISApp Server : Tomcat, IIS, IBM WebSphereDatabase : Oracle 10g, 11g, Microsoft SQL Server

9


Suggested optimizations on TomCat:

Have the latest OS patches, Service packs and JDKs installed as higher versions always have better performance and better security.

Have windows server instead of a non server version as non server version will have various limitations like number of users etc.

Connector tuning will yield good results – Generally Client creates the connection and once server sends response to the request connection is closed. One may land up opening 100s of connection to render a single page – Modify “MaxKeepAliveRequests” to a higher value option to avoid this.

Tomcat also has a thread pool. Make the best use of the same by adjusting minProcessors and maxProcessors values.10

Strategy 2: Infrastructure Optimization - Web Server


Suggested optimizations on IIS:

Configure the following parameters like Shutdown worker processes when idle for good performance

Managing log files will result in good performance in almost all the servers. Make sure that only required values are selected for logging and old log files are archived to avoid space issue. 11

Strategy 2: Infrastructure Optimization - Web Server … Contd

BFS-US SSDU ResearchBFS-US SSDU Research 12

Strategy 2: Infrastructure Optimization - Application Server

Suggested optimizations on WebSphere Application Server for better performance :

Modify Initial Heap Size and Maximum Heap size to 3096

Have Loglevel severity to “error” or “fatal”

“show application print statements” can be disabled

IBM Service Logs can be disabled (Process Defnitions -> Logging & Tracking -> IBM Service Logs)

Have a strategy around connection pool management in WAS


Strategy 2: Infrastructure Optimization – Database Server

Suggested optimizations on Oracle DB Server for better performance :

Ensure that the database is loaded to the required volume before Load testing.As in any other server system resources monitoring plays a vital role in DB server also.

Besides CPU, Memory, ensure that Disk I/ O is monitored. Ensure that all the disks are properly loaded.

Other major aspects of Oracle tuning are :- Denormalizing a database – If your database is doing too many joins then you

can consider this option to improve performance Indexing (B-Tree Concept)

- is used in databases to speed up retrieval process- is one of the most time tested technique that can be used to

improve performance Full Table Scan - it is better to go for full table scan for smaller tables


Strategy 2: Infrastructure Optimization – Database Server .. Contd

Tuning SQL itself is another interesting topic :

You can make use of AWR logs and other monitoring tools to know what queries take more time and what queries are called often and then select the queries that should be

tuned.

Following are few samples of SQL tuning : - If you are retrieving from multiple tables, the sequence of the tables matters

a lot for performance. Optimizers (if tuned properly – cost optimization)does help in making such decisions.

If it is a read only table read once and cache them so that we can avoid reading multiple times

Do not read the entire table when only few columns are required. DB filters can be handy here.


Strategy 3: Optimal combination and usage of tools

Selecting an optimal combination of tools plays a vital role in identifying bottlenecks.

While selecting load generation tools we may need to take into the account the following key factors:

Protocol Support required for your application like : Http, Win socket, SOAP, WAP, IMAP, POP3, EJB, RMI-Java, COM/DCOM, Tuxedo, Seibel, SAP R/3, Terminal Emulation, SSL, XML, PeopleSoft, VT100-520, ODBC, FTP, Streaming Media, LDAP, TCP/IP, UDP, MMS, Microsoft .NET SOAP Stack, CORBA, Oracle OCI, Ajax

Ease of Parameterization and Correlation Network address management User ramp up / down facility Scheduling tests License details Customer Support


Strategy 3: Optimal combination and usage of tools… Contd

While selecting monitoring tools we may need to take into the account the following key factors:

Technology support required for your application - .Net, J2EE, Mainframe etc Supported system resources monitoring Depth in analysis For e.g. does it help in identifying bottlenecks at code level ?

Query level ? License details Customer SupportThere are many Performance tools in the market but based on our experience in

Investment banking we find that the following tools are most commonly used.

Strategic Areas / Technology

.Net J2EE Mainframe

Load Generation Tools VSTS HP LR , Radview’s Web load, Compuware QALoad, Borland ‘s Silk Performer, Microsoft Web Application Stress Tool (WAS)

HP LR using RTE protocol

Monitoring Tools IIS Debug Diagnostics

HP Diagnostics, Wily Introscope, Dynatrace, Site scope

Strobe


Strategy 3: Optimal combination and usage of tools… Contd

There are two types of monitoring tools Agent based tools like HP Diagnostics ,Wily Introscope Agent less tools like Site Scope

Sample Agent based Monitoring Tool output – Wily Introscope

Detailed analysis up to code / query level can be done in agent based monitoring.


SLAs should be broken down during architecture phase for each of the components.

Components should be individually load/volume tested to make sure that they meet required SLA.

This will ease bottleneck identification and help in addressing the same at an earlier stage rather than struggling during final phase of the project.

18

Strategy 4: Component based Testing


In investment banking as availability at the right time matters most, failover testing is essential.

In failover testing , the behavior of the system will be tested by bringing down the servers in a logical way – web, app etc.

In many instances CPU utilization was high and later during analysis we realized that Load Balancers were not working as expected.

Make sure that your strategy covers testing load balancers also.

19

Strategy 5: Failover Testing


Designing for the right performance and ensuring that performance is engineered into the product / application is key.

Specifically in investment banking business , organizations cannot afford to incur higher expenditure on application failures and higher opex on unnecessary infrastructure, hence performance testing becomes an indispensable activity for optimizing the quality of the application

We recommend organizations to view performance quality from the bigger picture and adopt the best strategies which were discussed during these sessions so that business at large is benefited thereby making organizations less vulnerable to competitive pressures.

20

The Way ahead


Conclusion and Results

Performance testers should also focus on performance engineering standpoint in addition to load testing / volume testing strategies.

User experience Scenarios Strategy1

Session login failed.Load beyond expected may be the reason. Capacity Modeling could have avoided this situation – Strategy 1

2 Market related data reaching the end user late due to network traffic rendering the info useless

Response time not meeting SLAs – Strategy 2 & 3

3 Market data graphs failed to pull the real time data from the server

May be the server responsible for pulling the data is down. Strategy 5

4 Viewing the detailed transaction statements on the web application giving distorted or half information due to delay in response from server

Database Server response may be slow. Strategy 3

5 CMP(Current Market price) dashboard showing old market price for the stocks as the real time market update is slow

Possible reason could be bottleneck in one of the components – Strategy 4


Conclusion and Results … Contd

User experience Scenarios Strategy6 Placing a trade order from Stock trading

web application during market mayhem (lot of trading activities happening at the same time). Server timed out.

Load beyond expected may be the reason. Capacity Modeling / Proper Workload modeling could have avoided this situation – Strategy 1

7 Placed a trade order from web application during heavy traffic. Trade got executed but the confirmation never came back leaving the transaction in dilemma stage

Incomplete transaction due to poor response time. Possible reason could be bottleneck in one of the layers / components – Strategy 4

8 Making a critical payment from bank web app to a third party using secure payment network. The money got debited from my account but due to heavy traffic on the third party server the transaction failed.

Possible reason could be bottleneck in one of the layers / components – Strategy 4


References and Acknowledgements

References

http://msdn.microsoft.com

Tomcat – The definite guide by Jason Brittain & Ian F. Darwin

Performance Tuning for Apache Tomcat by Mark Thomas

Oracle Performance Tuning by Peter Corrigan and Mark Gurry

Performance Testing Advanced Certification material from Infosys

Acknowledgements

Vijay from Infosys Infrastructure Management Services

Sanjeeb Kumar from Financial Services team for sharing the user experiences

Finacle SPE team

http://msdn.microsoft.com/


Q&A:[email protected]

[email protected]

24

mailto:[email protected]

mailto:[email protected]

Documents

BFS-US SSDU Research Best Strategies to avoid worst performance failures in Investment Banking Sujatha Badri Narayanan Karthik Authoor Venkata Infosys