MySQL Empowers Mission
Critical Financial Application ~ MySQL Case study in financial industry ~
Ryusuke KajiyamaMySQL Senior Evangelist, APACSun Microsystems / MySQL
2
Agenda
1. The project background
2. System requirements
3. System architecture
4. DBMS selection
5. Detailed design and benchmark
6. Post project review
3
1. The project background
2. System requirements
3. System architecture
4. DBMS selection
5. Detailed design and benchmark
6. Post project review
4
Project Overview
Target systemTarget system Front office support application for asset management
companies (investment trusts/investment advisors, etc.)System for fund managerssoftware-as-a-service model application
The purposes of systematizationThe purposes of systematization Timely front positions understanding and fund planningsTo enhance the ability to manage financial products in
accordanceTo integrate information from various sources, both in
and outside the companyTo focus on core operations as business efficiency
improves
5
The main functions of the system
Front-office position managementFront-office position managementFund and capital managementFund and capital managementContract managementContract managementUser managementUser management
Application functions
Data integration via messaging infrastructureData integration via messaging infrastructure (used for linkage with systems of other companies)(used for linkage with systems of other companies)Client application development frameworkClient application development framework (tailored for the individual needs of fund managers)(tailored for the individual needs of fund managers)
Infrastructure functions
6
1. The project background
2. System requirements
3. System architecture
4. DBMS selection
5. Detailed design and benchmark
6. Post project review
7
2. System requirements
Data VolumeData VolumeThe number of funds: 2,500+The number of balances reported by account/brand: 250,000+Transaction data volume: 1TB+
Function requirements Function requirements Advanced user interface
Graphs and other GUI components as well as a high operability of drag and drop, and other functionsPublishing infrastructure for data to be reflected in real-time
Real-time/batch data linkage between systemsMessaging/file transfer
Basic requirementsBasic requirements Small start & high scalability
In terms of costIn terms of system configuration
Consideration for multi-customers
8
2. System requirements
Fault-tolerance requirementsFault-tolerance requirements Avioidng single failure point (Server/network duplexing)If server down: recovery time within 10 minutes
Operational requirementsOperational requirements Weekdays: 6:00-20:00 (online)Other days: application batch/infrastructure batchAll maintenance operations are automated
Performance requirements (online)Performance requirements (online)Number of transactions per second: 200/secondResponse time: 3 seconds
9
1. The project background
2. System requirements
3. System architecture
4. DBMS selection
5. Detailed design and benchmark
6. Post project review
10
System configuration overview
Orchestration InfrastructureOrchestration InfrastructureOrchestration InfrastructureOrchestration Infrastructure
Workflow definitionWorkflow definition Flow controlFlow control
Presentation InfrastructurePresentation InfrastructurePresentation InfrastructurePresentation Infrastructure
Messaging infrastructureMessaging infrastructureMessaging infrastructureMessaging infrastructure
Service component groupService component group
Position manageme
nt
Position manageme
nt
Fund manageme
nt
Fund manageme
ntRisk
management
Risk manageme
nt
OMSOMS
Compliance
Compliance
Database service groupDatabase service group
BrandBrand
CharacteristicsCharacteristics MarketMarket
BenchmarkBenchmarkAccountAccount
CompanyCompany
RoutingRouting Database linkageDatabase linkagePublishingPublishing
Screen layout definition
Screen layout definition
Screen - data map
Screen - data map
Data set definitionData set
definition Data loadData load Data cacheData cache
------------
------------
------------
System evidence / system auditSystem evidence / system audit
11
Servers supporting the architecture are as follows:Usage Role
Presentation infrastructure
Aggregation server ・ Accepting client requests, and via the EMS server requesting the service group to deal with them
・ Delivering real data on the server side to clients
Messaging infrastructure
EMS server ・ Serving as a messaging infrastructure to enable smooth communications between servers which are loosely coupled
Service group (In this project, DB function is available in each service server.)
Position management / fund management server
・ Providing business logic relating to position management/fund management
・ Using same configuration for business logic server and DB server
Authentication server ・ Providing an authentication function
Other (key usages)
File linkage server ・ A server used for file linkage with external systems
External EMS server ・ A server used for real-time linkage with external systems
Shared disk ・ Storage for sharing business DB, authentication DB and others
Server configuration overview
12
Hardware configuration
This program exclusively employs IA-based servers.Usage CPU Memory HDD capacity
Presentation infrastructure
Aggregation server Dual core Xeon 5160 (3.00GHz) 4GB 146GB(RAID1)
Messaging infrastructure
EMS server Dual core Xeon 5160 (3.00GHz) 4GB 146GB(RAID1)
Service group (In this project, DB function is available in each service server.)
Position management / fund management server
Quad core Xeon X535 (2.66GHz) 8GB 146GB(RAID1)
Authentication server Dual core Xeon 5110 (1.60GHz) 4GB 146GB(RAID1)
Others
File linkage server Dual core Xeon 5110 (1.60GHz) 4GB 146GB(RAID1)
External EMS server Dual core Xeon 5160 (3.00GHz) 4GB 146GB(RAID1)
Shared disk The 2C2D (C: controller, D: disk enclosure) configuration with a capacity of 146GB×16 HDDs. Possible to increase capacity up to 25TB (146GB×28 HDDs). By adding enclosures, can be up to 50TB with 2C4D (56 HDDs)
1238GB (RAID 5)
SAN switch - - -
13
Software configuration
Messaging infrastructureMessaging infrastructure
Presentation infrastructurePresentation infrastructure
RedHat Linux ES 4.0
Apache2.0.54
JBoss AP Server 4.0.5
JDK 5.0
EMSClient API
Strus In-company F/W
Application
Aggregation server
RedHat Linux ES 4.0
Cluster middleware
EMS serverExternal EMS server
TIBCO EMS 4.4.0
Scale-out Scale-out
configurationconfiguration
Scale-up + HA Scale-up + HA
configurationconfiguration
14
Software configuration
Service group infrastructureService group infrastructure
Cluster middleware
MySQL5.0.40
JBoss AP Server 4.0.5
JDK 5.0
EMSClient API
In-company F/W
Application
Position/fund management serverAuthentication server
RedHat Linux ES 4.0
Scale-up + HA Scale-up + HA
configurationconfiguration
15
Screen shots
User registration screenUser registration screen
16
Screen shots
Front position management screenFront position management screen
17
Screen shots
Transaction progress management screenTransaction progress management screen
18
1. The project background
2. System requirements
3. System architecture
4. DBMS selection
5. Detailed design and benchmark
6. Post project review
19
First thoughts
Commercial product ?
MySQL ?
20
Thoughts on “commercial product”
Points: consPoints: consCost (1)
Software license fees/software maintenance fees are required.Cost (2)
Higher spec hardware may be needed to run heavy DBMS. Possible higher hardware costs
Cost (3)While existing know-how can be applied, designing and implementation tend to be time consuming
Points: prosPoints: prosBoth in and outside the company it has produced proven results:
Stable product qualityMany engineers with know-how are in SIerMore know-how of application and system design in SIer
The product architecture can accommodate a large-scale project
21
Thoughts on MySQL
Points: consPoints: cons Proven results are relatively limited in business systems
Main proven results are in reference systems.No proven results with NRI’s internal use in update systems.It is uncertain whether the requirements and practical use conditions of this project would be met.
Limited design know-howNot much experience in HA designDifficult to find technical experts in other div of NRI
Points: prosPoints: prosExpectations from simple architecture of MySQL:
Easier in designing and implementations phaseFewer errors in designing and implementationsLess time for designing and implementations
Low costsNo software license fee; only an inexpensive maintenance feeEasy designing means SE cost reductions can be expected.
22
Final decision “MySQL”
Reasons behind adopting MySQLReasons behind adopting MySQL
MySQL meets basic requirements of this project.MySQL enables a small start both in investment costs and
systems scale.MySQL is flexible to expand both in terms of investment costs
and scale. * Cost advantage is outstanding, including preparing dev
environment.Rapid system design and implementation
Ease of expansion meets requirements in this multi-user model system.
Simplicity of MySQL’s implementation process is impressive.Advanced technical support from NRI’s OSS support team
Quick trouble shooting can be expected.Many experienced engineers went through number of tough
projectsChallenging something new!!
Using only proven technology will rust knowledge and solution capability
23
Comments in executive review
Response measures based on the suggestionsResponse measures based on the suggestionsFor (1)
The lack of design know-how was complemented by support from NRI’s OSS Support team and MySQL. Followed by a DBMS-related professional review
For (2)A simple prototype application was developed to confirm that at least there was no problem with function/performance.
For (3)As a backup plan, in the case of the system being found infeasible, replace with a commercial product.
Suggestions in internal design review meetingSuggestions in internal design review meeting
(1) Review by MySQL professional for system design is required, because team did not have much experience of using MySQL in mission critical and high transactional environment.
(2) Feasibility test should be conducted using real environment.(3) If the system malfunctioned, backup plans should be
discussed for worst case scenarios.
24
1. The project background
2. System requirements
3. System architecture
4. DBMS selection
5. Detailed design and benchmark
6. Post project review
25
System design/operation design
Adopting the InnoDB storage engine to handle transaction dataSingle MySQL server instance to be shared for multiple companies, not running multiple instances for each companies on the single server
To reduce daily operational tasks, use partitions (backup, monitoring)Minimizing downtime through server
Storing data/index in the common tablespace file
Tablespace(single file viewed from the OS)
Tablespace(single file viewed from the OS)
DB for Company A
DB for Company B
DB for Company N
MySQL
Multi-users supportMulti-users support
26
Backup with “Snapshot”
Adopting “Snapshot” feature of the storage device【 Processing steps 】 (1) Storage management server issues a Snapshot command. A new tablespace is cloned in storage device.
TablespaceTablespace
DB for Company A
Shared storage
DB for Company B
TablespaceTablespaceSnapshotSnapshot
Position/fund management
server
Storage management
serverNFS media server
Snapshot commandSnapshot commandUnmountUnmount
Snapshot operation performedSnapshot operation performed
Backup operations (Step 1)Backup operations (Step 1)
27
Backup with “Snapshot”
(2) When Snapshot operation is done, the position/fund management server mounts the shared storage, in preparation for making service available; in addition, in preparation for backup, the NFS media server mounts the copy area.
Shared storage
Copied tablespaceCopied tablespace
Position/fund management
server
Storage management
serverNFS media server
MountMount MountMount
TablespaceTablespace
DB for Company A
DB for Company B
Available
Backup operations (Step 2)Backup operations (Step 2)
28
Backup with “Snapshot”
(3)The backup server issues a backup command to the NFS media server and executes “LAN-free backup” to tape library is performed.
Shared storage
NFS media server Backup server
MountMount
Copied tablespaceCopied tablespace
Tape library
Backup command
Backup operations (Step 3)Backup operations (Step 3)
29
Infrastructure test
The basic features as a DBMS meets the conditions of practical use.New features of MySQL 5.0 was not used in this project. e.g. view, stored procedure and trigger
Avoiding the use of brand-new features to reduce risks: the application can be implemented without such featuresPreparing for performance tuning: View will not be usedFacilitating operations and maintenance: Avoiding using stored procedure and trigger
After series of tests, stability of products, ease of restore + recovery and simplicity of restore + recovery process met our requirements.TIPS: When configuring HA cluster, do NOT use `mysqld_safe` shell script to start the server. -> In case of failure , both mysqld_safe and clustering tool will try to restart MySQL server and won’t make proper fail-over or cause unstable condition.
Evaluation of featuresEvaluation of features
30
Performance tests
Doing benchmark tests to find out basic performance of MySQL server. Tests included Import, Export and Create Index performance.Using production environment to find “real” performance.
Evaluation of performanceEvaluation of performance
Tests Purposes
Select (FULL Scan)
- To obtain basic performance values for full-table scan in MySQL
- To find the difference of execution time and resources usage on:
data size; concurrency; existence of cache in InnoDB buffer pool.
Select (Index Scan)
- To obtain basic performance values in index search in MySQL
- To find the difference of execution time and resources usage on:
data size; concurrency; existence of cache in InnoDB buffer pool.
Insert - To obtain basic performance values in data insert in MySQL
- To find the difference of execution time and resources usage on:
data size; concurrency; existence of cache in InnoDB buffer pool.
31
Performance tests
Data size: 1GB, 5GBConcurrency: 1, 4, 8, and 16Rebooting OS before each test caseIssuing two sets of queries in a row per concurrency
1st : Right after booting the MySQL server2nd: After a 60-second interval following the first query
The SQL statement issued is shown below. Full table scan is forced by the HINT
SELECT SUM(gid) FROM http_auth IGNORE INDEX (primary key) ;
To assess the execution time, the date command was executed before and after processing and the difference was measured.
Select (FULL Scan) performanceSelect (FULL Scan) performance
32
Performance tests
Data transfer rate of storage : 14.5[MB/sec].With 5GB, data transfer rate was accelerated to 59.7[MB/sec]
with smart storage controller.No difference was found relating to concurrency, probably
because of one-table access. The effectiveness of the cache mechanism was confirmed.
Select (FULL Scan) performance : resultsSelect (FULL Scan) performance : resultsData size Concurrenc
yTrial Execution
time (hh:mm:dd)
1GB 1 1st 0:01:372nd 0:00:00
4 1st 0:01:372nd 0:00:00
8 1st 0:01:362nd 0:00:00
16 1st 0:01:392nd 0:00:00
33
Performance test
CPU utilization (1GB, 1 concurrency pattern)In the 1st trial, the process proved to be I/O bound. In the 2nd trial, CPU utilization proved to be close to zero.
The 2nd trialThe 1st trial
Select (FULL Scan) performance : resultsSelect (FULL Scan) performance : results
34
Performance test
Available memory size (1GB, 1 concurrency pattern)In the 1st trial, approximately 1.4 GB memory was used, probably because the retrieval data was cached.In the 2nd trial, no change was found.
The 2nd trial
The 1st trial
Select (FULL Scan) performance : resultsSelect (FULL Scan) performance : results
35
Performance test
Disk read size: 1GB, 1 concurrency patternIn the 1st trial, a disk read was found to have occurred with an
average of approx 15[MB/s].In the 2nd trial, no disk read was found.
The 2nd trial
The 1st trial
Select (FULL Scan) performance : resultsSelect (FULL Scan) performance : results
36
Performance test
Data size: 1GB, 5GBConcurrency: 1, 4, 8, and 16Rebooting OS before each test caseIssuing two sets of queries in a row per concurrency
1st : Right after booting the MySQL server2nd: After a 60-second interval following the first query
The SQL statement issued is shown below. Executing the program by specifying a unique and random number for the WHERE statement
SELECT * FROM http_auth WHERE uid = ‘ xxx’ ;
To assess the execution time, the date command was executed before and after processing and the difference was measured.
Select (Index Scan) performanceSelect (Index Scan) performance
37
Performance test
Retrieval performance is processed at the high speed of 10[ms/sec]. A 5GB item of data is processed at a speed of 20-30 [ms/sec].An increase in concurrency results in almost no decrease in retrieval speeds.It was confirmed that the cache mechanism functions effectively.
Select (Index Scan) performance : resultsSelect (Index Scan) performance : resultsData size
Concurrency Trial Execution time (hh:mm:dd)
Ave. throughputs (TPS)
Ave. response time (ms/process)
1GB 1 1st 0:00:01 100 10
2nd 0:00:00 - -
4 1st 0:00:01 400 10
2nd 0:00:00 - -
8 1st 0:00:01 800 10
2nd 0:00:00 - -
16 1st 0:00:01 1600 10
2nd 0:00:00 - -
38
Performance test
CPU utilization (1GB, 1 concurrency pattern)In the 1st trial, it was found that only a minimal amount of resources was required to complete the process.In the 2nd trial, CPU utilization proved to be close to zero.
The 2nd Trial
The 1st Trial
Select (Index Scan) performance : resultsSelect (Index Scan) performance : results
39
Performance test
Available memory size (1GB, 1 concurrency pattern)In the 1st trial, a memory of approx. 11MB was utilized. Index and retrieved data are considered to have been cached.In the 2nd trial, no reduction in free memory and no increase in cache was found.
The 2nd trial
The 1st trial
Select (Index Scan) performance : resultsSelect (Index Scan) performance : results
40
Performance test
Disk read size (1GB, 1 concurrency pattern)In the 1st trial, a disk read was found to have occurred (approx. 13[KB/s] on average).In the 2nd trial, no disk read was found.
The 2nd trialThe 1st trial
Select (Index Scan) performance : resultsSelect (Index Scan) performance : results
41
Performance test
Data size: 1GB, 5GBConcurrency: 1, 5, and 10Rebooting OS before each test caseInserting one record = 1KBIn the case of 5 or 10 multiplexes, a specified data size is to be built in throughout the entire number of processes operated in parallel.
Example) in the case of a 5 GB data size 1 concurency : 1 thread (5GB/thread) Insert 5 concurency : 5 threads (1GB/thread) Inserts 10 concurency : 10 threads (500MB/thread) Inserts
To assess the execution time, the date command was executed before and after processing and the difference was measured.
Inserts performanceInserts performance
42
Performance test
Insert performance of 450-480[processes/second] with single thread.In 5GB test case, 500+ [processes/second]
Particularly noteworthy was that regardless of multiplex or DB size, there was no change in throughputs. Although in this test, commit processing was performed for each insert, the performance in this method is considered to be reaching the processing limit.
Inserts performance : resultsInserts performance : resultsData size
Concurrency Execution time (sec.)
Throughputs/thread
Ave. throughputs per threads (TPS)
Total throughputs (processes/second)
1GB 1 2212 452.1 452.1 452.1
5 2089 96.0 95.9 478.796.095.895.895.8
10 2095 47.8 47.8 477.247.847.847.847.847.847.747.747.747.7
43
Performance test
CPU utilization (1GB, 1 concurrency pattern)An average CPU utilization is 13% with a maximum utilization of
37%.It was confirmed that programmatically, a delayed write occurred
even after the completion of the Insert process. It can be assumed that the MySQL system performs commit processing.
Delayed write
Inserts performance : resultsInserts performance : results
44
Performance test
Available memory size: 1GB, 1 concurrency patternIt is considered that through the Insert processing, the data for Insert was cached.
A reduction of approx. 1GB due to
caching
Inserts performance : resultsInserts performance : results
45
Performance test
Disk read/write size: 1GB, 1 concurrency patternFew disk reads occurred. An average disk read speed was 14[MB/s] with maximum 17[MB/s]. The bottleneck can be caused by the disk I/O to the Binlog.It was confirmed that a delayed write occurred in disk I/O.
Actual DB is updated in bulk?
Inserts performance : resultsInserts performance : results
46
1. The project background
2. System requirements
3. System architecture
4. DBMS selection
5. Detailed design and benchmark
6. Post project review
47
Benefits of using MySQL
No license fees for MySQLNo license fees for MySQLMinimum investment: Low initial investment costs in this service
delivery business model is important In most of projects, license fee of DBMS is relatively large
portion in hardware and software costs.No additional fees: No fee for options, No upliftLower cost of expansion: No additional fee for more CPUs,
memory or clients
Stable and high product qualityStable and high product qualityMySQL showed high data processing performance not only in data
reference application, but also in writing application. With no product deficiency detected during development and
production phase.Easy to confirm “real” performance in early stage, because MySQL
does not require high spec servers and testing on dev server shows similar results on production server
48
Conclusion
There were no technical problems in functions of MySQLStable and high product qualityHigh quality technical support from Sun
Now we are seeing more use of MySQL in multiple industryincluding Telco, Finance, Manufacturing, Healthcare and Government.
MySQL is now better than commercial products in both quality and performance.
With the accumulation of best-practice and know-how, better professional services and technical support are available.
“You can find many systems which you don’t haveto pay expensive “DBMS Tax” and you can get a lot of
benefits with understanding MySQL more.”
“Do feasibility tests before jump into project with MySQL.”
MySQL is more than cheap & easy-to-use!MySQL is more than cheap & easy-to-use!
Time to use MySQL in mission criticalTime to use MySQL in mission critical
MySQL Empowers Mission
Critical Financial Application ~ MySQL Case study in financial industry ~
Ryusuke [email protected] Senior Evangelist, APACSun Microsystems / MySQL