Upload
distributed-matters
View
390
Download
0
Embed Size (px)
Citation preview
SCALED RISK
Next Generation of Financial Platform
NoSQL in Financial IndustryDistributed Matters - Barcelona – 21 November 2015
Pierre Bittner - CTO
SCALED RISK
SCALED RISK 2
Integrated Big Data & Analytics Platform
SaaS or On-Premise
Hadoop/HBase + Low latency + External Consistency+ Flexible Data Schema + In-Memory OLAP
WHAT?
HOW?
FOR? Real-Time Risk Management
WHERE?
What is Scaled Risk?
SCALED RISK 3
Why NoSQL Matters in Financial Industry?• Volume / Velocity
§ New York Stock Exchange generates about 4−5 terabytes of data per day.§ Algo Trading, High Frequency Trading: In 2012, accounted for 50% of all US equity trading volume.
Trade execution milli- and even microseconds.
ġ
E
Y
G
• Coherency / Availability / Security§ Regulatory Report: Intraday Monitoring,§ MTTR, Data Spikes on Market Event, Disaster Recovery, ACL
• Mixed workloads: Streaming and Historical Analysis – Point In Time Comparison§ BackTesting, Replay (UTC Timestamping of all events, FIFO)§ Lambda-architecture, Kappa-architecture
• Needs for Multi-tenancy / Data & Process Governance (Data Lake / Data Centric Arch.)
SCALED RISK 4
G
Y
E
Real-Time Enterprise-Wide Risk ManagementImproved and trustable view of global risk and support implementation of next regulations
Real-TimeFraud DetectionPre-check, real-time and historical data verification for trades, payments, orders, …
Real-Time Market Analytics
On-demand live and historicaldata analysis on global market
Why NoSQL Matters in Financial Industry?
Customer Story: Market Exchange
Market Surveillance
SCALED RISK
OTC Market• Foreign Exchange• Debt Market (Bond)• Commodities• Bloomberg, FXAll,…• …
Regulated Market• Securities• Options• NYSE, Eurex• LSE• …
Structured Data Feeds
Booking Systems• Trader Positions• Intraday Events• Valorization• Volatility, Correlation
Referential Data• Counterparts• Analytical Structure• Products Definition• Mappings
Unstructured Data FeedsNews & Mkt Analysis• Reuters, BBG• Research
Social Media• Twitter• LinkedIn, …
Trading• Global Positions• Intraday funding & forecasting• Collateral Optimization
RT Aggregated Positions
Sales• Credit Line• Profitability Indicator• Customer Interests
Global• Market Flows• Analyst/Market Correlation
On-Demand Analysis
Market Risk Analysis• Stress per Counterparty
Sales• Customer alerts on
Market Trends• Recommendation &
Lead Generation
Live Report & Alerts• On Market Events• Custom scenario• Market Surveillance
5
Today’s Trading Challenge: On-Demand Live Analysis & Alerts
Risk• CVA, Counterparty Exposure• Limit, Stress Test
Intraday Limit Risk• Automatic Monitoring
SCALED RISK 6
ContextExtreme performance and resilience : Peak activity > 1M order p. second Low Latency
ObjectiveOn-demand market analytics out of real-time & historical dataResilient primary storage
ProblemsHigh volumes, difficult access to historySLAs for data & service availability
Customer Story: On Demand Market Surveillance for Exchange
SolutionScaled Risk at the outflow of the matching engine
BenefitsStreamline process, consistent viewHigh availability and scalabilityReduced TCO
lResultA single system for storage and computation of spot & historical data for market surveillance
SCALED RISK7
On Demand Market Surveillance: Pilot PerimeterHigh Level Architecture Candidate ģ
SCALED RISK8
On Demand Market Surveillance: Pilot Perimeter
Focus on evaluating HBase frameworkåØ HBase performance on Read/WriteØ HBase behavior during a node failureØ HBase process isolationØ Global consistency
Key parts of the architectureåØ Message Bus (Kafka)Ø Storage System (HDFS)Ø Operational Database (HBase)Ø Real-Time Analytics tool (Scaled Risk)Ø History & Data Analytics tool (SR & Spark)
Benefits of architecture (streamline process, cost, …) not covered in this step.
Confirm Hadoop/HBase technical Stackå
Evaluate Scaled Risk performanceå
Explore Scaled Risk featureså
Pilot PerimeterSuitability of HBase and Scaled Risk in term of properties and performance.
Pilot duration : 2 months
SCALED RISK9
HBase: Random Access to your Planet-Wide Data
Key-value data organization per row. Table is a namespace.å
Each cell is timestampedå
ACID per row; Rowkey for fast access and data distribution å
HBase in few wordsHBase is an open-source, distributed, versioned, non-relational, scalable, wide-column data store.Ø It is the Hadoop database, leveraging mainly on HDFS.Ø Based on Google BigTable storage system.
Four primary operations are Get, Put, Delete and Scan å
Server-side operations with Coprocessor (Observer, Endpoint)å
Linear scalability, automatic sharding and failover supportå
Strictly consistentå
Hadoop ecostem integration (YARN), MapReduce, Hive, Sparkå
Phoenix for SQL Flavorå
SCALED RISK
NoSQL Wide Column Store Real-Time Distributed OLAP
• Dynamic Data Schema• Schema on read and write• Fast, Random R/W access
• Fast In-Memory Data Processing• Full Consistency; Linear Scalability• Open API (Valuation)
On-Demand Market Surveillance: Functional Architecture
10
Low
Lat
ency
Inte
rnal
Bus
Read
-Iso
latio
ns
As Of DateHBase As Storage
Inje
ctor
(Thr
ift)
• Advanced Index and search for Data Classification and Correlation
• Semantic reconciliation
Real Time Indexing
Real-Time Alerting
0
1
2
3
4
5
6
Contrat 1 Contrat 2 Contrat 3 Contrat 4
Alert on Analytics
Volume Matching Cancel Rate
Alert on Data
REST
/ A
PI /
Web
Soc
ket
SCALED RISK
On-Demand Market surveillance: Technical Architecture
11
Head NodeName Node
Head NodeSecondary Name Node
Head NodeHbase Master
Worker NodeRegion Server
Data Node
Worker NodeRegion Server
Data Node
Worker NodeRegion Server
Data Node
Worker NodeRegion Server
Data Node
Worker NodeRegion Server
Data Node
Worker NodeRegion Server
Data Node
HP Loader
3 x Hadoop Head nodes: HP ProLiant DL360 Gen9 Server8x 900GB 10k rpm SAS, 128 GB RAM, 2 x (10 cores) Intel Xeon CPU E5-2660 v3 @ 2.60GHz, 4 x 1GbE ports and 2 x 10GbE ports
6 x Hadoop worker nodes: HP ProLiant DL380 Gen9 Server2 x 120GB SSD OS,15 x 3TB 7.2k rpm SATA, 128 GB RAM, 2 x (10 cores) Intel Xeon CPU E5-2660 v3 @ 2.60GHz, 4 x 1GbE ports and 2 x 10GbE ports1 x HP Smart HBA H240ar, 1 x HP Smart HBA H240
1 x HP Loader:HP ProLiant DL380 Gen814 x 1TB 7.2k rpm SAS, 128 GB RAM, 2 x (10 cores) Intel Xeon CPU E5-2670 v2 @ 2.50GHz
Cluster size and componentsHadoop cluster details :• Hadoop HDFS usable size : 60TB (Block replication 3, no compression)• Hadoop HDFS data disk RAW size : 241TB• Hadoop cluster memory : 6 x 128GB = 768GB
Hadoop components and associated services• Hadoop Distribution : HortonWorks HDP 2.2 Stack• Cluster management : HP Insight CMU v7.3• Hdfs v2.6.0• Hbase v0.98.4• Zookeeper v3.4.6
Other details :• OS : RHEL - RedHat Enterprise Linux v6.5 – 64bit• Linux filesystem for Hadoop data : ext4• JVM used for Hadoop : Oracle Java 1.7.0_67
SCALED RISK 12
On Demand Market Surveillance : Functional Consistency• Market Exchange Data types
§ A unique Data flow containing all types of message§ Order messages§ Trade messages§ Test injector generates 1,5m in 7’ (client limitation)
E
Y
G• Scaled Risk Data exhaustiveness control
§ Dynamic data model with two tables§ Trade and Order messages are split§ Test method: Messages count
Message Type Message sub type Count
Order
New 792,546Replace 645,889Status 40,821Others 80
(unique order ids)792,886
Cancel n/a 680,626Trade n/a 137,573
• Order and Trade Life-cycle Control§ Message fields consistency control§ Test method: Data sampling
Message Type CountOrder Table 792,886Trade Table 137,573
Order Id Trader Contract Qty Price SideA 6C9 JFFCE150500000F 1 49350 BuyB W90 JFFCE150500000F 2 49350 SellC MAT JFFCE150500000F 1 49345 Buy
Trade Id Trader Contract Qty Price Side1630 6C9 JFFCE150500000F 1 49350 Buy1630 W90 JFFCE150500000F 1 49350 Sell1631 MAT JFFCE150500000F 1 49345 Buy1631 W90 JFFCE150500000F 1 49345 Sell
SCALED RISK 13
On Demand Market Surveillance : Performance Indicators
Sender/Trade (per region)• 130 K trades per second• 800 K on cluster
Test Scenario• 7 minutes• 1,479,335 messages• Stats only on Order Table
End-to-end• Nominal Latency ~200ms• 90% of messages with <412ms
SCALED RISK
On Demand Market Surveillance : Fault Tolerance TestHBase is designed to be fault tolerant. • A node fails when the white stripe appears on the
whole width of the graph.
• All nodes are impacted by the failure, and not only thekilled node (as expected remember CP).
• Another white rectangle is displayed before the nodefailure.
• It represents all the messages that have beencorrectly inserted before the failure, but never flushedto disk.
• Because the WAL is deactivated by trade injector(option), those messages were lost when regions weremoved from the killed node to other nodes.
X axis is the rowkey prefix, to show the distribution of insertion on the cluster. The Y axis is the time.Points displayed over the entire width of the X axis means that the distribution is correct.
SCALED RISK
On Demand Market Surveillance : Fault Tolerance Test
X axis is the rowkey prefix, to show the distribution of insertion on the cluster. The Y axis is the time.Points displayed over the entire width of the X axis means that the distribution is correct.
A second test confirms that HBase remains available even if a node fails.Test consists in inserting data in HBase from both YCSB and trade injector clients.• YCSB inserts data in a table distributed on 5 nodes• Trade injector inserts data in a table distributed on 4 nodes.• The node killed does not impact trades injection.
SCALED RISK
On Demand Market Surveillance: Next Steps
Deeper evaluation of HBaseImpact of volumes on performanceEvaluation of HA Region Servers for data access
Wider view of the targeted architectureOverall resilienceOverall latencySimplificationHot zone/Cold zoneTCO
Business requirements of the project:MIFID II impactNew services
SCALED RISK 17
Extreme flexibility thanks to our OLAP cube and Data Schema• 360 view of the position (As Of Date, explain, multi-aggregation level)• In-memory distributed calculation• Sub-second end-to-end (push architecture)
Low latency internal bus• UDP unicast, acknowledgement by UDP• No region location pain• Exactly once delivery, no message resent, multicast storm prevention
Resiliency• HBase RPC poll on message losses• HDFS message storage on overflood and region events
Overview of Scaled Risk implementationĦ Open Architecture
• Open Standards: seamless integration to HBase (coprocessor) • Open API (Valuation, FIFO), Toolkit approach
SCALED RISK
www.scaledrisk.com
SCALED RISK