Upload
spark-summit
View
1.039
Download
1
Embed Size (px)
Citation preview
Monitoring Electronic Trading Environments using Spark
Fergal Toomey and Pierre LacaveCorvil
Monitoring Goals
•Performance and technical risk
•Client experience
•Compliance
Challenges
•Distributed across sites and firms
•Latency sensitive
•Multiple protocols and data formats
EXCHANGES
Matching Engine Trading
Gateways
Market Data
Validation/ Routing
Consolidation/ Distribution
MARKET DATA SERVICES
BROKERAGE SERVICES
TRADERS
•Transformations:•Change of custody•Different symbologies•Parent/child orders•Variable routing
Data Sources
Data Volumes
Platform
Corvil Streaming
DataCorvil
Streaming DataCorvil Streaming
Data
Message BusDistributed Real-Time Processing
StorageData Sources
Exploration and Visualization
Metrics Calculation
Windowing & Correlation
Raw Data•Enriched (correlated data)•Searchable
Custom Metrics•Latencies•Counts•Filterable by key dimensions•Rolled up
Clients MarketsA B C D
ID: 1
CLIENT: C1SYMBOL: ABC
ID: APARENTID: 1
SYMBOL: ABC
ID: ZPARENTID: A
SYMBOL: ABC
ID: 10PARENTID: Z
SYMBOL: ABCMARKET: NYSE
Clients Markets
Events Correlation
Filter by destination market in point A?
A B C D
ID: 1
CLIENT: C1SYMBOL: ABC
ID: APARENTID: 1
SYMBOL: ABC
ID: ZPARENTID: A
SYMBOL: ABC
ID: 10PARENTID: Z
SYMBOL: ABCMARKET: NYSE
Clients Markets
Events CorrelationA B C D
ID: 1
CLIENT: C1SYMBOL: ABCMARKET: NYSE
ID: APARENTID: 1
CLIENT: C1 SYMBOL: ABCMARKET: NYSE
ID: ZPARENTID: A
CLIENT: C1 SYMBOL: ABCMARKET: NYSE
ID: 10PARENTID: Z
CLIENT: C1SYMBOL: ABCMARKET: NYSE
Clients Markets
Events CorrelationA B C D
Events Windowing
Batch #1 Batch #2 Batch #3
window #1
window #2
A B C D
Spark Streaming window
Reconstruct batch based on original time
Spark Streaming micro-batch
Direct Data Access
Access metrics via SQL / JDBC
BI tools, self-developed application, etc
Calculation done in HBASE
Data Extracts
• Filtering pushed down to HBASE• Aggregation done In SPARK (SparkSQL)
CSV extraction to HDFS
Extract Consumption
THANK YOU.www.corvil.com