Embed Size (px)
Citation preview
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
SHOW ME THE METRICS
James Brooks
Born in 1999UK, Romania, Portugal, Ireland, Malta, Gibraltar, USAEngineering Blog: www.betsandbits.com700+ Engineers and Growing
ExchangeSportsbookGames
~1M active users~100M daily transactions~2.5Bn daily API calls~100k/s monitoring pointsE2E P95 transaction times <4ms
One of UK’s most successful .coms with ~£2Bn Market Cap2
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
OpenTSDB & OpenTSP at Betfair
3
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
4
What is OpenTSDB?
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
5
What is OpenTSDB?
A database optimised for storing and querying Time Series Metrics
os.filesystem.size
1413978032
912234340host=fe-108 cluster=bf_c mount=/
type=Free
Name Timestamp
Value
metadata
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
6
OpenTSDB Architecture
Hadoop
•TSDB Read•TSDB Write
Frontend Server
•TSDB Read•TSDB Write
Frontend Server
•TSDB Read•TSDB Write
Frontend Server
Load B
ala
nce
r
Metrics
Queries
TSDB
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
7
OpenTSDB Architecture
Metrics
Queries
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
8
OpenTSDB Architecture (ctd)
TSDB
Source :http://mo.nitor.me/hadoop-hbase-opentsdb/
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
10
OpenTSDB Architecture (ctd)
MetrilyxTSDB
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
11
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
12
OpenTSDB Architecture (ctd)
MetrilyxTSDB Visualiser
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
13
tcollector
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
14
OpenTSDB Architecture (ctd)
TSDBtcollector
collect-os
collect-jmx
tcollector
collect-os
collect-jmx
• TSDB Use Cases
• Troubleshooting / Root Cause analysis
• Performance Analysis
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
15
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
16
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
17
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
18
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
19
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
20
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
21
• TSDB Use Cases
• Troubleshooting / Root Cause analysis
• Performance Analysis
• Monitoring and Alerting
• Limitations
• Load
• Recency
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
22
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
23
• TSDB Use Cases
• Troubleshooting / Root Cause analysis
• Performance Analysis
• Monitoring and Alerting
• Limitations
• Load
• Recency
• TSP
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
24
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
25
OpenTSP
tcollector
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
26
OpenTSP
TSDBtcollector
collect-os
collect-jmx
forwarder
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
27
OpenTSP Architecture: Forwarder
TAG
FILTERcollect-statse
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
28
OpenTSP Architecture: Statse
application
collect-statse
countmeanmedia
np95p99
forwarder
collect-statse
countmeanmedia
np95p99
forwarder
application
collect-statse
countmeanmedia
np95p99
forwarder
application
poller
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
29
OpenTSP Architecture: Poller
collect-snmp
collect-f5collect-
netscaler
switchesroutersfirewalls
load balancersstorage
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
30
OpenTSP Architecture: Aggregator
forwarder
forwarder
poller
aggregator
collect-site
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
31
OpenTSP Architecture: Controller
controller
collectors
aggregator
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
32
[james@ohlookahostname~]$ CONTROL_HOST=tspconfig.inf.betfair tsp-forwarder -t{ "Filter": [ { "Match": [ "", "host", "^$" ], "Set": [ "", "host", “ohlookahostname.inf.betfair" ] } ], "LogPath": "/var/log/tsp/forwarder.log", "Relay": { "aggregator": { "DropRepeats": false, "Host": “tspaggregator.inf.betfair", "MaxConnsPerHost": 1, "OnQueueFull": "Drop" }, "tsd": { "DropRepeats": true, "Host": "tsdb.inf.betfair", "MaxConnsPerHost": 1, "OnQueueFull": "Drop" } }, "CollectPath": "/etc/tsp/collect.d"}
Visualiser
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
33
TSP at Betfaircon
troller collectors TSDB
Visualiser
tsdb2mysql
mysqld
feedhealth
Riemann
future?
aggregator
aggregator
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
34
Open Source• tsp-forwarder
• collect-statse• tsp-poller
• collect-f5• collect-netscaler• collect-snmp
• tsp-aggregator• tsp-controller
Apache Licence v2
http://github.com/betfair/opentsp
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
35
