Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
goProbe: A Scalable Distributed Network Monitoring Solution
Christian DeckerLennart Elsen
Fabian KohnRoger Wattenhofer
Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks
Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks
Scalability
ReportingDebugging/Operations
? ??
StoragePacket Capture
Acquisition of Traffic Data
StoragePacket Capture
Grouping
Information Reduction
Acquisition of Traffic Data
NetFlow
Source IPDestination IPNext Layer ProtocolIPv4/6 Next Hop …
Source PortDestination Port…
Packet Size Number of PacketsSampling IntervalTTLInterface Name…
Field N Length
Field N Type
…
…
Field 2 Length
Field 2 Type
Field 1 Length
Field 1 Type
Count
…
System Uptime
Sequence #
NetFlow Version
Net
wor
kTr
ansp
ort
Met
a In
fo
NetFlow Packet
Packet aggregation by set of shared attributes
Network packet headers & packet counters
Expiry time
NetFlow
NetFlow Exporter
NetFlow Exporter
Network A
Network BNetFlow Collector
Source IPDestination IPNext Layer ProtocolIPv4/6 Next Hop …
Source PortDestination Port…
Packet Size Number of PacketsSampling IntervalTTLInterface Name…
Field N Length
Field N Type
…
…
Field 2 Length
Field 2 Type
Field 1 Length
Field 1 Type
Count
…
System Uptime
Sequence #
NetFlow Version
Net
wor
kTr
ansp
ort
Met
a In
fo
NetFlow Packet
NetFlow
NetFlow Exporter
NetFlow Exporter
Network A
Network BNetFlow Collector
Source IPDestination IPNext Layer ProtocolIPv4/6 Next Hop …
Source PortDestination Port…
Packet Size Number of PacketsSampling IntervalTTLInterface Name…
Field N Length
Field N Type
…
…
Field 2 Length
Field 2 Type
Field 1 Length
Field 1 Type
Count
…
System Uptime
Sequence #
NetFlow Version
Net
wor
kTr
ansp
ort
Met
a In
fo
NetFlow Packet
d
Analysts
Current Network Monitoring System
Single Host
ExporterDB
Query Tool
Queries
Aggregated Results
Flow Data
Request Traffic Metadata
Formatted Results
FastBitnProbe
nProbeFastBit
Query Tool
Challenges Capturing Process
nProbeFastBit
Query Tool
Challenges Capturing Process
Immense memory footprint
Challenges Capturing Process
FastBitQuery Tool
One process per capture interfacenP
robe
nPro
be
nPro
be
FastBitQuery ToolnP
robe
nPro
be
nPro
be
Challenges Storage Backend
Challenges Storage Backend
FastBitQuery ToolnP
robe
nPro
be
nPro
be
Inefficient memory management
Challenges Storage Backend
FastBit
Query ToolnP
robe
nPro
be
nPro
be
No data compression
Challenges Storage Backend
FastBit
Query ToolnP
robe
nPro
be
nPro
be
Long query execution times
Challenges
FastBit
Query ToolnP
robe
nPro
be
nPro
be
Poor Scalability
Reduced Flow Format
Src IP Dst IP IP Protocol Src Port Dst Port Packets
RcvdPackets
SentBytes Rcvd
Bytes Sent
Shared Attributes Counters
Reduced Flow Format
Src Port Dst Port
Shared Attributes Counters
Appl. Layer
Protocol
Deep Packet
Inspection
Reduced Flow Format
Src Port Dst Port
Deep Packet
Inspection
Appl. Layer
ProtocolDst Port
Source Port Aggregation
✗
Appl. Layer
ProtocolFlow in goProbe
Stored Flow
Collection of Flow Information — goProbe
goProbe
Written in Google Go
One capture routine per interface
Packet capture using modified libpcap
Database flush in regular intervals
TimerData Channel
Data Prepare
Local Database
Aggregation…
goProbe – Concept (Multiple Interfaces)
DB
Flow Table
Interface
How does it Compare?
Database Performance Evaluation
Reference DB
Runtime
CPU utilization
Disk I/O
Memory
7.8 GB
Aggregation Queries
Conditional Queries120 Million Entries
Data Read From Disk [MB]
FastBit
InfoBright EE
InfiniDB 1405
105
5617
350
74
2200
AggregationConditional
Runtime [s]
FastBit
InfoBright EE
InfiniDB 23
10
63
17
9
60
Reserved Memory [MB]
FastBit
InfoBright EE
InfiniDB 668
387
1399
630
351
3300
CPU Utilization [%]
FastBit
InfoBright EE
InfiniDB 83
213
17
302
352
23
Results
InfiniDBInfobright EE
$
File Based
Compression
Concurrency
Independent Processing
Tailored Column Store
goDB
Tailored Column Store — goDB
File Based
Compression
Concurrency
Independent Processing
Day 1
Destination IP
Source IP
Destination Port
IP Protocol
Appl. Layer Protocol
Bytes Received
Bytes Sent
Packets Received
Packets Sent
One File per Attribute
64
64
64
Day 1
Destination IP
Source IP
Destination Port
IP Protocol
Appl. Layer Protocol
Bytes Received
Bytes Sent
Packets Received
Packets Sent
One File per Attribute
64
64
64
172.0.50.4 | 10.30.0.3 | 8145 | 6 | 128 | 1024 | 1 | 8
Block-wise Writing and Reading
5 min 5 min 5 min
Attribute File
Block Timestamps
Length of Uncompressed Block
Position
Header
Compressed Block
Day
1D
ay d
…
Full
Dat
abas
eConcurrent Processing
Day
1D
ay d
…
Concurrent Processing
Day
1D
ay d
Worker 1
Worker dsip dip counters
sip dip counters
Partial Result Block i, Day 1
Partial Result Block j, Day d
Concurrent Processing
Decompress Aggregate
Day
1D
ay d
Worker 1
Worker d
Partial Result Block i, Day 1
Partial Result Block j, Day d
sip dip counters
sip dip counters
sip dip counters
Combined Result
Merge Routine
Concurrent Processing
Decompress Aggregate
Day
1D
ay d
Worker 1
Worker d
Partial Result Block i, Day 1
Partial Result Block j, Day d
sip dip counters
sip dip counters
sip dip counters
Combined Result
Merge Routine
Format Sort Limit
Concurrent Processing
Decompress Aggregate
Data Read From Disk [MB]
FastBit
goDB760
5617
494
2200
AggregationConditional
Runtime [s]
FastBit
goDB20
63
13
60
Reserved Memory [MB]
FastBit
goDB50
1399
47
3300
CPU Utilization [%]
FastBit
goDB123
17
237
23
How does it Compare?
Traffic Portfolio of an NGO Customer
Global Breakdown of PortsEx
tern
al T
raffi
cIn
tern
al T
raffi
c
Global Breakdown of PortsEx
tern
al T
raffi
cIn
tern
al T
raffi
c
HTTPS
HTTP
SMBDNS
Global Breakdown of Ports European Hub Traffic UsageEx
tern
al T
raffi
cIn
tern
al T
raffi
c
https://github.com/open-ch/
Conclusion
Improved capturing and flow logic
High performance DB written from scratch
Global deployment
Open source: