View
1
Download
0
Category
Preview:
Citation preview
Software Defined Privacy-Preserving Measurement Instrument and Services
Yan Luo, Univ. of Massachusetts LowellCody Bumgardner, Univ. of KentuckyGabriel Ghinita, Univ. of Massachusetts BostonMichael McGarry, Univ. of Texas El Paso
Overview of IRNC AMIS ProjectDesign of IRNC AMIS Instrumentation Framework• Overview• AMIS measurement functions• Plugin management• Privacy preservation• Data analytics interface
Q&A
Outline
2IRNCAMIS,Internet2TechX,2016
Objectives• 40+Gbps flow-granularity network measurement
instrument• Software defined measurement• Preserving privacy of network flow info• In-depth flow analytics
NSF Funded Project Team: • Yan Luo, PI, University of Massachusetts Lowell• Cody Bumgardner, Co-PI, University of Kentucky• Gabriel Ghinita, Co-PI, Univ. of Massachusetts Boston• Michael McGarry, Co-PI, University of Texas El Paso
Overview of IRNC AMIS Project
3IRNCAMIS,Internet2TechX,2016
IRNC AMIS Deployment Plan
4IRNCAMIS,Internet2TechX,2016
Overview of IRNC AMIS Framework
5
PluginManagement
Web-basedNetworkData
VisualizationandAnalytics
PrivacyProtection
AMISMeasurementFunctions
UTEP1. Configureandmanage
measurementtasks2. Annotateinstrumentdata
withauxiliarydataforanalytics
3. Measurementdatavisualizationandanalytics
UKY1. Config management2. Dispactch mtask to AMIS nodes3. Query processing on netflow
records
UMB1. SyntacticPrivacy2. DifferentialPrivacywith
BigData tools
UML1. Netflow generation2. Packet tracing3. Packet validation4. Software defined
measurement5. Distributed
measurement
IRNCAMIS,Internet2TechX,2016
Current functions• Netflow
§ Generate NetFlow v5 record and push to NetFlow collector§ Analyze traffic characteristics from NetFlow records
• Packet Tracing§ Trace the occurrence of flows/packets on links monitored
by (distributed) AMIS instrument• Packet Validation
§ Validate the packets on the link (IP level validation) and report good/bad packets
Ongoing Work• Software Defined Measurement
§ P4 based protocol parser and matching table§ Measurement function composition
• Distributed and Collaborative Measurement
AMIS Measurement Functions
6IRNCAMIS,Internet2TechX,2016
AMIS Instrument Internal Architecture
7
1.NetFlow:OVS 2.PacketValidation:PCAP 3.PacketTracing:PCAP
veth0
veth1
PacketTracingCustomizedFunctions
pcap
Netflow
X86Server
Mellanox100GNIC
Packets
…
PacketValidation
veth2
veth3
OVSBridge
veth4
veth5
Controller
Policies&Rules
IRNCAMIS,Internet2TechX,2016
IRNC AMIS: Distributed Control System
Processing- Interaction DistributedControlSystem
-Project components managed through a distributed control system comprised of agents and plugins.
-Plugins implemented to manage control of data processing components.
-Agents allow for anonymous operation with centralized control.
8
Device View
9
mQueue_0
Probe
mQuery_0
Agent Controller
Central Services
Key:
-Control and Config Data
-Measurement Data
Service Module View:AMIS Device
mQuery_1mQueue_1
Measurements
Probe Agent
Local Processing
AMIS Device
lQueue_0
lQueue_1lQuery_1
Direct Client
eQueue_0
System View
10
lQueue_0
Probe Agent
Agent ControllerDashboard
/API
Query eQueue_1
Central Controller
Device 0
Key:
-Control and Config Data
-Measurement Data
lQueue_0
Probe Agent
Device 1
Remote Control and Aggregation
eQueue_0Direct Client
Privacy
OfflineModeOnlineMode
DifferentialPrivacyStrongProtectionModeratePerformanceStatisticalQueriesOnlyHighStorageRequirements
SyntacticPrivacyBest-effortProtectionFastPerformanceFlexibleQueryCapabilitiesLowStorageRequirements
AMIS: Supported Privacy Modes
11IRNCAMIS,Internet2TechX,2016
FlowWindow
FastIndexer
GeoMappingASMapping
k-anonymizerEngine
HilbertFractalsLinearRuntime
OnlinePrivacyModule
Sanitizedflows
Sensitive
Sanitized
Researcher
NetworkEngineer
Student
RabbitMQRawFlows
AMIS Privacy: Online Mode
12IRNCAMIS,Internet2TechX,2016
HBase Repository
DifferentialPrivacyEngine
HadoopAnalytics
OfflinePrivacyModule
Sensitive
Sanitized
QueryOptimizer
SanitizedQueryResults
RabbitMQRawFlows
Researcher
NetworkEngineer
Student
AMIS Privacy: Offline Mode
13IRNCAMIS,Internet2TechX,2016
Empower understanding of network exchange link utilization• who is communicating with who? what types of data are they communicating?• which institutions are utilizing the exchange links?
Empower network management (autonomous?)• FCAPS model of network management
• F – fault detection and correction– troubleshooting, monitoring, anomaly detection
• C – configuration and operation• A – accounting and billing• P – performance assessment and optimization
– performance monitoring, capacity planning
• S – security assurance and protection
Objectives for Visualization
14IRNCAMIS,Internet2TechX,2016
Most of our visualizations are driven by Netflowrecords
We annotate Netflow records with AS, geographic, and application data, Netflow+
We turn Netflow+ records into data viz objects to be rendered in the browser
Data Visualization Pipeline
15
AMISinstrument
AMISdatapipeline
URL:taskID
JSON:Ne1lowrecords
AMISNetFlowprocessing
RESTAPIURL:taskID
JSON:vizdata
RESTAPIWebserver
AMISNetFlowvisualiza<on
Webbrowser
AMISNetFlowannota<on
URL:taskID
JSON:NetFlowrecords
RESTAPIAMISdatapipeline
IRNCAMIS,Internet2TechX,2016
16IRNCAMIS,Internet2TechX,2016
17IRNCAMIS,Internet2TechX,2016
18
Thank You!
Q&A
IRNCAMIS,Internet2TechX,2016
Visualization of Anonymized Data
Hyper-RectangleView AnalyticsView
19IRNCAMIS,Internet2TechX,2016
Recommended