1
Unveiling Anomalies in Large-scale Networks via Sparsity and Low Rank
Morteza Mardani, Gonzalo Mateos and Georgios Giannakis
ECE Department, University of Minnesota
Acknowledgments: NSF grants no. CCF-1016605, EECS-1002180
Asilomar ConferenceNovember 7, 2011
22
Context
Backbone of IP networks
Traffic anomalies: changes in origin-destination (OD) flows
Motivation: Anomalies congestion limits end-user QoS provisioning
Goal: Measuring superimposed OD flows per link, identify anomalies
by leveraging sparsity of anomalies and low-rank of traffic.
Failures, transient congestions, DoS attacks, intrusions, flooding
33
Model Graph G (N, L) with N nodes, L links, and F flows (F >> L)
(as) Single-path per OD flow xf,t
є {0,1}
Anomaly
LxT LxF
Packet counts per link l and time slot t
Matrix model across T time slots
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
f1
f2
l
4
Low rank and sparsity
X: traffic matrix is low-rank [Lakhina et al‘04]
A: anomaly matrix is sparse across both time and flows
0 100 200 300 400 5000
1
2
3
4x 10
7
Time index (t)
|xf,
t|
0 200 400 600 800 10000
2
4x 10
8
Time index(t)
|af,
t|
0 50 1000
2
4x 10
8
Flow index(f)
|af,
t|
55
Objective and criterion
(P1)
Given and routing matrix , identify sparse when is low rank
R fat but XR still low rank
Low-rank sparse vector of SVs nuclear norm || ||* and l1 norm
66
Distributed approach
Goal: Given (Yn, Rn) per node n є N and single-hop exchanges, find
Y=n
Nonconvex; distributed solution reduces complexity: LT+FT ρ(L+T)+FT
Centralized
(P2)
XR=LQ’Lxρ
M. Mardani, G. Mateos, and G. B. Giannakis, ``In-network sparsity-regularized rank minimization: Algorithms and applications," IEEE Trans. Signal Proc., 2012 (submitted).
≥r
77
Separable regularization Key result [Recht et al’11]
New formulation equivalent to (P2)
(P3)
Proposition 1. If stationary pt. of (P3) and ,
then is a global optimum of (P1).
88
Distributed algorithm
Network connectivity implies (P3) (P4)
(P4)
Consensus with neighboring nodes
Alternating direction method of multipliers (AD-MoM) solver
Primal variables per node n :
n Message passing:
1010
Attractive features Highly parallelizable with simple recursions
Low overhead for message exchanges Qn[k+1] is T x ρ and An[k+1] is sparse
FxF
Recap(P1) (P2) (P3) (P4)
CentralizedConvex
LQ’ fact.Nonconvex
Sep. regul.Nonconvex
ConsensusNonconvex
Stationary (P4) Stationary (P3) Global (P1)
Sτ(x)
τ
1111
Optimality
Proposition 2. If converges to ,
and , then:
i)
ii)
where is the global optimum of (P1).
AD-MoM can converge even for non-convex problems
Simple distributed algorithm identifying optimally network anomalies
Consistent network anomalies per node across flows and time
1212
Synthetic data Random network topology
N=20, L=108, F=360, T=760 Minimum hop-count routing
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
False alarm probability
Det
ecti
on p
roba
bili
ty
PCA-based method, r=5PCA-based method, r=7PCA-based method, r=9Proposed method, per time and flow
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Pf=10-4
Pd = 0.97
---- True---- Estimated
1313
Real data Abilene network data
Dec. 8-28, 2008 N=11, L=41, F=121, T=504
0100
200300
400500
0
50
100
0
1
2
3
4
5
6
Time
Pf = 0.03Pd = 0.92Qe = 27%
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
False alarm probability
Det
ecti
on p
roba
bili
ty
r=1, PCA-based methodr=2, PCA-based methodr=4, PCA-based methodProposed, per time and flow
---- True---- Estimated