Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Introduction New Algorithm Final Remarks
Solving SDD Systems in Nearly-Linear Time
Pegah Pournajafi
École Normale Supérieure de LyonCR02 Combinatorial Scientific Computing
Lyon, Jan. 14, 2020
Pegah Pournajafi Solving SDD Jan. 14, 2020 1 / 44
Introduction New Algorithm Final Remarks
Table of Contents
1 Introduction
2 New Algorithm
3 Final Remarks
Pegah Pournajafi Solving SDD Jan. 14, 2020 2 / 44
Introduction New Algorithm Final Remarks
Table of Contents
1 Introduction
2 New Algorithm
3 Final Remarks
Pegah Pournajafi Solving SDD Jan. 14, 2020 3 / 44
Introduction New Algorithm Final Remarks
Ax = b
DefinitionAn n × n matrix A is symmetric diagonally dominant (SDD) if
AT = A,and Aii ≥
∑j 6=i |Ai j | for every i ∈ [n].
Definition
A function f (n) is called nearly linear, denoted by O(n) iff (n) = O(n poly log(n)).
Pegah Pournajafi Solving SDD Jan. 14, 2020 4 / 44
Introduction New Algorithm Final Remarks
SDD
Appears in applicationModeling electrical networksSimulating random walksImage Smoothing
Related to Laplacian matrices of graphs→ Graph AlgorithmsThe Jacobi and Gauss–Seidel methods for solving a linearsystem. . .
Pegah Pournajafi Solving SDD Jan. 14, 2020 5 / 44
Introduction New Algorithm Final Remarks
Previous Algorithms
Algorithms For SDD and Laplacian matrices:Iterative methods for PSD matrixes, e.g. Conjugate Gradient,Chebychev Method, . . . . mostly O(mn)Vaidya; 1990 : O(m1.75).Spielman, Teng; 2004 : O(mpoly log(n)).
Kelner, Miller, Peng; 2012 : O(m log n).
Kelner, Orecchia, Sidford, Zhu; 2013: O(m log2 n log 1ε ).
Pegah Pournajafi Solving SDD Jan. 14, 2020 6 / 44
Introduction New Algorithm Final Remarks
Advantages of This Algorithm
Why this new algorithm is good then?A new approachEasy to analyzeSimple and intuitiveAdaptableSize of words is log(n) not poly log(n)
Pegah Pournajafi Solving SDD Jan. 14, 2020 7 / 44
Introduction New Algorithm Final Remarks
Table of Contents
1 Introduction
2 New Algorithm
3 Final Remarks
Pegah Pournajafi Solving SDD Jan. 14, 2020 8 / 44
Introduction New Algorithm Final Remarks
The Problem
SDD Linear System
Given an n × n SDD matrix A, a non-zero vector b ∈ Rn, and apositive number ε we are going to solve the system
Ax = b
to find an approximation of the solution x s.t.
‖x − A†b‖A ≤ ε‖A†b‖A,
where ‖z‖A = ‖zT Az‖.
The following algorithm, solves a SDD linear system in nearly lineartime.
Pegah Pournajafi Solving SDD Jan. 14, 2020 9 / 44
Introduction New Algorithm Final Remarks
Graph Theory Background
Definition
Suppose G = (V ,E ,W ) is a weighted graph on n vertices and medges, and suppose W : E → (0,+∞) is its weight function. For anarbitrary orientation of edges of G, we can define its adjacency matrixU ∈ Rn×m, where the rows of U are indexed by the vertices and thecolumns of U are indexed by edges. If we denote
√W (e) by ρe, then
Uv ,e
−ρe if e = (v ,u) for some u ∈ V+ρe if e = (u, v) for some u ∈ V0 otherwise
Remark
We denote the Laplacian of G with L. Notice that L = UT U, no matterwhich orientation have been chosen.
Pegah Pournajafi Solving SDD Jan. 14, 2020 10 / 44
Introduction New Algorithm Final Remarks
The Algorithm: A Big Picture
SDD algorithm
Reduce the problem to Ly = d , where L is the Laplacian of aconnected graph.Represent the problem as Ly = UUT y = d .Construct a basis N for the null space of U.
Let K =
[UN
], and solve the system Kf =
[d0
](approximately).
Find y s.t. UT y = f (answer to Ly = d).Find x based on the reduction.Return x .
Pegah Pournajafi Solving SDD Jan. 14, 2020 11 / 44
Introduction New Algorithm Final Remarks
What to prove
So we need to prove that all the followings can be done in at mostnearly linear time:
1 Ax = b can be reduce to Ly = d .2 Kf = g can be solved.3 N can be found.4 y exists and can be found.
Pegah Pournajafi Solving SDD Jan. 14, 2020 12 / 44
Introduction New Algorithm Final Remarks
The Reduction
Given Ax = b , where A is SDD.Reduce it to Ly = d , where L is the Laplacian matrix of aconnected graph.
Pegah Pournajafi Solving SDD Jan. 14, 2020 13 / 44
Introduction New Algorithm Final Remarks
The Reduction
Reduction:Given SDD A and Ax = b
A = Ap + An + D1 + D2.
L =
(D1 + D2/2 + An −D2/2− Ap
−D2/2− Ap D1 + D2/2 + An
)Consider L
(x1
x2
)=
(b−b
)as Ly = d .
If(
x1
x2
)is a solution for Ly = d , then x1−x2
2 is a solution for Ax = b
If(
x1
x2
)is an ε-approximate solution for Ly = d , then x1−x2
2 is an
ε-approximate solution for Ax = b
O(nnz(A)) = O(m) time reduction.
L is 2n × 2n
Take connected components if necessary
Pegah Pournajafi Solving SDD Jan. 14, 2020 14 / 44
Introduction New Algorithm Final Remarks
Solving Kf = g: The Kaczmarz Method
We want to solve Kf = g.
K =
k1
k2...
km
and g =
g1g2...
gm
.
f ∗ is a solution of Kf = g ⇔ f ∗ ∈⋂m
i=1 〈ki , f 〉 = gi .
Pegah Pournajafi Solving SDD Jan. 14, 2020 15 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Kaczmarz Algorithm for solving Kf = g.
input: K ∈ Rm×m, g ∈ Rm
output: an approximation xT of the solution1 Pick some arbitrary f02 Pick some i ∈ {1, . . . ,m}3 update ft so it satisfies the i-th equation→ ft+1
"orthogonal projection"4 Repeat 2 and 3 for T times.5 Output fT
Pegah Pournajafi Solving SDD Jan. 14, 2020 16 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Question 1: Does it converge?
Question 2: What is the convergence rate?
Question 3: How to choose the hyperplanes?
Pegah Pournajafi Solving SDD Jan. 14, 2020 17 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Question 1: Does it converge?Yes, as long as every hyperplane is considered an infinite number oftimes.
Question 2: What is the convergence rate?
Question 3: How to choose the hyperplanes?
Pegah Pournajafi Solving SDD Jan. 14, 2020 17 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Question 1: Does it converge?Yes, as long as every hyperplane is considered an infinite number oftimes.
Question 2: What is the convergence rate?Could be slow. Bounds on the rate of convergence for general A existfor any deterministic hyperplane selection rule.
Question 3: How to choose the hyperplanes?
Pegah Pournajafi Solving SDD Jan. 14, 2020 17 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Question 1: Does it converge?Yes, as long as every hyperplane is considered an infinite number oftimes.
Question 2: What is the convergence rate?Could be slow. Bounds on the rate of convergence for general A existfor any deterministic hyperplane selection rule.
Question 3: How to choose the hyperplanes?Randomly.
Pegah Pournajafi Solving SDD Jan. 14, 2020 17 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Question 1: Does it converge?Yes, as long as every hyperplane is considered an infinite number oftimes.
Question 2: What is the convergence rate?Could be slow. Bounds on the rate of convergence for general A existfor any deterministic hyperplane selection rule.
Question 3: How to choose the hyperplanes?Randomly.
Question 4: From which probability distribution?
Pegah Pournajafi Solving SDD Jan. 14, 2020 17 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Question 1: Does it converge?Yes, as long as every hyperplane is considered an infinite number oftimes.
Question 2: What is the convergence rate?Could be slow. Bounds on the rate of convergence for general A existfor any deterministic hyperplane selection rule.
Question 3: How to choose the hyperplanes?Randomly.
Question 4: From which probability distribution?Probability of choosing the i-th hyperplane is proportional to ‖ki‖2.
Pegah Pournajafi Solving SDD Jan. 14, 2020 17 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Randomized Kaczmarz Algorithm for solving Kf = g.
input: K ∈ Rm×m, g ∈ Rm
output: an approximation xT of the solution1 Pick some arbitrary f0
2 Pick some i ∈ {1, . . . ,m} with probability distribution P(i) = ‖ki‖2
‖K‖2
3 update ft so it satisfies the i-th equation→ ft+1"orthogonal projection"
4 Repeat 2 and 3 for T times.5 Output fT
Pegah Pournajafi Solving SDD Jan. 14, 2020 18 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
What is the convergence rate of the randomized Kaczmarz?
DefinitionMoore–Penrose pseudo-inverseGiven matrix A, The inverse of the linear operator A, restricted to theimage of it, is well defined and is denoted by A+. This is called theMoore-Penrose pseudo-inverse of A.
Definition
The squared-Frobenius norm of a matrix A, denoted by ‖A‖2F , is
defined to be the sum all the entries of A.
Pegah Pournajafi Solving SDD Jan. 14, 2020 19 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
Definition
Given an m × n matrix A, let ‖A+‖2 be defined as follows
‖A+‖2 , supz∈Rn, Az 6=0
‖z‖2
‖Az‖2 .
The average condition number is defined to be
κ(A) , ‖A‖F‖A+‖.
Pegah Pournajafi Solving SDD Jan. 14, 2020 20 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method
TheoremThe Randomized Kaczmarz method for solving Kf = g, starting withf0, satisfies
‖fτ − f ∗‖2 ≤ ε2‖f0 − f ∗‖2
for τ = O(κ2(K ) log 1ε ) with probability at least 0.9.
Pegah Pournajafi Solving SDD Jan. 14, 2020 21 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method on this specific problem
Solving Kf = g in the algorithm:
K =
[UN
], where N is a basis for the null space of U.
Start with f0 s.t.
Kf0 =
[UN
]f0 =
[dz0
].
Run Kaczmarz.Claim: The following hold for each step:
Kft =[UN
]ft =
[d
zt → 0
].
Pegah Pournajafi Solving SDD Jan. 14, 2020 22 / 44
Introduction New Algorithm Final Remarks
The Kaczmarz Method on this specific problem
TheoremIf ft is the result of the t-th iteration of the Randomized Kaczmarzalgorithm on this specific problem, then
Kft =[UN
]ft =
[d
zt → 0
]where zt → 0 as t →∞.
"We converge on zt without changing Uft = d ."proof. On the board.
Pegah Pournajafi Solving SDD Jan. 14, 2020 23 / 44
Introduction New Algorithm Final Remarks
Each Iteration
On each iterationChosen Ki among N part,Time: O(poly log n) - not good.ft+1 = ft + αKi where
α =gi −
∑j Kij(f0)j −
∑j Kij(ft)j∑
j K 2ij
.
Some data structurerepresenting ft only on the U partevaluate
∑j Kij(ft)j is an easy way
Update ft+1 = ft + αKi
Output ffinal
Time: O(log n)
Pegah Pournajafi Solving SDD Jan. 14, 2020 24 / 44
Introduction New Algorithm Final Remarks
Number of Iterations
Kaczmarz algorithm never visits the U rows of K . It works onsum subspace of K .The number of iterations ≤ O(κ2(N))
In this case we can compute ‖N+‖2 as follows:
‖N+‖2 = supz∈Rm, Nz 6=0, Uz=0
‖z‖2
‖NZ‖2
Pegah Pournajafi Solving SDD Jan. 14, 2020 25 / 44
Introduction New Algorithm Final Remarks
Construction of N
In general, it is not easy to find the null space (Gaussianelimination)Here, we have an easy combinatorial way to construct N.This N has the property that κ(N)2 = O(m log(n)) which makesthe number of iterations of Kaczmarz good enough.
Pegah Pournajafi Solving SDD Jan. 14, 2020 26 / 44
Introduction New Algorithm Final Remarks
Construction of N
G: the graph corresponding to L = UUT
T : an spanning tree of GU =
[UT UN
], where UT represents T
For any e = (i , j) /∈ T , there is a unique path between i and jusing edges of TDenote this unique path by Pe
Denote the unique cycle Pe ∪ {e} by Ce
Pegah Pournajafi Solving SDD Jan. 14, 2020 27 / 44
Introduction New Algorithm Final Remarks
Construction of N
TheoremThere exists some WN , s.t. UN = UT WN .
proof. WN ∈ R(n−1)×(m−n+1).
(WN)e,e′ =
− ρe′ρe
if e ∈ Pe′ and e, e′ have the same direction on Ce′
+ρe′ρe
if e ∈ Pe′ and e, e′ do not have the same direction on Ce′
0 if e /∈ Pe′
Check UN = UT WN .
Pegah Pournajafi Solving SDD Jan. 14, 2020 28 / 44
Introduction New Algorithm Final Remarks
Construction of N
Theorem
N =[−W T
N I]∈ R(m−n+1)×m is a basis for the null space of U.
proof.
UNT =[UT UN
] [−WNI
]= −UT WN + UN = −UN + UN = 0.
Pegah Pournajafi Solving SDD Jan. 14, 2020 29 / 44
Introduction New Algorithm Final Remarks
Example
on the board.Electrical flow.
Pegah Pournajafi Solving SDD Jan. 14, 2020 30 / 44
Introduction New Algorithm Final Remarks
Choose a good N
Definition
Let T be a spanning tree of the graph G. Stretch of the edge e′ /∈ T ,is defined as
str(e′) =∑
e∈Pe′
(ρe′
ρe)2.
Also, we definestr(T ) =
∑e/∈T
str(e).
Pegah Pournajafi Solving SDD Jan. 14, 2020 31 / 44
Introduction New Algorithm Final Remarks
Choose a good N
Remember that the number of iterations is O(κ(N)2).We want to find a T s.t. κ(N)2 = ‖N‖2
F‖N+‖2 is small.
Pegah Pournajafi Solving SDD Jan. 14, 2020 32 / 44
Introduction New Algorithm Final Remarks
Choose a good T
Theorem
‖N+‖2 = supz∈Rm, Nz 6=0, Uz=0
‖z‖2
‖NZ‖2 ≤ 1.
proof. N =[−W T
N I]. z =
[zTzN
].
WNzN = zT . (cycle argument UZ = 0)
‖NZ‖2 = ‖ −W TN zT + zn‖2
= ‖W TN zT‖2 + 2zT
T WNzN + ‖zN‖2
= ‖W TN zT‖2 + ‖zT‖2 + ‖zN‖2
≥ ‖zT‖2 + ‖zN‖2 = ‖z‖2.
Pegah Pournajafi Solving SDD Jan. 14, 2020 33 / 44
Introduction New Algorithm Final Remarks
Choose a good T
Theorem
‖N‖2F = str(T ) + (m − n + 1).
proof.
‖N‖2F = ‖WN‖2
F + ‖I‖2F
=∑
e′ /∈T ,e∈Pe′
(ρe′
ρe)2 + (m − n + 1)
= str(T ) + (m − n + 1).
Pegah Pournajafi Solving SDD Jan. 14, 2020 34 / 44
Introduction New Algorithm Final Remarks
Choose a good T
Corollary
κ2(N) ≤ str(T ) + (m − n + 1).
So we have to find a low stretch tree.
Theorem (Abraham, Neiman; 2012)
For a graph G on n vertices and m edges, one can find a spanningtree T with str(T ) = O(m log log log n) in O(m log log log n) time.
They use Petal Decomposition in their algorithm.
Pegah Pournajafi Solving SDD Jan. 14, 2020 35 / 44
Introduction New Algorithm Final Remarks
y exists
Theorem
Let f be a solution for Kf =[UN
]f =
[d0
], then there is a y s.t.
UT y = f .
This y can be found easily due to the structure of U.
proof. on the board.
Pegah Pournajafi Solving SDD Jan. 14, 2020 36 / 44
Introduction New Algorithm Final Remarks
The Algorithm: In more details
SDD algorithm
Reduce the problem to Ly = d .Represent the problem as Ly = UUT y = d .Find a spanning tree of G with str(T ) = O(m log log log n).Using T, construct a basis N for the null space of U.
Let K =
[UN
], and consider the system Kf =
[d0
].
Solve the above system, using Kaczmarz method, withO(m log log log n) iterations.Find y s.t. UT y = f (answer to Ly = d).Find x based on the reduction.Return x .
Pegah Pournajafi Solving SDD Jan. 14, 2020 37 / 44
Introduction New Algorithm Final Remarks
Run Time of the Algorithm
SDD algorithm
Reduce the problem to Ly = d . O(m)
Represent the problem as Ly = UUT y = d . no costFind a spanning tree of G with str(T ) = O(m log log log n).
O(m log log log n)Using T, construct a basis N for the null space of U. O(m)
Let K =
[UN
], and consider the system Kf =
[d0
]. no cost
Solve the above system, using Kaczmarz method, withO(m log n log(n/ε)) iterations. O(m log n log(n/ε))×O(log(n))
Find y s.t. UT y = f (answer to Ly = d). < O(m)
Find x based on the reduction. no costReturn x .
Pegah Pournajafi Solving SDD Jan. 14, 2020 38 / 44
Introduction New Algorithm Final Remarks
Run Time of the SDD algorithm
The running time of the SDD algorithm is O(m log2(n) log(n/ε)).
Pegah Pournajafi Solving SDD Jan. 14, 2020 39 / 44
Introduction New Algorithm Final Remarks
Table of Contents
1 Introduction
2 New Algorithm
3 Final Remarks
Pegah Pournajafi Solving SDD Jan. 14, 2020 40 / 44
Introduction New Algorithm Final Remarks
Remarks
In the original paper, they have used electrical flows to explainand prove their theorem. - Similar to the example.This algebraic point of view of the algorithm is proposed by"Sivan Toledo, Haim Avron, and Haim Kaplan".Approach:
Representationexample: Cholesky factorizationIterationexample: Conjugate Gradient Method, Kaczmarz MethodCombination of the twoexample: this algorithm
Pegah Pournajafi Solving SDD Jan. 14, 2020 41 / 44
Introduction New Algorithm Final Remarks
Further Questions
Can the algorithm be parallelized?The constant in the running time is big. Could it become smaller?Can this method (combining good representation and gooditerative method) be used in other graph problems?
Pegah Pournajafi Solving SDD Jan. 14, 2020 42 / 44
Introduction New Algorithm Final Remarks
References
A Simple, Combinatorial Algorithm for Solving SDD Systems inNearly-Linear Time; Jonathan A. Kelner and Lorenzo Orecchiaand Aaron Sidford and Zeyuan Allen Zhu; 2013A talk by Sivan Toledo; available here.Lx = b Laplacian Solvers and Their Algorithmic Application;Nisheeth K. Vishnoi
Pegah Pournajafi Solving SDD Jan. 14, 2020 43 / 44
Introduction New Algorithm Final Remarks
Thank You.
Pegah Pournajafi Solving SDD Jan. 14, 2020 44 / 44