Extremum Properties of
Orthogonal Quotients Matrices
By
Achiya Dax
Hydrological Service, Jerusalem , Israel
e-mail: [email protected]
The Eckart – Young Theorem ( 1936 )
says that the “truncated SVD” matrix
Ak = Uk Dk VkT
solves the least norm problem
minimize F ( B ) = || A - B || F
2
subject to rank ( B ) k
( Also called the Schmidt-Mirsky Theorem. )
Ky Fan’s Maximum Principle ( 1949 )
S a symmetric positive semi-definite n x n matrix
Yk the set of orthogonal n x k matrices
1 + … + k = max { trace ( YkT S Yk ) | YkYk }
Solution is obtained for the Spectral matrix
Vk = [v1 , v2 , … , vk] .
giving the sum of the largest k eigenvalues.
Outline
* The Eckart – Young Theorem
and Orthogonal Quotients matrices.
* The Orthogonal Quotients Equality.
* The Symmetric Quotients Equality
and Ky Fan’s extremum principles.
* An Extended Extremum Principle.
The Singular Value Decomposition
A = U V T
= diag { 1 , 2 , … , p } , p = min { m , n }
U = [u1 , u2 , … , up] , U T
U = I
V = [v1 , v2 , … , vp] , VT
V = I
A V = U AT U = V
A vj = j uj , AT uj = j vj j = 1, … , p .
Low - Rank Approximations
Ak = Uk Dk VkT
Dk = diag { 1 , 2 , … , k } ,
Uk = [u1 , u2 , … , uk] , UkT Uk = I
Vk = [v1 , v2 , … , vk] , VkT Vk = I
The Eckart – Young Theorem ( 1936 )
says that the “truncated SVD” matrix
Ak = Uk Dk VkT
solves the least norm problem
minimize F ( B ) = || A - B || F
2
subject to rank ( B ) k
( Also called the Schmidt-Mirsky Theorem. )
Rank - k Matrices
B = Xk Rk YkT
where
Rk is a k x k matrix
Xk = [x1 , x2 , … , xk] , XkT Xk = I
Yk = [y1 , y2 , … , yk] , YkT Yk = I
The Eckart – Young Problem
can be rewritten as
minimize F ( B ) = || A - Xk Rk YkT || F
2
subject to XkT Xk = I and Yk
T Yk = I .
Theorem 1 : Given a pair of orthogonal
matrices, Xk and Yk , the related
“Orthogonal Quotients Matrix”
XkT
A Yk = ( xiTAyj )
solves the problem
minimize F ( Rk ) = || A - Xk Rk YkT || F
2
Notation:
Xk - denotes the set of all real m x k
orthogonal matrices Xk ,
Xk = [x1 , x2 , … , xk] , XkT Xk = I
Yk - denotes the set of all real n x k
orthogonal matrices Yk ,
Yk = [y1 , y2 , … , yk] , YkT Yk = I
Corollary 1 : The Eckart – Young Problem can be rewritten as
minimize F ( Xk, Yk ) = || A - Xk Rk YkT
|| F
2
subject to XkXk and YkYk ,
where Rk is the “Orthogonal Quotients Matrix”
Rk = XkT
A Yk .
The Orthogonal Quotients Equality
For any pair of orthogonal matrices,
XkXk and YkYk ,
|| A - Xk Rk YkT
|| F
2 = || A || F 2 - || Rk || F
2
where Rk is the orthogonal quotients matrix
Rk = XkT
A Yk .
Corollary : The Eckart – Young Problem
minimize F ( Xk, Yk ) = || A - Xk Rk YkT
|| F
2
subject to XkXk and YkYk .
is equivalent to
maximize || XkT
A Yk || F
2
subject to XkXk and YkYk ,
and the SVD matrices Uk and Vk solves both problems,
giving the optimal value of 12
+ 22 + … + k
2 .
Question : Is the related minimum problem
minimize || (Xk)T A Yk || F
2
subject to XkXk and YkYk ,
solvable ?
Using the Orthogonal Quotients Equality
the last problem takes the form
maximize F ( Xk, Yk ) = || A - Xk Rk YkT
|| F
2
subject to XkXk and YkYk .
Note that the more general problem
maximize F ( B ) = || A - B || F
2
subject to rank ( B ) k
is not solvable .
Returning to symmetric matrices
How we extend the
Orthogonal Quotients Equality
to symmetric matrices ?
The Spectral Decomposition
S = ( Sij ) a symmetric positive semi-definite n x n matrix
With eigenvalues n
and eigenvectors v1 , v2 , … , vn
S vj = j vj , j = 1, … , n . S V = V D
V = [v1 , v2 , … , vn] , VT V = V VT = I
D = diag { 1 , 2 , … , n }
S = V D VT = j vj vjT
Recall that
Rayleigh Quotient Matrices
Sk = YkT S Yk
play important role in
Ky Fan’s Extremum Principles .
Ky Fan’s Maximum Principle
S a symmetric positive semi-definite n x n matrix
Yk the set of orthogonal n x k matrices
1 + … + k = max { trace ( YkT S Yk ) | YkYk }
Solution is obtained for the Spectral matrix
Vk = [v1 , v2 , … , vk] .
which is related to the largest k eigenvalues.
Ky Fan’s Minimum Principle
S a symmetric n x n matrix .
Yk the set of orthogonal n x k matrices .
n - k+1 + … + n = min { trace ( YkT S Yk ) | YkYk }
Solution is obtained for the Spectral matrix
Vk = [vn-k+1 , … , vn] ,
which is related to the smallest k eigenvalues.
Question : Can we formulate the
Symmetric Quotients Equality
in terms of
trace ( Sk ) = trace ( YkT S Yk ) = yj
T S yj
The Symmetric Quotients Equality
S a symmetric n x n matrix
YkYk an orthogonal n x k matrix
Sk = YkT
S Yk the related “Rayleigh quotient matrix”
trace ( S - Yk Sk YkT
) = trace ( S ) - trace( Sk )
Corollary 1 : Ky Fan’s maximum problem
maximize trace (Yk S YkT )
subject to YkYk ,
is equivalent to
minimize trace ( S - Yk Sk YkT
)
subject to YkYk .
The Spectral matrix Vk = [v1 , v2 , … , vk]
solves both problems,
giving the optimal value of 1 + … + k .
Recall that : The Eckart – Young Problem
minimize F ( Xk, Yk ) = || A - Xk Rk YkT
|| F
2
subject to XkXk and YkYk .
is equivalent to
maximize || XkT
A Yk || F
2
subject to XkXk and YkYk ,
and the SVD matrices Uk and Vk solves both problems,
giving the optimal value of 12
+ 22 + … + k
2 .
Corollary 2 : Ky Fan’s minimum problem
minimize trace (Yk S YkT )
subject to YkYk ,
is equivalent to
maximize trace ( S - YkT
Sk Yk )
subject to YkYk .
The matrix Vk = [vn-k+1 , … , vn]
solves both problems,
giving the optimal value of n-k+1 + … + n .
Extended Exremum Principles
Can we extend these extremums
from eigenvalues of symmetric matrices
to singular values of rectangular matrices ?
Notation: 1 m* m , 1 n* n ,
Xm* - denotes the set of all real m x m*
orthogonal matrices Xm* ,
Xm* = [x1 , x2 , … , xm*] , Xm*T Xm* = I
Yn* - denotes the set of all real n x n*
orthogonal matrices Yn* ,
Yn* = [y1 , y2 , … , yn*] , Yn*T Y* = I
Notations :
Given Xm* Xm* and Yn* Ym* ,
the m* x n* matrix
( Xm*)TA Yn* = ( xiTAyj )
is called “Orthogonal Quotients Matrix” .
Notations :
The singular values of the
Orthogonal Quotients Matrix
( Xm*)TA Yn* = ( xiTAyj )
are denoted as
1 2 … k 0 , where
k = min { m*, n* } .
Questions :
Which choice of orthogonal matrices
Xm* Xm* and Yn* Ym* ,
maximizes (or minimizes ) the sum
(1)p + (2)p + … + (k)p
where p > 0 is a given positive constant .
An Extended Maximum Principle : The SVD matrices
Um* = [u1 , u2 , … , um*] and Vn* = [v1 , v2 , … , vn*]
solve the problem
maximize F ( Xm* , Yn* ) = (1)p + (2)p + … + (k)p
subject to Xm* Xm* and Yn* Yn* ,
for any positive power p > 0 ,
giving the optimal value of (1)p + (2)p + … + (k)p .
An Extended Maximum Principle
That is, for any positive power p > 0 ,
(1)p + (2)p + … + (k)p =
max{ (1)p + (2)p +…+ (k)p | Xm* Xm* and Yn* Yn*},
and the maximal value is attained for the matrices
Um* = [u1 , u2 , … , um*] and Vn* = [v1 , v2 , … , vn*] .
The proof
is based on “rectangular” versions of
Cauchy Interlace Theorem
and
Poincare Separation Theorem .
Corollary 1 : When p = 1 the SVD matrices
Um* = [u1 , u2 , … , um*] and Vn* = [v1 , v2 , … , vn*]
solve the “Rectangular Ky Fan problem”
maximize F ( Xm* , Yn* ) = 1 + 2 + … + k
subject to Xm* Xm* and Yn* Yn* ,
giving the optimal value of
1 + 2 + … + k .
Corollary 2 : When p = 1 and m* = n* = k
the SVD matrices
Uk = [u1 , u2 , … , uk] and Vk = [v1 , v2 , … , vk]
solve the maximum trace problem
maximize F ( Xk , Yk ) = trace ( (Xk)T
A Yk )
subject to Xk Xk and Yk Yk ,
giving the optimal value of
1 + 2 + … + k .
* See also Horn & Johnson, “Topics in Matrix Analysis”, p. 195.
Corollary 3 : When p = 2 the SVD matrices
Um* = [u1 , u2 , … , um*] and Vn* = [v1 , v2 , … , vn*]
solve the “rectangular Eckart – Young problem”
maximize F ( Xm* , Yn* ) = || (Xm*)T
A Yn* || F
2
subject to Xm* Xm* and Yn* Yn* ,
giving the optimal value of
(1)2 + (2)2 + … + (k)2 .
An Extended Minimum Principle
Question : Can we prove a similar
minimum principle ?
Answer : Yes, but the solution matrices
are more complicated .
An Extended Minimum Principle : Here we consider the problem
minimize F( Xm* , Yn* ) = (1)p + (2)p + … + (k)p
subject to Xm* Xm* and Yn* Yn* ,
for any positive power p > 0 .
The solution matrices are obtained by deleting some
columns from the SVD matrices
U = [u1 , u2 , … , um] and V = [v1 , v2 , … , vn] .
Summary
* The Eckart – Young Theorem
and Orthogonal Quotients matrices.
* The Orthogonal Quotients Equality.
* The Symmetric Quotients Equality
and Ky Fan’s extremum principles.
* An Extended Extremum Principle.