Embed Size (px)
Citation preview
Robust and Scalable Column/Row Sampling from Corrupted Big
Data
Mostafa Rahmani
University of Central Florida
George Atia
University of Central Florida
Abstract
Conventional sampling techniques fall short of draw-ing descriptive sketches of the data when the data isgrossly corrupted as such corruptions break the lowrank structure required for them to perform satisfacto-rily. In this paper, we present new sampling algorithmswhich can locate the informative columns in presenceof severe data corruptions. In addition, we developnew scalable randomized designs of the proposed algo-rithms. The proposed approach is simultaneously ro-bust to sparse corruption and outliers and substantiallyoutperforms the state-of-the-art robust sampling algo-rithms as demonstrated by experiments conducted usingboth real and synthetic data.
1. Introduction
Finding an informative or explanatory subset of alarge number of data points is an important task ofnumerous machine learning and data analysis appli-cations, including problems arising in computer vision[11], image processing [13], bioinformatics [2], and rec-ommender systems [17]. The compact representationprovided by the informative data points helps sum-marize the data, understand the underlying interac-tions, save memory and enable remarkable computa-tion speedups [14]. Most existing sampling algorithmsassume that the data points can be well approximatedwith low-dimensional subspaces. However, much ofthe contemporary data comes with remarkable corrup-tions, outliers and missing values, wherefore a low-dimensional subspace (or a union of them) may not wellfit the data. This fact calls for robust sampling algo-rithms, which can identify the informative data pointsin presence of all such imperfections. In this paper, wepresent an new column sampling approach which canidentify the representative columns when the data isgrossly corrupted and fraught with outliers.
1.1. Summary of contributions
We study the problem of informative column sam-pling in presence of sparse corruption and outliers. Thekey technical contributions of this paper are summa-rized next: I. We present a new convex algorithmwhich locates the informative columns in presence ofsparse corruption with arbitrary magnitudes. II. Wedevelop a set of randomized algorithms which providescalable implementations of the proposed method forbig data applications. We propose and implement ascalable column/row subspace pursuit algorithm thatenables sampling in a particularly challenging scenarioin which the data is highly structured. III. We developa new sampling algorithm that is robust to the simulta-neous presence of sparse corruption and outlying datapoints. The proposed method is shown to outperformthe-state-of-the-art robust (to outliers) sampling algo-rithms. IV. We propose an iterative solver for the pro-posed convex optimization problems.
1.2. Notations and data model
Given a matrix L, ‖L‖ denotes its spectral norm,‖L‖1 its ℓ1-norm given by ‖L‖1 =
∑
i,j
∣
∣L(i, j)∣
∣, and
‖L‖1,2 its ℓ1,2-norm defined as ‖L‖1,2 =∑
i
‖li‖2, where‖li‖2 is the ℓ2-norm of the ith column of L. In an N -dimensional space, ei is the ith vector of the standardbasis. For a given vector a, ‖a‖p denotes its ℓp-norm.For a given matrix A, ai and ai are defined as the ith
column and ith row of A, respectively. In this paper, Lrepresents the low rank (LR) matrix (the clean data)with compact SVD L = UΣVT , where U ∈ R
N1×r,Σ ∈ R
r×r and V ∈ RN2×r and r is the rank of L. Two
linear subspaces L1 and L2 are independent if the di-mension of their intersection L1 ∩ L2 is equal to zero.The incoherence condition for the row space of L withparameter µv states that max
i‖eTi V‖ ≤ µvr/N2 [7].
The proofs of all the lemmas are provided in the sup-plementary material. In this paper (except for Section
43211818
5), it is assumed that the given data follows the follow-ing data model.
Data Model 1. The given data matrix D ∈ RN1×N2
can be expressed as D = L+S. Matrix S is an element-wise sparse matrix with arbitrary support. Each ele-ment of S is non-zero with a small probability ρ.
2. Related Work
The vast majority of existing column sampling algo-rithms presume that the data lies in a low-dimensionalsubspace and look for few data points spanning thespan of the dominant left singular vectors [16,31]. Thecolumn sampling methods based on the low ranknessof the data can be generally categorized into random-ized [8–10] and deterministic methods [3,12,13,15,20].In the randomized method, the columns are sampledbased on a carefully chosen probability distribution.For instance, [9] uses the ℓ2-norm of the columns, andin [10] the sampling probabilities are proportional tothe norms of the rows of the top right singular vectorsof the data. There are different types of determinis-tic sampling algorithms, including the rank revealingQR algorithm [15], and clustering-based algorithms [3].In [12,13,20,22] , sparse coding is used to leverage theself-expressiveness property of the columns in low rankmatrices to sample informative columns.
The low rankness of the data is a crucial requirementfor these algorithms. For instance, [10] assumes thatthe span of few top right singular vectors approximatesthe row space of the data, and [12] presumes that thedata columns admit sparse representations in the restof the data, i.e., can be obtained through linear combi-nations of few columns. However, contemporary datacomes with gross corruption and outliers. [22] focusedon column sampling in presence of outliers. While theapproach in [22] exhibits more robustness to the pres-ence of outliers than older methods, it still ends upsampling from the outliers. In addition, it is not robustto other types of data corruption, especially element-wise sparse corruption. Element-wise sparse corruptioncan completely tear existing linear dependence betweenthe columns, which is crucial for column sampling al-gorithms including [22] to perform satisfactorily.
In [29], we proposed a new data sampling tool,dubbed Spatial Random Sampling (SRS). In contraryto the other column sampling methods, SRS does notrely on the low rankness of data and it is guaranteedto preserve the spatial distribution of data since it per-forms uniform random sampling in the spatial domain.In this paper, it is assumed that the sparse corruptionis sever such that it disturbs the spatial distribution oftrue data.
3. Shortcoming of Random Column Sam-pling
As mentioned earlier, there is need for sampling al-gorithms capable of extracting important features andpatterns in data when the data available is grossly cor-rupted and/or contains outliers. Existing sampling al-gorithms are not robust to data corruptions, hence uni-form random sampling is utilized to sample from cor-rupted data. In this section, we discuss and study someof the shortcomings of random sampling in the contextof two important machine learning problems, namely,data clustering and robust PCA.
Data clustering: Informative column sampling is aneffective tool for data clustering [20]. The representa-tive columns are used as cluster centers and the data isclustered with respect to them. However, the columnssampled through random sampling may not be suit-able for data clustering. The first problem stems fromthe non-uniform distribution of the data points. For in-stance, if the population in one cluster is notably largerthan the other clusters, random sampling may not ac-quire data points from the less populated clusters. Thesecond problem is that random sampling is data inde-pendent. Hence, even if a data point is sampled froma given cluster through random sampling, the sampledpoint may not necessarily be an important descriptivedata point from that cluster.
Robust PCA: There are many important applica-tions in which the data follows Data model 1 [4, 6, 7,18,19,27,30,32–34]. In [7], it was shown that the opti-mization problem
minL,S
λ‖S‖1 + ‖L‖∗ s. t. L+ S = D (1)
is guaranteed to yield exact decomposition of D intoits LR and sparse components if the column space (CS)and the row space (RS) of L are sufficiently incoherentwith the standard basis. However, the decompositionalgorithms directly solving (1) are not scalable as theyneed to save the entire data in the working memoryand have O(rN1N2) complexity per iteration.
An effective idea to develop scalable decompositionalgorithms is to exploit the low dimensional structureof the LR matrix [23–25,28]. The idea is to form a datasketch by sampling a subset of columns of D whose LRcomponent can span the CS of L. This sketch is decom-posed using (1) to learn the CS of L. Similarly, the RSof L is obtained by decomposing a subset of the rows.Finally, the LR matrix is recovered using the learnedCS and RS. Thus, in lieu of decomposing the full scaledata, one decomposes small sketches constructed fromsubsets of the data columns and rows.
43221819
Since existing sampling algorithms are not robustto sparse corruption, the scalable decomposition al-gorithms rely on uniform random sampling for col-umn/row sampling. However, if the distributions ofthe columns/rows of L are highly non-uniform, randomsampling cannot yield concise and descriptive sketchesof the data. For instance, suppose the columns of Ladmit a clustering structure as per the following as-sumption.
Assumption 1. The matrix L can be represented asL = [U1Q1 ...UnQn]. The CS of {Ui ∈ R
N1×r/n}ni=1
are random r/n-dimensional subspaces in RN1 . The
RS of {Qi ∈ Rr/n×ni}ni=1 are random r/n-dimensional
subspaces in {Rni}ni=1, respectively,∑n
i=1 ni = N2, andmini
ni ≫ r/n.
The following two lemmas show that the sufficientnumber of randomly sampled columns to capture theCS can be quite large depending on the distribution ofthe columns of L.
Lemma 1. Suppose m1 columns are sampled uni-formly at random with replacement from the matrix L
with rank r. If m1 ≥ 10µvr log2rδ , then the selected
columns of the matrix L span the CS of L with proba-bility at least 1− δ.
Lemma 2. If L follows Assumption 1, the rank of L isequal to r, r/n ≥ 18 logmax
ini and ni ≥ 96 r
n log ni, 1 ≤
i ≤ n, then P
[
µv < 1n
0.5N2
mini
ni
]
≤ 2∑n
i=1 n−5i .
According to Lemma 2, the RS coherency parameter µv
is linear in N2
mini
ni
. The factor N2
mini
ni
can be quite large
depending on the distribution of the columns. Thus,according to Lemma 1 and Lemma 2, we may need tosample too many columns to capture the CS if the dis-tribution of the columns is highly non-uniform. As anexample, consider L = [L1 L2], the rank of L = 60and N1 = 500. The matrix L1 follows Assumption 1with r = 30, n = 30 and {ni}30i=1 = 5. The matrixL2 follows Assumption 1 with r = 30, n = 30 and{ni}30i=1 = 200. Thus, the columns of L ∈ R
500×6150
lie in a union of 60 1-dimensional subspaces. Fig. 1shows the rank of randomly sampled columns of L ver-sus the number of sampled columns. Evidently, weneed to sample more than half of the data to span theCS. As such, we cannot evade high-dimensionality withuniform random column/row sampling.
4. Column Sampling from Sparsely Cor-rupted Data
In this section, the proposed robust sampling algo-rithm is presented. It is assumed that the data fol-
1000 2000 3000 4000 5000 6000Number of randomly sampled columns
20
30
40
50
60
Ran
k of
sam
pled
col
umns
Figure 1. The rank of randomly sampled columns.
lows Data model 1. Consider the following optimiza-tion problem
mina
‖di −D−ia‖0 subject to ‖a‖0 = r, (2)
where di is the ith column of D and D−i is equal toD with the ith column removed. If the CS of L doesnot contain sparse vectors, the optimal point of (2) isequivalent to the optimal point of
mina
‖si − S−ia‖0subject to li = L−ia and ‖a‖0 = r ,
(3)
where li is the LR component of di and L−i is the LRcomponent of D−i ( similarly, si and S−i are the sparsecomponent). To clarify, (2) samples r columns of D−i
whose LR component cancels out the LR componentof di and the linear combination si −S−ia is as sparseas possible.
This idea can be extended by searching for a set ofcolumns whose LR component can cancel out the LRcomponent of all the columns. Thus, we modify (2) as
minA
‖D−DA‖0 s. t. ‖AT ‖0,2 = r, (4)
where ‖AT ‖0,2 is the number of non-zero rows of A.The constraint in (4) forces A to sample r columns.Both the objective function and the constraint in (4)are non-convex. We propose the following convex re-laxation
minA
‖D−DA‖1 + γ‖AT ‖1,2 , (5)
where γ is a regularization parameter. DefineA∗ as theoptimal point of (5) and define the vector p ∈ R
N2×1
with entries p(i) = ‖a∗i‖2, where p(i) and a∗i are theith element of p and the ith row of A∗, respectively.The non-zero elements of p identify the representativecolumns. For instance, suppose D ∈ R
100×400 followsData model 1 with ρ = 0.02. The matrix L with rankr = 12 can be expressed as L = [L1 L2 L3 L4] wherethe ranks of {Lj ∈ R
100×100}4j=1 are equal to 5, 1, 5,and 1, respectively. Fig. 2 shows the output of the pro-posed method and the algorithm presented in [12]. As
43231820
shown, the proposed method samples a sufficient num-ber of columns from each cluster. Since the algorithmpresented in [12] requires strong linear dependence be-tween the columns of D, the presence of the sparse cor-ruption matrix S seriously degrades its performance.
Figure 2. The elements of the vector p. Te left plot corre-sponds to (5) and the right plot corresponds to [12].
4.1. Robust column sampling from Big data
The complexity of solving (5) is O(N32 +N1N
22 ). In
this section, we present a randomized scalable approachwhich yields a scalable implementation of the proposedmethod for high dimensional data reducing complexityto O(r3 + r2N2). We further assume that the RS ofL can be captured using a small random subset of therows of L, an assumption that will be relaxed in Section4.1.1. The following lemma shows that the RS can becaptured using few randomly sampled rows even if thedistribution of the columns is highly non-uniform.
Lemma 3. Suppose L follows Assumption 1 and m2
rows of L are sampled uniformly at random with re-placement. If the rank of L is equal to r and
m2 ≥ 10 c rϕ log2r
δ, (6)
then the sampled rows span the row space of L
with probability at least 1 − δ − 2N−31 , where ϕ =
max(r,logN1)r .
The sufficient number of sampled rows for the setupof Lemma 3 is roughly O(r), and is thus independentof the distribution of the columns. Let Dr denote thematrix of randomly sampled rows of D and Lr its LRcomponent. Suppose the rank of Lr is equal to r. Sincethe RS of Lr is equal to the RS of L, if a set of thecolumns of Lr span its CS, the corresponding columnsof L will span the CS of L. Accordingly, we rewrite (5)as
minA
‖Dr −DrA‖1 + γ‖AT ‖1,2 . (7)
Note that we still have an N2 × N2 dimensional op-timization problem and the complexity of solving (7)is roughly O(N3
2 + N22 r). In this section, we propose
an iterative randomized method which solves (7) with
complexity O(r3 + N2r2). Algorithm 1 presents the
proposed solver. It starts the iteration with few ran-domly sampled columns of Dr and refines the sampledcolumns in each iteration.
Remark 1. In both (7) and (8), we use the same sym-bol A to designate the optimization variable. However,in (7) A ∈ R
N2×N2 , while in (8) A ∈ Rm×N2 , where
m of order O(r) is the number of columns of Dsr.
Here we provide a brief explanation of the differentsteps of Algorithm 1:
Steps 2.1 and 2.2: The matrix Dsr is the sampled
columns of Dr. In steps 2.1 and 2.2, the redundantcolumns of Ds
r are removed.
Steps 2.3 and 2.4: Define Lsr as the LR component of
Dsr. Steps 2.3 and 2.4 aim at finding the columns of Lr
which do not lie in the CS of Lsr. Define dri as the ith
column of Dr. For a given i, if lri (the LR componentof dri) lies in the CS of Ls
r, the ith column of F =Dr−Ds
rA∗ will be a sparse vector. Thus, if we remove
a small portion of the elements of the ith column of Fwith the largest magnitudes, the ith column of F willapproach the zero vector. Thus, by removing a smallportion of the elements with largest magnitudes of eachcolumn of F, step 2.3 aims to locate the columns of Lr
that do not lie in the CS of Lsr, namely, the columns of
Lr corresponding to the columns of F with the largestℓ2-norms. Therefore, in step 2.5, these columns areadded to the matrix of sampled columns Ds
r.As an example, suppose D = [L1 L2] + S follows
Data model 1 where the CS of L1 ∈ R50×180 is inde-
pendent of the CS of L2 ∈ R50×20. In addition, assume
Dr = D and that all the columns of Dsr happen to be
sampled from the first 180 columns of Dr, i.e., all sam-pled columns belong to L1. Thus, the LR componentof the last 20 columns of Dr do not lie in the CS of Ls
r.Fig. 3 shows F = Dr −Ds
rA∗. One can observe that
the algorithm will automatically sample the columnscorresponding to L2 because if few elements (with thelargest absolute values) of each column are eliminated,only the last 20 columns will be non-zero.
Remark 2. The approach proposed in Algorithm 1 isnot limited to the proposed method. The same ideacan be used to enable more scalable versions of exist-ing algorithms. For instance, the complexities of [12]and [22] can be reduced from roughly N3
2 to N2r, whichis a remarkable speedup for high dimensional data.
4.1.1 Sampling from highly structured Big
data
Algorithm 1 presumes that the rows of L are welldistributed such that c1r randomly sampled rows of L
43241821
-1060
-5
200
0
40 150
F = Dr - D
rs A*
5
Rows columns
100
10
2050
0 0
Figure 3. Matrix F in step 2.3 of Algorithm 1.
Algorithm 1 Scalable Randomized Solver for (7)1. Initialization
1.1 Set c1, c2, and kmax equal to integers greater than 0. Setτ equal to a positive integer less than 50. r is a known upperbound on r.1.2 Form Dr ∈ R
m2×N2 by sampling m2 = c1r rows of D
randomly.1.3 Form Ds
r ∈ Rm2×c2r by sampling c2r columns of Dr ran-
domly.
2. For k from 1 to kmax
2.1 Locate informative columns: Define A∗ as the optimalpoint of
minA
‖Dr −DsrA‖1 + γ‖AT ‖1,2 . (8)
2.2 Remove redundant columns: Remove the zero rowsof A∗ (or the rows with ℓ2-norms close to 0) and remove thecolumns of Ds
r corresponding to these zero rows.2.3 Remove sparse residuals: Define F = Dr − Ds
rA∗. For
each column of F, remove the τ percent elements with largestabsolute values.2.4 Locate new informative columns: Define Dn
r ∈ Rm2×2r
as the columns of Dr which are corresponding to the columns ofF with maximum ℓ2-norms.2.5 Update sampled columns: Ds
r = [Dsr Dn
r ].2. End For
Output: Construct Dc as the columns of D corresponding to
the sampled columns from Dr (which form Dsr). The columns
of Dc are the sampled columns.
span its RS. This may be true in many settings wherethe clustering structure is only along one direction (ei-ther the rows or columns), in which case Algorithm
1 can successfully locate the informative columns. If,however, both the columns and the rows exhibit clus-tering structures and their distribution is highly non-uniform, neither the CS nor the RS can be capturedconcisely using random sampling. As such, in this sec-tion we address the scenario in which the rank of Lr
may not be equal to r. We present an iterative CS-RSpursuit approach which converges to the CS and RS ofL in few iterations.
The table of Algorithm 2, Fig. 4 and its captionprovide the details of the proposed sampling approachalong with the definitions of the used matrices. We
Algorithm 2 Column/Row Subspace Pursuit Algo-rithmInitialization: Set Dw equal to c1r randomly sampled rows ofD. Set X equal to c2r randomly sampled columns of Dw andset kmax equal to an integer greater than 0.
For j from 1 to jmax do
1. Column Sampling
1.1 Locating informative columns: Apply Algorithm 1 with-out the initialization step to Dw as follows: set Dr equal to Dw,set Ds
r equal to X, and set kmax = 1.1.2 Update sub-matrix X: Set sub-matrix X equal to Ds
r,the output of Step 2.5 of Algorithm 1.1.3 Sample the columns: Form matrix Dc using the columnsof D corresponding to the columns of Dw which were used toform X.
2. Row Sampling
2.1 Locating informative rows: Apply Algorithm 1 withoutthe initialization step to DT
c as follows: set Dr equal to DTc , set
Dsr equal to XT , and set kmax = 1.
2.2 Update sub-matrix X: Set sub-matrix XT equal to Dsr,
the output of step 2.5 of Algorithm 1.2.3 Sample the rows: Form matrix Dw using the rows of Dcorresponding to the rows of Dc which were used to form X.
End For
Output: The matrices Dc and Dw are the sampled columns
and rows, respectively.
start the cycle from the position marked I in Fig. 4.The matrix X is the informative columns of Dw. Thus,the rank of XL (the LR component of X) is equal tothe rank of Lw (the LR component of Dw). The rowsof XL are a subset of the rows of Lc (the LR com-ponent of Dc). If the rows of L exhibit a clusteringstructure, it is likely that rank(XL) < rank(Lc). Thus,rank(Lw) < rank(Lc). We continue one cycle of thealgorithm by going through steps II and 1 of Fig. 4 toupdate Dw. Using a similar argument, we see that therank of an updated Lw will be greater than the rankof Lc. Thus, if we run more cycles of the algorithm– each time updating Dw and Dc – the rank of Lw
and Lc will increase. While there is no guarantee thatthe rank of Lw will converge to r (it can converge toa value smaller than r), our investigations have shownthat Algorithm 2 performs quite well and the RS ofLw (CS of Lc) converges to the RS of L (CS of L) invery few iterations.
4.2. Solving the convex optimization problems
The optimization problem (8) can be rewritten as
minQ,A,B
‖Q‖1 + γ‖BT ‖1,2
subject to A = B and Q = Dr −DsrA ,
(9)
43251822
Figure 4. Visualization of Algorithm 2. I: Matrix Dc isobtained as the columns of D corresponding to the columnswhich form X. II: Algorithm 1 is applied to Matrix Dc toupdate X (the sampled rows of Dc). 1: Matrix Dw isobtained as the rows of D corresponding to the rows whichform X. 2: Algorithm 1 is applied to matrix Dw to updateX (the sampled columns of Dw).
which is equivalent to
minQ,A,B
‖Q‖1 + γ‖BT ‖1,2 +µ
2‖A−B‖2F
+µ
2‖Q−Dr +Ds
rA‖2Fsubject to A = B and Q = Dr −Ds
rA ,
(10)
where µ is the tuning parameter. The optimizationproblem (10) can be efficiently solved using the Alter-nating Direction Method of Multipliers (ADMM) [5].Due to space constraints, we defer the details of theiterative solver to the supplementary material. Theoptimization problem (5) can be solved using the sametechniques as well, where we would just need to sub-stitute Dr and Ds
r with D.
5. Robustness to Outlying Data Points
In this section, we extend the proposed sampling al-gorithm (5) to make it robust to both sparse corruptionand outlying data points. Suppose the given data canbe expressed as
D = L+C+ S , (11)
where L and S follow Data model 1. The matrix C
has some non-zero columns modeling the outliers. Theoutlying columns do not lie in the column space of Land they cannot be decomposed into columns of L plussparse corruption. Below, we provide two scenariosmotivating the model (11).
I. Facial images with different illuminations wereshown to lie in a low dimensional subspace [1]. Nowsuppose we have a dataset consisting of some sparsely
corrupted face images along with few images of ran-dom objects (e.g., building, cars, cities, ...). The im-ages from random objects cannot be modeled as faceimages with sparse corruption. We seek a sampling al-gorithm which can find informative face images whileignoring the presence of the random images to identifythe different human subjects in the dataset.
II. A users rating matrix in recommender systemscan be modeled as a LR matrix owing to the simi-larity between people’s preferences for different prod-ucts. To account for natural variability in user pro-files, the LR plus sparse matrix model can better modelthe data. However, profile-injection attacks, capturedby the matrix C, may introduce outliers in the userrating databases to promote or suppress certain prod-ucts. The model (11) captures both element-wise andcolumn-wise abnormal ratings.
The objective is to develop a column sampling al-gorithm which is simultaneously robust to sparse cor-ruption and outlying columns. To this end, we proposethe following optimization problem extending (5)
minA
‖D−DA+E‖1 + γ‖AT ‖1,2 + λ‖E‖1,2subject to diag(A) = 0 .
(12)
The matrix E cancels out the effect of the outlyingcolumns in the residual matrix D−DA. Thus, the reg-ularization term corresponding to E uses an ℓ1,2-normwhich promotes column sparsity. Since the outlyingcolumns do not follow low dimensional structures, anoutlier cannot be obtained as a linear combination offew data columns. The constraint plays an importantrole as it prevents the scenario where an outlying col-umn is sampled by A to cancel itself.
The sampling algorithm (12) can locate the repre-sentative columns in presence of sparse corruption andoutlying columns. For instance, suppose N1 = 50,L = [L1 L2], where L1 ∈ R
50×100 and L2 ∈ R50×250.
The ranks of L1 and L2 are equal to 5 and 2, respec-tively, and their column spaces are independent. Thematrix S follows Data model 1 with ρ = 0.01 and thelast 50 columns of C are non-zero. The elements of thelast 50 columns of C are sampled independently froma zero mean normal distribution. Fig. 5 compares theoutput of (12) with the state-of-the-art robust sam-pling algorithms in [22, 26]. In the first row of Fig. 5,D = L + S + C, and in the second row, D = L + C.Even if S = 0, the robust column sampling algorithm(12) substantially outperforms [22, 26]. As shown, theproposed method samples correctly from each cluster(at least 5 columns from the first cluster and at least 2columns from the second), and unlike [22,26], does notsample from the outliers.
43261823
Figure 5. Comparing the performance of (12) with the al-gorithm in [22,26]. In the first row, D = L+C+S. In thesecond row, D = L+C. The last 50 columns of D are theoutliers.
Figure 6. Left: Subspace recovery error versus the numberof randomly sampled columns. Right: The rank of sampledcolumns through the iterations of Algorithm 1.
6. Experimental Results
In this section, we apply the proposed samplingmethods to both real and synthetic data and the per-formance is compared to the-state-of-the-art.
6.1. Shortcomings of random sampling
In the first experiment, it is shown that random sam-pling uses too many columns from D to correctly learnthe CS of L. Suppose the given data follows Datamodel 1 with ρ = 0.01. In this experiment, D is a2000 × 6300 matrix. The LR component is generatedas L = [L1 ... L60] . For 1 ≤ i ≤ 30, Li = UiQi , whereUi ∈ R
2000×1, Qi ∈ R1×200 and the elements of Ui
and Qi are sampled independently from N (0, 1). For31 ≤ i ≤ 60, Li =
√10UiQi , where Ui ∈ R
2000×1,Qi ∈ R
1×10. Thus, with high probability the columnsof L lie in a union of 60 independent 1-dimensionallinear subspaces.
We apply Algorithm 1 to D. The matrix Dr isformed using 100 randomly sampled rows of D. Sincethe rows of L do not follow a clustering structure, therank of Lr (the LR component of Dr) is equal to 60with overwhelming probability. The right plot of Fig.6 shows the rank of the LR component of sampledcolumns (the rank of Ls
r) after each iteration of Al-
gorithm 1. Each point is obtained by averaging over20 independent runs. One can observe that 2 iterationssuffice to locate the descriptive columns. We run Al-
gorithm 1 with kmax = 3. It samples 255 columns on
average.We apply the decomposition algorithm to the sam-
pled columns. Define the recovery error as ‖L −UUTL‖F /‖L‖F , where U is the basis for the CSlearned by decomposing the sampled columns. If welearn the CS using the columns sampled byAlgorithm
1, the recovery error in 0.02 on average. On the otherhand, the left plot of Fig. 6 shows the recovery er-ror based on columns sampled using uniform randomsampling. As shown, if we sample 2000 columns ran-domly, we cannot outperform the performance enabledby the informative column sampling algorithm (whichhere samples 255 columns on average).
6.2. Face sampling and video summarization withcorrupted data
In this experiment, we use the face images in theYale Face Database B [21]. This dataset consists offace images from 38 human subjects. For each sub-ject, there is 64 images with different illuminations.We construct a data matrix with images from 6 humansubjects (384 images in total with D ∈ R
32256×384 con-taining the vectorized images). The left panel of Fig.8 shows the selected subjects. It has been observedthat the images in the Yale dataset follow the LR plussparse matrix model [6]. In addition, we randomly re-place 2 percent of the pixels of each image with ran-dom pixel values, i.e., we add a synthetic sparse matrix(with ρ = 0.02) to the images to increase the corrup-tion. The sampling algorithm (5) is then applied toD. The right panel of Fig. 8 displays the images cor-responding to the sampled columns. Clearly, the algo-rithm chooses at least one image from each subject.
Informative column sampling algorithms can be uti-lized for video summarization [11, 12]. In this exper-iment, we cut 1500 consecutive frames of a cartoonmovie, each with 320 × 432 resolution. The data ma-trix is formed by adding a sparse matrix with ρ = 0.02to the vectorized frames. The sampling algorithm(7) is then applied to find the representative columns(frames). The matrix Dw is constructed using 5000randomly sampled rows of the data. The algorithmsamples 52 frames which represent almost all the im-portant instances of the video. Fig. 7 shows some ofthe sampled frames (designated in color) along withneighboring frames in the video. The algorithm judi-ciously samples one frame from frames that are highlysimilar.
6.3. Sampling from highly structured data
Suppose L1 ∈ R2000×6300 and L2 ∈ R
2000×6300 aregenerated independently similar to the way L was con-structed in Section 6.1. Set V equal to the first 60
43271824
Figure 7. Few frames of the video file. The frames in color are sampled by the robust column sampling algorithm.
Figure 8. Left: 6 images from the 6 human subjects form-ing the dataset. Right: 8 images corresponding to thecolumns sampled from the face data matrix.
right singular vectors of L1, U equal to the first 60right singular vectors of L2, and set L = UVT . Thus,the columns/rows of L ∈ R
6300 lie in a union of 60 1-dimensional subspaces and their distribution is highlynon-uniform. Define rw and rc as the rank of Lw andLc, respectively. Table 1 shows rw and rc through theiterations of Algorithm 2. The values are obtained asthe average of 10 independent runs with each averagevalue fixed to the nearest integer. Initially, rw is equalto 34. Thus, Algorithm 1 is not applicable in thiscase since the rows of the initial Lw do not span theRS of L. According to Table 1, the rank of the sam-pled columns/rows increases through the iterations andconverge to the rank of L in 3 iterations.
Table 1. Rank of columns/rows sampled by Algorithm 2
Iteration Number 0 1 2 3rc - 45 55 60rw 34 50 58 60
6.4. Robustness to outliers
In this experiment, the performance of the samplingalgorithm (12) is compared to the state of the art ro-bust sampling algorithm presented in [22, 26]. Sincethe existing algorithms are not robust to sparse cor-ruption, in this experiment matrix S is set equal tozero. The data can be represented as D = [Dl Dc].The matrix Dl ∈ R
50×100 contain the inliers, the rankof Dl is equal to 5 and the columns of Dl are dis-tributed randomly within the CS of Dl. The columnsof Do ∈ R
50×no are the outliers, the elements of Dc aresampled from N (0, 1) and no is the number of outliers.We perform 10 independent runs. Define pa as theaverage of the 10 vectors p. Fig. 9 compares the out-
put of the algorithms for different number of outliers.One can observe that the existing algorithm fails notto sample from outliers. Interestingly, eve if no = 300,the proposed method does not sample from the out-liers. However, if we increase no to 600, the proposedmethod starts to sample from the outliers.
Figure 9. Comparing the performance of (12) with [22,26].The blues are the sampled points from inliers and blacksare sampled points from outliers.
Acknowledgment: This work was supported by NSFAward CCF-1552497 and NSF Grant CCF- 1320547.
References
[1] R. Basri and D. W. Jacobs. Lambertian reflectanceand linear subspaces. IEEE Transactions on Pat-tern Analysis and Machine Intelligence, 25(2):218–233, 2003.
[2] J. Bien and R. Tibshirani. Prototype selection forinterpretable classification. The Annals of AppliedStatistics, pages 2403–2424, 2011.
[3] C. Boutsidis, J. Sun, and N. Anerousis. Clustered sub-set selection and its applications on it service metrics.In Proceedings of the 17th ACM conference on Infor-mation and knowledge management, pages 599–608.ACM, 2008.
[4] T. Bouwmans, A. Sobral, S. Javed, S. K. Jung, and E.-H. Zahzah. Decomposition into low-rank plus additivematrices for background/foreground separation: A re-view for a comparative evaluation with a large-scaledataset. Computer Science Review, 23:1–71, 2017.
[5] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eck-stein. Distributed optimization and statistical learn-ing via the alternating direction method of multipli-ers. Foundations and Trends➤ in Machine Learning,3(1):1–122, 2011.
43281825
[6] E. J. Candes, X. Li, Y. Ma, and J. Wright. Robustprincipal component analysis? Journal of the ACM(JACM), 58(3):11, 2011.
[7] V. Chandrasekaran, S. Sanghavi, P. A. Parrilo, andA. S. Willsky. Rank-sparsity incoherence for ma-trix decomposition. SIAM Journal on Optimization,21(2):572–596, 2011.
[8] A. Deshpande, L. Rademacher, S. Vempala, andG. Wang. Matrix approximation and projective clus-tering via volume sampling. In Proceedings of the sev-enteenth annual ACM-SIAM symposium on Discretealgorithm, pages 1117–1126. Society for Industrial andApplied Mathematics, 2006.
[9] P. Drineas, A. Frieze, R. Kannan, S. Vempala, andV. Vinay. Clustering large graphs via the singular valuedecomposition. Machine learning, 56(1-3):9–33, 2004.
[10] P. Drineas, M. W. Mahoney, and S. Muthukrishnan.Subspace sampling and relative-error matrix approx-imation: Column-based methods. In Approximation,Randomization, and Combinatorial Optimization. Al-gorithms and Techniques, pages 316–326. Springer,2006.
[11] E. Elhamifar, G. Sapiro, and S. Sastry. Dissimilarity-based sparse subset selection. IEEE Transactions onSoftware Engineering, 38(11), 2014.
[12] E. Elhamifar, G. Sapiro, and R. Vidal. See all by look-ing at a few: Sparse modeling for finding representativeobjects. In IEEE Conference on Computer Vision andPattern Recognition (CVPR), pages 1600–1607, 2012.
[13] E. Esser, M. Moller, S. Osher, G. Sapiro, and J. Xin.A convex model for nonnegative matrix factorizationand dimensionality reduction on physical space. IEEETransactions on Image Processing, 21(7):3239–3252,2012.
[14] S. Garcia, J. Derrac, J. Cano, and F. Herrera. Proto-type selection for nearest neighbor classification: Tax-onomy and empirical study. IEEE Transactions onPattern Analysis and Machine Intelligence, 34(3):417–435, 2012.
[15] M. Gu and S. C. Eisenstat. Efficient algorithms forcomputing a strong rank-revealing QR factorization.SIAM Journal on Scientific Computing, 17(4):848–869, 1996.
[16] N. Halko, P.-G. Martinsson, and J. A. Tropp. Findingstructure with randomness: Probabilistic algorithmsfor constructing approximate matrix decompositions.SIAM review, 53(2):217–288, 2011.
[17] J. Hartline, V. Mirrokni, and M. Sundararajan. Opti-mal marketing strategies over social networks. In Pro-ceedings of the 17th international conference on WorldWide Web, pages 189–198. ACM, 2008.
[18] S. Javed, A. Mahmood, T. Bouwmans, and S. K. Jung.Spatiotemporal low-rank modeling for complex scenebackground initialization. IEEE Transactions on Cir-cuits and Systems for Video Technology, 2016.
[19] Q. Ke and T. Kanade. Robust l 1 norm factorizationin the presence of outliers and missing data by alterna-tive convex programming. In IEEE Computer Society
Conference on Computer Vision and Pattern Recogni-tion (CVPR), volume 1, pages 739–746, 2005.
[20] D. Lashkari and P. Golland. Convex clustering withexemplar-based models. Advances in neural informa-tion processing systems, 20, 2007.
[21] K.-C. Lee, J. Ho, and D. J. Kriegman. Acquiring linearsubspaces for face recognition under variable lighting.IEEE Transactions on Pattern Analysis and MachineIntelligence, 27(5):684–698, 2005.
[22] H. Liu, Y. Liu, and F. Sun. Robust exemplar ex-traction using structured sparse coding. IEEE Trans-actions on Neural Networks and Learning Systems,26(8):1816–1821, 2015.
[23] R. Liu, Z. Lin, S. Wei, and Z. Su. Solving principalcomponent pursuit in linear time via l 1 filtering. arXivpreprint arXiv:1108.5359, 2011.
[24] L. Mackey, A. Talwalkar, and M. I. Jordan. Dis-tributed matrix completion and robust factorization.arXiv preprint arXiv:1107.0789, 2011.
[25] L. W. Mackey, M. I. Jordan, and A. Talwalkar. Divide-and-conquer matrix factorization. In Advances inNeural Information Processing Systems (NIPS), pages1134–1142, 2011.
[26] F. Nie, H. Huang, X. Cai, and C. H. Ding. Efficient androbust feature selection via joint 2, 1-norms minimiza-tion. In Advances in Neural Information ProcessingSystems, pages 1813–1821, 2010.
[27] M. Rahmani and G. K. Atia. High dimensional lowrank plus sparse matrix decomposition. IEEE Trans-actions on Signal Processing, 65(8):2004–2019, 2017.
[28] M. Rahmani and G. K. Atia. Randomized robust sub-space recovery and outlier detection for high dimen-sional data matrices. IEEE Transactions on SignalProcessing, 65(6):1580–1594, 2017.
[29] M. Rahmani and G. K. Atia. Spatial random sampling:A structure-preserving data sketching tool. IEEE Sig-nal Processing Letters, 24(9):1398–1402, Sept 2017.
[30] A. Sobral and E.-h. Zahzah. Matrix and tensor comple-tion algorithms for background model initialization: Acomparative evaluation. Pattern Recognition Letters,2016.
[31] J. A. Tropp. Column subset selection, matrix factor-ization, and eigenvalue optimization. In Proceedings ofthe Twentieth Annual ACM-SIAM Symposium on Dis-crete Algorithms, pages 978–986. Society for Industrialand Applied Mathematics, 2009.
[32] J. Wright, A. Ganesh, K. Min, and Y. Ma. Compres-sive principal component pursuit. Information and In-ference, 2(1):32–68, 2013.
[33] T. Zhou and D. Tao. Godec: Randomized low-rank& sparse matrix decomposition in noisy case. In In-ternational Conference on Machine Learning (ICML),2011.
[34] Z. Zhou, X. Li, J. Wright, E. Candes, and Y. Ma.Stable principal component pursuit. In IEEE Interna-tional Symposium on Information Theory Proceedings(ISIT), pages 1518–1522. IEEE, 2010.
43291826
