Journal of Mathematical Imaging and Vision (2020) 62:1159–1172https://doi.org/10.1007/s10851-020-00967-4

Background Subtraction using Adaptive Singular Value Decomposition

Günther Reitberger1 · Tomas Sauer1

Received: 4 July 2019 / Accepted: 15 May 2020 / Published online: 5 June 2020© The Author(s) 2020

AbstractAn important taskwhen processing sensor data is to distinguish relevant from irrelevant data. This paper describes amethod foran iterative singular value decomposition that maintains a model of the background via singular vectors spanning a subspaceof the image space, thus providing a way to determine the amount of new information contained in an incoming frame. Weupdate the singular vectors spanning the background space in a computationally efficient manner and provide the abilityto perform blockwise updates, leading to a fast and robust adaptive SVD computation. The effects of those two propertiesand the success of the overall method to perform a state-of-the-art background subtraction are shown in both qualitative andquantitative evaluations.

Keywords Image processing · Background subtraction · Singular value decomposition

1 Introduction

With static cameras, for example in video surveillance, thebackground, like houses or trees, stays mostly constant overa series of frames, whereas the foreground consisting ofobjects of interest, e.g., cars or humans, causes differencesin image sequences. Background subtraction aims to distin-guish between foreground and background based on previousimage sequences and eliminates the background from newlyincoming frames, leaving only the moving objects containedin the foreground. These are usually the objects of interest insurveillance.

1.1 Motivation

Data-driven approaches are a major topic in image pro-cessing and computer vision, leading to state-of-the-artperformances, for example in classification or regressiontasks. One example is video surveillance used for secu-rity reasons, traffic regulation, or as information source inautonomous driving. The main problems with data-drivenapproaches are that the training data have to be well bal-anced and to cover all scenarios that appear later in the

B Günther Reitbergerreitberg@forwiss.uni-passau.de

Tomas Sauersauer@forwiss.uni-passau.de

1 FORWISS, University of Passau, Passau, Germany

execution phase and have to be well annotated. In contrast tocameras mounted at moving objects such as vehicles, staticcameras mounted at some infrastructure observe a scenery,e.g., houses, trees, parked cars, that is widely fixed or atleast remains static over a large number of frames. If one isinterested in moving objects, as it is the case in the afore-mentioned applications, the relevant data are exactly the onedifferent from the static data. The reduction of the input data,i.e., the frames taken from the static cameras, to the relevantdata, i.e., themoving objects, is important for several applica-tions like the generation of training data for machine learningapproaches or as input for classification tasks reducing falsepositive detections due to the removal of the irrelevant staticpart.

Calling the static part background and the moving objectsforeground, the task of dynamic and static part distinction isknown as foreground background separation or simply back-ground subtraction.

1.2 Background Subtraction as OptimizationProblem

Throughout the paper, we make the assumptions that thecamera is static, the background is mostly constant up torare changes and illumination, and the moving objects, con-sidered as foreground, are small relative to the image size.Then, background subtraction can be formulated as an opti-mization problem. Given an image sequence stacked in

123

1160 Journal of Mathematical Imaging and Vision (2020) 62:1159–1172

vectorized form into the matrix A ∈ Rd×n , with d being

the number of pixels of an image and n being the number ofimages, foreground–background separation can be modeledas decomposing A into a low-rank matrix L , the backgroundand a sparse matrix S, the foreground, cf. [8]. This leads tothe optimization problem

minL,S

rank(L) + λ‖S‖0 s.t. A = L + S. (1)

Unfortunately, solving this problem is not feasible. There-fore, adaptations have to bemade. Recall that a singular valuedecomposition (SVD) decomposes a matrix A ∈ R

d×n into

A = UΣV T (2)

with orthogonal matrices U ∈ Rd×d and V ∈ R

n×n and thediagonal matrix

Σ =[Σ ′ 00 0

]∈ R

d×n, Σ ′ ∈ Rr×r , r = rank A,

where Σ ′ has strictly positive diagonal values. The SVDmakes no relaxation of the rank, but, given � ≤ r , the best (inan �2 sense) rank-�, � ∈ N, estimate L of A can be obtainedby using the first � singular values and vectors, see [28,29].This solves the optimization problems

min ‖A − L‖F or ‖A − L‖2 s.t. rank L ≤ �. (3)

Weuse the followingnotation throughout our paper:U:,1:� :=U (:, 1 : �) := [u1, . . . , u�], with ui being the i th column ofU, i ∈ {1, . . . , �}.

The first � columns of the U matrix of the SVD (2) of A,i.e., the left singular vectors corresponding to the � biggestsingular values, span a subspace of the column space of A.The background of an image J ∈ R

d×1 is calculated by theorthogonal projection of J onU� := U:,1:� byU�(UT

� J ). Theforeground then consists of the difference in the backgroundfrom the image J −U�(UT

� J ) = (I −U�UT

�

)J .

The aim of a surveillance application is to subtractthe background from every incoming image. Modeling thebackground via (3) results in a batch algorithm, wherethe low-rank approximations are calculated based on some(recent) sample frames stacked together to thematrix A. Notethat this allows the background to change slowly over time,for example due to changing illumination or to parked carsleaving the scene. It is well known that the computationaleffort to determine the SVD of A with dimensions d � nis O(dn2) using R-SVD and computing only Un = U:,1:ninstead of the complete d × d matrix U , and the memoryconsumption is O(dn), cf. [10]. Especially in the case ofhigher definition images, only rather few samples n can be

used in this way. This results in a dependency of the back-ground model from the sample image size and an inabilityof adaption to a change in the background that is not coveredin the few sample frames. Hence, a naive batch algorithm isnot a suitable solution.

1.3 Main Contributions and Outline

The layout of this paper is as follows. In Sect. 2, we brieflyrevise related work in background subtraction and SVDmethods. Section 3 introduces our algorithm of iterativelycalculating an SVD. The main contribution here consistsin the application and adaption of the iterative SVD tobackground subtraction. In Sect. 4, we propose a concretealgorithm that adapts the model of the background in away that is dependent on the incoming data because ofwhich we call it adaptive SVD. A straightforward versionof the algorithm still has limitations, because of which wepresent extensions of the basic algorithm that overcomethese deficits. In Sect. 5, evaluations of the method give animpression of execution time, generality, and performancecapabilities of the adaptive SVD. Finally, in Sect. 6 our mainconclusions are outlined.

2 RelatedWork

The “philosophical” goal of background modeling is toacquire a background image that does not include any mov-ing objects. In realistic environments, the background mayalso change, due to influences like illumination or objectsbeing introduced to or removed from the scene. Consider-ing these problems as well as robustness and adaptation,background modeling methods can, according to the surveypapers [2,3,6], be classified into the following categories:statistical background modeling, background modeling viaclustering, background estimation, and neural networks.

The most recent approach is, of course, to model thebackground via neural networks. Particularly, convolutionalneural networks (CNNs) [15] have performed very well inmany tasks of image processing. These techniques, however,usually involve a labeling of the data, i.e., the backgroundhas to be annotated, mostly manually, for a set of train-ing images. The network then learns the background basedon the labels. Gracewell and John [12], for example, usean autoencoder network architecture for the training of abackground model based on labeled data. Background mod-eling is often combined with classification or segmentationtasks where every pixel of an image is assigned to one class.Based on the classes, the pixel can then be classified as back-ground or foreground, respectively. This task is often doneby means of transfer learning, cf. [11, p. 526]. Followingthis idea, Lim and Keles [17] add three layers to a pre-

123

Journal of Mathematical Imaging and Vision (2020) 62:1159–1172 1161

trained image content classifying CNN and post-train theirresulting network on few labeled foreground segmentations.Such techniques strongly depend on the training data andcan only be improved by adding further data. An evalua-tion of the network performance depending on the trainingdata is made in [20], and an overview of neural networksfor background subtraction is provided in [4]. Reinforce-ment learning [21] and unsupervised learning, on the otherhand, do not need labeled data for training. Sultana et al. [30]extend a pre-trained CNN with an unsupervised network togenerate background at the image positionswhere objects aredetected. Therefore, their approach does not require labeleddata, at least not for the post-training. Our algorithm, onthe other hand, is data driven as well, but it is flexible anddoes not have to be fully re-trained if the application dataare essentially different from the training data; this featurestrongly contrasts with the aforementioned neural networkapproaches.

Statistical background modeling includes Gaussian mod-els, support vector machines, and subspace learning models.Subspace learning originates from the modeling of the back-ground subtraction task as shown in (1). Our approachtherefore also belongs to this domain. Principal componentpursuit (PCP) [8] is based on the convex relaxation of (1) by

minL,S

‖L‖∗ + λ‖S‖1 s.t. A = L + S, (4)

with ‖L‖∗ being the nuclear norm of matrix L , the sumof the singular values of L . Relaxation (4) can be solvedby efficient algorithms such as alternating optimization. AsPCP considers the �1 error, it is more robust against out-liers or salt-and-pepper noise than SVD-based methods andthus more suited to situations that suffer of that type ofnoise. Since outliers are not a substantial problem in traf-fic surveillance, which is our main application in mind, wedo not have to dwell on this type of robustness. In addition,the pure PCP method also has its limitations such as beinga batch algorithm, being computationally expensive com-pared to SVD, andmaintaining the exact rank of the low-rankapproximation, cf. [6]. This is a problem when it comes todata that are affected by noise in most components, whichis usually the case in camera-based image processing. Weremark that to overcome the drawbacks of plain PCP, manyextensions of the PCP have been developed. Rodriguez andWohlberg introduce an incremental PCP algorithm in [24]and extend it in [25] by an optimization step to cope withtranslational and rotational jitter. Incremental PCP is able toadapt to a gradually changing low-rank subspace, which isthe case with video data in the field of background subtrac-tion.

Generally, the approach to solve (1) by some relaxation,like PCPdoes, is also called robust principal component anal-

ysis (RPCA). Static approaches assume a constant low-ranksubspace, whereas dynamic ones incorporate a graduallychanging low-rank subspace. Recursive projected compres-sive sensing (ReProCS) described by Guo et al. [13] andthe ReProCS-based algorithm MERoP described in [22] areexamples for a solution of the dynamic RPCA problem, justlike [24] mentioned above. An overview of RPCA methodsis given in [5,31].

As already mentioned, we want to focus on �2 reg-ularization and therefore SVD-based methods. In recentyears, several algorithms have been developed to speedup the calculation of the SVD, for finding the optimalrank-� matrix approximating a given data matrix that isusually high dimensional and dense, i.e., of high rank, inthe domain of background subtraction. Liu et al. [18] itera-tively approximate the subspace of dimension � via a blockKrylov subspace optimization approach. Although this isfast, convergence assumptions have to be met. Anotherpopular way for a fast SVD calculation is by random-ization. Erichson et al. use random test matrices in [9]followed by a compressed sensing technique to approxi-mate the dominant left and right singular vectors. Kalooraziand de Lamare offer in [14] a decomposition of the datamatrix into UZV with Z allowing only small off-diagonalentries in an �2 sense. This factorization is rank reveal-ing and can be efficiently calculated via random sampling.To further speed up the randomized approaches, whichtend to have good parallelization abilities, Lu et al. offera way to blockwisely move parts of the calculations ontothe GPU in [19]. For our application, however, it is impor-tant that the low-rank subspace calculation algorithm isextendable to an iteratively growing data matrix which isnot available in the aforementioned fast SVD calculationapproaches.

There is naturally a close relationship between our SVD-based approach and incremental principal component anal-ysis (PCA) due to the close relationship between SVD andPCA. Given a matrix A ∈ R

n×d with n being the num-ber of samples and d the number of features, the PCAsearches for the first k eigenvectors of the correlation matrixAT A which span the same subspace as the first k columnsof the U matrix of the SVD of AT , i.e., the left singularvectors of AT . Thus, usually the PCA is actually calcu-lated by an SVD; since AT = UΣV T gives AT A =UΣV T VΣUT = UΣ2UT , the PCA produces the samesubspace as our iterative SVD approach. One difference isthat PCA originates from the statistics domain and the appli-cations search for the main directions in which the datadiffer from the mean data sample. That is why, the matrixA usually gets normalized by subtraction of the column-wisemean and divided by the columnwise standard deviationbefore calculating the PCA which, however, makes no sensein our application. This is also expressed in the work by

123

1162 Journal of Mathematical Imaging and Vision (2020) 62:1159–1172

Ross et al. [26], based on the sequential Karhunen–Loevebasis extraction from [16]. They use the PCA as a fea-ture extractor for a tracking application. In our approach,we model the mean data, the background, by singular vec-tors only and dig deeper into the application to backgroundsubtraction, which has not been considered in the PCAcontext. Nevertheless, we will make further comparisonsto PCA, pointing out similarities and differences to ourapproach.

3 Update Methods for Rank RevealingDecompositions and Applications

Our background subtraction method is based on an iterativecalculation of an SVD for matrices augmented by columns,cf. [23]. In this section, we revise the essential statementsand the advantages of using this method for calculating theSVD.

3.1 Iterative SVD

The method from [23] is outlined, in its basic form, as fol-lows:

– Given: SVD of Rd×nk � Ak = UkΣkV Tk , nk d, and

rank(Ak) =: rk– Aim: Compute SVD for Ak+1 = [Ak, Bk],

Bk ∈ Rd×mk , mk := nk+1 − nk

– Update: Ak+1 = Uk+1Σk+1V Tk+1 with

Uk+1 = UkQ

[U 00 I

],

Vk+1 =[Vk 00 I

](P ′

k Pk)T

[V 00 I

],

where Q results from a QR-decomposition,Σk+1, U andV result from the SVD of a (rk +mk)×(rk +mk)matrix.Pk and P ′

k are permutation matrices.

For details, see [23]. In the original version of the iterativeSVD, the matrix Uk is (formally) of dimension d × d. Sincein image processing d captures the number of pixels of oneimage, an explicit representation of Uk consumes too muchmemory to be efficient which suggests to represent Uk interms of Householder reflections. This ensures that themem-ory consumption of the SVD of Ak is bounded by O(n2k +rkd), and the step k + 1 requires O(n3k+1 + d mk(rk + mk))

floating point operations.

3.2 Thresholding—Adaptive SVD

There already exist iterative methods to calculate an SVD,but for our purpose the approach from [23] has two favorableaspects. The first one is the possibility to perform blockwiseupdateswithmk > 1, that is, with several frames. The secondone is the ability to estimate the effect of appending Bk onthe singular values of Ak+1. In order to compute the SVD ofAk+1, Z := UT

k Bk is first calculated and aQRdecompositionwith column pivoting of Zrk+1:d,: = QRP is determined.The R matrix contains the information in the added data Bk

that is not already described by the singular vectors in Uk .Then, the matrix R can be truncated by a significance levelτ such that the singular values less than τ are set to zero inthe SVD calculation of

[Σ ′

k Z1:rk ,:PT

R

].

Therefore, one can determine only from the (cheap) calcu-lation of a QR decomposition, whether the new data containsignificant new information and the threshold level τ cancontrol how big the gain has to be for a data vector to beadded to the current SVD decomposition in an iterative step.

4 Description of the Algorithm

In this section,we give a detailed description of our algorithmto compute a background separation based on the adaptiveSVD.

4.1 Essential Functionalities

The algorithm in [23] was initially designed with the goalto determine the kernels of a sequence of columnwiseaugmentedmatrices using the V matrix of the SVDs. In back-ground subtraction, on the other hand, we are interested infinding a low-rank approximation of the column space of Aand therefore concentrate on theU matrix of the SVD whichwill allow us to avoid the computation and storage of V .

The adaptive SVD algorithm starts with an initializationstep called SVDComp, calculating left singular vectors andsingular values on an initial set of data. Afterward, dataare added iteratively by blocks of arbitrary size. For everyframe in such a block, the foreground is determined and thenthe SVDAppend step performs a thresholding described inSect. 3.2 to check whether the frame is considered in theupdate of the singular vectors and values that correspondingto the background.

123

Journal of Mathematical Imaging and Vision (2020) 62:1159–1172 1163

4.1.1 SVDComp

SVDComp performs the initialization of the iterative algo-rithm. It is given the matrix A ∈ R

d×n and a column number� and computes the best rank-� approximation A = UΣV T ,

U =: [U0,U′0], Σ =:

[Σ0 00 Σ ′

0

], V =: [V0, V ′

0],

by means of an SVD with Σ0 ∈ R�×�,U0 ∈ R

d×�, andV0 ∈ R

n�. Also, this SVD is conveniently computed bymeans of the algorithm from [23], as the thresholding of theaugmented SVDwill only compute and store an atmost rank-� approximation, truncating the Rmatrix in the augmentationstep to at most � columns. This holds both for initializationand update in the iterative SVD.

As mentioned already in Sect. 3.1,U0 is not stored explic-itly but in the form of Householder vectors h j , j = 1, . . . , �,stored in a matrix H0. Together with a small matrix U0 ∈R

��, we then have

U0 = U0

�∏j=1

(I − h j hTj ),

and multiplication with U0 is easily performed by doing �

Householder reflection and then multiplying with an � × �

matrix. Since V0 is not needed in the algorithm, it is neithercomputed nor stored.

4.1.2 SVDAppend

This core functionality augments a matrix Ak , given byUk,Σk, Hk , determined either by SVDComp or previousapplications of SVDAppend, by m new frames contained inthematrix B ∈ R

d×m as described in Sect. 3.1. The details ofthis algorithm based on Householder representation can befound in [23]. By the thresholding procedure from Sect. 3.2,one can determine, even before the calculation of the SVDif an added column is significant relative to the thresholdlevel τ . This saves computational capacities by avoiding theexpensive computation of the SVD for images that do notsignificantly change the singular vectors representing thebackground.

The choice of τ is significant for the performance of thealgorithm. The basic assumption for the adaptive SVD is thatthe foreground consists of small changes between frames.Calculating SVDComp on an initial set of frames and con-sidering the singular vectors, i.e., the columns ofU0, and therespective singular values give an estimate for the size of thesingular values that correspond to singular vectors describ-ing the background. With a priori knowledge of the maximalsize of foreground effects, τ can even be set absolutely to the

size of singular values that should be accepted. Of course,this approach requires domain knowledge and is not entirelydata driven.

Another heuristic choice of τ can be made by consider-ing the difference between two neighboring singular valuesσi − σi+1, i.e., the discrete slope of the singular values. Thelast and smallest singular values describe the least dominanteffects. These model foreground effects or small, negligibleeffects in the background. With increasing singular values,the importance of the singular vectors is growing. Basedon that intuition, one can set a threshold for the differ-ence of two consecutive singular values and take the firstsingular value exceeding the difference threshold as τ .Figure 1d illustrates a typical distribution of singular val-ues. Since we want the method to be entirely data driven,we choose this approach. The threshold τ is determined byi := min {i : σi − σi+1 < τ ∗} and τ = σi with the thresholdτ ∗ of the slope being determined in the following.

4.1.3 Re-Initialization

The memory footprint at the kth step in the algorithmdescribed in Sect. 3.1 is O(n2k + rk d) and grows withevery frame added in the SVDAppend step. Therefore, a re-initialization of the decomposition is necessary.

One possibility is to compute an approximation of Ak ≈UkΣkV T

k ∈ Rd×nk or the exact matrix Ak by applying

SVDComp to Ak with a rank limit of � that determines thenumber of singular vectors after re-initialization. This strat-egy has two disadvantages. The first one is that it needs Vk ,which is otherwise not needed for modeling the background,and hence would require unnecessary computations. Evenworse, though U0 ∈ R

�×�, Σ0 ∈ R�×�, and H0 ∈ R

d×� arereduced properly, the memory consumption of V0 ∈ R

nk�

still depends on the number of frames added so far.The second re-initialization strategy, referred to as (II),

builds on the idea of a rank-� approximation of a set offrames representing mostly the background. For every frameBi added in step k of the SVDAppend, the orthogonal projec-tion

Uk(:, 1 : i)(Uk(:, 1 : i)T Bi ),

i.e., the “background part” of Bi , gets stored successively.The value σi is determined in Sect. 4.1.2 as threshold for theSVDAppend step. If the number of stored background imagesexceeds a fixed size μ, the re-initialization gets performedvia SVDComp on the background images. No matrix V isnecessary for this strategy, and the re-initialization is basedon the background projection of the most recently appendedframes.

In the final algorithm, we use a third strategy, referred toas (III) which is inspired by the sequential Karhunen–Loeve

123

1164 Journal of Mathematical Imaging and Vision (2020) 62:1159–1172

Fig. 1 Example of artifacts due to a big foreground object that was added to the background. The foreground object in the original image (a)triggers singular vectors containing foreground objects falsely added to the background (b) in previous steps. These artifacts can thus be seen inthe foreground image (c)

basis extraction [16]. The setting is very similar, and the Vmatrix gets dropped after the initialization aswell. Theupdatestep with a data matrix Bk is performed just like the updatestep of the iterative SVD calculation in Sect. 3.1 based on thematrix [UkΣk, Bk]. The matrices Σk+1 and Uk+1 get trun-cated by a thresholding of the singular values at every updatestep. Due to this thresholding, the number of singular valuesand accordingly the number of columns ofUk have an upperbound. Therefore, the maximum size of the system is fixedand no re-initialization is necessary. Calculating the SVD of[UkΣk, Bk] is sufficient since due to

[UkΣk, Bk][UkΣk, Bk]T = UkΣkΣTk U

Tk + Bk B

Tk

= UkΣkVTk VkΣ

Tk U

Tk + Bk B

Tk

= [UkΣkVTk , Bk][UkΣkV

Tk , Bk]T

the eigenvectors and eigenvalues of the correlation matri-ces with respect to [UkΣk, Bk] and [UkΣkV T

k , Bk] are thesame. Therefore, the singular values of [UkΣk, Bk] and[UkΣkV T

k , Bk] are the same, being roots of the eigenvaluesof the correlation matrix. In our approach, we combine theadaptive SVD with the re-initialization based on UkΣk , i.e.,

we perform SVDComp onUkΣk , becausewewant to keep thethresholdingof the adaptiveSVD.This is essentially the sameas an update step inKarhunen–Loeve settingwith Bk = 0 andamore rigorous thresholding or a simple truncation ofUk andΣk . The thresholding strategy of the adaptive SVD Sect. 3.2is still valid, as the QR decomposition with column pivotingsorts the columns of the matrix according to the �2 norm andthe columns ofUkΣk are ordered by the singular values due to||UΣ:,i ||2 = σi .UkΣk already is in SVD form, and therefore,SVDComp at re-initialization is reduced to a QR decompo-sition to regain Householder vectors and a truncation of Uk

and Σk which is less costly than performing a full SVD.Since it requires the V matrix, the first re-initialization

strategy will not be considered in the following, where wewill compare only strategies (II) and (III).

4.1.4 Normalization

The concept of re-initialization via a truncation of Uk andΣk either directly through SVDComp of UkΣk or in theKarhunen–Loeve setting with thresholding of the singularvalues still has a flaw: The absolute value of the singularvalues grows with each frame appended to UkΣk as

123

Journal of Mathematical Imaging and Vision (2020) 62:1159–1172 1165

n∑i=1

σ 2i = ‖A‖2F .

This also accounts for

nk+1∑i=1

σ 2nk+1,i = ‖Uk+1Σk+1‖2F

≈ ‖[UkΣk, Bk]‖2F= ‖UkΣk‖2F + ‖Bk‖2F .

The approximation results from the thresholding performedat the update step.Asonly small singular values get truncated,the sum of the squared singular values grows essentially withthe Frobenius norm of the appended frames. Growing singu-lar values do not only introduce numerical problems, theyalso deteriorate thresholding strategies, and the influence ofnewly added single frames decreases in later steps of themethod. Therefore, some upper bound or normalization ofthe singular values is necessary.

Karhunen–Loeve [16] introduce a forgetting factor ϕ ∈[0, 1] and update as [ϕUkΣk, Bk]. They motivate this factorsemantically: More recent frames get a higher weight. Rosset al. [26] show that this value limits the observation history.With an appending block size of m, the effective number ofobservations ism/(1−ϕ). By the Frobenius norm argument,the singular values then have an upper bound. By the samemotivation, the forgetting factor could also be integrated intostrategy (III). Moreover, due to

‖(ϕUkΣk):,i‖2 = ∥∥ϕ σi U:,i∥∥2 = ϕσi ,

the multiplication with the forgetting factor keeps the orderof the columns ofUkΣk and linearly affects the 2-norm and isthus compliant with the thresholding. However, the concretechoice of the forgetting factor is unclear.

Another idea for normalization is to set an explicit upperbound for the Frobenius norm of observations contributingto the iterative SVD, or, equivalently, to

∑σ 2i = ‖A‖2F . At

initialization, i.e., at the first SVDComp, the upper bound is

determined by‖A‖2Fn η with n being the number of columns of

A and η being the predefined maximum size of the system.This upper bound is amultiple of themean squared Frobeniusnorm of an input frame, and we define a threshold ρ :=‖A‖F√

n

√η. If the Frobenius norm ‖Σ0‖F of the singular values

exceeds ρ after a re-initialization step,Σ0 gets normalized toΣ0

ρ‖Σ0‖F . One advantage of this approach is that the effective

system size can be transparently determined by the parameterη.

In data science, normalization usually aims for zero meanand standard deviation one. Zero mean over the pixels in theframes, however, leads to subtracting the rowwise mean of

A, replacing A by (I −11T )A. This approach is discussed inincremental PCA, cf. [26], but since the mean image usuallycontributes substantially to the background, it is not suitablein our application.

A framewise unit standard deviation makes sense sincethe standard deviation approximates the contrast in imageprocessing and we are interested in the image content regard-less of the often varying contrast of the individual frames.Different contrasts on a zero mean image can be seen as ascalar multiplication which also applies for the singular val-ues. Singular values differing with respect to the contrast arenot a desirable effect which is compensated by subtractingthe mean and dividing by the standard deviation of incomingframes B, yielding B−μ

σ. Due to the normalization of single

images, the upper bound for the Frobenius norm ρ is more amultiple of the Frobenius norm of an average image.

4.2 Adaptive SVD Algorithm

The essential components being described, we can nowsketch ourmethodbased on the adaptiveSVD inAlgorithm1.

Data: Images of a static camera and a matrix A of initializationimages.

Result: Background and foreground images for every inputimage.

1 U ,Σ, i ← SVDComp(A, �, τ ∗);2 while there are input images do3 B ← read, vectorize, and normalize the current image;4 // project B onto the current background model;5 J ← U:,1:i (U

T:,1:iB);

6 // subtract the background from B and use this as mask on theinput image;

7 F ← B · (|B − J| > θ);8 // build a block of input images;9 M ← [M, B];

10 // append a block of images;11 if M .cols == β then12 U ,Σ, i ← SVDAppend(U, Σ , M, τ ∗);13 M ← [ ];14 end15 // re-initialization if maximum size is exceeded;16 if U .cols > n∗ then17 U ,Σ ← SVDComp(UΣ , �);18 end19 end

Algorithm 1: Background Subtraction using adaptiveSVD.

The algorithm uses the following parameters:

– �: Parameter used in SVDComp for rank-� approxima-tion.

– η: Parameter for setting up the maximal Frobenius normas a multiple of the Frobenius norm of an average image.

123

1166 Journal of Mathematical Imaging and Vision (2020) 62:1159–1172

– τ ∗: Threshold value for the slope of the singular valuesused in SVDAppend.

– θ : Threshold value depending on the pixel intensity rangeto discard noise in the foreground image.

– β: Number of frames put together to one block Bk forSVDAppend.

– n∗: Maximum number of columns ofUk . If n∗ is reacheda re-initialization is triggered.

For the exposition in Algorithm 1, we use pseudo-codewith a MATLAB like syntax. Two further explanations arenecessary, however. First, we remark that SVDAppend andSVDComp return the updated matrices U and Σ and theindex of the thresholding singular value determined by τ ∗as described in Sect. 4.1.2. Using the threshold value θ , theforeground resulting from the subtraction of the backgroundfrom the input image gets binarized. This binarization is usedas mask on the input image to gain the parts that are con-sidered as foreground. |B − J | > θ checks elementwisewhether |Bjk − J jk | > θ and returns a matrix consisting ofthe Boolean values of this operation.

4.3 Relaxation of the Small Foreground Assumption

A basic assumption of our background subtracting algorithmis that the changes due to the foreground are small relativeto the image size. Nevertheless, this assumption is easilyviolated, e.g., by a truck in traffic surveillance or generallyby objects close to the camera which can appear in singu-lar vectors that should represent background. This has twoconsequences. The first is that the foreground object is notrecognized as such and the second one leads to ghostingeffects because of the inner product as shown in Fig. 1.The following modifications increase the robustness of ourmethod against these unwanted effects.

4.3.1 Similarity Check

Big foreground objects can exceed the threshold level τ

in SVDAppend and therefore are falsely included in thebackground space.With the additional assumption that back-ground effects have to be stable over time, frames with largemoving objects can be filtered out by utilizing the blockappending property of the adaptive SVD. There, a largemov-ing object causes significant differences in a block of imageswhich can be detected by calculating the structural similarityof a block of new images.Wang et al. propose in [33] the nor-malized covariance of two images to capture the structuralsimilarity. This again can be written as the inner product of

normalized images, i.e.,

s(Bi , Bj ) = 1

d − 1

d∑l=1

Bi,l − μi

σi

B j,l − μ j

σ j,

with Bi and Bj being two vectorized images with d pixels,mean values μi and μ j , and standard deviations σi and σ j .Considering that the input images already become normal-ized in our algorithm, see Sect. 4.1.4, this boils down to aninner product.

Given is a temporally equally spaced and ordered blockof images B := {B1, B2, . . . , Bm} and one frame Bi withi ∈ {1, 2, . . . ,m} =: M . The structural similarity of frameBi regarding the block B is the measure we search for. Thiscan be calculated by

1

m − 1

∑j∈M\{i}

s(Bi , Bj ),

i.e., the mean structural similarity of Bi regarding B. For therelatively short time span of one block, it generally holds thats(Bi , Bj ) ≥ s(Bi , Bk) with i, j, k ∈ M and i < j < k, i.e.,the structural similarity drops going further into the futureas motions in the images imply growing differences. Thiseffect causes the mean structural similarity of the first or lastframes of B generally being lower than of the middle onesdue to the higher mean time difference to the other frames inthe block.

This bias can be avoidedby calculating themean similarityregarding subsets of B. Let ν > 0 be a fixed number of pairsto be considered for the calculation of themean similarity and T ∈ N

+ be the fixed cumulative time difference. Calculatethemean similarity si of Bi regarding B by selecting pairwisedistinct { j1, j2, . . . , jν} from M\{i} withν∑

l=1

| jl − i | = T and si = 1

ν

(ν∑

l=1

s(Bi , Bjl )

).

If si is smaller than the predefined similarity threshold s,frame i is not considered for the SVDAppend.

4.3.2 Periodic Updates

Using the threshold τ speeds up the iterative process, butalso has a drawback: If the incoming images stay constantover a longer period of time, the background should mostlyrepresent the input images and there should be high singu-lar values associated with the singular vectors describing it.Since input images that can be explained well do not getappended anymore, this is, however, not the case. Anotherdrawback is that outdated effects, like objects that stayedin the focus for quite some time and then left again, have

123

Journal of Mathematical Imaging and Vision (2020) 62:1159–1172 1167

a higher singular vectors than they should, as they are notrelevant anymore. Therefore, it makes sense to periodicallyappend images although they are seen as irrelevant and do notsurpass τ . This also helps to remove falsely added foregroundobjects much faster.

4.3.3 Effects of the Re-Initialization Strategy

The re-initialization strategy (II) based on the background

imagesUk(:, 1 : i)(Uk(:, 1 : i)T Bi ) as described inSect. 4.1.3supports the removal of incorrectly added foregroundobjects.When such an object, say X , is gone from the scene, i.e.,

Bi does not contain X and Uk(:, 1 : i)(Uk(:, 1 : i)T Bi ) doesnot contain it either because a singular vector not con-taining X approximates Bi much better. As X was addedto the background, there must be at least one columnj∗ of Uk containing X , i.e., Uk(:, 1 : j∗)T X � 0. As

Uk(:, 1 : i)(Uk(:, 1 : i)T Bi )does not contain X , (Uk(:, 1 : i)TBi ) j∗ must be close to zero as otherwise the weighted addi-

tion of singular vectors Uk(:, 1 : i)(Uk(:, 1 : i)T Bi ) cancelsX out. The re-initialization is thus based on images not con-taining X , and the new singular vectors also do not containleftovers of X anymore.

Finally, the parameter ηmodifies the size of the maximumFrobenius norm used for normalization in re-initializationstrategy (III) from Sect. 4.1.4. A smaller η reduces the impor-tance of the already determined singular vectors spanningthe background space and increases the impact of newlyappended images. If an object X was falsely added, itgets removed more quickly if current frames not contain-ing X have a higher impact. A similar behavior like withre-initialization strategy (II) can be achieved. The disad-vantage is that the background model changes quickly anddoes not capture long time effects that well. In the end,it depends on the application which strategy performs bet-ter.

5 Computational Results

The evaluation of our algorithm is done based on animplementation in the C++ programming language usingArmadillo [27] for linear algebra computations.

5.1 Default Parameter Setting

Algorithm 1 depends on parameters that are still to be speci-fied. In the following, we will introduce a default parametersetting that works well in many different applications. Theparameters could even be improved or optimized for a spe-cific application using ground-truth data. Our aim here,however, is to show that the adaptive SVD algorithm is a

very generic one and applicable almost “out of the box” forvarious situations. The chosen default parameters are as fol-lows:

– � = 15,– n∗ = 30,– η = 30,– τ ∗ = 0.05 · ρ, with ρ = ||A||F√

n

√η of the initialization

matrix A ∈ Rd×n ,

– β = 6, ν = 3, T = 6, s = 0.97,– θ = 1.0.

The parameter � determines how many singular values andcorresponding singular vectors are kept after re-initialization.Setting � too low can cause a loss of background informa-tion. In our examples, 15 turned out to be sufficient not tolose information. The re-initialization is triggered when n∗relevant singular values have been accumulated. Choosingthat parameter too big reduces the performance, as the float-ing point operations per SVDAppend step depend cubicallyon the number of singular vectors and linearly on the numberof singular vectors times the number of pixels, see Sect. 3.1.The system size η controls the impact of newly appendedframes, and a large value of η favors a stable background.The threshold value τ ∗ for the discrete slope of singular val-ues in the SVDAppend step depends on the data. The heuristicfactor 0.05 proved to be effective to indicate that the curveof the singular values flattens out. The block size β and thecorresponding ν and T depend on the frame rate of theinput. The choice is such that it does not delay the update ofthe background space too much, which would be the effectof a large block size. Keeping it relatively small, we are ableto evaluate the input regarding similarity and stable effects.Due to the normalization of the input images to zero meanand standard deviation one, the similarity threshold s andthe binarization threshold θ are stable against different inputtypes.

5.2 Small Foreground Objects

The first example video for a qualitative evaluation is from awebcammonitoring the city of Passau,Germany, fromabove.The foreground objects, e.g., cars, pedestrians, and boats,are small or even very small. The frame rate of 2 framesper minute is relatively low, and the image size is 640 ×480px.This situation allows for a straightforward applicationof the basic adaptive SVDalgorithmwithout similarity checkand regular updates. The remaining parameters are as in thedefault setting of Sect. 5.1.

123

1168 Journal of Mathematical Imaging and Vision (2020) 62:1159–1172

Fig. 2 Example frame from a webcam video monitoring the city of Passau. In (a), the input image is shown and in (b) the foreground image as aresult of Algorithm 1

Fig. 3 Plot marking the true detections in the foreground image ofFig. 2b by green circles and incorrect detections by red circles withwhite stripes (Color figure online)

In Fig. 2, an example frame1 and the according foregroundimage from the webcam video are shown. The moving boatin the foreground, the cars in the lower left and right corners,and even the cars on the bridge in the background are detectedwell. Small illumination changes and reparking vehicles leadto incorrect detections on the square in the front. Figure 3depicts these regions.

5.3 Handling of Big Foreground Objects

Figure 1 is a frame from an example video1 including theprojection onto the background space, the computed fore-

1 The complete sample videos can be downloaded following https://www.forwiss.uni-passau.de/en/media_and_data/.

ground image, and the distribution of the singular values.To illustrate the improvements due to similarity checks andperiodic updates, the same frame is depicted in Fig. 4 wherethe extended version of our algorithm is applied. The arti-facts due to big foreground objects that were added to thebackground in previous frames are not visible anymore. Theperson in the image still gets added to the background, butonly after being stationary for some frames.

5.4 Execution Time

The performance of our implementation is evaluated basedon an Intel® Core™ i7-4790 CPU @ 3.60 Hz × 8. Theexample video from Sect. 5.3 has a resolution of 1920 ×1080px with 25 fps. For the application of our algorithmon the example video, the parameters are set as shown inSect. 5.1.

As the video data were recorded with 25 fps, there isno need to consider every frame for a background updatebecause the background is assumed to be constant overa series of frames and can only be detected consideringa series of frames. Therefore, only every second frameis considered for a background update, while a back-ground subtraction using the current singular vectors isperformed on every frame. Our implementation with thesettings from Sect. 5.1 handles this example video with8 fps.

For surveillance applications, it is important that the back-ground subtraction is applicable in real time for which 8 fpsare too slow. One approach would be to reduce the resolu-tion. The effects of that will be discussed in the followingsection. Leaving the resolution unchanged, the parametershave to be adapted. Setting � = 10 and n∗ = 25 signifi-

123

Journal of Mathematical Imaging and Vision (2020) 62:1159–1172 1169

Fig. 4 The same scene as in Fig. 1. The artifacts due to big foreground objects are reduced by similarity checks and regular updates. The currentforeground object gets added to the background only after being stationary for a series of frames

cantly reduces the number of background effects that can becaptured, but turns out to be still sufficient for this particu-lar scene. The number of images considered for backgroundupdates can be reduced as well. Downsampling the framesby averaging over a window size of 8 and setting � = 10 andn∗ = 25 leads to a processing rate of 25 fps which is realtime.

In Sect. 3.1, we point out that the number of floatingpoint operations for an update step depends linearly on thenumber of pixels d when usingHouseholder reflections.A re-initialization step is computationally even cheaper, becauseonly Householder vectors have to be updated. The followingexecution time measurements underline the theoretical con-siderations. Our example video is resized several times, 900images are appended, and re-initialization is performedwhenn∗ singular vectors are reached. Table 1 shows the summedup time for the append and re-initialization steps during itera-tion for the given image sizes. The number d of pixels equals2, 073, 600 = 1920 · 1080.

The factors td/i/(td/16 · 16i ) with i ∈ {1, 2, 4, 8, 16}

and total append or re-initialization times td/i are shown inTable 1 for image sizes d/i . These factors should be con-

Table 1 Execution time for performing an SVD update iteratively on900 frames for different image sizes and d = 2073600 = 1920 · 1080#Pixels Append (s) Factor Re-init. (s) Factor

d 69.30 1.90 16.46 1.37

d/2 34.28 1.88 8.25 1.38

d/4 16.67 1.83 3.79 1.26

d/8 6.72 1.47 1.68 1.12

d/16 2.28 1 0.75 1

stant for increasing image sizes due to the linear dependency.Still, the factors keep increasing, but even less than a loga-rithmic order. This additional increase in execution time canbe explained due to the growing amount of memory that hasto be managed and caching becomes less efficient as withsmall images.

5.5 Evaluation on Benchmark Datasets

The quantitative evaluation is performed on example videosfrom the background subtraction benchmark dataset CDnet

123

1170 Journal of Mathematical Imaging and Vision (2020) 62:1159–1172

Fig. 5 Example frame from the pedestrians video of the CDnetdatabase

2014 [32]. The first one is the pedestrians video belongingto the baseline category. It contains 1099 frames (360× 240px) of people walking and cycling in the public. An exampleframe is shown in Fig. 5.

For the frames 300 through 1099, binary ground-truthannotations exist that distinguish between foreground andbackground. From the first 299 frames, 15 frames areequidistantly subsampled and taken for the initial matrix M .Thereafter, Algorithm 1 is executed on all frames from 300through 1099. Instead of applying the binary mask in line 7of Algorithm 1 onto the input image, the mask itself is theoutput to achieve binary images.

With the default parameter setting of Sect. 5.1, the pixel-wise precision of 0.967 andF-Measure of 0.915 are achievedwith a performance of 843 fps. The thresholding leading tothe binary mask is sensitive to the contrast of the foregroundrelative to the background. If it is low, foreground pixelsare not detected properly. To avoid missing pixels withinforeground objects, themorphological close operation is per-formedwith a circular kernel.Moreover, a fixedminimal sizeof foreground objects can be assumed reducing the number offalse positives. These two optimizations lead to the precisionof 0.973 and F-Measure of 0.954 at 684 fps. The completeevaluation measures are given in Table 2. Default representsthe default parameter setting and morph the version with theadditional optimizations. In the following, themorphologicalpostprocessing is always included.

Table 3 Evaluation of the adaptive SVD algorithm on example videosfrom the CDnet dataset using precision and F-Measure. Prec∗ andF-Meas∗ give the precision and F-Measure of the best unsupervisedmethod of the benchmark regarding the test video

Video Prec F-Meas Prec* F-Meas*

highway 0.901 0.816 0.935 0.954

park 0.841 0.701 0.776 0.765

tram 0.957 0.812 0.838 0.886

turnpike 0.962 0.860 0.980 0.881

blizzard 0.919 0.854 0.939 0.845

streetLight 0.992 0.982 0.984 0.983

Our method delivers a state-of-the-art performance forunsupervisedmethods.Thebest unsupervisedmethod, IUTIS-5 [1] on the benchmark site could achieve the precisionof 0.955 and F-Measure of 0.969. It is based on geneticprogramming combining other state-of-the-art algorithms.The execution time is not given, but naturally higher thanthe execution time of the slowest algorithm used, assumingperfectly parallel execution. We introduced domain knowl-edge only in the morphological optimizations. Otherwise,there is no specific change toward the test scene. Even moredomain knowledge is used in supervised learning techniquesas object shapes are trained and irrelevant movements in thebackground are excluded due to labeling. They are able tooutperform our approach regarding the evaluation measures.An overall average precision and F-Measure of more than0.98 is, for example, achieved from FgSegNet [17]. Never-theless, the authors mention that their postprocessing is doneon benchmark data leading to a bias as the short sequenceslookverymuchalike.Moreover, the benchmark site itself dis-claims that the supervised methods may have been trained onevaluation data as ground-truth annotations are only availablefor evaluation data.

The positive effect of a blockwise appending of the datawith a similarity check and regular updates as shown abovealso applies here: Our adaptive SVD algorithm on the givenpedestrians video from the benchmark site without usingthe similarity checks and regular updates only leads to theprecision of 0.933 and F-Measure of 0.931.

The performance of our algorithm on more examplevideos from the CDnet dataset is listed in Table 3. The parkvideo is recorded with a thermal camera, the tram (CDnet:

Table 2 Evaluation of the pedestrians scene of the CDnet database with the benchmark evaluation metrics including FPR (false-positive rate),FNR (false-negative rate), PBC (percentage of wrong classifications)

Recall Specificity FPR FNR PBC Precision F-Measure

default 0.869 1.000 0.000 0.131 0.158 0.967 0.915

morph 0.936 1.000 0.000 0.063 0.088 0.973 0.954

123

Journal of Mathematical Imaging and Vision (2020) 62:1159–1172 1171

tramCrossroad_1fps) and turnpike (CDnet: turnpike_0_5fps)videos with a low frame rate, and the blizzard video whilesnow is falling. For highway and park, the best unsuper-vised method is IUTIS-5 and for tram, turnpike, blizzard,and streetLight that is SemanticBGS [7]. SemanticBGS com-bines IUTIS-5 and a semantic segmentation deep neuralnetwork, and the execution time is given with 7 fps for473 × 473px images based on a NVIDIA GeForce GTXTitan X GPU.

Besides the park video, the content is mostly vehiclesdriving by. The performance of our algorithm clearly dropswhenever the initialization image set contains a lot of fore-ground objects like in the highway video, where the street isnever empty. Moreover, a foreground object turns into back-ground when it stops moving which is even a feature of ouralgorithm. This, however, causes problems in a lot of thebenchmark videos of the CDnet dataset with vehicles stop-ping at traffic lights, like in the tram video, or people stoppingand starting to move again. There is a category of videoswith intermittent objectmotion in theCDnet benchmark. Ouralgorithm performs with an average precision of 0.752 andF-Measure of 0.385, whereas SemanticBGS reaches an aver-age precision of 0.915 andF-Measure of 0.788. The precisionof our algorithm tends to be higher than the F-Measure, asit detects motion very well and therefore is certain that ifthere is movement, it is foreground, but often foreground isnot detected due to a lack of motion. To delay the additionof a static object to the background, it is possible to reducethe regular updates, for example. But as this feature regu-lates the adaption of the background model to a change inthe background, this only enhances the performance for verystable scenes. In the streetLight video, no regular update wasperformed in contrast to the other videos. Including regularupdates, the precision is 0.959 and the F-Measure 0.622 dueto cars stopping at traffic lights. The only domain knowledgewe introduce is the postprocessing via morphological opera-tions. Otherwise, the algorithm has no knowledge about thekind of background it models. Therefore, not only vehicles orpeople are detected as foreground, but alsomovement of treesor the reflection of the light of the vehicles on the ground,which is negative for the performance regarding the CDnetbenchmark.

6 Conclusions

We utilized the iterative calculation of a singular valuedecomposition to model a common subspace of a series offrames which is assumed to represent the background ofthe frames. An algorithm, the adaptive SVD was developedand applied for background subtraction in image processing.The assumption that the foreground has to be small objectswas considered in more detail and relaxed by extensions of

the algorithm. In an extensive evaluation, the capabilities ofour algorithm were shown qualitatively and quantitativelyusing example videos and benchmark results. Compared tostate-of-the-art unsupervised methods, we obtain competi-tive performance with even superior execution time. Evenhigh-definition videos can be processed in real time.

The evaluation also showed that if an application to adomain such as video surveillance is intended, our algorithmwould need to be extended to also consider semantic infor-mation. Therefore, it can only be seen as a preprocessing step,e.g., reducing the search space for classification algorithms.In future work, we aim to evaluate the benefit of using ouralgorithm in preprocessing of an object classifier. Moreover,we will address the issue of foreground objects turning intobackground after being static for some time which is desir-able in some cases and erroneous in others. A first approachis to use tracking because objects do not disappear withoutany movement. In the end, there is also some paralleliza-tion ability in our algorithm separating the projection ontothe background of incoming images from the update of thebackground model. Further performance improvements willbe investigated.

Acknowledgements OpenAccess funding provided by Projekt DEAL.Our work results from the project DeCoInt2, supported by the GermanResearch Foundation (DFG) within the priority program SPP 1835:“Kooperativ interagierende Automobile,” Grant Numbers DO 1186/1-1, FU 1005/1-1, and SI 674/11-1.

Open Access This article is licensed under a Creative CommonsAttribution 4.0 International License, which permits use, sharing, adap-tation, distribution and reproduction in any medium or format, aslong as you give appropriate credit to the original author(s) and thesource, provide a link to the Creative Commons licence, and indi-cate if changes were made. The images or other third party materialin this article are included in the article’s Creative Commons licence,unless indicated otherwise in a credit line to the material. If materialis not included in the article’s Creative Commons licence and yourintended use is not permitted by statutory regulation or exceeds thepermitted use, youwill need to obtain permission directly from the copy-right holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

References

1. Bianco, S., Ciocca, G., Schettini, R.: Combination of video changedetection algorithms by genetic programming. IEEETrans. Evolut.Comput. 21(6), 914–928 (2017)

2. Bouwmans, T.: Recent advanced statistical background modelingfor foreground detection: a systematic survey. Recent Patents Com-put. Sci. 4, 147–176 (2011)

3. Bouwmans, T.: Traditional and recent approaches in backgroundmodeling for foreground detection: an overview. Comput. Sci. Rev.11–12, 31–66 (2014)

4. Bouwmans, T., Javed, S., Sultana, M., Jung, S.K.: Deep neuralnetwork concepts for background subtraction: a systematic reviewand comparative evaluation. Neural Netw. 117, 8–66 (2019)

123

1172 Journal of Mathematical Imaging and Vision (2020) 62:1159–1172

5. Bouwmans, T., Sobral, A., Javed, S., Jung, S.K., Zahzah, E.H.:Decomposition into low-rank plus additive matrices for back-ground/foreground separation: a review for a comparative evalua-tion with a large-scale dataset. Comput. Sci. Rev. 23, 1–71 (2017)

6. Bouwmans, T., Zahzah, E.H.: Robust PCAvia principal componentpursuit: a review for a comparative evaluation in video surveillance.Comput. Vis. Image Understand. 122, 22–34 (2014)

7. Braham, M., Piérard, S., Van Droogenbroeck, M.: Semantic back-ground subtraction. In: 2017 IEEE International Conference onImage Processing (ICIP), pp. 4552–4556 (2017)

8. Candès, E.J., Li,X.,Ma,Y.,Wright, J.: Robust principal componentanalysis? J. ACM 58(3), 1–37 (2011)

9. Erichson, N.B., Brunton, S.L., Kutz, J.N.: Compressed singularvalue decomposition for image and video processing. In: 2017IEEE International Conference on Computer Vision Workshops(ICCVW), pp. 1880–1888 (2017)

10. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. JohnsHopkins University Press, Baltimore (1996)

11. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MITPress (2016). http://www.deeplearningbook.org

12. Gracewell, J., John,M.:Dynamic backgroundmodeling using deeplearning autoencoder network.Multimed. ToolsAppl. 79(7), 4639–4659 (2020)

13. Guo, H., Qiu, C., Vaswani, N.: Practical ReProCS for separatingsparse and low-dimensional signal sequences from their sum—part1. In: 2014 IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP), pp. 4161–4165 (2014)

14. Kaloorazi, M.F., de Lamare, R.C.: Randomized rank-revealingUZV decomposition for low-rank approximation of matrices(2018). arXiv:1811.08597

15. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-basedlearning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)

16. Levey, A., Lindenbaum, M.: Sequential Karhunen–Loeve basisextraction and its application to images. IEEE Trans. Image Pro-cess. 9(8), 1371–1374 (2000)

17. Lim, L.A., Keles, H.Y.: Foreground segmentation using convo-lutional neural networks for multiscale feature encoding. PatternRecognit Lett. 112, 256–262 (2018)

18. Liu, X., Wen, Z., Zhang, Y.: Limited memory block Krylovsubspace optimization for computing dominant singular valuedecompositions. SIAM J. Sci. Comput. 35(3), A1641–A1668(2013)

19. Lu, Y., Ino, F., Matsushita, Y.: High-performance out-of-coreblock randomized singular value decomposition on GPU (2017).arXiv:1706.07191

20. Minematsu, T., Shimada, A., Uchiyama, H., Taniguchi, R.: Analyt-ics of deep neural network-based background subtraction. MDPIJ. Imag. 4(6), 78 (2018)

21. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J.,Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K.,Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou,I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.:Human-level control through deep reinforcement learning. Naturepp. 518–529 (2015)

22. Narayanamurthy, P., Vaswani, N.: A fast and memory-efficientalgorithm for robust PCA (MEROP). In: 2018 IEEE Interna-tional Conference on Acoustics, Speech and Signal Processing(ICASSP), pp. 4684–4688 (2018)

23. Peña, J.M., Sauer, T.: SVD update methods for large matrices andapplications. Linear Algebra Appl. 561, 41–62 (2019)

24. Rodriguez, P.,Wohlberg, B.: Incremental principal component pur-suit for videobackgroundmodeling. J.Math. Imag.Vis.55(1), 1–18(2016)

25. Rodríguez, P., Wohlberg, B.: Translational and rotational jitterinvariant incremental principal component pursuit for video back-

ground modeling. In: 2015 IEEE International Conference onImage Processing (ICIP), pp. 537–541 (2015)

26. Ross,D.A., Lim, J., Lin, R.S., Yang,M.H.: Incremental learning forrobust visual tracking. Int. J. Comput. Vis. 77(1), 125–141 (2008)

27. Sanderson, C., Curtin, R.: Armadillo: a template-basedC++ libraryfor linear algebra. J. Open Source Softw. 1, 26 (2016)

28. Schmidt, E.: Zur Theorie der linearen und nichtlinearen Integral-gleichungen. I. Teil. Entwicklung willkürlicher Funktionen nachSystemen vorgeschriebener. Math. Annalen 63, 433–476 (1907)

29. Stewart, G.W.: On the early history of the singular value decom-position. SIAM Rev. 35(4), 551–566 (1993)

30. Sultana, M., Mahmood, A., Javed, S., Jung, S.K.: Unsuperviseddeep context prediction for background estimation and foregroundsegmentation. Multimed. Tools Appl. 30(3), 375–395 (2019)

31. Vaswani, N., Bouwmans, T., Javed, S., Narayanamurthy, P.: Robustsubspace learning: robust PCA, robust subspace tracking, androbust subspace recovery. IEEE Signal Process. Mag. 35(4), 32–55(2018)

32. Wang, Y., Jodoin, P., Porikli, F., Konrad, J., Benezeth, Y., Ishwar,P.: CDnet 2014: an expanded change detection benchmark dataset.In: 2014 IEEEConference on Computer Vision and Pattern Recog-nition Workshops, pp. 393–400 (2014)

33. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image qual-ity assessment: from error visibility to structural similarity. IEEETrans. Image Process. 13(4), 600–612 (2004)

Publisher’s Note Springer Nature remains neutral with regard to juris-dictional claims in published maps and institutional affiliations.

Günther Reitberger received theB.Sc. and the M.Sc. degrees inComputer Sciences from the Uni-versity of Passau, Germany, in 2014and 2016, respectively. Currently,he is working on his PhD at theInstitute for Software Systems inTechnical Applications of ComputerScience (FORWISS) of the Uni-versity of Passau, Germany. Hisresearch interests include mathe-matical image processing, stereovision, artificial intelligence, objectdetection, and object tracking.

Tomas Sauer received his PhDdegree in mathematics from theUniversity of Erlangen-Nürnbergin 1993 for research in Approx-imation Theory. From 2000 to 2012,he was professor for NumericalMathematics/Scientific Computa-tion at the University of Gießen,and since 2012, he holds the chairfor Mathematical Image Process-ing at the University of Passauand is also director of the AppliedResearch Institute FORWISS andof the Fraunhofer IIS ResearchGroup on “Knowledge-Based Image

Processing.” His current research interests include, among others, sig-nal and image processing for huge data, tomography, machine learningand sparse reconstructions, and continued fractions.

123