Learning Visibility of Landmarks for Vision-Based Localization

Pablo F. Alcantarilla, Sang Min Oh, Gian Luca Mariottini, Luis M. Bergasa, Frank Dellaert

Abstract— We aim to perform robust and fastvision-based localization using a pre-existing largemap of the scene. A key step in localization is asso-ciating the features extracted from the image withthe map elements at the current location. Althoughthe problem of data association has greatly benefitedfrom recent advances in appearance-based matchingmethods, less attention has been paid to the effectiveuse of the geometric relations between the 3D mapand the camera in the matching process.

In this paper we propose to exploit the geometricrelationship between the 3D map and the camerapose to determine the visibility of the features. Inour approach, we model the visibility of every mapfeature with respect to the camera pose using a non-parametric distribution model. We learn these non-parametric distributions during the 3D reconstructionprocess, and develop efficient algorithms to predictthe visibility of features during localization. With thisapproach, the matching process only uses those mapfeatures with the highest visibility score, yielding amuch faster algorithm and superior localization re-sults. We demonstrate an integrated system based onthe proposed idea and highlight its potential benefitsfor the localization in large and cluttered environ-ments.

I. Introduction

We aim to perform fast and robust vision-based lo-calization, using an a-priori obtained accurate 3D mapof the environment. This map can be reconstructed bytechniques such as bundle adjustment [1], [2] or VisualSLAM [3]. Such a setup is feasible and would be useful inmany applications, among others to aid the navigation ofthe visually impaired, which is our primary motivation. Avision-based localization system would be able to providean accurate pose aligned with the head orientation, whichcan enable a system to provide the visually impaired withinformation about their current position and orientationand/or guide them to their destination through diversesensing modalities [4].

Pablo F. Alcantarilla and Luis M. Bergasa are with Departmentof Electronics, University of Alcala. Alcala de Henares, Madrid,Spain. e-mail: pablo.alcantarilla, bergasa@depeca.uah.es

Sang Min Oh is now with Kitware Inc., Clifton Park, NY 12065USA. This work was done when Sang Min Oh was at GeorgiaInstitute of Technology. e-mail: sangmin.oh@kitware.com

Gian Luca Mariottini is with Department of Computer Scienceand Engineering, University of Minnesota, Minneapolis, MN 55455,USA. e-mail: gianluca@cs.umn.edu

Frank Dellaert is with School of Interactive Computing, GeorgiaInstitute of Technology, Atlanta, GA 30332, USA. e-mail: del-laert@cc.gatech.edu

A key problem in obtaining accurate pose is associatingthe features extracted from the image with the mapelements at the current location. Solving this matchingproblem becomes very challenging when the map con-tains a very large number of 3D landmarks, e.g., a fewmillion, and/or has a complex spatial structure. Recentadvances in appearance-based methods have improvedthe matching process for localization substantially. Acommon trend has been the efficient use of invariantappearance descriptors [5], [6]. Less attention has beenpaid to the effective use of the geometric relations be-tween both the 3D landmarks and the camera pose toachieve this goal. In many cases, the only use of geometricinformation can be efficiently used to prune out the 3Dlandmarks which are beyond the proximity from thelatest known camera pose.

A promising avenue that has been explored in theliterature is predicting whether a feature will be visibleor not in the image, based on a rough estimate of thecamera pose. For example, in the visual SLAM work byDavison [3], the visibility of a 3D landmark is predictedbased on the similarity of the current camera pose w.r.t.(with respect to) a memorized reference pose, recordedwhen the landmark is first seen. Although this visibilitycriterium works reasonably well, each 3D landmark mem-orizes only one reference camera pose and more complexvisibility criteria can not be exploited. In the vein, Salaet al. [7] studied the optimal feature selection for robotnavigation by means of a graph theoretical formulationwhere the landmarks are divided into groups with similarvisibility. However, the visibility modeling for every fea-ture is still limited since, for a given visibility region onlythe same set of k features is visible (being k a parameterdefined by the pose-estimation algorithm). In [8] the au-thors showed an augmented reality application in whichthey model for every feature the probability distributionfrom which the feature can be tracked successfully bymeans of a finite set of Gaussian mixtures. The extensionof this method for larger environments is difficult and theuse of a Gaussian kernel makes this approach limited. Inaddition, in their visibility modeling they only considercamera translation and not orientation, which in factis more important than translation for modeling thevisibility of each feature. Zhu et. al. [9] studied howto build an optimal landmark database rapidly and usethis database online for real-time global localization. Itwas shown that by using an intelligent subsampling ofthe landmark database the size of the database can be

2010 IEEE International Conference on Robotics and AutomationAnchorage Convention DistrictMay 3-8, 2010, Anchorage, Alaska, USA

978-1-4244-5040-4/10/$26.00 ©2010 IEEE 4881

reduced without dropping the accuracy. In this approachlandmarks are characterized by its appearance usingHistogram of Oriented Gradients (HOG) [10] and theselection of putative matches for pose estimation reliesvastly on appearance descriptors, not exploiting all thegeometry information.

In this paper we generalize these approaches and intro-duce a memory-based learning framework to predict, foreach feature, its visibility w.r.t. the varying camera pose.Our approach efficiently captures the visibility of land-marks located in structurally complex or cluttered envi-ronments. This is made possible by memorizing multiplereference poses for each feature, using non-parametriclearning methods. In the remainder of the paper, theoverall framework of our 6-DoF vision-based localizationsystem is shown. Then, the importance of the visibilityprediction problem is discussed, and our solution basedon the non-parametric framework is presented along withan efficient implementation. Finally, the experimental re-sults demonstrate the benefits of the proposed approach.Our results show that with our method, the visibilityof 3D landmarks is predicted more accurately and thealgorithm improves the quality of the matches as well asof the localization algorithm accuracy.

II. Vision-Based Camera Localization

We describe the localization problem and our overallframework in this section. A map M is commonly definedas a set of high quality landmarks reconstructed fromthe images. Such a 3D map comprises of the location ofeach feature and can be obtained through visual SLAMtechniques, e.g., [11]. Even though stereo camera may beused for the 3D map reconstruction, we would focus onlocalization problem based on monocular vision in thispaper.

Our localization system can be described within theBayesian filtering framework [12]. The posterior proba-bility of the current camera pose θt (w.r.t. a world refer-ence frame) can be estimated from the prior probabilitydistribution recursively based on the motion model andmeasurement model as follows:

P (θt|Z1:t,M)︸ ︷︷ ︸Posterior

∝ P (zt|θt,M)︸ ︷︷ ︸Measurement

·

·∫θt−1

P (θt|θt−1)︸ ︷︷ ︸Motion

P (θt−1|Z1:t−1,M)︸ ︷︷ ︸Prior

dθt−1(1)

where Z1:t ≡ zt indicates the sequence of images up totime t. While the framework in Eq. 1 is very general, weimplemented the posterior as a single delta function, i.e.,only the most likely camera pose is estimated. More indetail, the overall localization system works through thefollowing steps:

a) While the camera is moving, the camera acquiresan image zt from which a set of image features Yt ≡yt,k|1 ≤ k ≤ |Yt| are detected by a corner detectorof choice, e.g., Harris corner detector [13].

b) Then, a promising subset of the map M ′t is chosenand re-projected onto the image plane based on theestimated previous camera pose θt−1 and knowncamera parameters.

c) Afterwards, a set of putative matches Ct areformed where the i-th putative match Ct,i is apair {yt,k, xj} which comprises of a detected corneryt,k and a map element xj . A putative match iscreated when the error in Euclidean norm betweena detected feature and a projected map element isvery low.

d) Finally, the posterior function in Eq. 1 is fed intoRANSAC along with the set of generated putativesCt, where a set of promising inlier putatives areselected to produce an optimal camera pose θt. TheRANSAC-based framework described above is verypopular and has been used successfully by manyauthors [14], [15].

As the map M becomes larger and denser, we need topay special attention in step (c) in order to maintainhigh-quality putative matches, since the high ratio ofoutlier putative matches will cause the RANSAC pro-cedure in step (d) to slow down substantially or evenfail to compute a correct estimate [16], which will resultin under-performing system. This is due to the fact thatthe number of all the possible putative match pairs (evenwithout assuming any spurious measurement), wouldincrease exponentially w.r.t. the size of the map |M |.Hence, unless only the high quality putative matches areselected, a large number of ill-formed putatives will be fedinto RANSAC. Traditionally, such high quality putativepairs can be found by using appearance or geometryinformation. For example, appearance descriptors suchas SIFT [5] are used to improve the matching accuracy,or only the map elements xi ∈ M in the proximityof the previous pose θt−1 are searched based on thedistance (geometry) to reduce matching candidates. Inmany implemented systems, both approaches are oftenused simultaneously.

III. Visibility

In this paper, we address the use of geometry infor-mation to improve the selection of a promising mapsubset M ′t in step (b), which turns out to be crucial toimprove localization. As mentioned earlier, the processof selecting a subset of map becomes more importantas the map becomes larger/denser, and as the envi-ronment with substantial degree of occlusions becomesstructurally more complex, which is often the case inindoor environments.

The subset of features M ′t can be selected optimally ifwe know the true visibility of features. In other words, asystem that can select only the features which are trulyvisible w.r.t. a pose θt in reality is optimal. The visibilityis determined by the joint function of the feature xj andthe query pose θt. Formally, the visibility vj(θt) of thej-th 3D landmark xj ∈ M w.r.t. a pose θt is a boolean

4882

variable defined as follows:

vj(θt) =

{1 if xj is visible from θt0 otherwise

(2)

A less optimal system would select a subset whichincludes additional non-visible features and miss someof the truly visible features. On the other hand, thefeatures Yt detected from an image would comprise ofthe features that are in the map or the ones that arenot (due to the change of the environment, or just noisyfeatures). Consequently, the putative match generationprocess in step (c) will create non-ideal matches as theyassociate detected image features with incorrect mapelements in M ′t . It can be usually assumed that thefeature detectors are engineered at its best, hence, theonly factor that remains to be improved is the sub-mapselection process which should eventually capture thevisibility. Otherwise, a naive approach to step (b) wouldintroduce a lot of invisible features into M ′t , which willonly become more severe as the map becomes larger anddenser. Consequently, if we know the true visibilities forall the features, we can generate high-quality putativematches by considering only the visible features againstthe detected features on the current image. Since the truevisibilities are unknown and should be predicted, we cantake a probabilistic modeling approach, and pursue tomodel the following probabilistic visibility distributionfj(θ) for every j-th map element w.r.t. a given camerapose θ:

fj(θ) ≡ P (vj |θ) (3)

A. Non-parametric Visibility Modeling

While there can be many different candidate modelsfor the visibility distribution fj(θ) in Eq. 3, we propose touse a non-parametric approach [17] where every featurein the map remembers the camera poses which have seenit previously. The use of the non-parametric approach ismotivated due to the complex nature of the visibility,which occurs due to the non-trivial geometric structuresthat exist in the environment. Once the visibility func-tion in Eq. 3 in a non-parametric form is obtained, ifa feature has been seen previously by a camera posesimilar to the current pose θt, such feature is likely tobe classified to be visible. However, a major drawbackof such non-parametric approach is the computationaldemand incurred due to the large size of the map M andthe size of the set of all the training camera poses storedfor every feature in the map.

A major contribution of this paper is a computation-ally tractable and scalable algorithm which allows us tocompute the visibilities for each feature efficiently underthe non-parametric framework otherwise infeasible. Forexample, Fig. 1 (a) depicts an example where the entiremap has been re-projected on the image plane. Onthe other hand, Fig. 1 (b) depicts only the projectionsof the map elements that are selected based on ournon-parametric framework. It can be clearly seen that

the often-used nearest-neighbor algorithm for putativematching would perform very poorly in generating high-quality putative matches since the number of back pro-jections dominate the entire image plane due to the densecharacteristics of the map. On the other hand, if we tryto lower the number of map elements naively by selectinga low distance threshold, it can potentially remove goodmap elements and may deteriorate the overall qualityof the matches nonetheless. In our work, the visibility

(a) (b)

Fig. 1. The blue crosses depict the re-projections of a 3D pointfrom the map onto the image plane. In (a) the whole map is re-projected, whereas in (b) only the most highly visible features fora given pose are re-projected.

function fj(θ) is learned by providing a set of trainingexamples and a similarity kernel function k. In moredetail, we collect a set of training data (camera poses)Θj for each map element where the pose set comprisesof both positive examples Θ1

j and negative examples Θ0j ,

i.e., Θj ≡{

Θ1j ,Θ

0j

}. By positive examples, we mean the

camera poses from which the particular map element wasvisible, and by negative examples the rest of the set. Withan appropriate kernel function k, the visibility at a querypose θt can be estimated using Bayes’ law, as shown inEq. 4:

P (vj = 1|θt) =P (θt|vj = 1)P (vj = 1)∑

vj

P (θt|vj)P (vj)(4)

P (θt|vj) =1∣∣Θvjj

∣∣ ∑i

k(θt,Θvjj,i) (5)

In particular, the likelihood factor P (θt|vj) is modeledby the non-parametric training examples where the like-lihood factor for the positive case is shown in Eq. 5.In general, the kernel function k should be chosen insuch a way that the similarity output value of the kernelfunction decreases as the query pose θt stays further awayfrom the training data.

Since the size of the training data Θj for every featurecan be still very large, which would incur substantialcomputational demand, we estimate an approximate vis-ibility based on K-Nearest-Neighbor (KNN) approachw.r.t. the latest pose estimate θt. This is possible dueto the fact that only the training data nearby the querypose θt are informative to correctly compute the visibilityin Eq. 4 in most cases. As a consequence, we come to havea succinct visibility estimation function in Eq. 6 where

4883

only the nearest K samples ΘKj =

{ΘK,1j ,ΘK,0

j

}would

be considered.

P (vj = 1|θt) ≈

∑i

k(θt,Θ

K,1j,i

)∑vj

∑i

k(θt,Θ

K,vjj,i

) (6)

While the problem of obtaining a training dataset forthe above problem does not look trivial, which seemsto be almost unrealistic to collect such data manuallyfor every single map elements, we can notice that thetraining data exists already from the moment we builta map M . As either the bundle-adjustment techniqueor a SLAM algorithm is used to create a map, theintermediate process of these algorithms is to providethe information on the features seen by every estimatedcamera poses in the dataset. Hence, for every feature, weexactly know which camera poses observed it, which arecollected as positive training data. The rest of the cameraposes are collected as negative training dataset. Note thatthe map building approaches above do not guarantee thatall the positive data are collected. In fact, it is expectedthat some portion of the positive camera poses actuallyslip into negative training set, since the map-buildingapproaches fails to find all the correct correspondences.

B. Learning Kernel Functions

In this section, we describe the necessary propertiesof a good kernel function for visibility modeling and thedetails of the function we have specially developed. Inparticular, our kernel function is designed to output highsimilarity between cameras with similar views, i.e., theimage features they see overlap substantially, and theparameters of the function can be optimized based onthe training dataset. Hence, the kernel function we usedincorporates (1) Euclidean distance between the camerasd(θ0, θ1) and (2) the normalized inner-product betweenthe viewing direction between the two cameras d′(θ0, θ1)as the two distance inputs. Then, a logistic function waslearned from the transformed two dimension data:

k(θ0, θ1) =1

1+exp(−(wd · d(θ0, θ1)+w

′d · d

′(θ0, θ1)− wo))

(7)Now, the parameters wd, wd′ and wo (which is an offsetparameter) of the kernel function can be learned from atraining dataset where each training datum comprisesof two camera poses with their view similarity scoreswhich ranges between 0 and 1. Such view similarityscores can be readily obtained as the proportion of thenumber of features observed by the two camera posesw.r.t. the total number of features seen by the cameras.For example, if θ0 observed features {x1, x2} and θ1observed features {x1, x3, x4}, then the view similarity

score is: |{x1,x2}∩{x1,x3,x4}||{x1,x2}∪{x1,x3,x4}| = 1

4 .

C. Fast Visibility Prediction for Large Maps

We introduce the core technique to decrease compu-tational demand to compute the visibilities for all the

map elements dramatically from the order of the mapsize O(M) to O(K) where K denotes the number ofthe neighbors in the KNN described. A naive way tocompute the visibilities for all the map elements is togo over every one of them one by one computing thevisibilities using Eq. 6. The computational demand forsuch approach increases linearly w.r.t. the size of the map|M |. However, the core observation is that the results ofthe visibility prediction using Eq. 6 will be mostly zero,since most of the map elements in a large map would notbe observed at all by the KNNs of the current query poseθt, effectively making the numerator in Eq. 6 to be zero.

As a consequence, once we find the KNNs of thecurrent query pose, we only need to predict the visibilitiesfor the subset of map elements which are at least seenonce by these KNNs. Then, we can set the visibilitiesto be zero for the rest of the map elements, even notcomputing them at all. More formally, the visibilitiesneed to be computed for such map elements which arein the union set {

⋃Xi} where Xi denotes the set of

map elements seen by the i-th neighbor pose amongKNNs. If we assume that the number of map elementsobserved by any pose is bounded by a constant c, thenthe overall number of map elements to be examined isO(cK) = O(K). In summary, we can prune out most ofthe map elements from the visibility computation processby considering only those which were seen by KNNs ofthe latest camera pose.

IV. Results and Discussion

We designed our experiments in such a way that wecan analyze the benefits that are obtained by the smartuse of the geometric information. In other words, anybenefits to be demonstrated in this paper are expectedto be obtained from the improvements on the geometricinformation use, isolated from any advantages due toappearance-based matching methods. A 3D map in adense cluttered environment was computed using Vi-sual SLAM techniques, which is used to provide test-beds for localization as well as to provide training datafor non-parametric visibility functions. During the mapcomputation process, the different training poses fromwhich each map element is seen, are memorized so asto be able to predict the visibility by means of a non-parametric modeling during the localization stage. Weshow monocular vision-based localization results for thetraining and test datasets. Additionally, to stand out thecontributions of our method, we compare our idea withtwo different methods:

• Brute Force: Under this assumption all the mapfeatures are re-projected onto the image plane fora given pose. Besides, only the features that arepredicted to lie within the image plane, are consid-ered to form the set of putatives to be used for poseestimation.

• Length and angle heuristic: Feature visibilityis calculated considering the difference between the

4884

viewpoint from which the feature was initially seenand a new viewpoint. This difference in viewpointhas to be below some length and angle ratio, andpredicted to lie within the image, in order to pre-dict the feature as visible. Usually the feature isexpected to be visible if the length ratio |hi|/|horig|is close enough to 1 (in practice between 5/7 and7/5 and the angle difference β = cos−1((hi ·horig)/(|hi||horig|)) is close to 0 (less than 45◦ inmagnitude).

Considering the first of the above approaches, a bruteforce approach will yield in a high number of outliersand localization errors under very dense maps (thousandsof features), whereas the second of the approaches canyield erroneous localization when the camera is facingoccluded areas. The second approach has been widelyused in the literature such as in [18], [3], since it is veryeasy to compute and provides good results, since afterthe visibility prediction, matching is performed using 2Dimage templates. However, one of the drawbacks of thiscriteria is that can not deal with occlusions since it as-sumes a transparent world. This in fact, can be a problemfor long term localization under cluttered environmentswith occlusions, such as the ones we are interested for thevisual impaired (e.g. cities, underground stations, offices,etc.)

The training dataset is a large sequence of 5319 framesin which a dense map with about 1316 3D points wasobtained. The test dataset is a separate small sequencerecorded in the same space and comprises of 2477 frames,including some camera views that were not fully capturedin the training dataset. Fig. 2 depicts the kind of environ-ment where we have tested our localization experiments.

In our experiments we use an adaptive threshold ver-sion of RANSAC to automatically determine the num-ber of RANSAC iterations needed [16]. The distancethreshold that is used to create a putative match isset to 4 pixels. In addition we only consider visibilityprediction of the 60 KNNs since we have found exper-imentally that this number of neighbors is enough forpredicting visibility. Our experimental results shown inFig. 3 highlights the benefits of our method where ourresult is shown to provide less number of higher-qualityputative matches, which eventually leads to faster andmore accurate RANSAC computation. In detail, Fig. 3depicts the inliers ratio (a), the number of putativesper frame (b) and the number of RANSAC iterations(c) during some of the first frames of the test sequence.The highest inliers ratio is obtained using our visibilityprediction approach and this ratio is normally above80%. We set the number of RANSAC iterations to amaximum of 500 for computational purposes. As it canbe seen in Fig. 3, the number of iterations for thebrute force case is very close to this bound, whereas forthe other experiments the number of iterations is muchlower, obtaining less than 20 iterations per frame for thevisibility case.

In Table I, the information about the mean inliersratio, mean number of putatives and RANSAC iterationsper frame is shown for the training and test datasetsrespectively where we can again observe the computa-tional benefits of our method. Fig. 4 and 5 show thelocalization results for the training and test sequencerespectively, w.r.t. the ground truth data obtained byVisual SLAM and bundle adjustment optimization. Map3D points are characterized by a vector Yi = {X Y Z}in a world coordinate frame (map 3D points are repre-sented by orange dots in the next figures). Each cam-era pose is parametrized by means of a vector θi ={X Y Z q0 qx qy qz} (translation and orientation givenby a unit quaternion). We don’t show any figure forthe brute force case, since this approach fails to provideaccurate localization for both datasets.

Finally, we show the overall localization accuracy ofour monocular-vision system for the camera locationsand rotations. In detail, Table II and III show the meansquared error with respect to the ground truth of theestimated localization for both translation and orienta-tion, training and test sequences respectively. In the testsequence, the camera goes straight during approximately1000 frames into a corridor where no occlusions arepresent, and after that it turns right into an area withsevere occlusions. Brute force and heuristic approachesyield a wrong localization result since these two ap-proaches are not able to estimate correctly the 180◦

rotation in the area with occlusions. On the contrary,with our approach, localization results are very similar tothe ones obtained in the ground truth even in areas of themap with a dense level of occlusions. Besides, the numberof RANSAC iterations per frame that are necessary areless than 20, showing that the method can work in realtime providing good localization estimates. It can beobserved in Fig. 5(b) a small gap in the camera position.This is due to the fact that in the test sequence there aresome camera views that were not fully captured in thetraining dataset, decreasing slightly overall localizationperformance. With our approach we have obtained thesmallest errors with respect to the other two methods,both in translation and rotation components. Results arequite satisfactory, taking into account that appearancedescriptors have not been used in the matching pro-cess. We think that the use of appearance descriptorscombined with our visibility prediction will improve andspeed up localization.

As it has been shown, our method depends on thequality of the input data. But how many training viewsare necessary to obtain accurate and fast localizationresults? Considering a huge number of training viewscan be an overwhelming computational burden for verylarge environments such as buildings, or even cities.We have done some experiments in which we reduceconsiderably the number of training views (samplinguniformly among the whole dataset of training views),and run our localization algorithm predicting features

4885

Fig. 2. Some frames of the analyzed environment

(a) (b) (c)

Fig. 3. Test sequence results: (a) inliers ratio, (b) number of putatives and (c) number of RANSAC iterations per frame

Training % Inliers # Putatives # IterationsBrute Force 0.6952 67.2181 461.7096Heuristic 0.7392 36.7744 52.7885

Visibility Prediction 0.8335 26.3897 6.4221Test % Inliers # Putatives # Iterations

Brute Force 0.6420 74.2782 439.0642Heuristic 0.6817 36.9931 84.3254

Visibility Prediction 0.7368 16.7587 17.6437

TABLE I

Inliers ratio, number of putatives per frame for training and test sequences

Case Training Training Training Test Test Testεx (m) εy (m) εz (m) εx (m) εy (m) εz (m)

Brute Force 3.4857 0.0974 1.8305 1.1825 0.1230 1.3366Heuristic 0.5642 0.0574 0.4142 1.1549 0.0954 0.5041

Visibility Prediction 0.0476 0.0243 0.0340 0.2781 0.0785 0.2736

TABLE II

Localization errors in translation with respect to ground truth

Training εq0 εqX εqY εqZBrute Force 0.2180 0.1030 0.3258 0.0497Heuristic 0.2222 0.0415 0.1911 0.0264

Visibility Prediction 0.0068 0.0106 0.0100 0.0091Test εq0 εqX εqY εqZ

Brute Force 0.2484 0.0488 0.2367 0.0346Heuristic 0.1826 0.0452 0.1561 0.0304

Visibility Prediction 0.0516 0.0366 0.0476 0.0237

TABLE III

Localization errors in rotation with respect to ground truth

4886

(a) (b)

Fig. 4. Comparison of Localization Results for Training Sequence: (a) Heuristic (b) With Visibility Prediction

(a) (b)

Fig. 5. Comparison of Localization Results for Test Sequence: (a) Heuristic (b) With Visibility Prediction

visibility. We have studied several localization runs witha different percentage of the total number of trainingviews: %100, %70, %50, %30 and %10. In Table IVratios about the inliers ratio, number of putatives andRANSAC iterations per frame are shown for both thetraining and test dataset respectively.

The inliers ratio is similar for all the experiments,being higher for the %100 of training views, since thisis the case in which we have the highest level of detailof the 3D structure. As long as we reduce the samplingrate, keeping the same number of KNNs, the distancebetween the current camera pose and its nearest neigh-bors increases so it is easier to predict a feature to bevisible when in fact it is not really visible from thecurrent camera pose. This is the reason why the numberof iterations that RANSAC needs to fit the best modelincreases as long as we reduce the sampling rate. In termsabout the difference in localization results is very similarbetween all the cases, and even localization results withonly a %10 of the training views are better than the bruteforce and heuristic approaches both for training and testsequence.

In our experiments the acquisition frame rate of thetraining sequence was 30 frames per second, the imageresolution was 320 × 240 pixels and camera was carried

in hand by a person at normal walking speeds (3Km/h−5Km/h). According to the results it seems that ourlocalization algorithm can provide good results with aconsiderably smaller number of views than the wholedataset of training views. This is an important factorfor large environments since we don’t have to keep inmemory the whole training dataset. It would be of benefitjust selecting the views that give more information aboutthe 3D reconstruction just in a similar way as it isexplored in [19] for data association.

V. Conclusions

We have presented an algorithm for predicting thehighly visible features for real time localization underindoor environments, in which every map feature modelsits visibility w.r.t. the camera poses via non-parametricdistributions. We have shown the benefits of our methodfor monocular vision-based localization. The localiza-tion errors considering our visibility prediction are verysmall compared to the ground truth and also the speed-up gain compared to the other analyzed methods isvery significant. In addition, our procedure can provideaccurate localization results even when a small set oftraining views is used. As future work we are interested inintegrating our approach into a real-time SLAM system,i.e. when the map has been computed and the remaining

4887

Sequence % Camera Poses % Inliers # Putatives # Iterations # Training ViewsTraining 100 0.8335 26.3897 6.4221 5319Training 70 0.8331 30.3055 6.8429 3723Training 50 0.8158 32.2688 8.0959 2661Training 30 0.7803 36.2432 8.6119 1569Training 10 0.7664 42.5013 11.2367 533Test 100 0.7368 16.7587 17.6437 5319Test 70 0.7194 20.0803 19.5891 3723Test 50 0.7170 25.0831 22.5293 2661Test 30 0.6983 26.3317 27.3240 1596Test 10 0.6510 29.4727 30.3448 533

TABLE IV

Inliers ratio, number of putatives and RANSAC iterations per frame considering different number of training views

uncertainty of the features are below a quality threshold,the map can be used for navigation purposes using thelearnt visibility from the mapping process. Moreover, theuse of powerful appearance descriptors and our visibilityprediction idea can provide very good localization resultsin cluttered environments with occlusions for indoor andoutdoor environments. Obtaining the camera views thathave more impact in the reconstruction, we can samplethe set of training poses reducing the total size of posesto be kept in memory. In the same way we are interestedin techniques such as the one described in [9] for reducingthe total number of landmarks that are necessary for anaccurate localization.

VI. Acknowledgements

This work was supported in part by the Span-ish Ministry of Education and Science (MEC) undergrant TRA2008-03600 (DRIVER-ALERT Project) andby the Community of Madrid under grant CM: S-0505/DPI/000176 (RoboCity2030 Project). The authorsare grateful to Kai Ni and Michael Kaess for fruitfuldiscussions and software collaboration.

References

[1] B. Triggs, P. McLauchlan, R. Hartley, and A. Fitzgibbon,“Bundle adjustment – a modern synthesis,” in Vision Algo-rithms: Theory and Practice, ser. LNCS, W. Triggs, A. Zis-serman, and R. Szeliski, Eds. Springer Verlag, Sep 1999, pp.298–375.

[2] K. Konolige and M. Agrawal, “Frame-frame matching forrealtime consistent visual mapping,” in IEEE Intl. Conf. onRobotics and Automation (ICRA), Apr 2007, pp. 2803–2810.

[3] A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse,“MonoSLAM: Real-time single camera SLAM,” IEEE Trans.Pattern Anal. Machine Intell., vol. 29, no. 6, 2007.

[4] B. N. Walker and J. Lindsay, “Navigation performance witha virtual auditory display: Effects of beacon sound, captureradius, and practice,”Human Factors, vol. 48, no. 2, pp. 265–278, 2006.

[5] D. Lowe, “Distinctive image features from scale-invariant key-points,” Intl. J. of Computer Vision, vol. 60, no. 2, pp. 91–110,2004.

[6] S. Se, D. Lowe, and J. Little, “Mobile robot localization andmapping with uncertainty using scale-invariant visual land-marks,” Intl. J. of Robotics Research, vol. 21, no. 8, pp. 735–758, Aug 2002.

[7] P. Sala, R. Sim, A. Shokoufandeh, and S. Dickinson, “Land-mark selection for vision-based navigation,” IEEE Trans.Robotics, vol. 22, no. 2, pp. 334–349, April 2006.

[8] H. Wuest, A. Pagani, and D. Stricker, “Feature managementfor efficient camera tracking,” in Asian Conf. on ComputerVision (ACCV), 2007.

[9] Z. Zhu, T. Oskiper, S. Samarasekera, R. Kumar, and H. Sawh-ney, “Real-time global localization with a pre-built visuallandmark database,” in IEEE Conf. on Computer Vision andPattern Recognition (CVPR), 2008, pp. 1–8.

[10] N. Dalal and B. Triggs, “Histograms of Oriented Gradients forHuman Detection,” in IEEE Conf. on Computer Vision andPattern Recognition (CVPR), 2005.

[11] L. M. Paz, P. Pinies, J. D. Tardos, and J. Neira, “Large scale6DOF SLAM with stereo-in-hand,” IEEE Trans. Robotics,vol. 24, no. 5, 2008.

[12] S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics.The MIT press, Cambridge, MA, 2005.

[13] C. Harris and M. Stephens, “A combined corner and edgedetector,” in Proc. Fourth Alvey Vision Conference, 1988, pp.147–151.

[14] D. C. K. Yuen and B. A. MacDonald, “Vision-based localiza-tion algortihm based on landmark matching, triangulation, re-construction, and comparison,” IEEE Trans. Robotics, vol. 21,no. 2, pp. 217–226, 2005.

[15] S. Tariq and F. Dellaert, “A multi-camera 6-DOF posetracker,” in IEEE and ACM Intl. Sym. on Mixed and Aug-mented Reality (ISMAR), 2004, pp. 296–297.

[16] R. Hartley and A. Zisserman, Multiple View Geometry inComputer Vision. Cambridge University Press, 2000.

[17] C. M. Bishop, Pattern Recognition and Machine Learning.Springer, 2007.

[18] A. J. Davison and D. W. Murray, “Simultaneous localizationand map-building using active vision,” IEEE Trans. PatternAnal. Machine Intell., vol. 24, no. 7, 2002.

[19] M. Kaess and F. Dellaert, “Covariance recovery from a squareroot information matrix for data association,” Journal ofRobotics and Autonomous Systems, 2009.

4888