Automatic Registration of Color Images to 3D Geometry

Computer Graphics International 2009

Yunzhen Li and Kok-Lim Low

School of ComputingNational University of Singapore

* Presented by Binh-Son Hua

Problem StatementRange images

Color images from untracked camera

. . .

3D model Colored 3D model

Automatically register color images to 3D model

2

MotivationsApplications of active range sensing

Manufacturing, cultural heritage modeling, etc.Photometric properties needed for visually-

realistic modelsOnly some range scanners can capture colorColor may not have required resolution

E.g. for close-up or zoomed-in views of paintingsView-dependent reflection requires many color

images from different directionsTherefore, better to capture color separately

However, impractical to manually register color images to 3D geometry

3

Previous WorkFeature-based approaches

Match corresponding features in both color images and 3D model

Can be fully automatedRestricted to certain types of objects[Stamos & Allen, ICCV 2001], [Liu & Stamos, CVPR 2005]

Statistics-based approachesUsed only if reflected intensities of range sensing light

were recorded with range dataSensing light often not in visible light spectrum

Compute statistical dependence between color images and sensing light intensitiesMutual information, chi-square, cross-correlation

Camera calibrated & tracked, or co-locate with scanner[Williams et al, 2004], [Hantak & Lastra, 3DPVT 2006]

4

Our ApproachColor images

. . .

Detailed scanned 3D model

Colored 3D model

Color mapping

Registration

Multiview geometry

reconstruction

Sparse 3D model

5

Steps

1. Data acquisition

2. Multiview geometry reconstruction

3. Approximate registration of sparse model to detailed model

4. Registration refinement

5. Color mapping

6

1. Data AcquisitionRange data

Laser range scanner

Color images Uncalibrated and untracked

digital cameraProject special light pattern

on large textureless surfacesImprove image feature

detection and MVG reconstruction

7

2. MVG ReconstructionDetect and match features in color images

Use SIFT

Compute MVGStructure-from-motionIncrementally add a new image and apply

sparse bundle adjustment (SBA)

Result is a sparse 3D model3D point cloudCamera parameters

8

2. MVG ReconstructionExample sparse 3D model

9

3. Approximate RegistrationTo align sparse model with detailed model

Unknown relative scale and poseRegister one image in MVG to 3D model

User input 6 point correspondencesEstimated transformation propagated to other

views and 3D points in MVGSparse model only approximately aligned to

detailed modelError in user inputsError in MVGGeometric distortion in detailed model

10

4. Registration RefinementNeed non-rigid alignment of MVG with detailed model

To overcome geometric distortion in range images

Registration refinementAutomatically detect planes in detailed modelIdentify 3D points in MVG near the planesRefine MVG to minimize distance

between 3D points and planesEasily incorporated into

sparse bundle adjustment

Better than using ICP algorithmTwo models are treated as rigid shapesCannot refine MVG

11

4. Registration RefinementExample result

12

Before registration refinement

Afterregistration refinement

5. Color MappingColors from different views can be used for

view-dependent renderingView-dependent texture mappingSurface light field

We simply want to assign a single color to each surface point, butSimple averaging blurs out detailsDifferent exposuresOcclusionsDepth boundariesVignetting and view-dependent reflection

13

5. Color MappingUse weighted blending

Use lower weights near image and depth boundaries

Preserve fine details

14

With details

preservation

Without details

preservation

5. Color MappingSmooth color and intensity transitions

With weighted blending

Without weighted blending

15

ResultOffice scene

30 color images (7 with projected pattern)

16

ConclusionAchieve accuracies within 3–5 pixels

everywhere on each imageNot reliant on detection of any specific type of

features in both color images and geometric model

Project light pattern to improve robustness of MVG

Better registration accuracy in face of geometric distortion

Effective color mapping method

17

AcknowledgementsThe Photo Tourism team

For sharing part of their code on MVGPrashast Khandelwal

For contribution to preliminary workSingapore Ministry of Education

For the funding

18