A Distributed Cooperative Framework for Continuous Multi- Projector Pose Estimation IEEE VR 2009 - March 16, 2009 Tyler Johnson, Greg Welch, Henry Fuchs,

A Distributed Cooperative Framework for Continuous

Multi-Projector Pose Estimation

IEEE VR 2009 - March 16, 2009

Tyler Johnson, Greg Welch, Henry Fuchs, Eric La Force, and Herman Towles

Department of Computer ScienceUniversity of North Carolina at Chapel Hill

A Distributed Cooperative Framework for Continuous Calibration

2

Funding

ONR: Behavior Analysis and Synthesis for Intelligent Training (BASE-IT), Dr. Roy Stripling, Program Manager

ONR: Virtual Technologies and Environments Program (VIRTE), CDR Dylan Schmorrow, Program Manager

ONR-NAVAIR: Deployable Intelligent Projection Systems for Training, SBIR contract with Renaissance Sciences Corporation

IARPA: Mockup Future Analyst Workspace (A-Desk), Dr. Jeff Morrison, Program Manager

NSF: Integrated Projector-Camera Modules for the Capture and Creation of Wide-Area Immersive Experiences, CRI:IAD grant


3

Adaptive Multi-Projector Displays

An Intelligent Projector Unit

(IPU)


4

Challenges

GeometricCompensating for display surface shapeCo-registration of projection images

PhotometricIntensity blending in image overlapsMatching colors between projectors

No Compensation

Compensation


5

Geometric Calibration

Calibrate Projectors

Estimate Display Surface

Before Display Use

Project Structured Light

During Display Use

Geometric Image

Correction

Render Imagery

Estimate Display Surface

Calibrate Projectors

Concurrently

Continuous

Calibration


6

Continuous Calibration

A Calibrated Two-

Projector Display

Projectors Moved or Bumped

A Recalibrated Two-Projector

Display


7

Cooperative Calibration

We propose a distributed, Kalman filter-based approach to continuous calibration where intelligent projector units interact as peers to cooperatively estimate the poses (orientation & position) of all projectors during actual display use


8

Related Work

Continuous CalibrationActive (Calibration Patterns)• Imperceptible Structured Light [Cotting04,05]

Passive (Application Imagery)• Continuous Display Surface Estimation [Yang&Welch01]• Single Projector Pose Estimation[Johnson&Fuchs07]• Multi-Projector Pose Estimation [Zhou08]

Hybrid• Automatic switch from passive to active [Zollmann06]

Distributed Upfront Calibration[Bhasker06]


9

Contributions

Our Kalman filter-based distributed cooperative framework provides

Continuous pose estimation for multiple projectors• Compatible with both active and passive feature

collection• All projectors may move simultaneously

Temporal filtering

Fault tolerance & scalability


10

Cooperative Calibration

Peer-to-Peer based, Single Program, Multiple Data

Each IPU considers itself to be the “local” IPUOther IPUs are considered “remote” IPUs

Each IPU is responsible for calculating its own pose using local and remote correspondences

AssumptionsThe internal calibration of each IPU is fixed and knownThe geometry of the display surface is static and knownProjectors remain mostly stationary, however they may drift over time or occasionally be moved by the user


11

Local Correspondences

Measured for each IPU between its primary and secondary cameras

Provides an estimate of pose


12

Remote Correspondences

Measured between the primary camera of the local IPU and a remote IPU

Remote IPU acts as a reference in estimating pose of local IPU


13

Collection of MeasurementsDisplay Surface


14

Pose of

Kalman Filter

Display Surface Model

Display Surface Model

Pose of

Intrinsics of

Intrinsics of

Measurements in

Predicted Measurements in

Measurements in

Measurement Function

Estimate , using


15

Kalman Filter

Error Covariance

State

Pose ofPose of

Pose of

Pose of


16

Filter Update

Predict IPUs remain stationary

Add additional uncertainty

Time Update

Measurement Update

Correct state based on residual

Measurement Jacobian

Measurement Noise

(t)


17

Distributed Operation

Each IPU has local access toIts own intrinsic calibration & poseIts own camera imagesDisplay surface model

Kalman filter update requires remote access to

Intrinsic calibration & poses of remote IPUsError & process noise covariance of remote IPUsImages from primary cameras of remote IPUs captured at time


18

Distributed Operation

Request/response mechanism for exchanging camera images, pose information etc


19

Results

x

y

z

Before Movement

During Movement

After Movement

Ψ

θ

φ

mm

rad

1240

1160

-95

-120

2640

2560

2.85

2.7

-0.3

-0.7

0.35

0.15


20

Video

Distributed Cooperative Pose Estimation in a Two-IPU display

P P


21

Discussion

Observability & DriftSurface geometry may not fully constrain posePossible to “anchor” solution in unobservable directions

Cooperative EstimationNot required for a working systemEnsures imagery is registered between projectors, especially when pose may be unobservable

P P


22

Future Work

Continuous calibration of display surface

Dynamic projector refocusing

Dynamic photometric blending

Improve scalability


23

Future Applications


24

In Conclusion

Documents

A Distributed Cooperative Framework for Continuous Multi- Projector Pose Estimation IEEE VR 2009 - March 16, 2009 Tyler Johnson, Greg Welch, Henry Fuchs,