Upload
lekiet
View
214
Download
0
Embed Size (px)
Citation preview
1
AR Display for Observing Sports Events based on Camera Trackingbased on Camera Tracking Using Pattern of Ground
Akihito Enomoto, Hideo Saito
HVRL H Vi i R h L bHVRL: Hyper Vision Research Lab.
Keio University
July 23, 2009
Recent Trend for Visual Media S th i d A l i
2
Synthesis and Analysis
Multiple CameraMultiple CameraMultiple Camera
Virtualized Reality (CMU)
Multiple Camera
Virtualized Reality (CMU)Virtualized Reality (CMU) Matrix
NFL EyeVision
Virtualized Reality (CMU) Matrix
NFL EyeVision 3D Display Technology
Free-Viewpoint VideoAuto Stereoscopic DisplayAuto-Stereoscopic Display
Volumetric 3D Display
Virtual Reality
Mixed Reality Display
Virtual Reality
Mixed Reality DisplayMixed Reality DisplayMixed Reality Display
3Virtualized Reality (CMU 1997 99)Example:
(CMU,1997-99)
Inputsequence
3-Men Basketballsequence
(These are movies.)
Synthetic court
4D Model
Synthetic court
4
For soccer scenes
cam 1 cam 4cam 2 cam 3
cam1 cam2 cam3 cam4
[Inamoto and Saito ICPR2002][Inamoto and Saito, ICPR2002][Inamoto and Saito, IEEE Trans. MM, 07]
5Calculation of Calculation of
Viewpoint PositionViewpoint PositionArbitrary View Synthesis Arbitrary View Synthesis
of Soccer Sceneof Soccer SceneRendering on Rendering on The StadiumThe StadiumViewpoint PositionViewpoint Position of Soccer Sceneof Soccer Scene The StadiumThe Stadium
・
・・・
Virtual Views of
Multiple View Images Captured at A Stadium
The Stadium
GUI
Ref.Cam1 Ref.Cam2
GUI
Overlay on The yStadium ImageVirtual View of The
Dynamic Regions
6
Example of Free Viewpoint ImagesExample of Free Viewpoint Images
Cam 1Cam 2Cam 2
Cam 3 Cam 4
7
Observe Soccer Match on the DesktopObserve Soccer Match on the Desktop[Inamoto, Saito ISMAR03]
User sees a desktop stadium model in the real world with video see-through HMD and observes dynamic objects of
Video See Through HMD
soccer scene overlaid onto the display.
Desktop Stadium Model
Video See-Through HMD
C t d S
Virtual View Virtual View GenerationGeneration
Captured Soccer Match at Stadium
8Determination of Determination of
Viewpoint PositionViewpoint PositionArbitrary View Synthesis Arbitrary View Synthesis
of Soccer Sceneof Soccer SceneRendering on Rendering on The StadiumThe StadiumViewpoint PositionViewpoint Position of Soccer Sceneof Soccer Scene The StadiumThe Stadium
・・・
M lti l Vi I
Virtual Views of The StadiumMultiple View Images
Captured at A Stadium
GUI
Stadium
GUI
Overlay on The Stadium ImageStadium ImageRef.Cam1 Ref.Cam2
Virtual View of The Dynamic RegionsHMD Camera Image
Overlay on Desktop Stadium Model
9
Experimental ResultsExperimental ResultsWe have implemented immersive observation systemWe have implemented immersive observation system for actual soccer matches.
Captured soccer images : 720×480 pixel, 24-bit-RGB color
Camera 1
Camera 2 Camera 3 Camera 4
Camera Configuration at A Stadium Canon Video See-Through HMDCamera Configuration at A Stadium Canon Video See Through HMD
10
ExampleExample
Stadium Camera 1 Stadium Camera 2Frame 335
( Camera 1-2 w = 0.47 z = 1.07 )( Camera 1-2 w = 0.5 )On Real Stadium Image On Tabletop Stadium Model
11
•Unstable Camera Tracking•Limited Camera Movement•Limited Camera Movement
12
AR Baseball Presentation Systemy[Uematsu, Saito, ICAT06, IVC09]
• Virtual baseball game is overlaid onto a real field model• Virtual baseball game is overlaid onto a real field model.• A user watches the game through a handheld monitor.
The baseball game is replayed from a scorebook data file– The baseball game is replayed from a scorebook data file.– Multiple planar markers are automatically integrated.
web-camera baseball field model
2D markers
handheld monitor
2D markers
user
13
Demo Video
In this presentation14
In this presentation…..
• Real soccer player captured with multiple cameras in stadium
• Observering cameraWid t– Wide-area movement
– Real-time, stable tracking
15
System ConfigurationSystem ConfigurationSystem ConfigurationSystem Configuration
Fi ld P d AR M kField Pattern and AR MarkerObserver’sViewView
ObserveringCameraCamera
16
Camera Tracking for RegistrationCamera Tracking for RegistrationCamera Tracking for RegistrationCamera Tracking for Registration
•AR Toolkit + Natural Features
[M t k ISMAR06][Motokawa ISMAR06]
17
18
Data Processing FlowData Processing FlowOff-Line On- Line
Observing Camera
Pose/Positionof Observing CameraCamera
Player ExtractionOverlay
Extracted
Stadium Camera (Fixed)
y ExtractedTextures
•Ball•Penalty Area
19
On Line PhaseOn Line PhaseCalibration Stadium Camera Selection AR Display
Observing Camera
Position of
Selection
Position of Projection
SelectionOverlayStadium Camera
Projection Matrices
20
CalibrationCalibrationCalibrationCalibration
Observing Camera
Position of
Selection
Position of Projection
SelectionOverlayStadium Camera
Projection Matrices
21
HomographyHomography Based CalibrationBased CalibrationHomographyHomography Based CalibrationBased CalibrationGilles Simon, et al.:Markerless tracking using planar
y
Markerless tracking using planar structures in the scene, ISAR2000
ZY
Z
X
xImage World
22
OnOn--Line Corner DetectionLine Corner DetectionOnOn Line Corner DetectionLine Corner Detection
Observing Camera
Intial Estimate
Template Matching(a) Initial Estimate
Refine
(b) Template (c) Refine4 Corner Positions
23
Selection of Stadium CameraSelection of Stadium CameraSelection of Stadium CameraSelection of Stadium Camera
Observing Camera
Position of
Selection
Position of Projection
SelectionOverlayStadium Camera
Projection Matrices
24
Stadium Camera with Closest Pan Angle (around Z axis) to Observing Camera is selected
YZ
YZObservingCamera
XX
Stadium Camera
25
AR DisplayAR DisplayAR DisplayAR Display
Observing Camera
Position of
Selection
Position of Projection
SelectionOverlayStadium Camera
Projection Matrices
26
Player PositionPlayer PositionyyPosition in Image
⎥⎥⎤
⎢⎢⎡
⎥⎥⎤
⎢⎢⎡
⎥⎥⎤
⎢⎢⎡
~ Y
X
y
xHomography
Position onGround
⎥⎥⎦⎢
⎢⎣⎥⎥⎦⎢
⎢⎣⎥
⎥⎦⎢
⎢⎣ 1
~
1
YyH3×3
G ou d
YZ
y
(X,Y,0)x X
(X,Y,0)(x,y)
Stadium Camera Image
27
Scaling Magnitude of PlayerScaling Magnitude of PlayerScaling Magnitude of PlayerScaling Magnitude of PlayerRatio of Translational Component of Both Cameras
Ratio =T
Ta
′×
a:Coefficient Ta:CoefficientT:Stadium CameraT’:Observing Camera
T T’T :Observing Camera
28
Ball PositionBall PositionBall PositionBall PositionStereo Matching
(X,Y,Z)(X,Y,Z)
(u,v)m (u’,v’)m’( )
v v’
P P’u u’
29
Overlaying Virtual Objects:Overlaying Virtual Objects:Overlaying Virtual Objects:Overlaying Virtual Objects:P
30
ResultsResults
Ajinomoto Stadium〈Stadium Capturing〉‐Ajinomoto Stadium‐Three CamerasVideo Size:960×540
10m 20m‐Video Size:960×540
〈AR S t 〉〈AR System〉‐CPU: Core2Duo 3.00GHzMemo 2GB‐Memory: 2GB
‐Video Size: 640×480
31
32
Effect of using Corners and MarkerEffect of using Corners and MarkerEffect of using Corners and MarkerEffect of using Corners and Marker
G d t th iGround truth ismanually measured
Average ( i l) Maximum ( i l)Average (pixel) Maximum (pixel)
Both 1.9 4.1Marker only 16.7 22.8
33
Additional FunctionalitiesAdditional FunctionalitiesAdditional FunctionalitiesAdditional Functionalities-Off-Side Detection-Ball Trajectory Display
Conclusion34
Conclusion
• AR Display for Observing Sports Events – Camera Tracking Using Pattern of GroundCamera Tracking Using Pattern of Ground
• Marker + Corner points tracking
35
• [email protected]• www hvrl ics keio ac jp• www.hvrl.ics.keio.ac.jp• Google : HVRL