2D to 3D conversion of formula 1 footage

2D to 3D conversion of Formula 1 footage

Serge Hendrickx

1

Presentation Overview

• Thesis objective

• Conversion process

• Conclusion and result 2

• 3D rotation

• Car detection and tracking

• Inpainting

• Overlay detection and conversion of background

Objective: 2D to 3D conversion of Formula 1 footage

2D recorded and broadcasted images 3D experience

3

Objective: 2D to 3D conversion of Formula 1 footage

4

What is special about Formula 1?

Undeformable objects with near perfect 3D models -> Investigate how these could be used in this conversion process -> Not just generating best-looking 3D

Introduction to Formula1 footage Example footage

5

Presentation Overview

• Thesis objective

• Conversion process

• Conclusion and result 6

• 3D rotation

• Car detection and tracking

• Inpainting

• Overlay detection and conversion of background

3D effect

• Translation + rotation

7

Retinal Disparity

3D effect

• Translation + rotation

8

Retinal Disparity

1) Nearest-neighbor

2) 3D Projection

3D effect

• Translation + rotation

9

Retinal Disparity

1) Nearest-neighbor

2) 3D Projection

Nearest neighbor rotation

10

For each pixel: 1) Find corresponding RGB 2) Detect nearest match in

rotated view 3) Copy pixelvalue

XYZ represented by RGB during rendering

11

Nearest neighbor rotation

3D effect

• Translation + rotation

12

Retinal Disparity

1) Nearest-neighbor

2) 3D Projection

3D projection

13

Traverse rendering pipeline

(X,Y) –coordinate in new view

(X,Y,Z) coordinate

(X,Y) –coordinate in known view

Using ModelView and Projection matrices of new view

Using ModelView and Projection matrices of known view

3D projection

14

Traverse rendering pipeline

Source:

Generated:

3D rotation

• 3D projection : + Fast + Produces clear image - Holes

15

• Neighborhood: - Very slow - Artifacts + Fills in holes

3D rotation

• 3D projection : + Fast + Produces clear image - Holes

16

• Neighborhood: - Very slow - Artifacts + Fills in holes

-> Combine both methods

17

Source:

3D rotation: final result

Rotated 10 degrees:

18

• Translation + rotation

Retinal Disparity

3D effect

19

• Translation + rotation

Retinal Disparity

3D effect

-> Exact position and pose of car needed

Presentation Overview

• Thesis objective

• Conversion process

• Conclusion and result 20

• 3D rotation

• Car detection and tracking

• Inpainting

• Overlay detection and conversion of background

Car detection and tracking

• Car detection

• Tracking throughout fragment

21

Car detection and tracking

• Car detection

• Tracking throughout fragment

22

Car detection

23

Sliding window -> binary classifier: contains car yes or no?

Car detection

24

Sliding window -> binary classifier: contains car yes or no?

Car detection

25

Sliding window -> binary classifier: contains car yes or no?

Car detection

26

Sliding window -> binary classifier: contains car yes or no?

Render car from all different angles

27

Car detection

Car detection

28

Sliding window -> binary classifier: contains car yes or no?

-> Does it match one of the renders?

Car detection

29

How to compare? Pixel per pixel -> Not resistant to illumination changes and other differences Solution: Histogram of Oriented Gradients (HoG)

HoG based object detection

30

Histogram of Oriented Gradients (HoG)

Gradient computation

Gradient binning

HoG based object detection

31

Parts-based detection

Star-based representation:

Real example:

HoG based object detection

32

Analysis of detection method on F1 footage

33

Analysis of distibution of certainty-scores of 1 particular render on 1 frame

Example: best certainty-score is 10 -> only 4% of matches score > 9 -> ony 14% of matches score > 5

Analysis of detection method on F1 footage

34

Score deduction in neighboring renders

Car detection and tracking

• Car detection

• Tracking throughout fragment

35

Car tracking

36

36,000 renders (360*100 angles) * 500 frames = 18,000,000 detections -> coarse selection of renders and frames for first pass.

Tier 1

Car tracking

37

36,000 renders (360*100 angles) * 500 frames = 18,000,000 detections -> coarse selection of renders and frames for first pass.

Tier 1

Car tracking

38

Tier 2 Starting from best detection: 1) Rerun detection on best frame 2) Detect surrounding frames

Surrounding frames: - Car is at nearly same position - Car viewed under nearly same angle

Car tracking

39

Tier 3: smoothing Smooth angles Smooth boudingbox location and size

Horizontal angle

Vertical angle

Car tracking

40

Tier 3: smoothing Smoothed angles can still be used Exact size and position needed

Horizontal angle

Vertical angle

Car tracking

41

Tier 3: smoothing Keypoint tracking

Track points using feature-tracking algorithm Calculate new positions using rendering pipeline

Car tracking

42

Tier 3: smoothing Keypoint tracking

Car tracking

43

Tier 3: smoothing Car rotation and template matching

1) Best match

2) Rotate to desired angle 3) Find rotated car in cropped image

Car tracking

44

Tier 3: smoothing Car rotation and template matching

Car tracking

45

Tier 3: smoothing Car rotation and template matching

1) Detect car 2) Rotate car to desired angle 3) Average multiple rotated cars

Car tracking

46

Tier 3: smoothing

Presentation Overview

• Thesis objective

• Conversion process

• Conclusion and result 47

• 3D rotation

• Car detection and tracking

• Inpainting

• Overlay detection and conversion of background

48

• Translation + rotation

Retinal Disparity

3D effect

Inpainting

49

Single frame methods:

Geometrical inpainting: Patch-based inpainting:

Inpainting

50

Single frame methods:

Geometrical inpainting: Patch-based inpainting:

Inpainting

51

Single frame methods:

Objective comparison Only outermost mixels important

Inpainting

52

Video inpainting

Inpainting

53

Video inpainting

Inpainting

54

Video inpainting

Inpainting

55

Video inpainting

Inpainting

56

Video inpainting

Presentation Overview

• Thesis objective

• Conversion process

• Conclusion and result 57

• 3D rotation

• Car detection and tracking

• Inpainting

• Non-object based 3D conversion

Non-object based 3D conversion

• Overlay detection

• Conversion of background to 3D

58

Non-object based 3D conversion

• Overlay detection

• Conversion of background to 3D

59

Overlay detection

60

Graphical overlay: - Not always present - If present, always at fixed location

Template: averaged over 100 frames

Non-object based 3D conversion

• Overlay detection

• Conversion of background to 3D

61

Conversion of background to 3D

62

Make3D algorithm - Texture gradients - Texture variations - Reduced contrast

Presentation Overview

• Thesis objective

• Conversion process

• Conclusion and result 63

• 3D rotation

• Car detection and tracking

• Inpainting

• Non-object based 3D conversion

Overview of 3D conversion process

64

1) Inpaint car and overlay 2) Generate depthmap 3) Shift background 4) Inpaint background 5) Add overlay 6) Add car 7) Merge to 3D

Presentation Overview

• Thesis objective

• Conversion process

• Conclusion and result 65

• 3D rotation

• Car detection and tracking

• Inpainting

• Non-object based 3D conversion

Result and conclusion

66

Is it a feasible method of 2D-3D conversion?

Yes, but 1) very time-consuming -> also the reason why most of this thesis featured 1 single fragment 2) differences between chosen render-viewpoint and car can become noticeable -> can be solved/prevented by using realistically rendered car instead of the actual car from the frame Car angle, size and location known at each frame -> could be used for other purposes, for example advanced graphical overlays

Result: video on 3D tv

Questions?

67