Augmented Reality Using High Fidelity Spherical Panorama with HDRI

Researcher: Zi Siang See

Augmented Reality using High FidelitySpherical Panorama with HDRI

Research Presentation: Zi Siang SeeHuman Interface Technology Laboratory New Zealand (HIT Lab NZ)

Email (Industrial collaboration): [email protected] (University Tunku Abdul Rahman) [email protected] (University of Canterbury): [email protected]

Zi Siang See, Mark Billinghurst, Adrian David Cheok (2015)SIGGRAPH Asia 2015 Symposium On Mobile Graphics And Interactive Applications

Zi Siang See, Mark Billinghurst, Adrian David Cheok (2015) Augmented Reality using High Fidelity Spherical Panorama with HDRI. SIGGRAPH Asia 2015 Symposium On Mobile Graphics And Interactive Applications.http://dx.doi.org/10.1145/2818427.2818445http://www.zisiangsee.com/research/zisiangsee_research_p009.pdf

http://dx.doi.org/10.1145/2818427.2818445

http://www.zisiangsee.com/research/zisiangsee_research_p009.pdf




Research Presentation

1) Introduction, Related Work

2) Method and Apparatus

3) Discussion


Figure: AR on Mobile

• Mobile Augmented Reality (AR) mixes a live real-world view with virtualinteractive content on a mobile device.

• One of the key enablers for this is tracking technology, such as computervision techniques for tracking off pre-defined markers or markerlessimages.

Introduction: Augmented Reality (AR) on Mobile

Source: google search (keyword “augmented reality”)


Augmented Reality using High Fidelity Spherical Panorama with HDRI

High fidelity requirements – mobile AR environment recognition

• Spherical Panorama

• High Dynamic Range Imaging (HDRI)

• Optimization involves obtaining accurately reproduced sphericalpanorama with least acquisition time - AR source tracking content hassufficient luminance by using HDRI having dynamic range extended fromsingle acquired digital negative for each angle in multirow configuration.


Figure: (a) AR using marker. (b) AR using panorama image, markerless experience.

• There have been previous studies on using pre-produced panoramaimages for AR tracking (Arth et al 2011; DiVerdi et al. 2008; Langlotz et al.2014; Ventura and Höllerer 2013; Wagner et al. 2010).

• Most of related studies describe how the panorama images can be usedfor AR tracking.

• Therefore, there is a need for specifying the method for high fidelityproduction of the source panorama images for mobile AR.

Introduction: AR using Panorama Image(a) (b)


Case Study: Corporate Project Requirements


Townships

Landscape

Augmented /Virtual

Other Scope

Construction

Panorama ImageReproduction, Projects

Heavy Requirement & Workload/Quantity

Industrial Case Study: Amount, Time, Quality


Figure: Panorama image reproduction process for AR /VR projects.




Introduction: Spherical Panorama Acquisition


Figure: reproduction options available for fast panorama image acquisition


LadyBug GoPro Rig Mobile (Handheld)


Figure: reproduction options available for fast panorama image acquisition


LadyBug GoPro Rig Mobile (Handheld)

Parallax errorShake /Blur

Resolution Limited Dynamic Range

Inconsistent lighting distribution for multiple angles

Nadir difficulty

Ghosting effect

(HDRI)

Inconsistent white balance for multiple angles

Time management


• Acquire wider panoramic visibility (Benosman and Kang, 2001)

• Major difficulty for spherical panorama reproduction may occur, this caninclude limited dynamic range reproduced and parallax error (Brown andLowe, 2006; Felinto et al, 2012)

Figure: panoramic imagery with visual abnormality - location in Sannomiya Kobe, Japan.


Camera shake

Limited dynamic range Visual abnormality

Parallax error


Figure: combined luminance (EV) range being covered by HDRI outcome.

Darker images(luminance) Brighter images

(luminance)

Introduction: HDRI (for multiple angle images)

• High Dynamic Range Imaging (HDRI) in photography technique has a goalto increase the dynamic range resulting an improved luminancereproduced in shadow and highlight areas (Reinhard et al, 2010; Debevecand Malik, 1997; Fairchild, 2007), usually by combining multiple exposuresconsist of darker and brighter images.


• Obstacles and issues of HDRI that involves multiple exposures can include lengthy acquisition (for each angle), ghosting effect and misalignment issue due to inconsistent acquisitions (Reinhard et al, 2010)

Figure: HDRI with visual abnormalities, ghosting.

Introduction: HDRI (for multiple angle images)

Multiple exposures for HDRISingle exposure (LDR)

Visual abnormality


• Spherical panorama can be used for AR as source tracking image.

• Various studies (Felinto et al 2012, Gawthrop 2007, Brown and Lowe2006; Schulz, 2012) have shown that photographic combination for HDRIand spherical panorama can be possible, however conditionally toleratingwith obstacles and issues.

Spherical Panorama (Multirow) HDRI requires multiple exposures

-4, -3, -2, -1, 0, +1, +2, +3, +4 (EV)

Difficulty: Spherical Panorama and HDRI for AR?


Figure: example of source tracking images or scenarios for AR

• For AR, Lieberknecht et al. [2009] has shown that it can be difficult tocreate source tracking images, or synthetic images that reproduce the realeffects of real-world phenomena such as lighting, noise, motion blur,discretization, blooming or limited color depths during real imageacquisition.

• Image elements that influence real-time tracking results include texturerichness, the texture repeatability of the objects to be tracked, the cameramotion and speed, the changes of the object scale, and variations of thelighting conditions over time.

Difficulty: Source Tracking Images for AR

Source: google search (keyword “augmented reality”)





3) Discussion


.. no photographic process is complete until the final images can be proudly displayed (Phase One, 2015).


Method: Spherical Panorama

Additional 3 nadir acquisition

Horizontal angles

Zenith

Nadir image acquired withoutincluding the base of the camera.

Stable mounting is required foravoiding drifting, shake and parallaxerror.

Nadir without obstacle,

allow sufficient space.


Method: High Dynamic Range Image (HDRI)Pix

el Valu

e

Sequences of extendable dynamic range from single RAW(photographic recording instrument Nikon D3x)

• Extending Dynamic Range from single acquired Digital Negative (RAW).

• Consistent to the idea brought up by Debevec and Malik (1997) aboutextending dynamic range can be possible from conventional single filmprocessing - multiple scanning method.


Method: High Dynamic Range Image (HDRI)

• -2 EV and +2 EV, resulting an additional increment from 8.5EV to 12.5EV.

• The finding in the study about native EV of dynamic range reproductionwas observed to have similar approximated usable exposure values withthe RAW produced by ISO100 according to the test by Rehm (2009).

Pix

el Valu

e

Sequences of extendable dynamic range from single RAW(photographic recording instrument Nikon D3x)



Figure: an image from multiple angle before applying HDRI.

without


Figure: proposed multiple angles HDRI for high fidelity AR.(HDRI method that requires a single acquisition that extendsthe dynamic range from a digital negative)


with HDRI



Figure: early attempt for HDRI computation from the source of single acquired digital negative, RAW.

HDRI Computation

Input: Source image in digital negative RAW format (e.g. NEF, DNG)

Output: The HDRI is reproduced from a single acquired RAW image having its dynamic

range extended from RAW

1. Three images, I, I+, I- are processed from the single source of RAW.

2. I+ and I- are respectively altered by +2EV and -2EV where dynamic range are extended.

That is, the luminance lx of each pixel x is recalculated as (lx)new = 2m (lx)

old, where m is the

change in EV.

3. A new image, I’+, is constructed out of I+, by preserving only the highlight regions of I+.

Similarly, a new image I’- is constructed out of I-, by preserving only the non-highlight

regions of I-.

4. I’+ is then overlaid on top of I’- with 50% opacity. The results of the two are then combined

to form the final image I’.


Source Image for AR: High Fidelity Spherical Panorama with HDRI

multiple angles facilitated by HDRI required for producing the spherical panorama(least or near-zero visual abnormality)


Multiple angles HDRI of multirow configuration (from the proposed method) with reduced error or mistakes that requires rectifications, including near-perfect nadir angle reproduction.

Nadir Angle:Reproduction from the original location-based scene, minimal post-production correction required

cubic projection



multiple angles facilitated by HDRI required for producing the spherical panorama(least or zero visual abnormality)



HDRI ghosting effectusing multiple exposures method.

HDRI reproduction from single acquisition, digital negative (RAW)

visual abnormality is eliminated or avoided (for multiple angles required for spherical panorama)



multiple angles facilitated by HDRI required for producing the spherical panorama(least or zero visual abnormality)



(a) LDR photo image (b) HDRI from single source acquisition (c) HDRI from multiple exposures

HDRI Panorama with reduced overall photographic production processes and time.






3) Discussion


Augmented Reality using High Fidelity Spherical Panorama with HDRI(Authoring process for AR environment recognition)

Spherical Panorama facilitated with HDRI(Cubic projection)

Cropped /selective cubic facadeworking with image-based AR

Figure: augmentation can be assigned on any cubic facades, speculated to work on most of the AR browsers /workflow.


AR for Mobile and Wearable

Figure: handheld experience of markerless AR.



Figure: handheld experience of markerless AR (video).



Figure: stereo image of AR – adaptable for HMD such as Oculus or Google Cardboard.



Figure: HDRI spherical panorama sampled at Sannomiya Station, Japan (01 Nov 2015).








Discussion

Potential obstacles and issues (before optimization):

• Parallax error (especially during hand-held)

• Unstable image acquisition, drifting motion (hand-held)

• Nadir angle difficulty

• Compromised geometrical registration in image

• Low dynamic range image (LDRI)

• High dynamic range requires multiple exposures

• Long acquisition time if multiple exposures

• High dynamic range ghosting with moving objects

• Multiple exposures misalignment

• Inconsistent lighting distribution for multiple angles

Benefit of using our method and apparatus:

• Free from parallax error

• Stable image acquisition

• Nadir angle with near-error-free authenticity

• Accurate geometrical registration in image

• High dynamic range image (HDRI)

• High dynamic range from single RAW acquisition

• High dynamic range with least acquisition time

• High dynamic range with no ghosting

• High dynamic range with perfect alignment

• Consistent high dynamic range for multiple angles


Discussion

Potential obstacles and issues (before optimization):

• Parallax error (especially during hand-held)

• Unstable image acquisition, drifting motion (hand-held)

• Nadir angle difficulty

• Compromised geometrical registration in image

• Low dynamic range image (LDRI)

• High dynamic range requires multiple exposures

• Long acquisition time if multiple exposures

• High dynamic range ghosting with moving objects

• Multiple exposures misalignment

• Inconsistent lighting distribution for multiple angles

Benefit of using our method and apparatus:

• Free from parallax error

• Stable image acquisition

• Nadir angle with near-error-free authenticity

• Accurate geometrical registration in image

• High dynamic range image (HDRI)

• High dynamic range from single RAW acquisition

• High dynamic range with least acquisition time

• High dynamic range with no ghosting

• High dynamic range with perfect alignment

• Consistent high dynamic range for multiple angles


Conclusion

• Markerless AR.

• High fidelity spherical panorama with HDRI can provide a near-error-freeand dynamic range enhanced source of image-based AR tracking content.

• Source tracking content is reproduced with little distortion, producing aresult very similar to the original scene condition.

• Without using extra sensors such as a GPS.

• Allows an AR experience to be delivered on a mobile device with a lowerprocessing requirement - adaptable in other wearable system or devices.


Implications

• Android-based mobile with RAW capturing capability.

• Approach adaptation in VR/AR workflow for mobile applications.


Future Study

• In the future we will conduct more extensive evaluation studies tocompare the tracking accuracy with the systems using our HDRIpanorama images to other traditional approaches.

• We will also explore other solutions suitable for HDRI panorama videoand hybrid approaches that combine panorama image tracking andsensor input.


Thank you

Research Discussion

collaboration /contact

www.zisiangsee.com

Zi Siang See, Mark Billinghurst, Adrian David Cheok (2015) Augmented Reality using High Fidelity Spherical Panorama with HDRI. SIGGRAPH Asia 2015 Symposium On Mobile Graphics And Interactive Applications.http://dx.doi.org/10.1145/2818427.2818445http://www.zisiangsee.com/research/zisiangsee_research_p009.pdf

http://www.zisiangsee.com/

http://dx.doi.org/10.1145/2818427.2818445

http://www.zisiangsee.com/research/zisiangsee_research_p009.pdf

Education

Augmented Reality Using High Fidelity Spherical Panorama with HDRI