Advanced point cloud processing

CURRENT DEVELOPMENTS IN AIRBORNE OBLIQUE IMAGING SYSTEMS AND AUTOMATED INTERPRETATION OF COMPLEX BUILDINGS Markus Gerke

THE FIRST AIRBORNE PHOTOS …WERE OBLIQUE

First recorded aerial photograph in the US (Boston),

by James Wallace Black, 1860, source Wikipedia

Intuitively the operator shot from a slanted angle

Recognition of buildings In the 1930’s: USGS and

military systematically captured oblique images

THE FIRST AIRBORNE PHOTOS …WERE OBLIQUE

First recorded aerial photograph in the US (Boston),

by James Wallace Black, 1860, source Wikipedia

Fairchild T3-A camera, source

Petrie (2009)

1. Properties, configurations and systems

2. Applications, products and research

3. Conclusions and Outlook

CONTENT




CONTENT

Vertical/nadir images: good observation of ground features and roof structures, assume constant scale Oblique images: observation of vertical structures, but occlusion more dominant, varying scale

PROPERTIES SCENE OBSERVATION

PROPERTIES SCALE

According to Shufelt (1999): Vertical images: t < 5° Low oblique: 5°< t <30° High oblique: t > 30° If (t+α) > 90°: horizon visible In this presentation: concentration on high oblique images which do not include the horizon

(symbols from Höhle (2008))

PROPERTIES OCCLUSION

“Self-” occlusion

visible

invisible


“Self-” occlusion Occlusion by other objects

visible

invisible

Mitigation through multiple view and overlap


visible

invisible

PROPERTIES OCCLUSION - EXAMPLE

All images © Slagboom en Peeters











According to Petrie (2009) we can distinguish between Fan and Maltese Cross configuration Some examples will follow

PROPERTIES CONFIGURATIONS

Fan configuration: increase swath width across/along track: more effective area coverage, reconnaissance applications/corridor mapping. Example: IGI Dual CAM

PROPERTIES FAN CONFIGURATION

Source: Homepage IGI www.igi.eu

Visionmap A3: two stepping frame cameras: increasing the FOV through sweeping.

PROPERTIES FAN CONFIGURATION

Source: Pechatnikov et al., 2009

Source: Homepage Visionmap

PROPERTIES MALTESE CROSS CONFIGURATION

Source: Petrie, 2009

IGI Penta DigiCAM, Source: Homepage IGI

Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation.


Leica Oblique 5-camera head, source homepage Leica

Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies offering such systems, eg: Track‘Air MIDAS IGI Penta DigiCam Hexagon/Leica 5 camera head, mid format camera RCD30

(multispectral)

Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies using such a configuration, eg: Pictometry / BlomOblique


Microsoft Bing maps

Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies using such a configuration, eg: Pictometry / BlomOblique Google


Microsoft Bing maps

Google maps

Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies using such a configuration, eg: Pictometry / BlomOblique Google Slagboom en Peeters (NL)


Microsoft Bing maps

Google maps




CONTENT

APPLICATIONS, PRODUCTS AND RESEARCH COMPLEXITY OF TASK VS. DEGREE OF AUTOMATION

Low Degree of automation High

Low

C

ompl

exity

of t

ask

High

Manual inspection/visualisation

Monoplotting

3D city modeling Supervised classification

Scene Interpretation

Dense Matching/Meshing

APPLICATIONS, PRODUCTS AND RESEARCH

Degree of automation

Com

plex

ity o

f tas

k


APPLICATIONS, PRODUCTS AND RESEARCH MANUAL INSPECTION/VISUALISATION

Purpose: manually check images, also compare to existing (cadastre) maps Slagboom en Peeters GIS interface

APPLICATIONS, PRODUCTS AND RESEARCH MANUAL INSPECTION/VISUALISATION

Purpose: manually check images, also compare to existing (cadastre) maps Pictometry ESRI interface



Com

plex

ity o

f tas

k


Monoplotting

APPLICATIONS, PRODUCTS AND RESEARCH MONOPLOTTING

Purpose: 2.5D mapping in images, e.g. for topographic maps, or height measurements, e.g. of houses, trees, lamp-posts etc. Tools e.g. Pictometry EFS (Electronic Field Study) or Pictometry online BlomDESKTOP Viewer or BlomWEB Viewer


Principle of height measurement in oblique images: If a DTM is known, the X-Y location of T can be computed through ray intersection with the DTM. Fixing the X-Y location and measuring dr’ in the image leads to an estimate of dh.


Tool: EFS

Source: IFA Sydney Togarepi, 2009

Own assessment of EFS (Enschede images): 2D accuracy: RMSE = 0.5m height accuracy: RMSE = 0.6m Problem: DTM specs not disclosed, DTM not exchangeable Camera calibration and orientation parameters cannot be modified



Com

plex

ity o

f tas

k


Monoplotting

3D city modeling

APPLICATIONS, PRODUCTS AND RESEARCH 3D CITY MODELING/CAD-LIKE RECONSTRUCTION

Purpose: interactive building modeling Texture mapping (Frueh et al., 2004, Wang et al., 2008) Pre-compiled collections like BlomUrbex 3D Interactive image-based modeling Eagleview


BlomUrbex 3D: Access to 400 city models in different LODs

LOD 2, textures from Blom oblique images. Source: Homepage Blom.


Interactive image-based modeling

Autodesk Imagemodeler. Image ©Blom.


Interactive image-based modeling

Final model.


Roof/Wall report, e.g. from Eagleview/Pictometry

Source: www.eagleview.com



Com

plex

ity o

f tas

k


Monoplotting

3D city modeling


APPLICATIONS, PRODUCTS AND RESEARCH DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION

Purpose: visualization of 3D structures, simple modeling, derivation of height models Open source/research: Semi-Global matching (Hirschmüller 2008), PMVS2 (Furukawa and

Ponce, 2010) Meshlab (meshlab.sourceforge.net)

Commercial software: EADS Streetfactory (http://www.astrium-geo.com/street-factory)


Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, SGM result from IFP Stuttgart (Fritsch et al., 2012)


Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, SGM result from IFP Stuttgart (Fritsch et al., 2012)


Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, PMVS2 point cloud


Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, PMVS2 point cloud


Purpose: visualization of 3D structures, simple modeling, derivation of height models IGI, Lünen, PMVS2 point cloud, meshed (Poisson reconstruction)


Purpose: visualization of 3D structures, simple modeling, derivation of height models Slagboom en Peeters, Enschede, PMVS2 point cloud, meshed


Purpose: visualization of 3D structures, simple modeling, derivation of height models Commercial system: EADS Street Factory

Source: EADS http://www.astrium-geo.com/street-factory



Com

plex

ity o

f tas

k


Monoplotting

3D city modeling Supervised classification


APPLICATIONS, PRODUCTS AND RESEARCH SUPERVISED (3D) CLASSIFICATION

Purpose: semi-automatically detect objects in the scene Especially vertical structures like walls interesting for object recognition Clear advantage over nadir images Example: supervised classification in object space: 3D points are classified instead of image pixels information fusion


Purpose: semi-automatically detect objects in the scene Disaster mapping (here: after 2010 Haiti earthquake) Facade (health) status important to assess building damage: advantage

of oblique images over nadir images

original images © Pictometry

Vertical


Purpose: semi-automatically detect objects in the scene Disaster mapping (here: after 2010 Haiti earthquake) Facade (health) status important to assess building damage: advantage

of oblique images over nadir images

original images © Pictometry 3D point cloud from image

matching (PMVS2)

Vertical

North

East


Purpose: semi-automatically detect objects in the scene Disaster mapping (here: after 2010 Haiti earthquake) Classifying trees/vegetation, facades, intact roofs, destroyed roofs 3D points with label information

Completeness reached in several

experiments: Trees/Vegetation: 63%

Facades: 82% Roof intact: 80%

Roof destroyed: 75%

source: Gerke 2011


Purpose: semi-automatically detect objects in the scene Same approach, but “intact” scene Enschede, overall accuracy ~80%



Com

plex

ity o

f tas

k


Monoplotting

3D city modeling


Supervised classification

Scene Interpretation

APPLICATIONS, PRODUCTS AND RESEARCH SCENE INTERPRETATION

Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification.

APPLICATIONS, PRODUCTS AND RESEARCH SCENE INTERPRETATION: BUILDING DETECTION

Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Step 1: façade detection

see Xiao et al., 2012


Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Step 2: outlining

see: Xiao 2012


Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Results: -detection(building level): correctness 95% , completeness: 70 to 90% -outlining: accuracy of outline (RMSE): 85cm – 1.10m Quality of detection and delineation depend largely on image resolution and overlap source: PhD thesis Jing Xiao, unpublished yet

APPLICATIONS, PRODUCTS AND RESEARCH SCENE INTERPRETATION: BUILDING VERIFICATION

Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building verification: Given a cadaster map the walls are verified using different evidence measures extracted from oblique images Eg: lines projected: if wall exists facade lines match in object space (blue)

source: Nyaruhuma et al., 2012

APPLICATIONS, PRODUCTS AND RESEARCH SCENE INTERPRETATION: BUILDING VERIFICATION

Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building verification: Given a cadastre map the walls are verified using different evidence measures extracted from oblique images Results: -correctness of decisions (building still existing/building demolished): 90-100% -potential to go for a per-wall check

source: Nyaruhuma et al., 2012




CONTENT

CONCLUSIONS

Oblique airborne photogrammetry in analogue times was too expensive, but with digital image evolution it comes back into focus

Oblique airborne photogrammetry is becoming mature shift to mid format cameras and sophisticated integrated sensor orientation

Applications in practice so far mainly visualization and (interactive) city modeling, automatic approaches (low level, high level) promising: Images implicitly contain 3rd dimension; no need for additional data Facades are very important to identify buildings However: large image overlap preferred (mitigation of occlusion

effects, multiple views per façade)

OUTLOOK

What the future might bring (personal view) More airborne surveying companies will acquire cameras (see mid format

heads from Leica/IGI) Oblique airborne images will become standard (soon in NL: capturing of

entire country by AeroData and Cyclomedia in Spring 2013 http://www.nederlandobliek.nl)

Boost of usage, mostly for urban applications: map update, 3D modeling Automation: standard software needs to be extended, starting with

bundle block adjustment/integrated sensor orientation

http://www.nederlandobliek.nl/

LITERATURE

Fritsch, D., Kremer, J., Grimm, A. 2012. A Case Study of Dense Image Matching Using Oblique

Imagery – Towards All-in-one Photogrammetry. GIM International. 26(4): 18-23.

Frueh, C., Sammon, R., Zakhor, A., 2004. Automated texture mapping of 3D city models with oblique

aerial imagery. Proceedings of the 2nd International Symposium on 3D Data Processing,

Visualization, and Transmission (3DPVT'04), 396-403.

Furukawa, Y., and J. Ponce, 2010. Accurate, Dense, and Robust Multi-View Stereopsis. Pattern

Analysis and Machine Intelligence, IEEE Transactions on, 32(8): 1362-1376.

Gerke, M., 2011. Supervised classification of multiple view images in object space for seismic

damage assessment. Photogrammetric image analysis : ISPRS conference, PIA 2011, Lecture Notes

in Computer Science

Hirschmüller, H., 2008. Stereo Processing by Semi-Global Matching and Mutual Information. IEEE

Transactions on Pattern Analysis and Machine Intelligence, 30(2): 328-341.

6952: 221-232.

LITERATURE

Höhle, J., 2008. Photogrammetric measurements in oblique aerial images. Photogrammetrie

Fernerkundung Geoinformation, 2008(1): 7-14.

Nyaruhuma, A.P., Gerke, M., Vosselman, G., Mtalo. E.G., 2012. Verification of 2D building outlines

using oblique airborne images. ISPRS Journal of Photogrammetry and Remote Sensing, 71: 62-75.

Petrie, G., 2009. Systematic oblique aerial photography using multi frame cameras. Photogrammetric

Engineering & Remote Sensing,75(2): 102-108.

Shufelt, J.A., 1999. Performance evaluation and analysis of monocular building extraction from aerial

imagery. IEEE Transactions on Pattern Analysis and Machine Intelligence 21 (4): 311-326. Wang, M., Bai, H., Hu, F., 2008. Automatic texture acquisition for 3D model using oblique aerial

images. The First International Conference on Intelligent Networks and Intelligent Systems (ICINIS

'08), 495-498.

LITERATURE

Xiao, J., Gerke, M. and Vosselman, G., 2012. Building extraction from oblique airborne imagery

based on robust facade detection. ISPRS Journal of Photogrammetry and Remote Sensing, 68: 56-

68.

Xiao, J., 2012. Automatic building outlining from multi-view oblique images. XXII ISPRS Congress :

Imaging a Sustainable Future, ISPRS Annals, I-3, 2012 ed. by M. Shortis, N. Paparoditis and C.

Mallet, 323-328

Thank you for your attention. Questions?

Documents

Advanced point cloud processing