Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
…
…
…
…
…
…
…
…
…
…
Patrick Helber, Benjamin Bischke, Damian Borth, Andreas Dengel
Deep Learning Competence Center & Smart Data and Knowledge Services
German Research Center for Artificial Intelligence (DFKI)
Earth Observation From SpaceDeep Learning Based Satellite Image Analysis
…
…
…
…
Satellite Data Becomes Publicly Available
2
© ESA / Astrium
Global Challenges
3
Satellite Analysis -> Global Challenges
4
Infrastructure & Traffic MonitoringPollution Monitoring: Oil Spills and Pipelines
Land Change Detection, DeforestationAgriculture
Agriculture: Monitoring Farm Land
5
© ESA
Climate Change: Deforestation
6
Monitoring of Economic Factors
7
• Economic Growing and Shrinking
© DigitalGlobe
Natural Disaster: Flooding, Wildfire, Earthquake …
8
Earth Observing SatellitesOptical and Radar
9
Satellites – Optical View
• Sentinel-2
– 10 meters per pixel
– Two-satellite constellation
– 5 days revisit time
– Global land surface coverage• Onshore, large islands, inland and coastal waters
• Worldview, GeoEye, SkySat
– Up to ca. 0.30 meters per pixel
– High-resolution images
10
Author: Raman
Optical View: 10 meters/pixel
11
Optical View: High-resolution up to ca. 30cm/pixel
12
Radar Satellites – Sentinel 1
• Synthetic Aperture Radar (SAR)
• Sentinel-1
– 6 day repeat cycle
– Two satellite constellation
• For the first time continuous
radar scanning
13
© ESA
Radar-Based Earth Observation
14
Satellite Image ClassificationOptical Images
15
EuroSAT Publicly Released by DFKI
16
Ground Truth Data
• European Urban Atlas
– Detailed mapping for 695 cities distributed over 30 Europeancountries
– Released August 2016
– Covered time period: 2011-2013
17
P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” arXiv preprint arXiv:1709.00029, 2017.
EuroSAT - Distributed Over 30 Countries
• 10 classes
• 27,000 images
• 64 x 64 images
• 2,000 – 3,000 images per class
• 13 spectral bands• Spatial resolution: 10 m per pixel
18
P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” arXiv preprint arXiv:1709.00029, 2017.
Land-Use and Land-Cover Classification
19
Industrial Residential Highway Annual Crop Permanent Crop Pasture
Sea & Lake RiverForestHerbaceous vegetation
Classification with ResNet and GoogLeNet
• Dataset split
– 80% Training and 20% Testing
• Convolutional Neural Networks
– ResNet-50
– GoogLeNet
20
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9, 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
Inception Module
ResNetGoogLeNet
Residual Building Block
Models Pre-trained on ImageNet
21
• Transfer learning
– Fine-tuning of pre-trained networks
• Pre-trained on ImageNet
– ILSVRC-2012
Band combination
RGB SWIR CI
Accuracy 0.9857 0.9705 0.9830
SWIR = Short-Wave-Infrared
CI = Color-Infrared
Classification Results
22
• Transfer learning
– Fine-tuning of pre-trained networks
• Pre-trained on ImageNet
– ILSVRC-2012
Band combination
RGB SWIR CI
Accuracy 0.9857 0.9705 0.9830
SWIR = Short-Wave-Infrared
CI = Color-Infrared
Spotting Land Use Changes
23
Residential Area Built Upin Dallas, USA
August 2015 March 2017
• Large-scale scanning and monitoring
• Time component• High frequency
• Near-real-time
• Future availability
• Innovative applications
• Building systems for future
real-time applications
Spotting Land Use Changes
October 2015 September 2016
Deforestation (Forest Clearing)in Villamontes, Bolivia
24
Industrial Buildings Demolished in Shanghai, China
December 2015 December 2016
Support Mapping Services
25
Melbourne, Australia Shanghai, China
Look More CloselyClassification with High-Resolution Optical Images
26
Functional Map of the World
• Four different satellites
• Different spatial and spectralresolutions
– 4-band and 8-band
• Temporal image sequences
• 63 land-use/-cover and building types
– Security & Defense
– Emergency Response
– Infrastructure
27
Christie, Gordon and Fendley, Neil and Wilson, James and Mukherjee, Ryan, “Functional Map of the World” CVPR, 2018.
Airport Car DealershipBorder Checkpoint
Police Station Shopping MallFlooded Road
Lighthouse DamSolar Farm
Task: Sequence Classification
• Sequences determine the underlying class
– Construction sites
– Flooded road
28
Classification and Fusion Approaches
• DenseNet-161
– CNN Features (I)
– Metadata Features (M)
– CNN + Metadata Features (IM)
• Sequence Classification
with LSTM
29
DenseNet
Approach LSTM-M CNN-I LSTM-I CNN-IM LSTM-IM
F1-Score 0.193 0.679 0.688 0.722 0.734
Christie, Gordon and Fendley, Neil and Wilson, James and Mukherjee, Ryan, “Functional Map of the World” CVPR, 2018.
CNN (Features)
Metadata Features
Misclassification
Hard to classify
• Port Shipyard
• Office building Fire station, police station
• Hospital Educational institution
• Police station Educational institution
• Multi unit residential Single unit residential
30
Shipyard Port
HospitalEducational Institution
Approach Port Shipyard Police station
Office Building
Hospital
F1-Score 0.193 0.351 0.329 0.225 0.458
Satellite Image SegmentationHigh-Resolution Optical Images (Aerial)
31
Fine-Grained Segmentation With 10cm/Pixel Images
32
Fine-Grained Cadastral Map for Land Change Detection
33
Work in progress!
Netherlands: High-Resolution RGB+NIR
34
Leverage Learning with Altitude Measurements
35
Beyond the Visible SpectrumMultispectral Signatures
36
Multispectral Classification
37
EuroSATRGB
FMoW
Multispectral Segmentation
• Damage Estimation Caused by Wildfires
– Dataset• Satellite: Sentinel-2
• Mapping: EMS Coperncicus
• 13 spectral bands
• 1.564 scenes from 5 different countries
• 256px images
– Band combinations• Visible spectrum: RGB
• Infrared spectrum: Red Edge
• Near-Infrared: NIR
38
RGB + Red Edge + NIR Spectral Information
39
SegNet
SegNet Input Pix. Accuracy mIOU
RGB 0.9878 0.7657
Red Edge 0.9885 0.7592
NIR 0.9815 0.4964
RGB + NIR 0.9864 0.7844
RGB + Red Edge 0.9855 0.7928 (+2,71)
RGB + Red Edge + NIR Prediction
loss
epoch
Qualitative Results
40
GT RGB RGB + Red Edge
RGB Red Edge10m 20m
Beyond the Visible SpectrumRadar Waves
41
Optical Satellite Images - Obstructed View
42
Optical Satellite Images - Obstructed View
43
Optical Satellite Images - Obstructed View
44
Optical Satellite Images - Obstructed View
45
Vision Independent of Clouds
46
Radar-Based Earth Observation Imagery
47
VVVH
© ESA
• Synthetic Aperture Radar (SAR)
• Satellite: Sentinel-1
• Dual-Polarization
– VV and VH
• Scenarios• Deforestation
• Find safest route for icebreakers
• Land monitoring
• Ship traffic monitoring
• Natural disasters like flooding and earthquake
Dual-Polarized SAR
48
Flooding
• Satellite: Sentinel-1
• Mapping: EMS Copernicus
• 9.426 radar image scenes
• 56 flooding events in 12 countries
• Globally spread• Australia, USA, France, UK, …
Segmentation with Dual-Polarized Radar Waves
• Convolutional Networks
– SegNet: Convolutional Encoder-Decoder Network
49
SegNet PredictionInput
VV + VH
Model Pix. Accuracy mIOU
FCN-6 0.9496 0.6772
VGG16 0.9413 0.6740
SegNet 0.9530 0.6541
Segmentation
Natural Disasters: Damage Maps
Flooding: Damage Estimation Fire: Damage Estimation
• Biomass by radar waves
50
• Government, Insurance, Investment
Copernicus Emergency Management Service (© European Union, 2012-2017)
Destroyed
Highly damaged
Slight damage
Multimedia Satellite Task 2018
• Emergency Response for Flooding Events
• Focus on Road Passibility (Impact of Infrastructure) (Road Access, blocked Road) with two subtasks
• Classification of Road-Access & Passability– in Social Multimedia
– in multiple Satellite Images (Radar, Optical)
• Registration open (http://www.multimediaeval.org)
51
Images in Tweets (Hurricane Harvey)
Satellite Images of Houston (US) for Hurricane Harvey
CombineResultvia Geo-Location
Thanks!
52
NVAIL partner
all networks trained on DGX-1