Deep Learning Competence Center & Smart Data and Knowledge … · 2018. 4. 19. ·...

Patrick Helber, Benjamin Bischke, Damian Borth, Andreas Dengel

Deep Learning Competence Center & Smart Data and Knowledge Services

German Research Center for Artificial Intelligence (DFKI)

Earth Observation From SpaceDeep Learning Based Satellite Image Analysis

Satellite Data Becomes Publicly Available

Global Challenges

Satellite Analysis -> Global Challenges

Infrastructure & Traffic MonitoringPollution Monitoring: Oil Spills and Pipelines

Land Change Detection, DeforestationAgriculture

Agriculture: Monitoring Farm Land

Climate Change: Deforestation

Monitoring of Economic Factors

• Economic Growing and Shrinking

Natural Disaster: Flooding, Wildfire, Earthquake …

Earth Observing SatellitesOptical and Radar

Satellites – Optical View

• Sentinel-2

– 10 meters per pixel

– Two-satellite constellation

– 5 days revisit time

– Global land surface coverage• Onshore, large islands, inland and coastal waters

• Worldview, GeoEye, SkySat

– Up to ca. 0.30 meters per pixel

– High-resolution images

Author: Raman

Optical View: 10 meters/pixel

Optical View: High-resolution up to ca. 30cm/pixel

Radar Satellites – Sentinel 1

• Synthetic Aperture Radar (SAR)

• Sentinel-1

– 6 day repeat cycle

– Two satellite constellation

• For the first time continuous

radar scanning

Radar-Based Earth Observation

Satellite Image ClassificationOptical Images

EuroSAT Publicly Released by DFKI

Ground Truth Data

• European Urban Atlas

– Detailed mapping for 695 cities distributed over 30 Europeancountries

– Released August 2016

– Covered time period: 2011-2013

P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” arXiv preprint arXiv:1709.00029, 2017.

EuroSAT - Distributed Over 30 Countries

• 10 classes

• 27,000 images

• 64 x 64 images

• 2,000 – 3,000 images per class

• 13 spectral bands• Spatial resolution: 10 m per pixel

P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” arXiv preprint arXiv:1709.00029, 2017.

Land-Use and Land-Cover Classification

Industrial Residential Highway Annual Crop Permanent Crop Pasture

Sea & Lake RiverForestHerbaceous vegetation

Classification with ResNet and GoogLeNet

• Dataset split

– 80% Training and 20% Testing

• Convolutional Neural Networks

– ResNet-50

– GoogLeNet

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9, 2015.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.

Inception Module

ResNetGoogLeNet

Residual Building Block

Models Pre-trained on ImageNet

• Transfer learning

– Fine-tuning of pre-trained networks

• Pre-trained on ImageNet

– ILSVRC-2012

Band combination

RGB SWIR CI

Accuracy 0.9857 0.9705 0.9830

SWIR = Short-Wave-Infrared

CI = Color-Infrared

Classification Results

• Transfer learning

– Fine-tuning of pre-trained networks

• Pre-trained on ImageNet

– ILSVRC-2012

Band combination

RGB SWIR CI

Accuracy 0.9857 0.9705 0.9830

SWIR = Short-Wave-Infrared

CI = Color-Infrared

Spotting Land Use Changes

Residential Area Built Upin Dallas, USA

August 2015 March 2017

• Large-scale scanning and monitoring

• Time component• High frequency

• Near-real-time

• Future availability

• Innovative applications

• Building systems for future

real-time applications

Spotting Land Use Changes

October 2015 September 2016

Deforestation (Forest Clearing)in Villamontes, Bolivia

Industrial Buildings Demolished in Shanghai, China

December 2015 December 2016

Support Mapping Services

Melbourne, Australia Shanghai, China

Look More CloselyClassification with High-Resolution Optical Images

Functional Map of the World

• Four different satellites

• Different spatial and spectralresolutions

– 4-band and 8-band

• Temporal image sequences

• 63 land-use/-cover and building types

– Security & Defense

– Emergency Response

– Infrastructure

Christie, Gordon and Fendley, Neil and Wilson, James and Mukherjee, Ryan, “Functional Map of the World” CVPR, 2018.

Airport Car DealershipBorder Checkpoint

Police Station Shopping MallFlooded Road

Lighthouse DamSolar Farm

Task: Sequence Classification

• Sequences determine the underlying class

– Construction sites

– Flooded road

Classification and Fusion Approaches

• DenseNet-161

– CNN Features (I)

– Metadata Features (M)

– CNN + Metadata Features (IM)

• Sequence Classification

with LSTM

DenseNet

Approach LSTM-M CNN-I LSTM-I CNN-IM LSTM-IM

F1-Score 0.193 0.679 0.688 0.722 0.734

Christie, Gordon and Fendley, Neil and Wilson, James and Mukherjee, Ryan, “Functional Map of the World” CVPR, 2018.

CNN (Features)

Metadata Features

Misclassification

Hard to classify

• Port Shipyard

• Office building Fire station, police station

• Hospital Educational institution

• Police station Educational institution

• Multi unit residential Single unit residential

Shipyard Port

HospitalEducational Institution

Approach Port Shipyard Police station

Office Building

Hospital

F1-Score 0.193 0.351 0.329 0.225 0.458

Satellite Image SegmentationHigh-Resolution Optical Images (Aerial)

Fine-Grained Segmentation With 10cm/Pixel Images

Fine-Grained Cadastral Map for Land Change Detection

Work in progress!

Netherlands: High-Resolution RGB+NIR

Leverage Learning with Altitude Measurements

Beyond the Visible SpectrumMultispectral Signatures

Multispectral Classification

EuroSATRGB

Multispectral Segmentation

• Damage Estimation Caused by Wildfires

– Dataset• Satellite: Sentinel-2

• Mapping: EMS Coperncicus

• 13 spectral bands

• 1.564 scenes from 5 different countries

• 256px images

– Band combinations• Visible spectrum: RGB

• Infrared spectrum: Red Edge

• Near-Infrared: NIR

RGB + Red Edge + NIR Spectral Information

SegNet

SegNet Input Pix. Accuracy mIOU

RGB 0.9878 0.7657

Red Edge 0.9885 0.7592

NIR 0.9815 0.4964

RGB + NIR 0.9864 0.7844

RGB + Red Edge 0.9855 0.7928 (+2,71)

RGB + Red Edge + NIR Prediction

Qualitative Results

GT RGB RGB + Red Edge

RGB Red Edge10m 20m

Beyond the Visible SpectrumRadar Waves

Optical Satellite Images - Obstructed View

Vision Independent of Clouds

Radar-Based Earth Observation Imagery

• Synthetic Aperture Radar (SAR)

• Satellite: Sentinel-1

• Dual-Polarization

– VV and VH

• Scenarios• Deforestation

• Find safest route for icebreakers

• Land monitoring

• Ship traffic monitoring

• Natural disasters like flooding and earthquake

Dual-Polarized SAR

Flooding

• Satellite: Sentinel-1

• Mapping: EMS Copernicus

• 9.426 radar image scenes

• 56 flooding events in 12 countries

• Globally spread• Australia, USA, France, UK, …

Segmentation with Dual-Polarized Radar Waves

• Convolutional Networks

– SegNet: Convolutional Encoder-Decoder Network

SegNet PredictionInput

VV + VH

Model Pix. Accuracy mIOU

FCN-6 0.9496 0.6772

VGG16 0.9413 0.6740

SegNet 0.9530 0.6541

Segmentation

Natural Disasters: Damage Maps

Flooding: Damage Estimation Fire: Damage Estimation

• Biomass by radar waves

• Government, Insurance, Investment

Destroyed

Highly damaged

Slight damage

Multimedia Satellite Task 2018

• Emergency Response for Flooding Events

• Focus on Road Passibility (Impact of Infrastructure) (Road Access, blocked Road) with two subtasks

• Classification of Road-Access & Passability– in Social Multimedia

– in multiple Satellite Images (Radar, Optical)

• Registration open (http://www.multimediaeval.org)

Images in Tweets (Hurricane Harvey)

Satellite Images of Houston (US) for Hurricane Harvey

CombineResultvia Geo-Location

Thanks!

NVAIL partner

all networks trained on DGX-1

Deep Learning Competence Center & Smart Data and Knowledge … · 2018. 4. 19. ·...

Documents

Visual SLAM using DenseNet Features - SKKU...of the input image, where 7 7 is the size of feature-maps and 1024 is the number of channels. The reason for choosing the ReLu layer is

CNN-PS: CNN-based Photometric Stereo for General Non ...openaccess.thecvf.com/...CNN-PS_CNN-based_Photometric_ECCV_2… · CNN-PS: CNN-based PhotometricStereo for General Non-Convex

P-CNN: Pose-based CNN Features for Action Recognition · MPII Cooking Improved ranking worsen ranking P-CNN outperforms state-of-the-art DT-FV with manually annotated human pose (GT)

ICTCS'2017 Accepted Papers Abstracts · Data mining. Statistical Selection of CNN-Based Audiovisual Features for Instantaneous ... suitable affective features to develop a regression

Performance Comparison of Hybrid CNN-SVM and CNN-XGBoost

Joint Object Detection and Viewpoint Estimation using CNN ... · Joint Object Detection and Viewpoint Estimation using CNN features Carlos Guindel · ICVES 2017 Logistic loss for

Deep Convolutional Neural Fields for Depth Estimation … · volutional neural networks (CNN). CNN features have been setting new records for a wide variety of vision applica-tions

COVID-DenseNet: A deep learning architecture to detect

ELF: Embedded Localisation of Features in pre-trained CNN · ELF: Embedded Localisation of Features in pre-trained CNN Assia Benbihi1,2, Matthieu Geist3, and Cedric Pradalier´ 1,4

Image Quality Assessment by Comparing CNN Features ...Amirshahi, Pedersen, and Yu: Image quality assessment by comparing CNN features between images Figure 1. Our new image quality

Iris Recognition With Off-the-Shelf CNN Features: A Deep ...rossarun/pubs/NguyenIrisRecognitionCNN_ACC… · K. Nguyen et al.: Iris Recognition With Off-the-Shelf CNN Features B

R-CNN By Zhang Liliang. Main idea: good features are no enough VOC07: mAP:35.1% -> 58.5%

CNN Tutorial

Faster R-CNN Features for Instance Search · 2016-05-30 · Faster R-CNN Features for Instance Search Amaia Salvador, Xavier Giro-i-Nieto, Ferran Marqu´ es´ Universitat Politecnica

Linked Dynamic Graph CNN: Learning on Point Cloud via ... · Linked Dynamic Graph CNN: Learning on Point Cloud via Linking Hierarchical Features Kuangen Zhang 1;2, Ming Hao3, Jing

2sRanking-CNN: A 2-stage ranking-CNN for …2.2.1 1st-stage Ranking-CNN Before explaining 1st-stage ranking-CNN, let’s brieﬂy explain how ranking-CNN works. Ranking-CNN was proposed

P-CNN: Pose-Based CNN Features for Action Recognitionopenaccess.thecvf.com/content_iccv_2015/papers/C... · Guilhem Cheron´ ∗† Ivan Laptev ... Removing one part at a time from

CNN | by CNN Staff

D2-Net: A Trainable CNN for Joint Description and ...openaccess.thecvf.com/content_CVPR_2019/papers/... · D2-Net: A Trainable CNN for Joint Description and Detection of Local Features

Towards Closing the Energy Gap Between HOG and CNN ...Towards Closing the Energy Gap Between HOG and CNN Features for Embedded Vision (Invited Paper) Amr Suleiman, Yu-Hsin Chen,