Introduction to Planetary Updates and Tips for using GDAL€¦ · 6/23/2020 · • QGIS / Saga GIS/ Mirone • Esri’s ArcMap/ArcGIS Pro • MapServer (WMS server) • NASA Ames

Introduction to Planetary Updates and Tips for using GDAL

OPvCON 1, June 2020

Trent Hare

[email protected]

Introduction to Planetary Updates and Tips for using GDAL

Time: Jun 23, 2020 10:00 AM Eastern Time (US and Canada)

Join Zoom Meetinghttps://us02web.zoom.us/j/83968101170?pwd=N3M0TnJRUThtRGxVeFdUOEFpcStIUT09

Meeting ID: 839 6810 1170Password: 571305One tap mobile+13126266799,,83968101170#,,,,0#,,571305# US (Chicago)+16465588656,,83968101170#,,,,0#,,571305# US (New York)

Dial by your location+1 312 626 6799 US (Chicago)+1 646 558 8656 US (New York)+1 301 715 8592 US (Germantown)+1 346 248 7799 US (Houston)+1 669 900 9128 US (San Jose)+1 253 215 8782 US (Tacoma)

Meeting ID: 839 6810 1170Password: 571305Find your local number: https://us02web.zoom.us/u/kcvn5G3K6j

June 2020 OPvCON 1 2

https://us02web.zoom.us/j/83968101170?pwd=N3M0TnJRUThtRGxVeFdUOEFpcStIUT09

https://us02web.zoom.us/u/kcvn5G3K6j

Workshop materials for demos


https://1drv.ms/u/s!AvjktWYZfSNqgs4c5x6ZhRlOjBN0Lw?e=5WXTse (~200 Mb)

See slide 18 for installing recommended environment

https://1drv.ms/u/s!AvjktWYZfSNqgs4c5x6ZhRlOjBN0Lw?e=5WXTse

Resources• GDAL Conversion Tips (planetary) - https://astrodiscuss.usgs.gov/t/gdal-conversion-tips/101

• GDAL/OGR Quickstart - https://live.osgeo.org/en/quickstart/gdal_quickstart.html

• GDAL Cheat Sheet - https://github.com/dwtkns/gdal-cheat-sheet

• GDAL Tutorials - https://gdal.org/tutorials/

• A Gentle Introduction to GDAL – Part (1, 2, 3, 4) , from https://medium.com/@robsimmon

• Youtube - https://www.youtube.com/watch?v=N_dmiQI1s24

• Workshop: Raster and vector processing with GDAL -https://download.osgeo.org/gdal/workshop/foss4ge2015/workshop_gdal.html

• RasterIO “raw-STIER-ee-o” - https://rasterio.readthedocs.io/

• Using GDAL and/or NASA ASP to create a colorize hillshade -https://planetarygis.blogspot.com/2017/06/using-gdal-andor-nasa-asp-to-create.html

• Map Projections used by the USGS - https://pubs.usgs.gov/bul/1532/report.pdf

• Spatial Reference (org) - https://spatialreference.org/ref/iau2000/

• Projection Wizard - http://projectionwizard.org/


https://astrodiscuss.usgs.gov/t/gdal-conversion-tips/101

https://live.osgeo.org/en/quickstart/gdal_quickstart.html

https://github.com/dwtkns/gdal-cheat-sheet

https://gdal.org/tutorials/

https://medium.com/planet-stories/a-gentle-introduction-to-gdal-part-1-a3253eb96082

https://medium.com/planet-stories/a-gentle-introduction-to-gdal-part-2-map-projections-gdalwarp-e05173bd710a

https://medium.com/planet-stories/a-gentle-introduction-to-gdal-part-3-geodesy-local-map-projections-794c6ff675ca

https://medium.com/planet-stories/a-gentle-introduction-to-gdal-part-4-working-with-satellite-data-d3835b5e2971

https://medium.com/@robsimmon

https://www.youtube.com/watch?v=N_dmiQI1s24

https://download.osgeo.org/gdal/workshop/foss4ge2015/workshop_gdal.html

https://rasterio.readthedocs.io/

https://planetarygis.blogspot.com/2017/06/using-gdal-andor-nasa-asp-to-create.html

https://pubs.usgs.gov/bul/1532/report.pdf

https://spatialreference.org/ref/iau2000/

http://projectionwizard.org/

AgendaMinutes Topic

10 Introduction to workshop

10 Introduction to GDAL

10 Installing GDAL – quick demo

• Conda (and QGIS)

15 What’s new

• ISIS3 https://gdal.org/drivers/raster/isis3.html

• VICAR https://gdal.org/drivers/raster/vicar.html

• PDS4 (GeoCSV) https://gdal.org/drivers/raster/pds4.html

• Plans for PROJ7 https://github.com/PlanetMap/csvForWKT

20 Format conversions – demo / optional follow-along

• Delayed processing (VRTs)

20 Map projections (GDAL’s Python API) - demo / optional follow-along

• Resources and tips

• Matching projections

20 Rasterio - demo / optional follow-along

• Rasterio Python API and plotting in Jupyter


https://gdal.org/drivers/raster/isis3.html

https://gdal.org/drivers/raster/vicar.html

https://gdal.org/drivers/raster/pds4.html

https://github.com/PlanetMap/csvForWKT

• Who am I

• At Astrogeology, USGS for 30 years

• Technical support for geologic “mapping”, science, PDS

• Interoperability evangelist, works with:• Open Geospatial Consortium

• Mapping and Planetary Spatial Infrastructure Team (MAPSIT)

• Community Sensor Model Working Group

• Mars Cartography Planning Working Group

• Current projects:• MRCTR GIS Lab (geospatial formats, web services, tools, training)

• Cartography and Imaging PDS Node

• Support Mars2020 landing site selection

• Socet GXP for topography creation (CSM)

OPvCON 1 6June 2020

What does interoperability mean…

• Sharing data (formats, services, metadata)

• Sharing ideas and methods (algorithms)

• Across applications but also “science” domains.

• Examples (start off easy):

• Text files (ASCII)

• Web sites (e.g. http)

• Graphical images (jpeg, png, gif)

OPvCON 1 7June 2020

What does interoperability mean…

• Examples (quickly getting tougher):• Software code (C++, Python, JAVA…)

• Raster Data (PDS, GeoTIff, VICAR, FITS, JPEG, …)

• Vector (CSV, Shapefile, GML, WKT, GML, GeoJson)

• Map Projections (Cylindrical, Mercator, Polar)

• Metadata (PDS, FGDC, NASA DIF, ISO)

• Symbologies (black solid line means, dashed means….)

• Methods (Slope -> Horn, 1, 2, 5 baseline, …)

• Applications (GIS, Remote Sensing, Graphical, Statistical)

goal = sharing

OPvCON 1 8June 2020

OPvCON 1 9June 2020

Support for GDAL

2006 - added PDS3 / ISIS2 read support to GDAL

2007 – ISIS 3/4 read support added

2014 – VICAR read support added

2017 – ”Geo”FITS to be added (github also fits2vrt script)

2017 – ISIS 3/4 writer

2017, 2019 – PDS4 (reader/writer) – raster/vector (table)

2019 – VICAR re-write (reader/writer)

2020 – PROJ7 (release tbd)

https://epn-vespa.github.io/geofits/


Convert to GIS/RS formats like GeoTiff, GeoJp2, ENVI, ASCII

Specifically to support:

• Derived map projected data (not raw “EDRs” PDS)

• Planetary datums and map projections

• Large files (cloud-access)

• Ability to use processing / Python tools

But as GDAL grew in use so did direct support for planetary formats

• QGIS / Saga GIS/ Mirone

• Esri’s ArcMap/ArcGIS Pro

• MapServer (WMS server)

• NASA Ames Stereo-pipeline

• …

• More recent

• Generic Mapping Tools

• GRASS

• TuiView – simple viewer

Main Goals for GDAL

AgendaMinutes Topic





15 What’s new

















What is GDAL

GDAL, in short, may not be a household name, but it’s everywhere and helps us to map everything. Or, as the folks at Google Earth put

it, “GDAL makes the world go round.”

-Matt Asay, AWS

https://thenewstack.io/gdal-the-open-source-technology-behind-google-maps/


More info

https://twitter.com/googleearth/status/1258140107552391168

https://thenewstack.io/gdal-the-open-source-technology-behind-google-maps/

https://download.osgeo.org/gdal/presentations/GDAL%202.2%20(FOSS4G-E%202017).pdf

What is GDAL

Geospatial Data Abstraction Library

• GDAL is a “translator library for raster geospatial data formats”

• Open source (and community driven)

• Used in many “geo” applications: QGIS, UDIG, SAGA, ARCMAP, GMT, MapServer (WMS), Google Earth, AMES Stereo-pipeline, SOCET GXP, …

• Handles many image formats for read and slightly less for writing: PDS3, ISIS2/3, VICAR, FITS (via CFITSIO), ENVI, GeoTiff, Jpeg2000, PNG, cloud-based,

NetCDF, … - over 150 formats


More info


What is GDAL

Geospatial Data Abstraction Library

• Started in 1998 by Frank Warmerdam

• A project of OSGeo since 2006 (~v1.3.2)

• MIT/X Open Source license (permissive)

• > 1M lines of code for library and utilities

• > 150K lines of tests in Python


From: Even Rouault (link)


What Domain?

• Planetary Research (PDS) supports dozens of domains• Atmospheres (GCMs)

• Planetary Interiors

• Astronomy

• Astrobiology

• etc., etc., etc.


GDAL is Geospatial – “tied to a planetary surface”

AgendaMinutes Topic





15 What’s new

















Installing GDAL - https://gdal.org/download.html

• All OSs• Anaconda - https://www.anaconda.com/products/individual

• MiniConda - free minimal installer for conda,https://docs.conda.io/en/latest/miniconda.html

• Docker (Ubuntu, Alpine) - https://github.com/OSGeo/gdal/tree/master/gdal/docker

• OS Specific• Mac OSX - Kyng Chaos, https://www.kyngchaos.com/software/frameworks/

• Windows - https://www.osgeo.org/projects/osgeo4w/

• Linux – (Debian unstable)

• Build yourself - https://github.com/OSGeo/GDAL


https://gdal.org/download.html

https://www.anaconda.com/products/individual

https://docs.conda.io/en/latest/miniconda.html

https://github.com/OSGeo/gdal/tree/master/gdal/docker

https://www.kyngchaos.com/software/frameworks/

https://www.osgeo.org/projects/osgeo4w/

https://github.com/OSGeo/GDAL

Anaconda(1) install Anaconda

(2) Start a conda terminal and install GDAL:

conda create -n gdal3 # creates a stand-alone environment

conda activate gdal3 # activates the environment

# for this workshop use this line:

conda install -c conda-forge gdal=3 jupyter basemap mathplotlib xerces-c rasterio qgis

# to get latest version (PDS4/VICAR updates):

conda install -c conda-forge gdal=3.1

Note: every time you start up a new terminal remember to type:

conda activate gdal3


Anaconda

(3) To test new environment

gdalinfo –version

python --version

# to test a GDAL python routine

gdal_edit.py


Docker GDAL

docker pull osgeo/alpine-small-3.1.0

docker run --rm -v $(pwd):/data osgeo/gdal:alpine-small-latest gdalinfo /data/my.tif

docker run --rm -v $(pwd):/data osgeo/gdal:alpine-small-latest gdalinfo --version

Important:

I am mapping my parent working directory $(pwd)to a linux virtual (made up) “/data” directory. Said again, that means the docker image thinks my $(pwd)is currently /data/. On Windows use ${PWD}

--convert a FITS to PDS4:

docker run --rm -v $(pwd):/data osgeo/gdal:alpine-small-latest gdal_translate -of PDS4 /data/ldem_16.fit /data/ldem_16.xml


QGIS

QGIS can help learn GDAL command-line options from the interface.

gdal_translate

-ot UInt16 -of GTiff

-co COMPRESS=PACKBITS

JEZ_ctx_0_soc_002_dtm_Ortho_flight.tif

output.tif


AgendaMinutes Topic




• Conda (Docker and QGIS)

15 What’s new• ISIS3 https://gdal.org/drivers/raster/isis3.html
















ISIS 3/4

Overview:

• Incorporate ISIS 3/4 write capabilities into existing GDAL C/C++.

• ISIS keywords to be accessed programmatically (C++ or Python).

• 31+ unit tests included in code delivery• https://github.com/OSGeo/gdal/blob/master/autotest/gdrivers/isis.py

• Allow detached ISIS label that is pointed into a NEW “raw” GeoTiff.• Also allows for TIFF compression (but not yet supported in ISIS)

• Virtual file systems (e.g. network, compressed – tar, zip)

Virtual file systems: many GDAL raster and vector drivers use a GDAL-specific abstraction to access files. This makes it possible to access less-standard types of files, such as in-memory files, compressed files (.zip, .gz, .tar, .tar.gz archives), encrypted files, files stored on network (either publicly accessible, or in private buckets of commercial cloud storage services). /vsis3/ (AWS S3), /vsigs/ (Google Cloud Storage), /vsiaz/ (Microsoft Azure), /vsioss/ (Alibaba Cloud), /vsihdfs/ (Hadoop File System), /vsiswift/ (OpenStack Swift), more.


https://github.com/OSGeo/gdal/blob/master/autotest/gdrivers/isis.py

VICAR

Overview:

• Large refactor of existing version (by Sebastian Walter, based on PDS/ISIS2)

• Incorporate VICAR write capabilities into existing GDAL C/C++.

• VICAR keywords to be accessed programmatically (C++ or Python).

• 17+ unit tests included in code delivery

• https://github.com/OSGeo/gdal/blob/master/autotest/gdrivers/vicar.py

• Supports most all VICAR bit types back to VAX

• Supports VICAR compression (Basic, Basic2)


Driver doesn’t currently support everything JPL needs, but I am hopefully updates can be provided.


https://github.com/OSGeo/gdal/blob/master/autotest/gdrivers/vicar.py

PDS4

Overview:

• Incorporate PDS4 read and write capabilities into existing GDAL C/C++.

• PDS4 keywords to be accessed programmatically (C++ or Python).

• Allow detached PDS4 label that is pointed into a NEW “raw” GeoTiff.

• Allow detached PDS4 label pointing into an EXISTING “raw” GeoTiff.• just in - GeoTIFF (raw) “almost” approved for PDS archives (last week)

• All code available and supported under the standard GDAL release.



PDS4 released for rasters (Sep 2017)

• Available in GDAL 2.3.x release

• 18+ Unit tests provided with delivery: https://github.com/OSGeo/gdal/blob/master/autotest/gdrivers/pds4.py

• Initial help page:

• http://www.gdal.org/frmt_pds4.html


https://github.com/OSGeo/gdal/blob/master/autotest/gdrivers/pds4.py

http://www.gdal.org/frmt_pds4.html

PDS4 update released for vectors (Feb. 2019)

• Available in GDAL 3.x release

• 15+ Unit tests provided with delivery: https://github.com/OSGeo/gdal/blob/master/autotest/ogr/ogr_pds4.py

• Initial help page:

• http://www.gdal.org/frmt_pds4.html


https://github.com/OSGeo/gdal/blob/master/autotest/ogr/ogr_pds4.py

http://www.gdal.org/frmt_pds4.html

PDS4 - Reign in expectations

• A full "GDAL" solution relies on a good PDS4 template. • Similar how FGDC metadata works in the real world. Metadata is never

fully automatable.

• Only the image's physical parameters (lines, samples, bit type) and map projection are automated.

• But GDAL supports template variables, so scripting can support a full EDR label using Python, PERL, etc.


PDS4 - simple scripting examples on GitHub

• Written In Python, but shows how Bash, PERL, etc. could be used

https://github.com/USGS-Astrogeology/GDAL_scripts/tree/master/PDS4gdal

observeID = getkey(from_=inputlbl, keyword='InstrumentId', grp='Archive')

theLine = '-co VAR_OBSERVING_SYSTEM_NAME={}'.format(observeID)

fileConfig.write(theLine)


https://github.com/USGS-Astrogeology/GDAL_scripts/tree/master/PDS4gdal

PDS4 Vector (GeoCSV)

• PDS4 binary/ASCII read table support.

• PDS ASCII GeoCSV write support

• Vector support1. If PDS4 table has Latitude / Longitude field (acts as point)

2. If PDS4 table has Well-Known-Text (WKT) field then supports points, lines, polygons


Next contract

• Add PDS4 table support.

• Write multiple image arrays in one file (called sub-datasets).

• Add units within “image” label section

• Vector support1. If PDS4 table has Latitude / Longitude field (acts as point)

2. If PDS4 table has Well-Known-Text (WKT) field then supports points, lines, polygons


Proposed PDS4 ASCII Table example:

PDS4 XML examples

• Very minimal PDS4 XML template (ships with GDAL binaries)• https://github.com/OSGeo/gdal/blob/master/gdal/data/pds4_template.xml

• Raster and Vector simple introduction• https://github.com/USGS-Astrogeology/GDAL_scripts/wiki


https://github.com/OSGeo/gdal/blob/master/gdal/data/pds4_template.xml

https://github.com/USGS-Astrogeology/GDAL_scripts/wiki

Wait - what…

• PDS4 supports Well-Known-Text (WKT) ?

Why WKT

• ISO standard

• Simple but allows for multiple shapes and holes

• Broad use across applications


Proj 7

Overview:

• Two new projections added• Point Perspective (aka Near-sided perspective)

• Oblique Cylindrical (used in Cassini archives, some-what unique version)

• “IAU” 2015 - code refactor - https://github.com/PlanetMap/csvForWKT

• uses new WKT v2 definitions (OGC/ISO standard).

• TO be built into GDAL release using SQLite database

• Useful for web services (WMS, WFS, WMTS, WCS, …)

• Allows for East/West Longitudes

• Allows for ocentric/ographic (almost)

• Defines the “W0” locations (e.g. Airy-0 on Mars, Hun Kal on Mercury)

• Code base for “IAU” WKT v2 definitions can support triaxial (GDAL doesn’t)



Point Perspective

Used in Earth-basedtelescopic views of theMoon (ROLO project)


AgendaMinutes Topic





15 What’s new





20 Format conversions – demo → (run: jupyter notebook 01_format_conversion.ipynb )



• Resources and tips → (run: jupyter notebook 02_projection_tips.ipynb )

• Matching projections → (run: jupyter notebook 03_match_projection.ipynb)


• Rasterio Python API and plotting in Jupyter → (run: jupyter notebook 04_rasterio.ipynb)






QGIS help make color ramps for GDAL


Please email if you have specific requirements.

[email protected], Trent Hare


mailto:[email protected]

Documents

Introduction to Planetary Updates and Tips for using GDAL€¦ · 6/23/2020 · • QGIS / Saga GIS/ Mirone • Esri’s ArcMap/ArcGIS Pro • MapServer (WMS server) • NASA Ames