a quick introduction to RL toolsadvanced level (1)

You are supposed to have already completed the related first tutorial (RL tools – basic level).So we'll assume you are already familiar with such concepts as geographic imagery, georeferencing, Spatial Reference System, Worldfile, GeoTIFF and so on.

This tutorial simply explains a single category of geographic imagery, i.e. grids: and it introduces a further RL tool: rl_grid_paint, which is exactly intended to support grid handling.

But this specific topic is complex enough to require a separate tutorial by its own.

Session #2: using GRID imageryETOPO1 is a commonly used global relief model of Earth's surface that integrates land topography and ocean bathymetry with a spatial resolution of 1 arc minute (about 2 km).As many other datasets produced by U.S. Federal Agencies its use is absolutely free and costless.http://www.ngdc.noaa.gov/mgg/global/global.html

GTOPO30 is another commonly used global digital elevation model (DEM) with a horizontal grid spacing of 30 arc seconds. This too is absolutely free and costless.http://eros.usgs.gov/#/Find_Data/Products_and_Data_Available/gtopo30_info

Both them are excellent examples of grids. May well be that you are completely ignoring anything about grids, so we'll proceed very slowly, in a step by step fashion.

Here is a preview of ETOPO1_Ice_g.tif as shown by GIMP: this one looks like a really odd kind of image ..

Exercise #1:

>rl_info -i ETOPO1_Ice_g.tif

Inspecting file: ETOPO1_Ice_g.tif Image type: GeoTIFF Width: 21601, Height: 10801 ColorSpace: GRID BitsPerSample: 16 SamplesPerPixel: 1 SampleFormat: INT Tiled: 64 x 64 Compression: NONE Georeferencing SRID: 4326 SrsName: WGS 84 Proj4Text: '+proj=latlong +ellps=WGS84 +to_meter=1.0000000000' UpperLeft: (-180.008333333335, 90.008333369334991) UpperRight: (180.00833340533501, 90.008333369334991) LowerLeft: (180.00833340533501, 90.008333369334991) LowerRight: (180.00833340533501, -90.008333333335003) Center: (0.000000036000002979, 0.000000017999994384) PixelSize: 0.01666666667

>

We can now check the image's attributes using the rl_info tool:• this one is absolutely normal GeoTIFF with complete georeferencing: nothing strange in

this, it's more or less the same of TrueMarble imagery.• the odd thing is related to pixel format: each pixel is represented as a signed 16 bit integer.• that's very odd, because you where expecting a pixel to be an unsigned 8 bit integers, as in

commonly used RGB, Palette256 or Grayscale color spaces.

All right, this one surely is an image, but of a very peculiar kind: actually this one is a grid.

Please note: you can now fully understand why GIMP (and by many others generic image-oriented tools) will show a really odd-looking preview for this peculiar image:

• GIMP (wrongly) assumes this one to be a Grayscale image.• GIMP is unable to handle signed values, so it will represent only negative values, ignoring at

all any positive value.• accordingly to this, you'll see the Ocean's floor (negative values) to be represented as

Grayscale, and the Continent's surface as deep black (i.e. zero).

In ordinary images a given pixel value corresponds to a color intensity.But this assumption isn't at all true for grids. A grid's pixel value corresponds to some arbitrary physical measure, not to a color intensity.

Both ETOPO1 and GTOPO30 actually are DEMs (Digital Elevation Models).Accordingly to this, each pixel value corresponds to elevation (measured in meters) above the sea level (which is assumed to be exactly zero).

• Continental areas are usually represented by positive values (with the exception represented by depressions, such as Death Sea, Afar and so on ...)

• The Oceanic floor is then represented by negative values.

>rl_grid_paint -i ETOPO1_Ice_g.tif -o etopo.tif -cr etopo2 -nd -99999< IN [GeoTIFF]: ETOPO1_Ice_g.tif Width:21601 Height:10801 BitsPerSample: 16 SamplesPerPixel:1 SampleFormat: INT Layout by tiles: 64 x 64 ColorSpace: GRID Compression: NONE Georeferenced: YES

> OUT [GeoTIFF]: etopo.tif Width:21601 Height:10801 BitsPerSample: 8 SamplesPerPixel: 3 SampleFormat: UINT Layout by tiles: 256 x 256 ColorSpace: TRUECOLOR Compression: NONE Georeferenced: YES

Ok, image successfully copied into 'etopo.tif'Ok, Worldfile successfully generated: 'etopo.tfw'

>

You can use the rl_grid_paint tool to get an ordinary visible image corresponding to some grid.• you already know the -i filename and -o filename arguments, because they exactly have the

same identical meaning as in any other RL tool.• the -cr filename argument is intended to select a color rules definition file. (we'll see better

what a color rules definition files is in the next section … wait, please).• the -nd value argument is intended to set the No Data value. i.e. the value corresponding to

absence of any measured value (this more or less corresponds to the NULL value concept as defined for DBMS).

Please note: you already know that GeoTIFF is a wondeful image format supporting geographic imagery. Anyway GeoTIFF has a conspicuous flaw related to grids: there is no way to include a No Data value.So you are forced to manually enter this value each time you'll process a GeoTIFF grid.Usually setting a reasonably big negative number is the most obvious thing to be done.

And here is a preview of ETOPO1 when properly rendered.

>rl_grid_paint -i ETOPO1_Ice_g.tif -o etopo.tif -cr srtm -nd -99999>rl_grid_paint -i ETOPO1_Ice_g.tif -o etopo.tif -cr terrain -nd -99999>rl_grid_paint -i ETOPO1_Ice_g.tif -o etopo.tif -cr ryg -nd -99999

Nothing prevents you using any other different color rules definition file at your will …

Here is an example of how it looks when using the srtm color rules

Yet another fancy example using the ryg color rules.

I suppose it will be now quite easy for you to understand how grid rendering does actually works:• color rules simply are mathematical rules establishing some correspondence criteria

between values and colors.• obviously there is nothing forcing you to use a determinate rule: you are absolutely free to

use the one that best matches your very specific requirement.• you'll give anyway a false colors (aka pseudo-colors) visible image.• grids are really interesting, because they are highly flexible, and they allow you to

represent in an really effective and easy-to-understand fashion a wide range of physical phenomena.

Color rules definition files simply are plain text files.In an attempt to avoid reinventing the whell yet another time, rl_grid_paint simply supports the same rules as defined for the well known GRASS GIS package.Here you can get a full reference: http://grass.osgeo.org/gdp/html_grass64/r.colors.html

-11000 0 0 0-5000 0 0 100-1000 50 50 200-1 150 150 2550 0 150 0270 90 165 90300 90 175 90500 50 180 50500 70 170 701000 70 145 751000 70 155 752000 150 156 1003000 220 220 2204000 245 245 2458850 255 255 255nv white

this one is the contenent of etopo2 color rules

0% red50% yellow100% green

and this is the ryg color rules definition

This first exercise is completed.Now you have acquired a basic comprehension about grids and color rules used to perform visual rendering for grid data.And you have learnt too that GeoTIFFs can store grids, and not only trivial images.

Exercise #2:

>rl_grid_paint -i etopo1_bed_c_i2.bin -o etopo.tif -cr etopo2 -srid 4326>rl_grid_paint -i etopo1_bed_c_f4.flt -o etopo.tif -cr etopo2 -srid 4326>rl_grid_paint -i etopo1_ice_g_i2.bin -o etopo.tif -cr etopo2 -srid 4326>rl_grid_paint -i etopo1_ice_g_f4.flt -o etopo.tif -cr etopo2 -srid 4326

Go back to the NOAA ETOPO download site: as you can notice the ETOPO1 dataset isn't simply offered in the GeoTIFF format: several binary formats are supported as well.And rl_grid_paint has the capability to process such binary grid format as well.

Please note: when using GeoTIFF grids we where forced to explicitly define a No Data value, because this kind of information isn't supported by the GeoTIFF format.

When using binary grids there is no need at all to define a No Data value, because this format supports this information: but on the other side you are required to explicitly define an SRS reference, because this information isn't supported by the format.

Moral lesson to learn: nobody's perfect.

>rl_info -i etopo1_ice_g_f4.flt

Inspecting file: etopo1_ice_g_f4.flt Image type: FLT-GRID Width: 21601, Height: 10801 ColorSpace: GRID BitsPerSample: 32 SamplesPerPixel: 1 SampleFormat: FLOAT Compression: NONE NoDataValue: -99999 MinCellValue: -10898 MaxCellValue: 8271 Georeferencing SRID: -1 UpperLeft: (-180.008333333335, 90.025000036004997) UpperRight: (180.00833340533501, 90.025000036004997) LowerLeft: (180.00833340533501, 90.025000036004997) LowerRight: (180.00833340533501, -89.991666666664997) Center: (0.000000036000002979, 0.016666684669999654) PixelSize: 0.01666666667

>

You can use rl_info to check image attributes for binary grids too:• only a partial georeferencing is supported: the SRID is undefined • not only No Data value is supported: an explicit range extent definition is supported as weel

(Min Value / Max Value)• binary grids (and GeoTIFF too) con store floating point values (both in single and double

precision). But this is not at all surprising, if you consider that they are intended to support an accurate representation of physical measured values.

Binary grids does actually consist in a couple of file:• a separate header file defines the grid attributes• and a distinct data file stores the binary data, strictly packed into consecutive columns for

each row.

Quite obviously the header file must define the matrix dimensions, the sample type, and the adopted endianness. The following one is an example of header file content:

ncols 21601nrows 10801xllcenter -180.00yllcenter -90.00cellsize 0.01666666667NODATA_value -99999byteorder LSBFIRSTNUMBERTYPE 4_BYTE_FLOATMIN_VALUE -10898.0MAX_VALUE 8271.0

Oddly enough, the header file format used for floating point data is very slightly different from the one used for integer data.

NCOLS 21601NROWS 10801XLLCENTER -180.000000YLLCENTER -90.000000CELLSIZE 0.01666666667NODATA_VALUE -32768BYTEORDER LSBFIRSTNUMBERTYPE 2_BYTE_INTEGERZUNITS METERSMIN_VALUE -10898MAX_VALUE 8271

Correspondence between a data file and the associated header file is established by suffix:

Type Data suffix Header suffix Example

INTEGER .bin .hdr int_grid.binint_grid.hdr

FLOAT .flt .hdr flt_grid.fltflt_grid.hdr

This second exercise is completed.Now you are aware that grid data can be stored using the binary grid format.

Exercise #3:

It's now time to begin exploring the GTOPO30 dataset: so you must go to the USGS GTOPO30 site and download some sample.

>rl_info -i W020N40.DEM

Inspecting file: W020N40.DEM Image type: DEM-GRID Width: 4800, Height: 6000 ColorSpace: GRID BitsPerSample: 16 SamplesPerPixel: 1 SampleFormat: INT Compression: NONE NoDataValue: -9999 Georeferencing SRID: -1 UpperLeft: (-19.99583333333333, 39.99583333333333) UpperRight: (20.004166666650676, 39.99583333333333) LowerLeft: (20.004166666650676, 39.99583333333333) LowerRight: (20.004166666650676, -10.004166666646675) Center: (0.004166666658672824, 14.995833333343327) PixelSize: 0.008333333333330001

>rl_grid_paint -i W020N40.DEM -o image.tif -cr srtm -srid 4326< IN [DEM-GRID]: W020N40.DEM Width:4800 Height:6000 BitsPerSample: 16 SamplesPerPixel:1 SampleFormat: INT ColorSpace: GRID Compression: NONE NoDataValue: -9999 Georeferenced: YES

> OUT [GeoTIFF]: image.tif Width:4800 Height:6000 BitsPerSample: 8 SamplesPerPixel: 3 SampleFormat: UINT Layout by strips: 1 RowsPerStrip ColorSpace: TRUECOLOR Compression: NONE Georeferenced: YES

Ok, image successfully copied into 'image.tif'Ok, Worldfile successfully generated: 'image.tfw'

>

All right, it doesn't looks too much different from any previous example. After all, a grid simply is a grid. Anyway, as you can easily notice, this time you encounter yet another different grid format, i.e. a DEM grid.

Some kind of perverse and insane fantasy seems to be the font of inspiration for sw developers working on grids: lots and lots of slightly different formats … that in conclusion are more or less exactly the same one, for any practical purpose.

Even showing a visual preview for GTOPO30 has little new to be discovered: this too is an elevation based dataset (although produced using a different approach), so you must not be surprised at all when discovering that both look very alike.

• ETOPO1 has a spatial resolution of 1 arc minute: this allows to implement an huge monolithic grid (21,600 x 10,800 pixel)

• GTOPO30 on the other side has a better spatial resolution of 30 arc seconds: this produces an increment by four times in allocation, so a single monolithic grid isn't possible, simply due to practical size considerations.

• consequently GTOPO30 is split in a mosaic of several smaller elementary tiles.• the Ocean looks full black on GTOPO30 simply because this dataset ignores bathymetry.

BYTEORDER MLAYOUT BILNROWS 6000NCOLS 4800NBANDS 1NBITS 16BANDROWBYTES 9600TOTALROWBYTES 9600BANDGAPBYTES 0NODATA -9999ULXMAP -19.99583333333333ULYMAP 89.99583333333334XDIM 0.00833333333333YDIM 0.00833333333333

DEM grids are based on the usual coupled files: the .DEM suffix identifies the data file, and the .HDR suffix identifies the corresponding header file.This latter has yet another different format (as shown on the above example) … but frankly all this is becoming quite boring and very few interesting.

This third example is finished.Now you are aware of exsistence of DEM+HDR grid format as well.And, may be, you are now conscious of what GTOPO30 really is.

Exercise #4:

Another freely accessible and costless source of highly appreciated elevation data is represented by the SRTM dataset. There are several noticeable differences distinguishing SRTM on one side, and ETOPO1, GTOPO30 on the other side:

• ETOPO1 and GTOPO30 are the result of a very careful collection of historical (conventional) data. The spatial resolution is limited to about 1 km (GTOPO30) or 2 km (ETOPO1).

• SRTM is directly produced by processing radar reflectivity data gathered from the space by the orbiting Space Shuttle.

• So SRTM is actually based on objective and very accurate physical measures.• And its intrinsic spatial resolution is by far higher: about 90m worldwide (3 arc seconds),

30m (1 arc second) for the U.S. coverage.• Anyway the SRTM dataset is plagued by a generalized issue: there are lots and lots of black

holes, i.e. areas lacking any measured elevation simply because radar reflectivity was anomalous, thus forbidding any appreciable echo to be measured. You can easily find some small black hole quite everywhere, but oceans, high mountain chains and sand deserts really represents an awful nightmare.

This one is an example showing a perfect SRTM tile (no black holes here)

This other is an example showing an SRTM tile plagued by lots and lots of black holes

You can freely download SRTM tiles from: http://dds.cr.usgs.gov/srtm/

>rl_info -i N42E012.hgt

Inspecting file: N42E012.hgt Image type: HGT-GRID Width: 1201, Height: 1201 ColorSpace: GRID BitsPerSample: 16 SamplesPerPixel: 1 SampleFormat: INT Compression: NONE NoDataValue: -32768 Georeferencing SRID: 4326 SrsName: WGS 84 Proj4Text: '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs' UpperLeft: (11.999583333333334, 43.000416666666666) UpperRight: (13.000416666666666, 43.000416666666666) LowerLeft: (13.000416666666666, 43.000416666666666) LowerRight: (13.000416666666666, 41.999583333333334) Center: (12.5, 42.5) PixelSize: 0.000833333333333333

>rl_grid_paint -i N42E012.hgt -o image.tif -cr srtm< IN [HGT-GRID]: N42E012.hgt Width:1201 Height:1201 BitsPerSample: 16 SamplesPerPixel:1 SampleFormat: INT ColorSpace: GRID Compression: NONE NoDataValue: -32768 Georeferenced: YES

> OUT [GeoTIFF]: image.tif Width:1201 Height:1201 BitsPerSample: 8 SamplesPerPixel: 3 SampleFormat: UINT Layout by tiles: 256 x 256 ColorSpace: TRUECOLOR Compression: NONE Georeferenced: YES

Ok, image successfully copied into 'image.tif'Ok, Worldfile successfully generated: 'image.tfw'

>

Please note: yet another different grid format (.hgt). Are you minimally surprised ?

The .hgt format adopted by SRTM introduces a different implementation criterium:• there is no header file at all• the .hgt file simply represents the data file• georeferencing attributes are directly encoded within the tile name• any other relevant attribute is well known simply by convention.• each single tile covers exactly a square region corresponding to an extension of 1 degree x

1 degree.• reduced resolution tiles (worldwide) always have 1,201 x 1,201 pixels• high resolution tiles (U.S. only) always have 3,601 x 3,601 pixels

This fourth exercise is completed.Now we are conscious of SRTM data availability: and this one really is a big useful resource.And you are now aware of existence of the .hgt format.

Exercise #5:

I suppose your patience and attention will reach its limit very soon … but we haven't yet completed to examine any possible grid format. There is another SRTM closely related dataset we must absolutely explore.CGIAR distributes an enhanced SRTM dataset, containing no black holes at all: all them has been carefully filled applying sophisticated interpolation algorithms.http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp

Very important notice: the CGIAR dataset isn't compeletely free.Yes, it's true, you are allowed to download at no cost the complete dataset.But their disclaimer is absolutely clearly in fordidding you to:

• distribute CGIAR data to third parties (i.e. any other people execpt yourself)• use CGIAR data for any commercial purpose• use CGIAR data for web sites• shortly said: use CGIAR data for anything different from your personal study.

So this one cannot be considered to be a free datasource. You are warned.

CGIAR distributes GeoTIFFs, but they are of no interest for you, because you already know anything about GeoTIFF grids. But CGIAR supports another different format as well: ASCII grid: and this one is exactly the one you'll use in this exercise.

ncols 6001nrows 6001xllcorner 9.9995836238757yllcorner 39.999583575447cellsize 0.00083333333333333NODATA_value -999937 39 40 39 41 40 44 42 40 37 40 41 38 36 34 ......38 39 36 37 40 38 38 38 39 39 39 39 37 39 39 ......38 39 38 39 38 36 36 37 40 40 40 39 41 39 39 ......39 39 39 40 39 40 40 41 40 38 40 41 41 39 38 ......40 40 40 40 38 40 41 39 37 37 39 39 39 38 39 ......40 42 42 39 38 40 39 38 38 41 41 40 39 38 38 ..........

An ASCII grid precisely is what it pretends to be: i.e. a plain text file (usually very hugely sized), completely ASCII encoded.

• the first 6 lines act into the header role• any other subsequent line corresponds to a grid scanline• pixel values are expressed as plain text as well• lines are marked by terminators (LF, CR+LF) as usual• pixels are delimited by space

ASCII grids are intended to ensure full portability across different systems in a cross-platform fashion: but actually, they really are intrinsically very inefficient, and their processing performances are by far worse than using the corresponding binary grid.

>rl_info -i srtm_39_04.asc

Inspecting file: srtm_39_04.asc Image type: ASCII-GRID Width: 6001, Height: 6001 ColorSpace: GRID BitsPerSample: 32 SamplesPerPixel: 1 SampleFormat: FLOAT Compression: NONE NoDataValue: -9999 Georeferencing SRID: -1 UpperLeft: (9.9995836238757008, 45.000416908780309) UpperRight: (15.000416957209014, 45.000416908780309) LowerLeft: (15.000416957209014, 45.000416908780309) LowerRight: (15.000416957209014, 39.999583575446998) Center: (12.500000290542356, 42.500000242113657) PixelSize: 0.00083333333333333

>rl_grid_paint -i srtm_39_04.asc -o image.tif -cr srtm -srid 4326< IN [ASCII-GRID]: srtm_39_04.asc Width:6001 Height:6001 BitsPerSample: 32 SamplesPerPixel:1 SampleFormat: FLOAT ColorSpace: GRID Compression: UNKNOWN NoDataValue: -9999 Georeferenced: YES

> OUT [GeoTIFF]: image.tif Width:6001 Height:6001 BitsPerSample: 8 SamplesPerPixel: 3 SampleFormat: UINT Layout by tiles: 256 x 256 ColorSpace: TRUECOLOR Compression: NONE Georeferenced: YES

Ok, image successfully copied into 'image.tif'Ok, Worldfile successfully generated: 'image.tfw'

>

From the RL tools own perspective there is nothing strange in ASCII grids: you'll simply notice that processing an ASCII grid is a really slow and sluggish process, due to the intrinsic inefficiency of this format, but anything goes as you are expecting.

Now you are become familiar with lots of different grid format, for any flavor and taste.

None of them is perfect, more or less they are exactly the same.• GeoTIFF is interesting because it's more standard• Binary grids are interesting because they are most efficient to be processed• ASCII grids are interesting because they offers an excellent interoperability, but at the

expense of a really poor performance.

And this explains how and why all them still exists: simply because there is not a clear winner, and deficiencies of the one are compensated by other deficiencies of the others.

Exercise #6:

Don't be worried: this one simply is a very quick addendum.

>rl_copy -i srtm_39_04.asc -o image.tif -srid 4326< IN [ASCII-GRID]: srtm_39_04.asc Width:6001 Height:6001 BitsPerSample: 32 SamplesPerPixel:1 SampleFormat: FLOAT ColorSpace: GRID Compression: UNKNOWN NoDataValue: -9999 Georeferenced: YES

> OUT [GeoTIFF]: image.tif Width:6001 Height:6001 BitsPerSample: 32 SamplesPerPixel: 1 SampleFormat: FLOAT Layout by tiles: 256 x 256 ColorSpace: GRID Compression: NONE Georeferenced: YES

Ok, image successfully copied into 'image.tif'Ok, Worldfile successfully generated: 'image.tfw'

>rl_copy -i srtm_39_04.asc -o image.flt< IN [ASCII-GRID]: srtm_39_04.asc Width:6001 Height:6001 BitsPerSample: 32 SamplesPerPixel:1 SampleFormat: FLOAT ColorSpace: GRID Compression: UNKNOWN NoDataValue: -9999 Georeferenced: YES

> OUT [FLT+HDR GRID]: image.flt Width:6001 Height:6001 BitsPerSample: 32 SamplesPerPixel: 1 SampleFormat: FLOAT ColorSpace: GRID Compression: NONE NoDataValue: -9999 Georeferenced: YES

Ok, HeaderFile successfully generated: 'image.hdr'Ok, image successfully copied into 'image.flt'

>

I suppose this must be absolutely clear for you: anyway an explicit statement sometimes may help.There is absolutely nothing magic about grids: you can apply any (reasonable) format conversion for grids too using rl_copy as usual.

>rl_subset -i srtm_39_04.asc -o elba.flt -ulx 10.0 -uly 42.9 -lrx 10.5 -lry 42.7< IN [ASCII-GRID]: srtm_39_04.asc Width:6001 Height:6001 BitsPerSample: 32 SamplesPerPixel:1 SampleFormat: FLOAT ColorSpace: GRID Compression: UNKNOWN NoDataValue: -9999 Georeferenced: YES

> OUT [FLT+HDR GRID]: elba.flt Width:600 Height:239 BitsPerSample: 32 SamplesPerPixel: 1 SampleFormat: FLOAT ColorSpace: GRID Compression: NONE NoDataValue: -9999 Georeferenced: YES

Ok, HeaderFile successfully generated: 'elba.hdr'Ok, image successfully copied into 'elba.flt'

>rl_grid_paint -i elba.flt -o image.tif -cr srtm -srid 4326 -rs 1 -bk 0x80c0d0< IN [FLT-GRID]: elba.flt Width:600 Height:239 BitsPerSample: 32 SamplesPerPixel:1 SampleFormat: FLOAT ColorSpace: GRID Compression: NONE NoDataValue: -9999 MinCellValue: -9999 MaxCellValue: 989 Georeferenced: YES

> OUT [GeoTIFF]: image.tif Width:600 Height:239 BitsPerSample: 8 SamplesPerPixel: 3 SampleFormat: UINT Layout by strips: 1 RowsPerStrip ColorSpace: TRUECOLOR Compression: NONE Georeferenced: YES

Ok, image successfully copied into 'image.tif'Ok, Worldfile successfully generated: 'image.tfw'

>

And you can use rl_subset and then rl_grid_paint in order to get a sub-image to be extracted and rendered from a grid.

here is an Elba island sub-image rendered preview

if you'll find that a blue sea looks better, you simply have to set an appropriate background color using the rl_grid_paint -bk color argument

And that's really all you need absolutely to know about grids.