WorkshopWorkshop:

Using Streaming Geospatial Data Using Streaming Geospatial Data ResourcesResources

IASSIST / IFDO 2005May 24, 2005Edinburgh, ScotlandUK

Workshop: Objectives

Objectives Introduction to some key Open Geospatial Consortium (OGC)

technologies Hands-on use of some OGC services and clients Highlight problems related to integration of these services with

data services operations & libraries Not Objectives

Learn all the relevant technologies in detail Learn how to use specific proprietary desktop GIS software or

how to “do GIS” Provide real concrete answers about how to integrate these

services with data services operations & libraries

Proprietary web map services ArcIMS, etc. -- map servers delivering mapping functionality to

Web browser clients or to desktop GIS software

Proprietary spatial database services ArcSDE, Oracle spatial, etc. – database servers delivering data to

desktop GIS software … or to map server applications

Open Geospatial Consortium-compliant services Web Map Services, Web Feature Services, etc. Focus on data and service interoperability OGC specifications also cover other services besides data delivery

Overview: What is streaming geospatial data?

Why Streaming Data?

Size of geospatial data resources Size of aerial photography, OSMM GML

Frequency of data update Seconds, hourly, daily, weekly, monthly, annual …

Flexible access to data Low (er) bandwidth access Subset, analyse, inspect data

New licensing methods BUT – DRM issues (see later)

Why Interoperability?

Integrate existing systems

Create distributed network of interoperable data stores and application services

Distributed multiagency/ multiorganizational/ multidepartmental sharing (ideally)

Application fusion, bring together multiple GIS applications using geography as the integrated framework (ideally)

Non OGC-compliant map servers are data islands Data from different servers cannot be combined

Data available in many vendor or non-proprietary formats Conversion time consuming, confusing

Data available in different coordinate systems Conversion time consuming, confusing

Service chaining Make services that add value to other services

Why Interoperability?

Web Services Architecture

ServiceRegistry

ServiceProvider

ServiceConsumer

PublishPublish FindFind

BindBind

WebWebBrowserBrowser

UDDIUDDIWSDLWSDL

WebWebServerServer

How?

OGC and Web Services

Open Geospatial Consortium was working on developing geospatial Web services before “Web Services” (SOAP/WSDL/UDDI/…) existed – focus on modular, reusable components

Because of early start OGC is now working to adapt OGC specifications to mainstream Web Services architecture (or vice versa), i.e. making OGC services build on top of standard Web Services frameworks

OGC use of HTTP GET & POST for invocations is a constraint on acceptance of OGC specs in Web Services community

See OGC Web Services, Phase 2 (OWS-2) & Phase 3 (OWS-3):

  http://www.opengis.org/initiatives/?iid=7

Exercise One

Using OGC-compliant map servers

OGC Technology Overview/Timeline

Early 1990’s: “Sticky tape and pipe cleaners” map server implementations—homegrown

Circa 1995-1996: Commercial software products become available—no interoperability

Circa 1998-2000: Commercial map server products stream data to workstation software (no interoperability) e.g. ArcIMS

Circa 2000-2001: WMS 1.x implementations, first home-grown, later commercial and open source

Circa 2002-2003: WFS 1.0 implementations

Open Geospatial Consortium Overview

The Open Geospatial Consortium (OGC) is a not-for-profit, international consortium whose many members work to make GIS an integral part of information systems of all kinds.

Operates a Specification Development Program that is similar to other Industry consortia (W3C, OMG, etc.).

Also operates an Interoperability Program (IP), a partnership-driven engineering and testing program designed to deliver proven specifications into the Specification Development Program.

OGC used to talk about “web-enabling GIS”, now they talk about “geo-enabling the web.”

Advantages of OGC Spec Adoption

OGC interface specifications are open, freely downloadable documents that can be applied and tested widely.

Commercial and non-commercial software can support these interfaces for interoperability and information exchange (and many now do).

Adopting common service interfaces lets solutions use the same software to connect to systems back-ended by different GIS vendor solutions (write once use many times).

Significant vendor participation in specification development: they get to help drive direction of specification development—and interoperability helps them to vertically integrate in industry i.e. good for vendors and good for consumers.

OGC Specifications (some!!)

Web Map Service (WMS) Specifies the behavior of services that return georeferenced

maps Web Feature Service (WFS)

Interface that supports query level access to vector data repositories

Web Coverage Service (WCS) Supports networked interchange of geospatial data as

‘coverages’ containing values or properties of geographic locations

Styled Layer Descriptor (SLD) Styling language for WMS map servers, allow for pre-defined

and user defined styles

OGC Technology Overview

Catalog Services (CAT) Support the ability to publish and search collections of metadata

for data, services, and related information objects Web Registry Services (WRS)

Defines a common mechanism to classify, register, describe, search, maintain and access information about OGC Web resources

Web Map Context (WMC) (Ver 1.1 now an adopted specification) Like a ‘spatial bookmark’, allows recreation of a map using OGC

services More (lots!)… see: http://www.opengis.org/specs/

Web Map Server Specification 1.1.1

GetCapabilitiesReturns server-level metadata, description of services and content,

acceptable request parameters GetMap

Returns map image whose geospatial and dimensional parameters are well-defined

GetFeatureInfoReturns information about particular features shown on map (optional)

Specification defines syntax for URLs to invoke map operations. XML encoding defined for service

level metadata.

Rendered Map Image

Returns info about feature

Capabilities XML Document with

Layer Information

Legend Roads Water BuildingsMap Layers

Web Map Service

GetCapabilities

GetMap

WMS Interface: Operation

Client

Adapted from Web Map Server Demonstration presentation © 2000 OGC

GetFeatureInfo

WMS Map Requests

Layers

Styles (possibly)

Bounding Box

Projection or geographic coordinate reference system

Desired output format

Output size (width, height)

Background transparency or color

Sample WMS Query

http://anyoldserver.com/WebMapService?version=1.1.1&

layers=Raster50K&styles=default&

srs=EPSG:27700&bbox=287000,692001,290000,695001&

width=400&height=400&

format=image/gif&

request=GetMap

Exercise Two

Examine a WMS Map Request

Asks the mapserver what functionalities a particular WMS map services is capable of providing, such as:

– supported interfaces (does it support GetFeatureInfo?)– map layers (what are they?)– symbolization (are there named styles available?)– geographic extent (what part of the world is this?)– Spatial Reference System (SRS) (which are supported?)– keywords (for topic searches)– supported image formats (supports transparency?)

The Capabilities Interface is required.

WMS GetCapabilities

Capabilities function as “service” metadata, providing information about what the WMS map service offers.

Clients determine whether or not they can work with that server based on its capabilities. WMS servers should be compatible with all versions of the standard earlier than the one it advertises.

The standard supports a lot of optional tags that enhance the usefulness of a Capabilities file - they are not all mandatory.

http://anyoldserver.com/WebMapService?request=GetCapabilitiesrequest=GetCapabilities&version=1.1.1&service=WMS

WMS GetCapabilities

Exercise Three

Examine a WMS Capabilities File

Support of this interface is optional

Returns information about a point on the map, similar to an “identify” operation in GIS

Layer must be set to queryable in Capabilities XML

Default format of request returned should be the MIME format for GML – application/vnd.ogc.gml

WMS is stateless so original GetMap request parameters are included

WMS service provider has some flexibility in how to return results – lots of variation here

WMS GetFeatureInfo

URL Prefix VERSION REQUEST Map-Request-Copy Query_Layers X Y

(Info_Format) (Feature_Count) (EXCEPTIONS) (Vendor Specific Parameters - VSP)

GetFeatureInfo Request Parameters

(Optional parameters in parentheses)

GetFeatureInfo Request

http://anyoldserver.com/WebMapService?version=1.1.1&

request=GetFeatureInfo&layers=cs_ng_10560_1&

styles=&srs=EPSG:27700&

bbox=332319,502462,333656,503799&width=600&height=600&

format=image/gif&query_layers=cs_ng_10560_1&

info_format=text/xml&feature_count=999&

x=258&y=270&

http://dearg.ucs.ed.ac.uk:8081/opengis/wms/historic/WebMapService?version=1.1.1&request=GetFeatureInfo&layers=cs_ng_10560_1&styles=&srs=EPSG:27700&bbox=332319.067428162

85,502462.06742816285,333656.93257183715,503799.93257183715&width=600&height=600&format=image/gif&query_layers=cs_ng_10560_1&info_format=text/

xml&feature_count=999&x=258&y=270&exceptions=application/vnd.ogc.se_xml

The optional DescribeLayer operation applies only to a WMS that supports Styled Layer Descriptors (SLD)

It allows the user to request an XML description of one or more map layers

DescribeLayer Request

Image Services vs. Feature Services

Image Services vs. Feature Services

Image Services Deliver image only (snapshot of the data), not actual feature (vector)

data More limited functionality—can’t export locally; changing symbology

more limited Lower bandwidth requirement

Feature Services Streams actual feature data over the Internet, available as individual

feature classes Allows extra functionality of changing symbology or exporting the data

locally; cleaner maps Higher bandwidth requirement Filtering; more sophisticated linking

Exercise Four

Image Services vs. Feature Services

Geography Markup Language 2.x-3.x

XML encoding for transport and storage of geographic information, including spatial and non-spatial properties of features

Defines the XML schema syntax, mechanisms, and conventions that provide open, vendor-neutral framework for the definition of geospatial application schemas and objects

Allows profiles that support proper subsets of GML framework descriptive capabilities

Support schemas for specialized domains

Support storage and transport of application schemas and datasets

Increase ability to share schemas and information

3.0 latest; 2.1.2 used with WFS (and in OSMM)

Web Feature Service 1.0

A WFS allows a client to retrieve actual geospatial data (vectors, not just an image) encoded in Geography Markup Language (GML 2.1) from multiple Web Feature Services.

A WFS request consists of a description of query or data transformation operations that are to be applied to one or more features.

The request is generated on the client and is posted to a WFS server using HTTP.

The WFS server then reads and (in a sense) executes the request and returns the resulting dataset in GML format.

Basic WFS servers provide Read-Only access to the data.

Transactional WFS servers actually allow clients to manipulate the data, including:

Create a new feature instance Delete a feature instance Update a feature instance Get or Query features based on spatial and non-spatial constraints

Web Feature Service 1.0

Differences between WFS and WMS capabilities (Nuke Goldstein Oct 2004)

http://www.directionsmag.com/article.php?article_id=686&trv=1

Exercise Five

OGC CompliantImage Services and Feature Services

Data Services Opportunities – Consuming Services

Conventional Library Geospatial Information Services

MapCollections

DataCollections

MapServers

MapPortals

• Map Collections– Paper Maps – Digital maps (e.g. Digimap)

• Data Collections– CD-ROMs, File server & FTP access– Web based (e.g. Census Dissemination Unit)

• Map Servers– Integrate collected data, Web-based mapping (e.g. EDINA)

• Coming: Map Portals– ‘Discovery’ portals (e.g. Go-Geo!)– Front end to distributed, streaming data (OpenGIS)

User Considerations for OGC Services

• Advantages– Time of need access– Streaming access (needed subset only, by location)– Most current data– Combine data from multiple sources

• Disadvantages– Reliability?– Effect of ongoing changes in underlying data on project work?– Image services (WMS) lack data intelligence, use as background

but not for analysis– Directory and catalog services not yet mature (services are often

moving targets)

When are streaming data services useful?

• By content type– Very large data files (e.g., ortho-imagery)– Very quickly changing data (e.g. ward boundaries)– Data not typically acquired by affiliated institution (other

regions/countries)• By user role

– Use of data as background or reference layer (e.g., ortho imagery, digital topo / historic maps)

– Preliminary analysis of data prior to acquisition or download– Casual user access and vertical integration– Geospatial web services as part of service chain (e.g., use to

geocode/geoparse documents)

Exercise Six

Using Desktop Tools to accessWeb Map Services (WMS)

So whats out there?

Exercise Seven

Another perspective - NC Local Government Mapping Services

Another perspective - NC Local Government Data Download

Problem: Service Discoverability/Selection

Geospatial Web Services are in a “Wild West” stage• Difficult to locate them• Difficult to assess content• Services come and go (demonstration services)• Variable adherence to standard; proprietary hooks

Disconnect between object metadata (e.g., FGDC metadata for dataset) and service metadata (e.g. OGC Capabilities file)-can have unidirectional link from service metadata to object

Catalog Services ( CAT )

Support the ability to publish and search collections of metadata for data, services, and related information objects

Discovery Services allows clients to locate metadata that describes GIS data

Access Services provides the client with methods to request services on the data, such as get a map

or download data, etc

Management Services Management Service defines methods for a client to change the metadata held by a

catalog.

Catalog Interface enables diverse but conformant applications to perform discovery, browse and query

operations against distributed and potentially heterogeneous catalog servers. 

Problem: Numeric Data Integration

Early OGC specifications and services are relatively map- and feature-centric—service content relatively “dumb” from the point of view of attribute/tabular data integration

How can geospatial web services be integrated with numeric data stores in a web services environment?

Discussion docs: Geo-linked Data Access Service (GDAS) (http://portal.opengis.org/files/?artifact_id=5858)

& associated Geo-Linking Service (GLS) (http://portal.opengis.org/files/?artifact_id=5859) Demo implementation at: http://wms1.agr.gc.ca/gdas/gdas.phtml

Exercise Eight

A Web Processing Example Using GDAS

(http://wms1.agr.gc.ca/GeoPS/GeoPSPage.html)

Problem: Who Will Preserve the Data?

History of lack of interest in long term access and data preservation on the part of data producers

Market for older data versions too small to be driving force in time series development (they don’t ask for it until its too late)

NC OneMap: “Historic and temporal data will be maintained and accessible”—not yet clear how that will be done

Context of current JISC repository call and ongoing Go-Geo! Phase 4b work

Historical Resource: Somerton, Monmouthshire

1843 to 1893 1891 to 1912

1904 to 1939 1919 to 1939

Future Historical Resource: Local GIS Data

Preservation

Secondary data archives are often developed as a by-product of providing data access (needed to get the data to give the data)—and secondary archives are often the ones that survive over time

If we no longer need to get the data to provide access to the data, is there as much incentive to build secondary archives

… on the other hand, data acquisition is time consuming, especially for local data. Can we use emerging Web Services technologies to automate the process of secondary archive construction? (e.g. agent-based trolling of capabilities files for new/updated data, then agent-negotiated transfer)

An Opportunity: Cascading Map Servers

Web Mapping Service that behaves like a client of other WMS servers and behaves like a server to other clients

When two or more maps are produced with same bounding box, spatial reference system, and output size, results can be accurately layered to produce a composite map

Can perform additional functions such as output format conversion or coordinate conversion for other servers

Enables creation of distributed map servers from which clients can build customized maps

Cascading Map ServersCascading Map Server

Web Browser

MapServer

Cascading Map Server MapServer

MapServer

MapServer

CapabilitiesIntegrator

MapIntegrator

JPEGWGS 84

GIFNAD 83

GIFWGS 84

Different viewer client in browser!

inte

rnet

inte

rnet

Adapted from Web Map Server Demonstration presentation © 2000 OGC

Why Cascading Map Servers?

Running a map server that locally hosts other peoples data is a recipe for stale data

Cascading map server leaves data management to data producer—deliver most current data

Library or data service defined cascading map servers might allow us to do with we do best: build a window to the world of information where the window is configured to best meet the needs of our target community

Cascading Map Servers: Software

Emerging software products from companies like Cubewerx, Ionic, Compusult (limited cascading support in open source Minnesota Map server)

Software not yet mature; niche companies leading the way on strictly OGC-compliant software (e.g. Cubewerx CubeSERV) http://www.cubewerx.com/main/products/CubeSERV_WMS.html

Proprietary hooks in some major vendor OGC connectors might complicate consumption of services by cascading map server

There are technical reservations however, e.g. how to specify plot order and reliably fuse 3rd party servers (SLAs/SLDs?)

Community Building: NC OneMap

… As informally discerned from NC OneMap training materials Agreements

• WMS version: 1.1.0 (for now)• Background color and transparency handling• Image preference: GIF followed by PNG• Make data available in geographic coordinates (most originates in some flavor of State

Plane)• Agree to call parcels “Parcels” and to use specific symbology• Registration process: Manual, with Capabilities files sent to USGS

Guidance• 1:12,000 to 1:15,000 scale cutoff for orthos and parcels (in process of working out

detailed guidance)• Encouragement to include layer metadata• Single word layer names• Load Capabilities DTD locally (service chain failure issues)• Security recommendations• Working on procedure for creating legend image swatches and method for making

legends available (no SLD support in phase 1)

Data Services Opportunities – Building Services

Building WMS Servers

Add-ons to commercial map server software (e.g. ArcIMS from ESRI)

Dedicated commercial software (e.g. Cubewerx)

Open source/freeware (e.g University of Minnesota Map Server)

Write your own (we did!!)

Follow the yellow brick road… (http://www.intl-interfaces.com/cookbook/WMS/)

+See Selected Resource handout for resources/links

OGC Directions

OGC Web Services 2 Initiative Web Services integration - SOAP/WSDL,UDDI, etc Technical baseline maturation

OGC Web Services 3 Initiative ‘Threads’ OGC Location Services (Open LS) OGC Web Sensor Enablement (SWE) OGC Common Architecture Geo-Decision Support (GeoDSS) Geo-Digital Rights Management (GeoDRM)

geoDRM – conceptual deployment

WMS

I&A

WFSClient

GetMap

GetFeature

Register

Login

Accept License

Accept License

GetMap

WMS

GetIdentity

VerifyIdentity

GetFeature

LicenseManagement

LicenseManagement

Conclusions

Conclusions: Technology

OGC technologies quickly evolving and gaining wide vendor acceptance (if sometimes inconsistently)

Reliability is still a question, especially in service chaining

With regard to service discovery and selection its still sort of like the Wild West—need to see Catalog Services and Web Registry Services fleshed out and widely adopted—need to flesh out library/data services roles in technical developments

Integration of OGC technologies with mainstream Web Services an important step

Conclusions: Data Services Implications

To what extent are streaming data services useful to users? Service reliability is a question Access to very large image resources as background is convenient—

in general convenient for background

Geospatial web services links with numeric data not well developed Opportunities exist to help make services more discoverable

(registries; catalog services) Need to integrate services with collections How do we link objects with services that act upon them? (parallels

in numeric data community with SDA’s etc.?) Cascading map servers as new service opportunity

Questions?

OGC Client Software – ESRI-centric list

 OGC Interoperability Add-on for ArcGIS

http://www.esri.com/software/standards/interopdownload.html   Cubewerx Extensions for ArcGIS and MapInfo

http://www.cubewerx.com/main/about_us/news/WMS_ext.html

 ARC3WMS (ArcView 3.x WMS Client)http://www.refractions.net/arc3wms/

Demo EDINA adapter

 ArcExplorer Webhttp://www.esri.com/software/standards/interopdownload.html#aeweb

CadCorp MapBrowserhttp://www.cadcorp.com/products_geographical_information_systems/map_browser.htm

UDIG (User-friendly Desktop Internet GIS)

http://udig.refractions.net/downloads/udig1.0.RC4.exe

Gaiahttp://www.thecarbonproject.com/products/gaia.html

MapInfo (7.8)http://extranet.mapinfo.com/products/Overview.cfm?productid=1044

AutoCAD

http://www.ime.co.uk/WMS/

OGC Client Software – some others

Catalog Services: Products

Red Spider Catalog2 (Ionic)http://www.ionicsoft.com/products/main_catalog.jsp

deegree (open source)http://deegree.sourceforge.net/

terraCatalog (Con terra)http://www.conterra.de/software/terraCatalog/index_engl.shtm

Indicio (Galdos)Http://www.galdosinc.com/index.html

Registries / Catalogs

Cascading Map Servers: Examples

USGS National Map

http://nationalmap.usgs.gov

NC OneMap

http://www.nconemap.com

FEMA Hazardmaps.gov

http://hazardmaps.gov

OGC Server Software -- Commercial

 Cubeserve WMS Cascading Map Server (Cubewerx)

http://www.cubewerx.com/main/products/CubeSERV_WMS.html

 Cartalinea (Galdos)

http://www.galdosinc.com/index.html

 Red Spider (Ionic)

http://www.ionicsoft.com/products/specRedSpiderEnterprise.jsp

 Moximedia Internet Mapping Framework

http://www.moximedia.com/ows.html

Cadcorp SIS

http://www.cadcorp.com/index.htm

Intergraph WMS Clientwww.wmsviewer.com

OGC Server Software -- Connectors

 WMS Bridge – ArcIMS WMS Connector (Compusult)

http://www.compusult.net/cslt_prod_dm_wmsb.html

 ArcIMS WMS and WFS Connectors (ESRI)http://www.esri.com/software/standards/product-support.html

OGC Server Software – Open Source

 Minnesota Map Server

http://mapserver.gis.umn.edu

 Geoserver

http://geoserver.sourceforge.net/html/index.php

 deegree

http://deegree.sourceforge.net/index.html

For a list of products compliant to or implementing OGC specs or interfaces see:

http://www.opengis.org/resources/?page=products

Example Server Implementations

 Federal• National Map http://www.nconemap.com

• FEMA HazardMaps http://hazardmaps.gov

 State and Local• NC OneMap http://www.nconemap.com

• NCMapNet http://www.ncmapnet.com

• Louisiana Atlashttp://atlas.lsu.edu/central/displayOGCWFSFAQ.htm

 University• Harvard University WMS (Distributed) Servers http://portal.opengis.org/files/files/?artifact_id=5908