Web Services Survey an Inventory

Background, Goals and Status

14th ESIP Federation Assembly MeetingRenaissance Mayflower Hotel

Washington, DCJanuary 4-6, 2005

Presented by Rudolf Husar, Technology Infusion Workgroup, rhusar@me.wustl.edu

OpenDAP CatGCMD

ESIP CatOGC Cats

ECHO Google

Background and Purpose

There is a rapidly growing array of distributed services for accessing, processing, visualizing, cataloging, discovering or otherwise manipulating Earth Science information through computer-computer interfaces.

The landscape is currently obscured by the lack of a suitable inventory and by the hype associated with web services

Purposes of the WS Inventory/Survey• assess the size and characteristics of the computer-accessible (WS) resource pool• expose the main computer-computer WS linking protocols used and • get a rough assessment of the current WS usage environments.

Beneficiaries:ESIPInfusion DSWG workgroup

other DSWG workgroups, (e.g. Reuse and Standards)

It should also aid the creation and diffusion of ES knowledge for Achieving a Sustainable Planet (ESIP)

WS Inventory: Features and Approach

WS Inventory Survey Features

• Flexible to capture the varied WS technologies and their use environments

• Simple for easy participation

Approach to the WS Inventory Content Creation• The content of the WS Inventory collected from the community through a web-form.• Subsequent versions of this inventory could incorporate more specifics for service binding• The content could migrate to formal WS Discovery services such as GCMD, ECHO and

others.

Current task: Designing the web-form to be filled out by the participating members

Latest draft Web Service Inventory page on the Infusion WG websitehttp://www.sciencedatasystems.org/seeds/wg/infusion/Lists/Service%20Inventory/AllItems.aspx

Discussion thread for web-form design WS Invedntory Form Design by the WGhttp://www.sciencedatasystems.org/seeds/wg/infusion/Lists/Web%20Services%20Discussion/AllItems.aspx

Select the node in the thread Press Post Reply (remember to log in, so we know who you are).

WS Inventory as an Open, Living Resource?

• Relationship to other Catalogs (Social behavior)

– Not to be rude (either you or I)

– Not just coexisting (either you or I)

– Cooperate, be part of a value-chain

OpenDAP CatGCMD

ESIP CatOGC Cats

ECHO Google

Cooperation of service catalog ‘vendors’ is achievable by

Opening the contents of one’s catalog to access by value-adding partnersWeb services offer an simple, safe and agile technology for creating value chains

Catalog Linking DemoA web service chaining technology demonstration

A template for linking GCMD, ECHO and other catalog chains?

Test client CATALOG, WashU http://datafed2.seas.wustl.edu/dvoy_services/sds.aspx

Client sends SOAP request to SherPoint server

Clients receives SOAP-wrapped catalog content

The XML data looks like this (SOAP envelope stripped away)

Client adds value, e.g. filter/transform/render add more metadata

Technology Infusion Websitehttp://www.sciencedatasystems.org/seeds/wg/infusion/Lists/Service%20Inventory/AllItems.aspx

Catalog data are collected through a form on SDS SharePoint server

Catalog content is maintained in a SharePoint database

SharePoint exposes data through SOAP web services

SOAP

web services

WS Catalog/Survey Discussion Thread

Scope of Web Services Subgroup, 2005WG Purpose: Find and demonstrate ways to infuse WS Technologies

1. Prepare an inventory of web services and their interfaces – The WS landscape is changing rapidly. New services are emerging – We need a snapshot of the service landscape, particularly in the REASoN projects – But what ‘services’ do we include? SOAP, OPeNDAP, OGC, files?

• New developments: – We have prepared a Ws service survey form at the WG website, populated with test survey entries – At ESIP meeting, we presented the the idea of the WS survey to the combined WGs, Rob Raskin’s Data Discovery and Brian Wilson’s Web

services.– The response was not too favorable: do we need yet another catalog/survay? Whats wrong with GCMD, ECHO and other catalog/dicovery

efforts– In the ESIP working sessions we have explored the possibility of using the GCMD SERF registration for general service description and ECHO

as a formal WS registry. – Both GCMD and ECHO representatives recognized that lining the two catalog efforts was meaningful and some changes will be necessary in

each catalog to formalize the link– Fat is that GCMS has very few formal SOAP web services registered– Brian Wilson has volunteered to be the ESIP rep. for this effort

• New ideas for WS status assessment– So, we still need some other low resistance approaches to gather the list of formal web services

• At the ESIP discussion Jim Frew has suggested tha we send out an emal to the SEEDs community simply requesting URLs of the web description documents, the WSDL from any one who has a service.

• Brian Wilson suggested to do a Google search on WSDL..I did that an viola, there were at least a dozen Earth Science WSDLs• I am in favor of tese light-handed approaches to explore the currently functional WS landscape

• Wilson - more documentation for the WSDL – • Next WG meeting Michael Burnett – UDDI -

• Demonstrate web service linking– Several REASoNs have a goal of facilitating satellite data flow and processing – A narrow pilot project could demonstrate satellite data access/processing/analysis/delivery – Issues: Which REASoN projects? Data products? Services? Architecture(s)?

• New developments:– At ESIP, Brian Wilsons WS WG cluster has set itself a goal to organize WS chaining demo projects– Our WS can actively participate in that prototyping exploration

1. Assess WS linking architectures for data access AND analysis– We don't quite know how to link distributed WSs into coherent science applications – An open dialog on the linking/architectural issues would speed up the group learning– Issues: e.g. Architecture without rigid architecture

• James, I did want to get the SOAP stuff over early in my track, and then move on to Grid stuff. But if this doesn't fit in with your plans, we can start with Grid in my track, then do the joint SOAP talk, and then return to Grid.

• Tuesday• 1:30-2 PM -- SOAP Protocol, Brian Wilson (joint with Data Protocol track)• 2-2:30 PM -- Structuring Services with SOAP as Universal Glue, Brian Wilson (could be joint with Data Discovery, but not crucial)• 2:30 - 3 PM -- Data Grid using SRB and Globus (Mike Smorul, Gary Jackson)• 3-3:30 PM -- Break• 3:30-5 PM -- More Grid Services stuff by Mike and Gary• 5-5:30 PM -- Richard Troy on Grid Technologies

• Note that I would still like to put Troy's talk on the first day, even though we have a half-hour less time than I thought. Is it okay to go until 5:30

• Wednesday• 8:30-9 AM -- Orientation Talk, Goals of Track, Brian Wilson• 9-9:30 AM -- Gathering an Inventory of Fed. Services, Rudy Husar• 1st Discussion Period -- collaborating on services, service interop, experiences with using SOAP, etc.• Perhaps have a joint discussion with the Data Protocol track about inter-operability. James, should we try to agree on a time?

• Grid Discussions (following above)• Lead off with half-hour talk by Paul Davis (or someone else?) --• Capabilities Matrix for SRB & Globus,• How does one join the Federation Data Grid?• Ensuing discussion

• Rest of Wed. & Thursday morning• Formulate recommendations for both Web & Grid Services use within the Federation.

• I'm still working on filling in a more detailed but tentative schedule for Wed. and Thu. I'm going to try to come prepared with some partially filled out comparison tables such as a Capabilities Matrix for SRB and Globus (single sign-on, data replication, job submission, bulk data transfer, etc.) I will talk to Paul and company about this.

• I suspect that we will have more *informal* presentations on Wed. to structure the discussions. Please feel free to suggest particular discussions you would like to see happen or like to lead. If you want to make further presentations on Wed. to provoke & facilitate discussion, let me know.

• Every function can be looked at as a "service“• The key protocol(s) for such distributed services are SOAP and REST (one-line URL's).

• If every data provider put up a SOAP service that took startTime, endTime, lat/lon box and physicalVariableName(s) as inputs, and returned a list of unique object ID's (i.e., granule file ID's), then we would almost be done. Then data discovery down to the inventory level would be reduced to "discovering" a list or catalog of all of the relevant SOAP services for a particular scientific domain. A catalog that we are starting to assemble via the survey form.

• The SOAP service could also provide a translation table from generic variableNames (atmosphericTemperature) to dataset-specific names, to make the discovery more generic.

• I would like to inject these ideas into the Data Discovery track. Perhaps I can say a few words during the joint half-hour, before or after Rudy's talk.

• I could also be convinced that the effort to catalog the available services via a survey is not, per se, that relevant to the Data Discovery topic.Perhaps Rudy will have to add a part to his talk that responds to the Data Discovery problem. Alternatively or additionally, we could have a joint discussion period later in the day.

• Rob, I'm already preparing too many talks so I'm not signing up for another joint talk. My joint SOAP protocol talk will address how Data Discovery can be solved by using SOAP/REST services. If folks miss that talk, then I can inject the idea in later discussions.