11

ACES Scholars’ Grid

5th ACES International WorkshopMaui Prince Hotel, Island of Maui, Hawaii

April 6 2006

Geoffrey FoxMarlon Pierce

Community Grids laboratoryComputer Science, Informatics, PhysicsIndiana University Bloomington IN 47401

http://grids.ucs.indiana.edu/ptliupages/presentations/gcf@indiana.edu http://www.infomall.org

2

Semantically Rich Services with a Semantically Rich Distributed Operating Environment

Database

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

FS

FS

FS

FS

FS

FS

FS

FS

FS FS

FS

FS

FS

FS

FS

FS

FS

FS FS

FS

FS PortalFS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

MD

MD

MD

MD

MD

MD

MD

MD

MD

MetaData

Filter Service

Sensor Service

OtherService

SOAP Message Streams

SOAP Message Streams

Raw Data Raw Data

Raw Data

Raw Data

Data

Data Data

Data

Information

Information

Knowledge

Knowledge

Wisdom

Decisions

Information

AnotherService

AnotherService

AnotherGrid

AnotherGrid

Grids of Grids Architecture

is same as outwardfacing application

service

33

ACES Grid and Services Services receive data in SOAP messages, manipulate it and

produce transformed data as further messages Meta-data is carried in SOAP messages but stored in databases

with XML-defined interfaces Meta-data controls processing and transport of SOAP Messages Meta-data describes what Quake information there is and how it

was created (provenance) Knowledge is created from data by services The Grid enhances Web services with semantically rich system

and application specific management One must exploit and work around the different approaches to

meta-data and their manipulation in Web Services• Just as we work around Job submission, security etc. choices that 5 years

from now will be clearly irrelevant in the big Service Architecture picture Grids of Grids: Compose Grids from smaller Grids and a service

is “just” a special case of a Grid• Sub-Grids could make GEON Globus SCEC SERVOGrid idiosyncratic

choices

44

What Type of Services are there? There are a horde of support services supplying security,

collaboration, database access, user interfaces The support services WS-* and GS-* are either associated with

system or application• Globus, Apache, OMII, EGEE, and many Grid Project produce these• Microsoft IBM Amazon Google will be major players

There are generalized filter services which are applications that accept messages and produce new messages with some data derived from that in input• Simulations (Such as PDE’s)• Data-mining• Transformations• Agents• Reasoning are all termed filters here

Note databases, sensors and simulations are sort of same thing: they are services that produce (Web Feature Service WFS formatted) Earth Science relevant messages – We call them ACESNodes

All services and their interactions are bathed in sea of meta-data and so implicitly need the Semantic Grid

55

WMS uses WFS that uses data sources

Railroads

RiversBridges

Interstate Highways

90

WFS Server

SQL Query

Railroads

[a-b]

SQ

L Q

uery

Riv

er [a

-d]

Bri

dge

[1-5

]

SQL QueryHigway [12-18]

`

ClientWMS

GetFeature

FeatureCollection

Get

Feat

ure

Feat

ureC

olle

ctio

n

<gml:featureMember> <fault> <name> Northridge2 </name> <segment> Northridge2

</segment> <author> Wald D. J.</author> <gml:lineStringProperty> <gml:LineString

srsName="null"> <gml:coordinates>

-118.72,34.243 -118.591,34.176 </gml:coordinates>

</gml:LineString> </gml:lineStringProperty> </fault> </gml:featureMember>

6

Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.

Integrating Archived Web

Feature Services and Google Maps

7

Typical use of Grid Messaging in NASA

Datamining Grid

Sensor Grid

Grid Eventing GIS Grid

WFS is Universal Interface

8

Real Time GPS and Google Maps

Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients.

Select and zoom to GPS station location, click icons for more information.

99

ACESNodes: ACESSensors, ACESRepositories, ACESFilters

Sensors are real-time and typically get their data from the “edge of the Grid”

Repositories are typically databases storing ACESData Filter are Simulations and transformations ACESNodes (Skynodes in Astronomy Virtual

Observatory) accept and produce messages in the same ACESFS Syntax – an enhanced Web Feature Service WFS that knows about faults, plates etc (ADQL, SIA, SSA in astronomy)• Copy VOTable use from Astronomy for all output

ACES should agree on ACESFS and the partners should agree that all Sensors, Repositories and Filters will be presented to world as ACESNodes • Astronomy has IVOA masterminding this

1010

Coupled Simulations

From Earthquake Occurrence with aftershocks to

Wave Motion to Directly damaged infrastructure to Behavior of people, traffic, telephony,

energy (14 critical infrastructures) ….

• These use “activity data” of where people are at a given time to model transportation, energy and phone use etc.

Package as a training game either on Xbox or TeraGrid

• Get FEMA officials to play it!

Electric Power and Natural Gas systems from LANL Interdependent Critical Infrastructure Simulations using SERVOGrid GIS sub-Grid

1111

ACESNodes IntegrationCountry Data Earthquake

Forecast/Model

Wave

Motion

Critical

Infrastructure

Australia Finley, LSM

PANDAS

Canada Polaris Radarsat

P.I.

P.R. China Seismic LURR Tsinghua (CNG)

Shanghai Grid

Japan GPS

Seismic

Daichi (InSAR)

GeoFEM Matsu’uraTalk

Taiwan Chen talk Chen talk

U.S.A. QuakeTables

Sesismic

InSAR

PBO (GPS)

P.I.

ALLCAL

GeoFEST, PARK,

VirtualCalifornia

TeraShake DoE NISACD DivisionLANL

International IMS

12

The Core Service Areas IService or Feature WS-* GS-

*NCES(DoD)

Comments

A: Broad Principles

FS1: Use SOA: Service Oriented Arch.

WS1 Core Service Model, Build Grids on Web Services. Industry best practice

FS2: Grid of Grids Strategy for legacy subsystems and modular architecture

B: Core Services

FS3: Service Internet, Messaging

WS2 NCES3 Streams/Sensors

FS4: Notification WS3 NCES3 JMS, MQSeries

FS5 Workflow WS4 NCES5 Grid Programming

FS6 : Security WS5 GS7 NCES2 Grid-Shib, Permis Liberty Alliance ...

FS7: Discovery WS6 NCES4

FS8: System Metadata & State

WS7 Globus MDSSemantic Grid

FS9: Management WS8 GS6 NCES1 CIM

FS10: Policy WS9 ECS

13

The Core Service Areas IIService or Feature WS-* GS-* NCES Comments

B: Core Services (Continued)

FS11: Portals and User assistance

WS10 NCES7 Portlets JSR168, NCES Capability Interfaces

FS12: Computing GS3

FS13: Data and Storage GS4 NCES8 NCOW Data Strategy

FS14: Information GS4 JBI for DoD, WFS for OGC

FS15: Applications and User Services

GS2 NCES9 Standalone ServicesProxies for jobs

FS16: Resources and Infrastructure

GS5 Ad-hoc networks

FS17: Collaboration and Virtual Organizations

GS7 NCES6 XGSP, Shared Web Service ports

FS18: Scheduling and matching of Services and Resources

GS3

1414

SERVOGrid http://www.servogrid.org Services I

Area Service Name Description

FS3 Messaging Service This is used to stream data in workflow fed by real-time sources. It is based on NaradaBrokering which can also be used in cases just involving archival data

FS3 Sensor Grid Services

We are developing infrastructure to support streaming GPS signals and their successive filtering into different formats. This is built over NaradaBrokering (see messaging service). This does not use Web Services as such at present but the filters can be controlled by HPSearch services.

FS4 Notification Service

This supplies alerts to users when filters (data-mining) detects features of interest

FS5FS9

Workflow /Monitoring /Management Services

The HPSearch project uses HPSearch Web Services to execute JavaScript workflow descriptions. It has more recently been revised to support WS-Management and to support both workflow (where there are many alternatives) and system management (where there is less work). Management functions include life cycle of services and QoS for inter-service links

FS6 Authentication and Authorization

This uses capabilities built into portal. Note that simulations are typically performed on machines where user has accounts while data services are shared for read access

FS7 Information Service

We have built data model extensions to UDDI to support XPath queries over Geographical Information System capability.xml files. This is designed to replace OGC (Open Geospatial Consortium) Web registry service

1515

SERVOGrid http://www.servogrid.org Services II

Area Service Name Description

FS8 Context Data Service

We store information gathered from users’ interactions with the portal interface in a generic, recursively defined XML data structure. Typically we store input parameters and choices made by the user so that we can recover and reload these later. We also use this for monitoring remote workflows. We have devoted considerable effort into developing WS-Context to support the generalization of this initial simple service.

FS11 Portal We use an OGCE based portal based on portlet architecture

FS11Appl.

Web Map Service We built a Web Service version of this Open Geospatial Consortium specification. The WMS constructs images out of abstract feature descriptions.

FS11Appl.

Scientific Plotting Services

We are developing Dislin-based scientific plotting services as a variation of our Web Map Service: for a given input service, we can generate a raster image (like a contour plot) which can be integrated with other scientific and GIS map plot images.

FS12 File Services We built a file web service that could do uploads, downloads, and crossloads between different services. Clearly this supports specific operations such as file browsing, creation, deletion and copying.

FS13Appl.

QuakeTables Database Services

The USC QuakeTables fault database project includes a web service that allows you to search for Earthquake faults.

1616

SERVOGrid http://www.servogrid.org Services III

Area Service Name Description

FS13 Data Tables Web Service

We are developing a Web Service based on the National Virtual Observatory’s VOTables XML format for tabular data. We see this as a useful general format for ASCII data produced by various application codes in SERVO and other projects.

FS14Appl.

Application and Host Metadata Service

We have an Application and a Host Descriptor service based on XML schema descriptors. Portlet interfaces allow code administrators to make applications available through the browser.

FS14Appl.

Web Feature Service

We’ve built a Web Service version of this OGC standard. We’ve extended it to support data streaming for increased performance.

FS15 Specific Applications:

Virtual California,Geofest, Park, RDAHMM .. These can be all launched by a single Job Management service or by custom instances of this with metadata preset to a particular application

Key interfaces/standards/software Used

GML WFS WMS WSDL XML Schema with pull parser XPP, SOAP with Axis 1.xUDDI WS-Context, JSR-168 JDBC ServletsWS-Management VOTables in Research

Key interfaces/standards/software NOT Used (often just for historical reasons as project predated standard)

WS-Security JSDL WSRF BPEL OGSA-DAI

1717

Delicious ACES http://del.icio.us purchased by Yahoo for ~$30M http://www.CiteULike.org http://www.connotea.org (Nature) http://www.bibsonomy.org/

• Associate metadata with Bookmarks specified by URL’s, DOI’s (Digital Object Identifiers)

• Users add comments and keywords (called tags)• Users are linked together into groups (communities)• Information such as title and authors extracted automatically

from some sites (PubMed, ACM, IEEE, Wiley etc.)• Bibtex like additional information

This is de facto Semantic Web – remarkable for its simplicity

1818

Connotea

1919

Connotea queried by SERVOGrid

2020

Provenance and Delicious ACES All ACESData should be associated with provenance

that describes its lineage• How and when it was created• Compiler options used in simulation• ACESFS query used on what ACESNodes

Provenance produced by computer automatically and/or by user

All ACESData can and should be labeled by a URIaces://acesnodenumber.xx.yy.whathaveyou

We can use del.icio.us style interface to annotate ACESData with missing provenance and user comments of any type (describing quality of data or a keyword relating different data etc.)

2121

Semantic Scholar Grid Citeseer and Google Scholar scour the Internet and

analyze documents for incidental metadata Title, author and institution of documents Citations with their own metadata allowing one to

match to other documents These capabilities are sure to become more powerful

and to be extended• Give “Citation Index” in real time• Tell you all authors of all papers that cite a paper that cites

you etc. (Note it’s a small world so don’t go too far in link analysis)

• Tell you all citations of all papers in a workshop Such high value tools will appear on “publisher” sites

of future (or less publishers will disappear)

2222

OSCAR2 Chemistry Document analysis

It detects “magic” chemical strings in text and then• Stores them as

metadata associated with document

Queries ChemInformatics repositories to tell you lots of information about identified compounds

Tells you which other documents have this compound

2323

ACES Version of OSCAR Some of the ACESNodes will store metadata associated

with ACESData – including documents• Note documents could be anywhere on the Internet – the

ACESNode may choose to store (a copy of) document or just its metadata

• Note all ACESNodes are federated i.e. there is no “one central” store of any type of data

Metadata will be user annotations including tags, Citeseer style citation information for all scientific fields

Then each scientific field has its own version of OSCAR tuned to extract natural metadata for science – for ACES this is GML (Chemistry is CML …) and ACESFS extensions

24

Semantic Scholars’ Grid I

Local MDStore

Local HarvestStore

Gatherer

AnalyzerIndexer

Query andGet list

Fetch MD and Documents

Run filter such asOSCAR2 on

harvested MDand documentsStore new MD

Index allLocal MD

Science.gov

PubMed

Google Scholar

etc.

Dspace

e-Prints

25

Semantic Scholars’ Grid II

Local MDStore

Updater

CiteULike

Connotea

Del.icio.us

etc.

ForeignUser Interface

Update and viewforeign MD

SSGViewer

Update local MDControl foreign interactions

View all MD’Access Community Tools

SynchronizeSSG and

foreign MD

ACM

IEEE

Google Scholar

etc.

Wiley

CommunityTools

Instant CitationIndex etc.

Plug-in