AstroGrid and OGSA-DAI: Early experiences

A PPARC funded project

AstroGrid and OGSA-DAI: Early experiences

Kona AndrewsGuy Rixon

Institute of Astronomy University of Cambridge

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Data Set Access: The DataCenter

DataCenter component

DataAccessPlug-in

Data

Web Services

Tomcat / Axis

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

DataCenter JDBC plug-in

DataCenter component

JDBC

DBMS

Web Services

Tomcat / Axis

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

DataCenter custom plug-in

DataCenter component

File querier

FITS files

Web Services

Tomcat / Axis

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

DataCenter proxy plug-in

DataCenter component

Service proxy

Vizier

Web Services

Tomcat / Axis

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

What we imagined

OGSA-DAI

Web Services

DataCenter Component

OGSA-DAI client plug-in

Tomcat / Axis

Grid Service Ports

Tomcat / OGSA

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Problem

OGSA-DAI

Web Services

DataCenter Component

OGSA-DAI client plug-in

Tomcat / Axis

Grid Service Ports

Tomcat / OGSA

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Problem

OGSA-DAI

Web Services

DataCenter Component

OGSA-DAI client plug-in

Tomcat / Axis

Grid Service Ports

Tomcat / OGSA

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Nasty solution

OGSA-DAI

Web Services

DataCenter Component

OGSA-DAI “client” plug-in

Tomcat / Axis

Grid Ports

Tomcat / OGSA

OGSA-DAIclient

Separate JVM

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMSADQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQLADQL

SQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

ADQL

SQL SQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

ADQL

SQL

Result rows

SQL

(to XML RowSet)

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

deliverFromURL

(to XML RowSet)

ADQL

SQL

Result rows

SQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

xslTransform

deliverFromURL

(to XML RowSet)

ADQL

SQL

Result rows

SQL

VOTable

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

xslTransform

deliverToFile / deliverToGFTP

deliverFromURL

(to XML RowSet)

ADQL

SQL

VOTable

Result rows

SQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

xslTransform

deliverToFile / deliverToGFTP

deliverFromURL

(to XML RowSet)

ADQL

SQL

VOTable

Result rows

SQL

Deliver results

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Comments

Astronomers often need LARGE result sets

Thousands to millions of rows

Memory usage is a big issue

OGSA-DAI 3.1 much better than 3.0

xslTransform activity hogging memory

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

OGSA-DAI speed penalty

0

10

20

30

40

1e3 1e4 1e5 2e5 4e5 1e6 2e6

Number of result rows

Rough figures only

Measured without xslTransform activity

Will likely vary with available memory

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Planned extension: User tables

OGSA-DAI

DBMSDataCenter

R/ODatasets

Query Query results

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Planned extension: User tables

OGSA-DAI

DBMSDataCenter

R/ODatasets

Usertables

Query Query results Data flow

GDS to GDS transfer

Bulk load

OGSA-DAI

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

DBMS

Planned extension: User tables

OGSA-DAI

DBMSDataCenter

DataCenter

OGSA-DAI

Table Manager

R/ODatasets

Usertables

Query Query results Data flow

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

DBMS

Planned extension: User tables

OGSA-DAI

DBMSDataCenter

DataCenter

OGSA-DAI

Table Manager

Client

User data

R/ODatasets

Usertables

Query Query results Data flow

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Comments

Will be relying on sqlBulkLoadRowSet (!)

Table management facilities currently thin

Not using X.509 auth at user granularity

Still require W/S-based access to G/S

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Our needs

Ability to handle very large result sets - Don’t rely on loading whole result set into memory!

Suite of DB/table management activities- List tables in DB

- Describe table column types / space usage

- Last access/modification time

- Etc etc etc

Ability to create and delete indices on tables

Ultra-simple installation procedures

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

The End

www.astrogrid.org