Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University

Web Services for the National Virtual Observatory

Tamás Budavári Johns Hopkins University

3/31/2005 Tamás Budavári (JHU) 2

Outline

Motivations and trendsHow will the VO work?The NVO (hi)storySample VO servicesWhere we are going

3/31/2005 Tamás Budavári (JHU) 3

The World is Exponential

Astrophysical data is growing exponentiallyDoubling every year (Moore’s Law+): both data sizes and number of data sets

Computational resources scale the same way

Constant $$$ will keep up with the data

Main problem is the software componentCurrently components are not reusedSoftware costs are increasingly larger fractionAggregate costs are growing exponentially

3/31/2005 Tamás Budavári (JHU) 4

Making Discoveries

When and where are discoveries made?Always at the edges and boundariesGoing deeper, using more colors…

Metcalfe’s lawUtility of computer networks grows as the number of possible connections: O(N2)

VO: Federation of N archivesPossibilities for new discoveries grow as O(N2)

Current sky surveys have proven thisVery early discoveries from SDSS, 2MASS, DPOSS

3/31/2005 Tamás Budavári (JHU) 5

Publishing Data

Roles

Authors

Publishers

Curators

Consumers

Traditional

Scientists

Journals

Libraries

Scientists

Emerging

Collaborations

Project website

Bigger archives

Scientists

3/31/2005 Tamás Budavári (JHU) 6

Changing RolesExponential growth: data never centralized

A project last 3-5 years (linear rate but new projects)Data sent upwards only at the end of the project

More responsibility on projectsBecoming Publishers and CuratorsLarger fraction of budget spent on software

More standards are neededEasier data interchange, fewer tools

More templates are neededRecycling software components

3/31/2005 Tamás Budavári (JHU) 7

Emerging New Concepts

Standardizing distributed dataWeb Services, supported on all platformsCustom configure remote data dynamicallyXML: Extensible Markup LanguageSOAP: Simple Object Access ProtocolWSDL: Web Services Description Language

Standardizing distributed computingGrid ServicesCustom configure remote computing dynamicallyBuild your own remote computer (and discard)Virtual Data: new data sets on demand

3/31/2005 Tamás Budavári (JHU) 8

Shielding Users

Users do not want to deal with… XML – they just want their dataConfiguring grid computing – want results

Web servicesData appears in user memory, XML is invisibleSOAP: just a remote procedure call

3/31/2005 Tamás Budavári (JHU) 9

NVO: How Will It Work?Define commonly used ‘core’ servicesBuild higher level toolboxes/portals on topWe do not build ‘everything for everybody’Use the 90-10 rule:

Define the standardsand interfacesBuild the frameworkBuild the 10% of services that are used by 90%Let the users build the rest from the components

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5

# of services

# o

f u

sers

3/31/2005 Tamás Budavári (JHU) 10

Atomic Services

Metadata information about resourcesWavelength: optical, ultraviolet, etc.Sky coverage: where on the sky?Translation of names to universal dictionary (UCD)

Simple search patterns on the resourcesCone SearchImage mosaicUnit conversions

Simple filtering, statistics, histogramsOn-the-fly recalibrations

3/31/2005 Tamás Budavári (JHU) 11

Higher Level Services

Built on basic Core ServicesPerform more complex tasks, e.g.

Automated resource discoveryCross-identificationsPhotometric redshiftsOutlier detectionsVisualization facilities

Expectation:Build custom services in days from existingbuilding blocks (like today in IRAF or IDL)

3/31/2005 Tamás Budavári (JHU) 12

Early Standards

VOTableUniversal container for tables (in XML)First VO standard (from the DTD era)

ConeSearchSimple catalog access based on locationFirst VO standard interface (http get)

Many implemented them!

3/31/2005 Tamás Budavári (JHU) 13

VO Registry

Searchable (fast)Various resourcesOAI harvesting

Web site andWeb services

5,459 resources

3/31/2005 Tamás Budavári (JHU) 14

Sloan Digital Sky Survey

The cosmic genome projectMap the northern sky ~7000 sq.degSophisticated science archive (SQL)Many ways to query and filter dataVarious data types: Images, spectra, catalogs

http://skyserver.sdss.org/

3/31/2005 Tamás Budavári (JHU) 15Courtesy of Robert Lupton

3/31/2005 Tamás Budavári (JHU) 16

SDSS Image Cutout

Build JPEG mosaicsFor given (RA, Dec)Arbitrary zoom levelsOverplot catalog dataOverplot geometryMapQuest like navigationList of objects

Web servicesJPEG returned in DIMEor byte array in XML Kudos to Maria

Nieto-Santisteban

Kudos to Maria Nieto-

Santisteban

3/31/2005 Tamás Budavári (JHU) 17

Spectrum Services

Public repository500,000 spectra (2B row)

Spatial search and more

Register & submit yours

Web siteOn-the-fly plotting

Building composites

Web servicesMore options and tools

Kudos to László Dobos

Kudos to László Dobos

3/31/2005 Tamás Budavári (JHU) 18

Catalog Access

ConeSearchVery limited but in placeMany resources available

Need simple queriesSQL-like queries, e.g.

SELECTSELECT <columns> FROMFROM <table> WHEREWHERE <conditions>

What else we need?

To specify area on sky• Could be very complicated

Standard interface to DBs

• XML Web Services for interop

Tools to federate archives

• Dynamically for flexibility

3/31/2005 Tamás Budavári (JHU) 19

Catalog Access: New Standard

Virtual Observatory Query Language

Level 3. Semantically rich XML language (TBD)

Level 2. Portals combining astro archivese.g. SkyQuery and Open SkyQuery

Level 1. Query a single databaseAstronomical Data Query Language VOTable

3/31/2005 Tamás Budavári (JHU) 20

Open SkyQuery

SkyNode interface to archives

Implements ADQL returns VOTableBasic node understands “REGION”Full node understands “XMATCH”

SkyQuery portalKnows the SkyNodes from RegistryUnderstands federated query

http://openskyquery.net/

3/31/2005 Tamás Budavári (JHU) 21

Architecture

Image cutout

SkyNodeSDSS

SkyNode2MASS

SkyNodeFIRST

SkyQuery

Web App

3/31/2005 Tamás Budavári (JHU) 22

SkyNode

Metadata functionsTables, Columns, FunctionsUptime, Footprint, etc…

Query functionsPerformQuery (ADQL q)

• Simple query interface

QueryCost (ADQL q)• Cost estimate for query

ExecutePlan (XPlan plan)• Recursive probabilistic join

3/31/2005 Tamás Budavári (JHU) 23

Data flow

SkyQuerySkyQuery

querySkyNode 1SkyNode 1

SkyNode 2SkyNode 2

SkyNode 3SkyNode 3

http://openskyquery.net

Registry

3/31/2005 Tamás Budavári (JHU) 24

SkyQuery portal

Hides complexitySimple web services

MethodsGet to metadata of nodesFind location of nodesSend query to one nodeSend federated query

• Create execution plan• Submit execution plan

3/31/2005 Tamás Budavári (JHU) 25

3/31/2005 Tamás Budavári (JHU) 26

Other VO Services

Density map plottingCosmic distance calc.Footprint intersectionWrappers for legacy

TCP services, e.g. NEDAnalysis tools & libraries e.g. CFITSIO (in C#)

http://voservices.org/

3/31/2005 Tamás Budavári (JHU) 27

NVO Summer School

Aspen, Colorado13-17 September 2004

Full house~40 “students”

3/31/2005 Tamás Budavári (JHU) 28

QuestionnaireVery good scores !!!Needs to be longer (10d)Great software package

• ZIP file on the NVO twiki

ProceedingsOn the Web

HERE

NVO Summer SchoolAll possible OS

Real test of interoperabilityChallenging to teach

WS learning curveDepending on background

Winning projectsC. Miller & S. KrughoffExtinction map serviceR. Lucas and friendsVLA science project

3/31/2005 Tamás Budavári (JHU) 29

NVO Comes of Age

Special NVO session at AAS ‘05Featuring ~10 science talks using NVO technology

First refereed astronomy publicationsUsing the NVO framework

Next Summer School is being organized

NASA+NSF: Long Term NVO FacilityStarting in 2006

3/31/2005 Tamás Budavári (JHU) 30

NVO is here!

3/31/2005 Tamás Budavári (JHU) 31

Where to go…

Links in the talkhttp://www.us-vo.org/http://www.ivoa.net/http://voservices.org/http://openskyquery.net/http://skyserver.sdss.org/ http://nvo.stsci.edu/voregistry/