Embed Size (px)
Citation preview
VO Standards – Catalog Access
Tamás Budavári Johns Hopkins University
9/14/2004 NVO Summer School, Aspen 2004
2
Background
Virtual Observatory Query LanguageLevel 3. Semantically rich XML language (TBD)
Level 2. Portals combining astro archivese.g. SkyQuery and Open SkyQuery
Level 1. Query a single databaseAstronomical Data Query Language VOTable
Start with something simpleDecided not to reinvent the wheel
9/14/2004 NVO Summer School, Aspen 2004
3
Outline
SQL PrimerHow to query databases
ADQLExtensions for astronomyDistributed resources
SkyQuerySkyNode interfaceSkyQuery portal
9/14/2004 NVO Summer School, Aspen 2004
4
Structured Query Language
Simple syntaxSELECTSELECT <columns> FROMFROM <table> WHEREWHERE <conditions>
For example: select ra, dec
from galaxy where r<19
Joining tablesMore tables with common column (link, relation)Select from multiple tables with join criteriaE.g. an object id:
select g.ra, g.dec, s.z from galaxy g, spec s where g.objid=s.objid and g.r<19
9/14/2004 NVO Summer School, Aspen 2004
5
Astronomical Data Query Language
SQL extended for astronomyAdded survey name in from clausee.g. SDSS:Galaxy g, DLS:obj o
Specify area on skyAdded “REGION” keyworde.g. Region('CIRCLE J2000 181.3 -0.76 6.5')
Formulate federated queryAdded “XMATCH” keyworde.g. XMatch(g,o) using aliases
9/14/2004 NVO Summer School, Aspen 2004
6
ADQL Translator
RepresentationsString for humans /sXML for machines /x
Helper serviceConverts both ways
• String to XML• XML to String
XML is pre-parsed• So your service could
accept both if needed
9/14/2004 NVO Summer School, Aspen 2004
7
ADQL representations
ADQL/s
ADQL/x
<?xml version="1.0" encoding="utf-16"?><Select xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.ivoa.net/xml/ADQL/v0.7.4"> <SelectionList> <Item xsi:type="columnReferenceType" Table="s" Name="objd" /> <Item xsi:type="columnReferenceType" Table="t" Name="id" /> </SelectionList> <From> <Table xsi:type="archiveTableType" Archive="SDSS" Name="Galaxy" Alias="s" /> <Table xsi:type="archiveTableType" Archive="TWOMASS" Name="PhotoObj" Alias="t" /> </From> <Where> <Condition xsi:type="intersectionSearchType"> <Condition xsi:type="xMatchType"> <Table xsi:type="includeTableType" Name="s" /> <Table xsi:type="includeTableType" Name="t" /> <Nature><</Nature> <Sigma xsi:type=“realType" Value="3.5" /> </Condition> <Condition xsi:type="regionSearchType"> <Region xmlns:q1="urn:nvo-region" xsi:type="q1:circleType" coord_system_id=""> <q1:Center ID="" coord_system_id=""> <Pos2Vector xmlns="urn:nvo-coords"> <Name>Ra Dec</Name> <CoordValue> <Value> <double>180</double> <double>0</double> </Value> </CoordValue> </Pos2Vector> </q1:Center> <q1:Radius>6.5</q1:Radius> </Region> </Condition> </Condition> </Where></Select>
<?xml version="1.0" encoding="utf-16"?><Select xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.ivoa.net/xml/ADQL/v0.7.4"> <SelectionList> <Item xsi:type="columnReferenceType" Table="s" Name="objd" /> <Item xsi:type="columnReferenceType" Table="t" Name="id" /> </SelectionList> <From> <Table xsi:type="archiveTableType" Archive="SDSS" Name="Galaxy" Alias="s" /> <Table xsi:type="archiveTableType" Archive="TWOMASS" Name="PhotoObj" Alias="t" /> </From> <Where> <Condition xsi:type="intersectionSearchType"> <Condition xsi:type="xMatchType"> <Table xsi:type="includeTableType" Name="s" /> <Table xsi:type="includeTableType" Name="t" /> <Nature><</Nature> <Sigma xsi:type=“realType" Value="3.5" /> </Condition> <Condition xsi:type="regionSearchType"> <Region xmlns:q1="urn:nvo-region" xsi:type="q1:circleType" coord_system_id=""> <q1:Center ID="" coord_system_id=""> <Pos2Vector xmlns="urn:nvo-coords"> <Name>Ra Dec</Name> <CoordValue> <Value> <double>180</double> <double>0</double> </Value> </CoordValue> </Pos2Vector> </q1:Center> <q1:Radius>6.5</q1:Radius> </Region> </Condition> </Condition> </Where></Select>
select s.objd, t.id from TWOMASS:PhotoObj t, SDSS:Galaxy s where XMatch(s,t)<3.5 and Region('Circle J2000 180 0 6.5')
select s.objd, t.id from TWOMASS:PhotoObj t, SDSS:Galaxy s where XMatch(s,t)<3.5 and Region('Circle J2000 180 0 6.5')
XML
9/14/2004 NVO Summer School, Aspen 2004
8
Open SkyQuery
SkyNode interface to archives
Implements ADQL returns VOTableBasic node understands “REGION”Full node understands “XMATCH”
SkyQuery portalKnows the SkyNodes from RegistryUnderstands federated query
9/14/2004 NVO Summer School, Aspen 2004
9
Architecture
Image cutout
SkyNodeSDSS
SkyNode2Mass
SkyNodeFirst
SkyQuery
Web App
9/14/2004 NVO Summer School, Aspen 2004
10
Step-by-step
Analyze queryCost estimates
Sort by size
Execution planCross-match
Recursively from small to large db
Propagate necessary attributes onlyReturn output in VO format
SELECT o.objId, o.r, o.type, t.objId, t.j_m FROM SDSS:PhotoPrimary o, TWOMASS:PhotoObj t WHERE XMatch(o,t)<3.5 AND Region(‘CIRCLE J2000…’) AND (o.i-t.j_m)>2 AND o.type=3
SELECT o.objId, o.r, o.type, t.objId, t.j_m FROM SDSS:PhotoPrimary o, TWOMASS:PhotoObj t WHERE XMatch(o,t)<3.5 AND Region(‘CIRCLE J2000…’) AND (o.i-t.j_m)>2 AND o.type=3
9/14/2004 NVO Summer School, Aspen 2004
11
SkyNode
Metadata functionsTables, Columns, FunctionsUptime, Footprint, etc…
Query functionsPerformQuery (ADQL q)
• Simple query interface
QueryCost (ADQL q)• Cost estimate for query
ExecutePlan (XPlan plan)• Recursive probabilistic join
9/14/2004 NVO Summer School, Aspen 2004
12
Data flow
SkyQuerySkyQuery
querySkyNode 1SkyNode 1
SkyNode 2SkyNode 2
SkyNode 3SkyNode 3
http://openskyquery.net
Registry
9/14/2004 NVO Summer School, Aspen 2004
13
Cross-matching
Probabilistic spatial joinLikelihood that all measurements are actually observations of the same objectConsider all objects from all surveys
• For surveys A, B and C, there are NANBNC combinations,NX is the number of objects that meet criteria in survey X
Massive optimization problem for true positionson the sky (x,y,z)
)1()()()(1 222222
22 zyxzzyyxx
iiii
i
)1()()()(1 222222
22 zyxzzyyxx
iiii
i
,...},{ DLSSDSSi
9/14/2004 NVO Summer School, Aspen 2004
14
XMatch variants
XMatch (r, g, w)Mandatory matches in all surveys
XMatch (r, g,!!w)Look for dropouts in certain surveysRemove objects if found in w
XMatch (r, g,??w)Return matches in w if found but don’t discard if notEssentially an “outer join”
9/14/2004 NVO Summer School, Aspen 2004
15
SkyQuery portal
Hides complexitySimple web services
MethodsGet to metadata of nodesFind location of nodesSend query to one nodeSend federated query
• Create execution plan• Submit execution plan
9/14/2004 NVO Summer School, Aspen 2004
16
9/14/2004 NVO Summer School, Aspen 2004
17
And more…
Cross-match your own catalogUpload file to temporary table, e.g. MyTableUse MyData:MyTable in federated query
Browse metadata on SkyNodesSample queries of all types
9/14/2004 NVO Summer School, Aspen 2004
18
Build your own SkyNode
Download templateSkyNode written in C#Uses SQL Server 2000HTM for fast searches
Where?http://openskyquery.net
• Ready to use web service
http://skyserver.org • Templates for database
From scratchWrite it in any language
• C++, Java, C#, Perl, Python
Use ADQL Translator?
Open source .NEThttp://go-mono.org
• C# w/ DB access and WS • Runs on your favorite
OS
9/14/2004 NVO Summer School, Aspen 2004
19
Summary
Astronomical Data Query LanguageWorking VO standard for catalog accessConstructs for area and cross-match
Open SkyQueryDynamically federates astronomical archivesUses VO Registry to lookup SkyNodesOn-the-fly probabilistic cross-matchingw/ dropout and outer join capabilities
9/14/2004 NVO Summer School, Aspen 2004
20
Future Works
Things to addImplement missing outer joinFull featured region specs
Footprint servicesTest survey boundariesAutomated intersection of footprints
And more…
9/14/2004 NVO Summer School, Aspen 2004
21
Where to go
Links in the talkhttp://openskyquery.net/http://openskyquery.net/adqltranslator/ http://skyserver.org/ http://skyquery.net/
Other relevant linkshttp://us-vo.org/http://voservices.org/http://www.ivoa.net/
