2004-09-15NVO Summer School, Aspen Center for Physics1 Publishing and Resource Discovery with Registries Ray Plante Gretchen Greene

2004-09-15NVO Summer School, Aspen Center for Physics 1

Publishing and Resource Discovery with Registries

Ray PlanteGretchen Greene


All about Registries

• Overview of the Registry Framework

• Publishing to the NVO

• Visits to Registries (publishing)

• VOResource: Resource Metadata in XML

• Visits to Registries (searching)

• IVOA Standard Registry Interface

• Exercise: query registry in an application


The role of Resource Registries

• Used to discover and locate resources—data and services—that can be used in a VO application

• Resource: anything that is describable and identifiable.– Besides data and services: organizations, projects,

software, …– Presently concerned with simple set of resource types

• Registry: a list of resource descriptions– Expressed as structured metadata

to enable automated processing and searching


An Overview of Data Discovery

• You can search the main NVO registry to find resources based on descriptive criteria

• NVO Registries are “coarse-grained”– You can find organizations, archives, catalogs– Won’t find images, celestial objects, table

records

• Registry framework contains multiple registries:– searchable registries– publishing registries


Local PublishingRegistry Local

SearchableRegistry

FullSearchableRegistry

Local PublishingRegistry


DataCenters

VOProjects

SpecializedPortals & Services

Registry Framework



SearchableRegistry




DataCenters

VOProjects


Registry Framework

harvest(pull)



SearchableRegistry




DataCenters

VOProjects


Registry Framework

harvest(pull)

Cross-harvest



SearchableRegistry




DataCenters

VOProjects


Registry Framework

harvest(pull)

selectiveharvesting

Cross-harvest



SearchableRegistry




DataCenters

VOProjects


ClientApplications

searchqueries

Registry Framework



SearchableRegistry




DataCenters

VOProjects


ClientApplications

searchqueries

Registry Framework


NVO Public Registries

Registry URL Searchable?

Publishing?

STScI/JHUNVO Registry

http://nvo.stsci.edu/voregistry/ Yes Yes

Caltech Carnivore http://nvo.caltech.edu:8080/carnivore/ Yes Yes

NCSARegistrationPortal

http://nvo.ncsa.uiuc.edu/nvoregistration.html

No Yes

Private Publishing Registries• HEASARC• CDS

Only support harvesting protocol


Overview of Publishing

• Resources are published if one can use NVO facilities to find them.

• How to Publish to the NVO http://us-vo.org/pub/files/PublishHowto.html

– Multiple layers of publishing• Starts with registry description of resource• Data Access Services

Incremental exposure for incremental effort

– Who are you? How you publish depends on what you want to publish.

• An individual with a small data collection• An archive center• Someone with a cool service


Small collections:VO-ready Repositories

• Repositories that allow users to deposit data to share with community– Guarantee long-term storage, availability

• Automatic support for VO publishing mechamisms– Entries into NVO Registry– Support for standard services:

Cone Search, SIA, SSA, SkyNode

• Currently available Repositories– Images: NCSA Astronomy Digital Image Library

http://adil.ncsa.uiuc.edu/– Spectra: Spectrum Service for the VO

http://voservices.net/spectrum/

• More public repositories are expected to emergeCheck NVO website (http://us-vo.org/) for latest

http://adil.ncsa.uiuc.edu/

http://voservices.net/spectrum/

http://us-vo.org/


Persistent Archives:Tools for Federation

• Registering your resources with a VO publishing registry– Enter description into registration form at one of the

available NVO registries:• STScI/JHU Registry: http://nvo.stsci.edu/voregistry/• NCSA Registration Portal:

http://nvo.ncsa.uiuc.edu/nvoregistration.html• Caltech Carnivore:

http://nvo.caltech.edu:8080/carnivore/

– If you have a large number of resources to register, you can run your own registry on your own site

• Caltech Carnivore: http://nvo.caltech.edu:8080/carnivore/

• NCSA VORegistry-in-a-Box http://nvo.ncsa.uiuc.edu/VO/software/


• Which registry should I register with?– Behavior is slightly different– Different strategies for making it “easier” to

register multiple resources– More uniform support is underway

• Support for standard services is strongest– Cone Search, SIA, SkyNode– SkyNode: go to STScI or Carnivore– SSA: go to NCSA

• Your feedback is valuable!

Caution: Construction Ahead


• What can/should you register?– Should: your Organization

• Declares yourself as a publisher with an ID– Should: your Data Collection

• Users at least know how to access it via a Browser– Can: your existing services

• Browser-based services: e.g. search page• Traditional CGI services• Web Services

The next level…• Implement and register one or more standard services

– Cone Search– Simple Image Access– SkyNode– Simple Spectral Access*

*standard still in development

Persistent Archives:Tools for Federation


Cool Services:Integrating with the VO

1. Register your service at a registry• Currently as a generic resource• Improved support for non-standard services coming• (immediate future: contact NVO project)

2. Integrate support for standard VO formats, schemas• FITS and VOTable• Standard Data Model schemas (emerging)

• VOResource, Space-time Coordinates, Spectra

3. Implement Standard Support Interface• a standard in development for:

Self-description, tracking health and usage


A word about Identifiers…

• IVOA Identifier: a globally-unique URI identifying a resource

Ex: ivo://adil.ncsa/targeted/SIA

• Required as part of a registered resource description

• As publisher, you control what it looks like• Two components:

– Authority ID: e.g. adil.ncsaDefines a namespace for identifiersOwned by a single publishing organization

– Resource Key: e.g. targeted/SIAName for the resource unique within the namespaceEncourage re-use of local identifiers


Visits to the Publishing Registries

• Publishing Registries– STScI/JHU Registry:

http://nvo.stsci.edu/voregistry/

– NCSA Registration Portal: http://nvo.ncsa.uiuc.edu/nvoregistration.html

– Caltech Carnivore: http://mercury.cacr.caltech.edu:8080/carnivore/

• Recommend you stick with one registry– Authority IDs are currently “stuck” to a single registry

• Can’t do something? Contact us!


Resource Metadata: XML Schema

• Classes of Resources– Generic Resource– Extensions: e.g.

DataCollection, Service, ConeSearch, SimpleImageAccess, SkyNode

• Organized into separate schemas:– Core resource metadata: VOResource

– Various extensions schemas containing specific types

• Capable of describing…– Data centers, research organizations, missions, observatories– Data collections, archives – VO standard services: Cone Search, Simple Image Access,

Simple Spectral Access, SkyNode– Existing Browser/CGI-based services– Web Services


Searching the Registry

Available Interfaces:– Keyword Search– Intermediate Interfaces

• Form-based• Under-development• e.g. http://nvo.caltech.edu:8080/carnivore/search

– Advanced SQL/ADQL/XQuery interfaces

Registry URL Searchable?

Publishing?

STScI/JHUNVO Registry

http://nvo.stsci.edu/voregistry/ Yes Yes

Caltech Carnivore http://nvo.caltech.edu:8080/carnivore/ Yes Yes

NCSARegistrationPortal

http://nvo.ncsa.uiuc.edu/nvoregistration.html No Yes


IVOA Standard Registry Interface

• IVOA Working Draft: 2 parts– Harvesting:

sending descriptions from publishers to searchable registry– Searching

• Searching– methods

• keywordSearch(string words, boolean combineByOr)• search(ADQLWhere constaints)

– Returns a list of VOResource descriptions– Advanced searching with ADQL:

• Just the “where” part—i.e. search constraints--of ADQL• In place of column names, use XPath to VOResource

element– Curation/Publisher like ‘%NASA%’– Query extensible to any VOResource XML extension– Maps readily to registry implementations based on RDBMS

or XML-DB.


Why might a client use the standard search interface?

• Uniform interface to all VO Registries

– Not dependant on a single registry

• Direct relationship between information you are querying and information you get back.

• Extensible to any type of resource description

• Re-use of ADQL

Custom interfaces to Registries:

• Registries can provide extended functionality

– More advanced capabilities

– Simpler interfaces for specialized purposes

• Client toolkits can provide simplifying interfaces

– Support for ADQL/s

– XPath aliases


What you might use this week

• STScI or Carnivore Registry portals to search for resources

• Any of the registration portals to register resources

• Software for querying registry from an application– Java: see java/dev/nvoregistry– IDL: see idl/VOlib_0.1 (call_registry.pro)– Python: see python/PythonVOClientLibrary.html

(VORegistry.py)– PHP: see in php/web: kwdSearch.php, phpVOlib

Documents

2004-09-15NVO Summer School, Aspen Center for Physics1 Publishing and Resource Discovery with Registries Ray Plante Gretchen Greene