MadrigalA Virtual ObservatoryforAMISR ScienceJohn Holt and Bill RideoutMIT Haystack ObservatoryAMISR Science Workshop, 2006

What is Madrigal?A collection of standards-based local databases that share metadata:

What is the local Madrigal database?UserWebinterfaceWeb services APIPython APIMatlab APIScriptsFrom anywhere on internetDatabase standard Cedar file formatMetadata standard Madrigal standardMadrigal derivation engineReal-time and historical data

How is Madrigal a Virtual Observatory?Searches from local Madrigal site can link to any other siteUser chooses whetherthe search is local orincludes all MadrigalsitesImplemented via shared,standard metadata

In what other way is Madrigal database a virtual observatory?Entire database of multiple types of instruments can be searched at once using a complex queryExample: Give me all measurements of Te and Ti on April 3, 2004 between longitudes 280 and 310 and on closed field linesQueries can be built on any parameter, measured or derived (such as Tsyganenko model)

How is this done?All registered Madrigal databases share standard metadataQueries use metadata to filter out experiments that dont need to be analyzed (fast)Then the remaining experiment is analyzed at the data level (slower)How much can be filtered quickly depends on whats in the metadata

Madrigal can hold data from many types of instrumentsISR

GPS TEC

Madrigal derivation engine adds to VO feelMadrigal data file Spatial and temporal conversions

Geophysical data (IMF, Kp, DST. etc)

Magnetic field models / MSIS

Empirical ISR modelsMadrigalderivationengineThe user sees the file as containing many additional parametersExample: Te from ISR (measured) Te from empirical model (derived)

Underpinnings of MadrigalCedar database formatDefines parameters allows instruments to add parameters unique to their instrumentsFile format requires start time, end time, instrumentFor each measurement time, 1D or 2D dataNew parameters can be added to format by working with NCARMadrigal metadataOnly instrument and time range no spatial or parameter information

Strengths of Cedar database formatUnambiguous time in the prologUse of standard parameters allows the derivation of many other parameters not in the data file - *very important*Human intervention not needed to interpret dataMadrigals core is a list of methods in a standard form with input parameters and output parameters, and an engine that chooses which methods to callEasily extensible by adding new methodsExample less than 48 hours after request, Tsyganenko field model added to MadrigalAllows model, geophysical data to appear to be in every data fileIncludes IGRF, MSIS, CGM, Tsyganenko, Kp, Dst, F10.7, IMF, etc.Makes the normally hard metadata problem easy

How hard is Madrigal to install?Arecibo experience Madrigal installed and data imported in < 2 weeks

SummaryA key to the success of AMISR will be prompt, unified and user-friendly access to data from the radar, its supporting instrumentation and other radar facilitiesMadrigal is a virtual observatory with numerous desirable featuresOpen Source you can contributeWell-defined metadata standardReal-time capabilityinteractive Web interfaceGlobal search capabilityProvision for including information such as html pages, publications, figures, movies and images Interactive plottingComplete Web-services interfaceHigh-level user friendly tools for entering new datasetsHelp in setting up a new Madrigal site or entering new datasets into Madrigal is available

Thank You

Summary of lessons learnedA robust, well-defined data format with time and space information part of the standard makes building virtual observatories far easier than the top down approachThe Cedar database format is a strong platform to build onThe existence of such a standard makes possible an open-source tool such as Madrigal to be easily deployed for any new instrument, saving development time

Two approaches to VOsTop down:Build custom interfaces to existing databasesBottom up: Encourage adoption ofstandard local database software (i.e., Madrigal)VOVOCustom interfacesUniform interfacesBoth approaches can be pursued