IQS Data Model R3 Design Specification - noao.edu Data Model R3 Design Specification ... A. “Raw” data products (via DTS) ... MEF FITS format

NOAO IQS R3 Data Model Design Specification

IQS Data Model R3Design Specification

[NSA Document Ref #?]

Brian Thomas, Rob Seaman, Chris Miller, Tod Lauer, Phil Warner, Rob Swaters,

Andrew Cooke, Sonya Lowry, Nelson Zárate

Draft Version: 0.90Date: Jun 18, 2006

Table of Contents

0. Overview..........................................................................12. Targeted Scope.................................................................2

2.1. Targeted Data Products............................................22.2. Query/Metadata Ranking........................................5

3. IQS Data Model and Framework......................................74. Resources........................................................................15Glossary of Acronyms........................................................15Appendix A. Data Dictionary.............................................16

A1. General Data Product (“DP”) Classes....................16A.2. Image, and ImageDataProduct Descendant Classes.......................................................................................20A.3. Observation and Related Classes..........................25

A.4. Instrument Classes................................................28A.5. Device Classes.......................................................29A.6. Collection Classes.................................................33A.7. MOSAIC Classes..................................................33A.8. Other NOAO Classes............................................35A.9. VO Classes............................................................36A.10. NOAOwide Classes (meant to be interfaces)....40A.11. Some v1.1 Classes...............................................41

...........................................................................................41Appendix B. Query Usecases............................................41Appendix C. R4 model UML component figures..............42

0. Overview

This document details the primary requirements and design of the IQS (Information Query Service) DM (data model) for the R3 development cycle of the NSA (NOAO Science Archive).

The main requirements for the IQS DM are easily enumerated:

1


1. The IQS data model shall represent information held by the NSA. ‘Information’ includes all data products, scientific catalogs and proposals which the NSA holds. It also includes any security information necessary to support the NSA security service.

2. The IQS data model shall be searchable. Searchability means that the model contains all of the metadata needed to support query usecases. This metadata may be in addition to the metadata needed by requirement 1.

3. The IQS data model shall be extensible. Extensibility means that the DM framework will support a phased implementation in which changes which add greater detail to existing components or entirely new components are both possible without breaking existing functionality in the NSA.

In the following section 2, “Targeted Scope”, we shall describe in greater detail the information which the data model will model in the R3 cycle, and the priorities for the modeling of that information. In section 3, “IQS Data Model and Framework”, we shall present a general design for the IQS data model. Section 4, “Resources”, related material to this work are described.

Appendix A describes the primary query usecases for the IQS DM.

2. Targeted Scope

Requirements 1 and 2 define the general scope of the content of the IQS data model, which is at its core that the IQS data model must represent the various data products, catalogs and proposals of the NSA. Unfortunately, we face a poverty of riches in this regard, as there exists far more information of this sort available than there is available manpower to model and implement (metadata remediation is a critical issue in this regard). Thus, it is incumbent on us, the designers of the IQS, that the creation of the data model be a done in a phased implementation (Requirement 3), and that we make hard choices for which information is most important to model. Simply put, the choices boil down to selecting which data products are the most important (section 2.3) and which metadata are the most important for use in searches (section 2.2). We present first a survey of all reasonable data products which might be included in R3.

2.1. R3 Data Product Survey

2


An inventory of data holdings required for R3 is detailed in this section. To better describe these holdings, we may subdivide them into five broad categories by their source:

A. “Raw” data products (via DTS)B. Legacy holdings from SavetheBits (STB)C. Pipeline reduced data productsD. Catalog/Survey data productsE. Ancillary Data Products.

As for the data format of products in each of these categories, all data from 14 will be in either SIF or MEF FITS format. There currently is no format restriction on ancillary data sets at this time, although the IQS could (and should) enforce some minimal metadata that includes subtype of ancillary product (e.g. 'observing log', 'weather') and related data product id(s).

2.1.1. Category A Data Products

Data products in this category includes OIR imaging and spectrographic data sets plus some that we will likely best continue to describe as "other" (e.g., FabryPerot). IFUs *are* part of this category, but are not considered for inclusion in R3, in fact, only directimaging and spectrographic images are in scope for R3.

Our primary query requirement for raw data products is to be able to rapidly and reliably provide proprietary access to these products to persons designated by the PI of the corresponding proposal. The IQS data model should focus on whole night data sets from the individual telescopes. Only infrequent searches will ever occur on associated science metadata. Must have near perfect confidence in associating telescope, calendar date, and proposal with each file.

Raw data products are almost by definition file oriented.

2.1.2. Category B Data Products

STB holdings have been identified as the obvious source for highvalue postproprietary input for pipeline commissioning and associated "demos". Thus, the scope of desired category 2 data includes any items held in STB which would support these usecases.

3


While our interest in supporting raw STB holdings for R3 still remains slight, we will likely ingest raw data associated with postproprietary pipeline reduced data streams (if any). Category B data are otherwise similar to category A data, although may contain obsolete metadata formulations requiring substantial effort to remediate into a common, modern model. In order to minimize the remediation effort, only targeted query metadata shall be tackled in R3. These metadata include the “minimal” set described below.

2.1.3. Category C Data Products

These data products originate from inhouse processing pipeline and may be subdivided by their source instrument. Two instruments are in play for R3, the dual MOSAIC cameras ("MOSAIC I, II"), and NEWFIRM.

More specifically these data products include pipeline produced reduced images, catalogs, and associated masks and other processing artifacts (e.g. these later items are often referred to as “calibration data products”). The majority of these data products are formatted as FITS files. Pipelines also output catalogs.

Query for these data products includes retrieval as a group, for example, as a collection of data products for a given observing run or night, PI, core object or sky location. It is also desirable that the detected sources in the catalogs be searchable as well. Metadata on the catalog (such as telescope, observatory, PI, original image id) as well on the individual sources (sky coordinates, brightness) should be available for search.

2.1.4. Category D Data Products

The R2 holdings are all derivative of the NOAO Survey program with one or two additional data sets. The quality of the metadata is abysmal.

Query on these data products is identical to the catalog data products in category C with the proviso that the quality of the metadata in this category is abysmal. That said, one deliverable from R2 was an abbreviated guide for survey teams to produce NSA vetted metadata (i.e., measures of the PSF and photometric depth). Therefore, the IQS data model should focus on those metadata most likely to be accurate.

2.1.5. Category E Data Products

4


Ancillary data is a completely openended category. Each new category of ancillary data (a very unhelpful name) will likely require a different metadata concept let alone a different child data model. That said, we should aim to include some data streams in R3 as moreorless purely human readable content. At this time, there are NO promised data products from this category which will be included in the IQS data model.

Query requirements for this type of information includes being able to pull back humanreadable content from associated observation structure. Hence, no actual search will be done on the category E component, it will be located by finding the appropriate observation to which it corresponds. Metadata for this category are simple and boil down to providing a simple link to text (or filebased) content.

2.2. Query/Metadata Ranking

In this section, we distill from the prior section the metadata that might serve to constrain queries on information on the IQS data model into prioritized categories. The following categories are recognized :

"Minimal"

This category means that the data model has sufficient description as to allow searchand retrieval of the underlying data product based on the following query constraints(as appropriate) :

• PI Name• Proposal ID• Telescope & Observatory• Instrument/Detector• RA (specified RA falls in image area)• DEC (specified DEC falls in image area)• "Observation Time" : Epoch, DateObs, UT• Reduction level (e.g. level 1 – 4)• Data type (a 'classification' of the data product, e.g. Observation/Flat/Bias, etc.)

"Intermediate"

This category means the data model extends the content of the minimal model with the following

5


further metadata description, as appropriate, e.g.

• Bandpass or filter• Exposure time• Object name• Seeing• Photometric depth

"Full"

This model means that there are, finally, the following additional (to the intermediate model) metadata in the description:

• Capture FITS keywords in nonsearchable manner• Links to ancillary products

2.3. R3 IQS Data Model Priorities

Distilling all of the previous work, we come now to the prioritization of the data products which should be represented by the IQS data model, and the type of metadata which should describe them. 3 priority categories are recognized: “High”, “Medium” and “Low”. Details are given below:

High:

Data in this category must be done in R3. This means that manpower will be transferred to the IQS project as needed to accomplish this goal. Data products which fall into this category are the following:

i. An "Intermediate" data model for Pipelineprocessed Mosaic frames.ii. "Minimal" data model for all raw data products of NOAO direct imaging instruments (NEWFIRM, MOSAIC).iii. "Intermediate" catalog model (Survey + Pipeline Catalogs) (file retrieval only).

Middle:

Data in this category are desired for R3. Existing IQS DM team (Brian, Rob, Nelson, Chris M?) will have to tackle, but if manpower limits are exceeded these *may* be dropped.

6


i. "Intermediate" data model (level 1) for Mosaic I, Mosaic II, and NEWFIRM raw dataii. "Minimal" data model for raw spectroscopic images.iii. "Full" catalog data model (for Survey + Pipeline holdings) (has searchable items).iv. "Full" data model for Pipelineprocessed Mosaic frames.

Low:

Data products in this category are considered to be ‘gravy’. It is unlikely that the IQS team will manage to implement these, but if they do, they are swell, hard working dudes.

i. "Full" Raw data model for both direct and spectroscopic images.ii. "Minimal" data model(s) for Ancillary data (Logs, sky pix, LTO reports).iii. "Full" data model for Pipelineprocessed NEWFIRM framesiv. All other NOAO pipeline processed data (with 'minimal' model).

3. IQS Data Model and Framework

Using the rankings/choices provided in section 2, we have developed an IQS data model which appears in UML class diagrams (figures 1 through 8) below. Core classes within the framework map to the main information that the IQS must hold, e.g. “Data Product”, “Observation”, “Instrument”, “Collection” and “Proposal”. Security structures remains to be done at this point.

Various namespaces appear within the framework, and are present to denote the semanitic scope of each of the components. “NOAO” domain should apply to NOAOwide missions (at the least, all of DPP), “NOAO.NSA” domain denotes archivespecific concepts and “NOAO.MOSAIC” denotes an example project domain within NOAO but not within the NSA.

This framework is designed to encompass extension largely through the addtion of new namespaces. In the example figures of this document, the NOAO.MOSAIC namespace represents one possible extension (colors within the data model figures denote the various namespaces). An analysis to map the various FITS keywords within the data products of R3 does not appear in these diagrams, and remains to be done at a later stage.

To the extent that the model can do so, we will attempt to use the various VO standards. These appear

7


in the model figures as a separate “VO” namespace. One critical issue with adoption of VO models is the apparent conflict between the coordinate descriptions which the NOAO and VO use (e.g. WCS versus STC). Given sufficient restrictions, it may be straightforward to map from WCS into STC, however, this analysis has not yet been performed.

{NOTES on this (unfinished) section:

• Description of main components such as “Instrument”, “Observation”, “Proposal”, etc.• Table schema for IQS not part of this document (perhaps another appendix??)}

Figure 1. Data Product and related classes. Green color denotes NOAO.NSA and NOAO.NSA.Generic namespaces.

8


Figure 2. Collection class diagram. Reddish color denotes VO namespace. Green color as in figure 1.

Figure 3. Observation class diagram. Colors as in earlier figures.

9


Figure 4. Image class diagram.

10


Figure 5. Instrument classes.

11


Figure 6. Device classes.

Figure 7. Detector classes. Only CCDbased classes are currently considered.

12


Figure 8A. MOSAIC classes, subclassed from standard NOAO.NSA ones. Colors as in figure 5.

Figure 8B. MOSAIC classes, part II, subclassed from standard NOAO.NSA ones. Colors as in figure

13


5.

Figure 9. Use of VO model components : Abbreviated description of the Quantity data model.

14


Figure 10. Use of interfaces, namespace is “NOAO” versus “NOAO.NSA”.

4. Resources

This document and related document materials may be found at :http://dpopsn.tuc.noao.edu:8080/DPP/NSA/IQS/datamodel/

Glossary of Acronyms

DTS Data Transport System

DS Data Service.

15


IQS Information Query Service.

MOSAIC An imaging camera experiment (2 instances, one each located north and south at NOAO observatories).

NSA NOAO Science Archive.

STB Save The Bits.

VO Virtual Observatory.

Appendix A. Data Dictionary

Appendix A. IQS Data Dictionary

v. 0.94 – June 16 2006

This dictionary indicates the list of data elements that will be stored in the searchable datastore for data. It includes the source the source of the information (FITSHeader, DataStore, PropDB, IQS[Remediation] or Pipeline), the source fields, the data type, and whether the field is allowed null/has default value.

DPP Data Level is implicit in the class type for Data Product classes. The mapping is indicated in the description for relevant classes.

A1. General Data Product (“DP”) Classes

A.1.1. DataProduct – Describes any filerelated data held by the NSA. There are many descendant types from this class (below), but all will use the same general scheme to classify ingested data products. Classification will rely on use of a combination of FITS keywords to disambiguate incoming data into particular IQS classes. Central to this task are the FITS:PIPELINE, FITS:PROCTYPE, FITS:CALTYPE, FITS:OBSTYPE and FITS:IMGTYPE.

All Dataproduct classes share a (static) association to a particular DPP Data Level. The actual value of the DPLevel is indicated in the description for data product classes.

16


Field(source:field)

Data type(units/key)

Allowed Null?

Default Value

Description

FileName(DS:fileName)

String No N/A Name of the file the data product corresponds to. Alternatively can use the FITS:DTNSANAM field as a source.

CheckSum(DS:checkSum)

CheckSum (*checkSumId*)

Yes NULL Contains information about checksum of DP file.

UncapturedMetaData(FITS)

TextBlock (*textBlockId*)

Yes NULL Contains information about uncaptured metadata. Most likely source is FITS file header keywords. Should exclude duplicate, outdated keywords, we know of (ex. ‘PROPID’ outdated vs ‘DTPROPID’ so PROPID won’t appear)

ENID(DS:ENID)

String No N/A DS lookup collection id for the data product.

ResourceId(DS:resourceId)

String No N/A DS lookup id for the data product.

ByteSize(DS:byteSize)

Double No 0.0 Number of bytes of the data product file.

DPLevel(IQS)

Integer No 0 Data product level classifier. This is a “static” field whose value is tied to the subclass of data product.

MimeType(IQS)

String Yes NULL The mime type the data product file corresponds to (see document ref?)

Table 1: Data Product Fields

A.1.2. ObservedDataProduct (implements DataProduct) – Describes any data product that derive from an observation of the sky.

17


Field(source:field)


Allowed Null?

Default Value Description

ParentObservation(Pipeline)

Observation (*obsid*)

No N/A Contains information about the parent observation.

RightAscension(FITS:RA)

Double (degrees)

No N/A The right ascension of data product in decimal degrees for J2000 coordinate system (2000.0 epoch). Describes the centroid of field of the data product.

Declination(FITS:DEC)

Double (degrees)

No N/A Declination of the data product in decimal degrees for J2000 coordinate system (2000.0 epoch). Describes the centroid of field of the data product.

Epoch(FITS:RADECEQ)

Double(year)

No 2000.00 The epoch of the observed data product.

Region(FITS:WCS)

Object Yes NULL Description of observed sky region(from resampled image). This information will likely be fed into a characterization class/object.

ObservationDate(FITS:DATENEW, FITS:DATEOBS, FITS:OBSTIME.1)

Double (mjd) No N/A The observation date (as Modified Julian Date) at the start of the exposure. Use DATENEW field preferentially. Remediation to check that the time is actually within the declared parent observation window is needed.

ExposureTime(FITS:EXPTIME or FITS:OEXPTIME2)

Double (sec) No N/A The number of seconds of exposure time.

OriginalExposureTime Double (sec) No ExposureTime The original exposure time before reduction.

DarkTime(FITS:DARKTIME)

Double (sec) No ExposureTime The amount of dark time for the exposure.

Airmass(FITS:AIRMASS)

Double Yes NULL The airmass at the start of the exposure.

ZenithDistance Double No N/A The zenith distance at the start of the

1 Choose one, in order of preference given (DATENEW is most preferable)2 For MOSAIC processed data products, this is normalized and the original EXPTIME value is written to OEXPTIME.

18


Field(source:field)


Allowed Null?


(FITS:ZD) exposure.

EnvironTemp(FITS:ENVTEMP)

Double (C) No N/A The environmental temperature at the time the data product was obtained.

Bandpass(FITS:LAMSTART, FITS:LAMEND)

Object Yes NULL The characteristic bandpass of the data product. This information will (likely) be fed into the VO:Characterization class/object.

Table 2: Observed Data Product Fields

A.1.3. FITSDataProduct (implements DataProduct) – a data product which is a type of FITS file. For these data products the mimetype should be restricted to “file/fits”.

Field(source:field)

Data type

Allowed Null?

Default Value

Description

NumberOfExtensions(FITS)

Integer No 0 Number of FITS extensions this data product has.

Extension(FITS)

Integer Yes NULL The extension to which this data product corresponds. A value of “NULL” means the entire file.

Table 3: FITS Data Product Fields

A.1.4. ImageDataProduct (implements Image, ObservedDataProduct, MatrixQuantity) – Describes any image type data product. This class adds no new field, however, for these data products mimetype should be restricted to “image/<type>”.

A.1.5. SpectrumDataProduct (implements Spectrum, ObservedDataProduct, ListQuantity) – Describes any spectral type data product.

Field(source:field)

Data type Allowed Null?


TDB TBD TBD TBD TBD

Table 4: Spectrum Data Product Fields

19


A.1.6. TableDataProduct (implements Table, DataProduct, CompositeQuantity) – Describes any tabular data product.

Field(source:field)

Data type (units/key)

Allowed Null?

Default Value

Description

Column(FITSTABLE:<column>)

ListOf<ListQuantity> (*qid*)

No N/A Enumerated list of Quanities which hold the column name, data type ,units and values from the original data product.

Table 5: Table Data Product Fields

A.1.6. TextBlock – Describes any block of text (nonbinary characters). In v1.0 only uncaptured metadata (header information) utilizes this structure.

Field(source:field)

Data type

Allowed Null?

Default Value

Description

Value(IQS)

String No <Empty String>

The actual text block.

Encoding(IQS)

String No ASCII The encoding standard for the text.

MimeType(IQS)

String No file/text The mime type if this text.

Table 6: TextBlock Fields

A.1.7. CheckSum – Describes the md5 checksum of any data product. Field

(source:field)Data type Allowed

Null?Default Value?

Description

Value(DS:Checksum)

String No N/A The md5 checksum value.

Table 7: Checksum Fields

A.2. Image, and ImageDataProduct Descendant Classes

20


A.2.1. SingleImageDataProduct (implements ImageDataProduct) – describes any single image data product. A.2.2. CompositeImageDataProduct (implements ImageDataProduct) – Describes any image which is derived from combining other images in some fashion.

Field(source)

Data Type(units/key)

Allowed Null?

Default Value

Description

ChildImages(Pipeline)

ListOf<Image>(*qid*)

No N/A Pointer to images from which this one is derived.

Table 8: Composite Image Data Product Fields

A.2.3. AvgDomeFlatImageDataProduct (implements CompositeImageDataProduct, DomeFlatImage) – describes the average dome flat data product. This class adds no new fields but restricts ChildImages field to only hold objects of type “DomeFlatImage”.

A.2.4. AvgBiasImageDataProduct implements CompositeImageDataProduct, BiasImage) – describes the average bias image data product. This class adds no new fields but restricts ChildImages field to only hold objects of type “BiasImage”.

A.2.5. AvgSkyFlatImageDataProduct implements CompositeImageDataProduct, SkyFlatImage) – describes the average sky flat data product. This class restricts ChildImages field to only hold objects of type “SkyFlatImage”.

A.2.6. ReducedObjectImageDataProduct (implements ObjectImage, SingleImageDataProduct) – describes a reduced (Dplevel == 1) data product. This class restricts the DPLevel attribute to be ‘1’.

Field(source:field)

Data Type (units/key)

Allowed

Null?

Default Value

Description

OriginalImage(IQS)

ObjectImage (*qid*)

Yes NULL Pointer to the original image.

Table 9: Reduced Object Image Data Product Fields

21


A.2.7. RawObjectImageDataProduct (implements ObjectImage, SingleImageDataProduct) – describes raw, unreduced object (science) frame. This class adds no new fields but restricts the DPLevel field to be ‘0’.

A.2.8. DomeFlatImageDataProduct (implements DomeFlatImage, SingleImageDataProduct) – describes a single dome flat image data product. This class adds no new fields but restricts the DPLevel field to be ‘0’.

A.2.9. BiasImageDataProduct (implements BiasImage, SingleImageDataProduct) – describes a single bias image data product. This class adds no new fields but restricts the DPLevel field to be ‘0’.

A.2.10. SkyFlatImageDataProduct (implements SkyFlatImage, SingleImageDataProduct) – describes a single sky flat image data product. This class adds no new fields but restricts the DPLevel field to be ‘0’.

A.2.11. ZeroImageDataProduct (implements ZeroImage, CompositeImageDataProduct) – describes a single zero image data product. This class adds no new fields but restricts the DPLevel field to be ‘0’ and all associated child images must be of type “BiasImage”.

A.2.12. BadPixelMaskImageDataProduct (implements BadPixelImage, SingleImageDataProduct) – This class adds no new fields.

A.2.13. Image describes all images held by the NSA.

22


Field(source:field)


Allowed Null?

Default Value

Description

BadPixelMask(IQS, FITS:BPM)

BadPixelImage Yes NULL Pointer to the bad pixel mask for this image.

Bitpix(FITS:BITPIX, IQS)

Integer No 8 Number of bits per pixel in the image.

ResamplingFactor(IQS)

Integer No 1 The factor by which all pixels have been equally rebinned in each dimension.

ResamplingAlgorithm(IQS)

String No “None” The name of the resampling algorithm used.

Table 10: Image Fields

A.2.13. ObjectImageDataProduct (implements Image) – describes all images which contain science data.

Field(source:field)


Allowed Null?

Default Value

Description

DetectionCatalog(IQS)

DetectionCatalog Yes NULL Pointer to the catalog of detections from this image (post v1.0).

PhotometricDepth(FITS:PHOTDPTH)

Double (magnitude) Yes NULL The magnitude of a 5sigma detectable point source.

MagnitudeZeroPoint(FITS:MAGZERO)

Double (magnitude) Yes NULL The conversion of image flux (in counts/sec) to magnitude.

MagnitudeZeroPointErr(FITS:MAGZERR)

Double (magnitude) Yes NULL The error on the Magnitude zero point.

SkyLevel(FITS:SKYMAG)

Double (magnitudes/arcsec^2))

Yes NULL The mean sky background including the RMS variability about that mean.

SkyNoise(FITS:SKYADU)

Double (counts) Yes NULL The RMS of the sky after point sources are subtracted.

Seeing( FITS:SEEING)

Double (arcsec) Yes NULL The measured seeing for this image.

Table 11: Object Image Fields

23


A.2.14. BiasImage (implements Image) – describes all images which contain bias data.

A.2.15. ZeroImage (implements Image) describes all images which contain zero image data.

Field(source:field)


Allowed Null?

Default Value

Description

BiasImages(Pipeline)

ListOf<BiasImage> (*qid*)

Yes NULL Pointer to the bias images from which this zero is derived.

Table 12: Zero Image Fields

A.2.16. SkyFlatImage (implements Image) describes all images which contain sky flat data. This class adds no new fields.

A.2.17. DomeFlatImage (impelements Image) describes all images which contain dome flat data.

A.2.18. BadPixelMaskImage (implements Image) – describes all bad pixel mask images. One day, if there are standard flags for these masks, they will be part of this definition.

A.2.19. PupilGhostImage (implements Image) – describes Pupil Ghost calibration images.

A.2.20. FringeImage (implements Image) – describes Fringe calibration images.

A.2.21. FocusImage (implements Image) describes Focus (calibration) images.

A.2.22. PupilGhostImageDataProduct (implements PupilGhostImage, SingleImageDataProduct) – describes a single pupil ghost image data product. This class adds no new fields but restricts the DPLevel field to be ‘2’.

A.2.23. FringeImageDataProduct (implements FringeImage, SingleImageDataProduct) – describes a single fringe image data product. This class adds no new fields but restricts the DPLevel field to be ‘2’.

A.2.24. FocusImageDataProduct (implements FocusImage, SingleImageDataProduct) – describes a

24


single focus image data product. This class adds no new fields but restricts the DPLevel field to be ‘1’.

A.3. Observation and Related Classes

A.3.1. Observation (implements Collection) – describes all observations. Restricts the objects within the collection to include at least 1 ObservedDataProduct.

25


Field(source:field)


Allowed Null?


Id(FITS:OBSID)

String No N/A The observation id.

Title(FITS:OBJECT)

String No EmptyString The title of the observation.

StartDate(FITS:DATENEW, FITS:DATEOBS)

Date No N/A The start date of the observation. TBD may go into Characterization at future version.

EndDate(IQS:StartDate+ FITS:EXPTIME)

Date No N/A The end date of the observation (e.g. same as last exposure end time) – TBD may go into Characterization at future version.

Observer(FITS:OBSERVER)

Person (*pid*) No UnknownPerson Pointer to the object which holds information about the observer.

Proposal(FITS:DTPROPID)

Proposal (*propId*) No UnknownProposal Pointer to the proposal which spawned this observation.

Instrument(FITS:DTINSTRU)

Instrument (*instrId*) No UnknownInstrument Pointer to the instrument which recorded this observation.

Observatory(FITS:DTSITE)

Observatory (*observatoryId*)

No N/A Pointer to the observatory where the observation was taken.

Characterization(IQS)

Characterization (*qid*)

Yes NULL VO Characterzation of the observation. These are metadata which allow for rapid summary of the observational characteristics (v1.1)

Pipeline(Pipeline)

Pipeline Yes NULL A pointer to the pipeline which was used to reduce these data. (v1.1)

Table 13: Observation Fields

A.3.2. Pipeline – describes any pipeline used to process data products.

26


Field(source:field)

Data Type Allowed Null? Default Value Description

TBD TBD TBD TBD TBD

Table 14: Pipeline Fields

A.3.4. Proposal – describes any proposal to the NOAO.

Field(source:field)

Data Type

Allowed Null?


Title(PropDB:title)

String No <EmptyString> The title of this proposal.

ProposalId(PropDB:PropID, FITS:DTPROPID)

String No N/A Remediation to check file designation versus declared ID’s must be done by the IQS.

PI(PropDB:PI, FITS:DTPI)

Person No UnknownPerson The Principle Investigator on this proposal. As for the Proposal ID, remediation by the IQS is needed to ensure data integrity.

Abstract(PropDb:abstract)

String No <emptyString> The abstract of the proposal.

ReleaseDate Date No N/A The date upon which the data may be publically released.

Table 15: Proposal Fields

A.3.5. Observatory – describes a facility where observations take place.

27


Field(source:field)


Allowed Null?

Default Value

Description

ObservatoryId(FITS:DTSITE, DTTELESC)

String No N/A The unique identifier for this observatory.

Name(IQS)

String No N/A The humanreadable name of this observatory.

Location(IQS)

Coordinates Yes NULL The geographic location of the observatory.

Instruments(IQS)

ListOf<Instrument> (*instrumentId*)

Yes NULL The instruments which are located at this observatory.

Table 16: Observatory Fields

A.4. Instrument Classes

A.4.1. Instrument – describes any scientific instrument for obtaining data.

Field(source:field)


Allowed Null?


InstrumentId(FITS:DTINSTRU)

String No N/A The unique id of this instrument.

Name(IQS)

String No <EmptyString> The humanreadable name of this instrument.

Description(IQS)

String No <EmptyString> A description of this instrument.

HostObservatory(IQS,FITS:DTSITE)

Observatory (*observatoryID*)

Yes NULL The observatory where the instrument is located. Remediation needed to ensure file matches lookup of allowed instruments at given observatory.

Software(,FITS:CONTROLR,FITS:IMAGESWV)

ListOf<Software> (*softwareId*)

Yes NULL The list of software used to run the instrument. Source may vary based on file type.

Table 17: Instrument Fields

28


A.4.2. EMInstrument – describes an instrument which measures electromagnetic radiation. The fields of this class generally have to be looked up from FITS:DTINSTRU mapping within the IQS. Warning perhaps may be thrown if lookup value is not defined.

Field(source:field)


Allowed Null? Default Value Description

Resolution(IQS)

EMResolution (*enid*) Yes NULL Pointer to the object which describes the resolution of this instrument.

FieldOfView(IQS)

EMFieldOfView (*enid*)

Yes NULL Pointer to the object which describes the field of view of this instrument.

Sensitivity(IQS)

EMSensitivity (*enid*) Yes NULL Pointer to the object which describes the sensitivity of this instrument.

Table 18: EMInstrument Fields

A.4.3. EMResolution (implements DataProduct) – description of the resolution of an EMInstrument. This class adds no fields but DPLevel is restricted to be ‘4’.

A.4.4. EMFieldOfView (implements DataProduct) – description of the field of view of an EMInstrument. This class adds no fields but DPLevel is restricted to be ‘4’.

A.4.5. EMSensitivity (implements DataProduct) – description of the electromagnetic sensitivity of an EMInstrument. This class adds no fields but DPLevel is restricted to be ‘4’.

A.4.6. QuantumEfficiency (implements DataProduct) description of the quantum efficiency of an EMInstrument. This class adds no fields but DPLevel is restricted to be ‘4’.

A.5. Device Classes

A.5.1 Device – description of any device which might be involved in obtaining scientific data. The

29


fields of this class generally have to be looked up from FITS:DTINSTRU, FITS:CCDNAME, FITS:ADC or FITS:AMPNAME mapping within the IQS. Warning perhaps may be thrown if lookup value is not defined.

Field(source:field)

Data Type Allowed Null?


DeviceId(IQS)

String No N/A The unique id of the device.

Name(IQS)

String No <EmptyString> The human readable name of this device.

Description(IQS)

String No <EmptyString> The description of this device.

Table 19: Device Fields

A.5.2. Detector (implements Device) – description of any detector. This class adds no new fields.

A.5.3. CCDAmplifier – description of an individual amplifier within a CCD detector.

30


Field(source:field)

Data Type (units)

Allowed

Null?

Default Value

Description

Gain (FITS:GAIN)

Double (e/ADU)

No N/A The gain on the amplifier.

OriginalGain(FITS:OGAIN)

Double (e/ADU)

No Gain The original gain (before nomalization) of the amplifier.

SampleTime(FITS:AMPINTEG)

Double (ns) No N/A Double Correlated Sample time.

ReadTime(FITS:READTIME)

Double (ns) No N/A The unbinned pixel read time.

ReadNoise(FITS:RDNOISE)

Double (e) No N/A The measured read out noise from the amplifier.

Seeing(FITS:DQSEAMP)

Double (arcsec)

Yes NULL The measured seeing for this amplifier. TBD for v1.1: this may not make sense to have here rather than “ObjectImage”.

Table 20: CCD Amplifier Fields

A.5.4. CCDChip – description of individual chip within a CCD detector. This class adds no new fields.

A.5.5. CCDDetector (implements Detector) description of CCD array detectors.Field

(source:field)Data Type

(units)Allowed Null?

Default Value

Description

Temperature(FITS:CCDTEM)

Double (Celsius) No N/A The temperature of the detector.

Size(FITS:CCDSIZE)

String No N/A The description of the size of the detector (in pixels).

QuantumEfficiency(IQS)

QuantumEfficiency (*enid*)

Yes NULL A description of the quantum efficiency of the detector.

Table 21: CCD Detector Fields

A.5.6. ADC (implements Device) –

31


Field(source:field)


Default Value

Description

TrackState(FITS:ADCSTAT)

String Yes NULL The tracking state may be “null|track|preset”.

PrismAngle(FITS:ADCPAN1, FITS:ADCPAN2)

<ListOf>Double (Degrees)

No N/A The prism angles of the ADC (sequential ordering depending on instrument).

Table 22: ADC Fields

A.5.7. Dewar – (implements Device) – describes a dewar for cooling other devices within an instrument.

Field(source:field)


Default Value

Description

Temperature(FITS:DEWTEMP)

Double (Celsius) No N/A The temperature of the dewar.

FillNeckTemp(FITS:DEWTEMP2)

Double (Celsius) No N/A The fill neck temperature of the detector.

Table 23: Dewar Fields

A.5.8. FilterDevice (implements Device) – describes a device which filters signal detected in an instrument.

Field(source:field)

Data Type

Allowed Null?

Default Value

Description

Position(FITS:FILPOS)

Integer Yes NULL The filter position on the filter wheel.

Transmission(IQS)

Transmission (*enid*)

Yes NULL The object which describes the transmission of the filter. This field will have to be looked up from FITS:FILTER mapping within the IQS. Warning perhaps may be thrown if lookup value is not defined.

Table 24: Filter Device Fields

A.5.9. ProcessingDevice (implements Device) – describes a device which modifies a signal detected by an instrument. This class adds no new fields.

32


A.5.10. DispersiveDevice (implements Device) describes a device which uses dispersive process to modify a signal detected by an instrument. This class adds no new fields.

A.5.11. MagnifyingDevice (implements Device) – describes a device which magnifies signal detected by an instrument. This class adds no new fields.

A.5.12. Telescope (implements MagnifyingDevice) – describes any (groundbased) telescope. Use the DTTELESC keyword to populate the ID field for this device.

Field(source:field)


Default Value

Description

FocalLength(FITS:TELFOCUS)

Double (cm) No N/A The focal length of the telescope.

Coorector(FITS:CORRECTOR)

String (*enid*) Yes NULL The object which describes the corrector on the telescope.

Table 25: Telescope Fields

A.6. Collection Classes

A.6.1. DPCollection (implements VO.Collection class) – describes a collection of data products. This class adds no new fields but restricts member objects to be of type DataProduct.

A.7. MOSAIC Classes

A.7.1. AmpNamesCatalog (implements FITSDataProduct, TableDataProduct) – describes the MOSAIC amp names catalog. This class adds no new fields.

A.7.2. CrossTalkCatalog (implements FITSDataProduct, TableDataProduct) – describes the crosstalk catalog. This class adds no new fields.

A.7.3. StandardStarsCatalog (implements FITSDataProduct, TableDataProduct) – describes the crosstalk catalog. This class adds no new fields.

33


A.7.4. WCSDB (implements FITSDataProduct, TableDataProduct) – describes the WCS solution as a table . This class adds no new fields.

A.7.5. CrossTalkImage (implements FITSDataProduct, RawObjectImageDataProduct) – describes the crosstalk image from a MOSAIC camera. This class adds no new fields (Crosslink with Xtalk catalog??)

A.7.6. WCSImage (implements FITSDataProduct, RawObjectImageDataProduct) – describes the crosstalk image from a MOSAIC camera. This class adds no new fields (Crosslink with WCSDB catalog??)

A.7.7. ReducedImage (implements FITSDataProduct, ReducedObjectImageDataProduct) – describes a reduced single image from a MOSAIC camera. This class adds no new fields.

A.7.8. RawImage (implements FITSDataProduct, RawObjectImageDataProduct) – describes a raw single image from a MOSAIC camera. This class adds no new fields.

A.7.9. BiasImage (implements FITSDataProduct, BiasImageDataProduct) – describes a single bias image from a MOSAIC camera. This class adds no new fields.

A.7.10. DomeFlatImage (implements FITSDataProduct, DomeFlatImageDataProduct) – describes a reduced single dome flat image rom a MOSAIC camera. This class adds no new fields.

A.7.11. SkyFlatImage (implements FITSDataProduct, SkyFlatImageDataProduct) – describes a reduced single sky flat image rom a MOSAIC camera. This class adds no new fields.

A.7.12. ZeroImage (implements FITSDataProduct, ZeroImageDataProduct) – describes a reduced single zero image from a MOSAIC camera. This class adds no new fields.

A.7.13. AvgSkyFlatImage (implements FITSDataProduct, AvgSkyFlatImageDataProduct) – describes a reduced veraged (stacked) sky flat image from a MOSAIC camera. This class adds no new fields.

A.7.14. AvgDomeFlatImage (implements FITSDataProduct, AvgDomeFlatImageDataProduct) – describes a reduced averaged (stacked) dome flat image from a MOSAIC camera. This class adds no

34


new fields.

A.7.15. AvgBiasImage (implements FITSDataProduct, AvgBiasImageDataProduct) – describes a reduced averaged (stacked) bias image from a MOSAIC camera. This class adds no new fields.

A.7.16. Camera (implements EMInstrument) – a MOSAIC camera. There are two possible general subtypes, but these are realized by differing restrictions on the components which belong to it.

Field(source:field)

Data type

(units)

Allowed Null?

Default Value

Description

TelescopeFocus(FITS:TELFOCUS)

Double (cm)

No N/A The distance of the camera from the telescope focal plane (TBD: Who knows this, Frank, Rob Seaman??)

Table 26: Camera Fields

A.7.17. BadPixelMask (implements BadPixelMaskImageDataProduct, FITSDataProduct) – Indicates a MOSAIC badpixel mask, where the cells of the image have an encoded bitflag with the following values: '0/NULL' == OK pixel, '1' == Saturated; '2' == bleed trails; '4' == cosmic ray hit; '8' == nonlinear pixel. A.7.18. PupilGhostImage (implements PupilGhostImageDataProduct, FITSDataProduct) – Indicates a MOSAIC pupil ghost calibration image.

A.7.19. FringeImage (implements FringeImageDataProduct, FITSDataProduct) – Indicates a MOSAIC fringe calibration image.

A.8. Other NOAO Classes

A.8.1. Software (inherits from Agent) – a class to describe basic information about software. Identification of the software may come from a variety of keywords, depending on the software. Other Fields may either be derived from a mapping in the IQS and the software ID.

35


Field(source:field)

Data Type

Allowed Null?


SoftwareId(IQS)

String No N/A The unique id of an instance of this class.

Name(IQS)

String No <emptystring> Human readable name of software.

Description(IQS)

String No <emptystring> Description of the software.

CompileDate(IQS, FITS:ARCONWD)

Date Yes NULL The date of the last compile of the software.

CompileParams(IQS)

String Yes NULL A descriptive string of the relevant parameters used to compile the software.

RuntimeParams(IQS, FITS:ARCONWM, FITS:ARCONGI)

String Yes NULL A descriptive string of any runtime parameters used.

Table 27: Software Fields

A.9. VO Classes

A.9.1. AstroObject (implements Object) – any astronomy object (viewable in the sky).

Field(source:field)


Default Value

Description

CoordinateSystem CoordinateSystem (*coordId*)

Yes NULL The coordinate system that descirbes the location of this object in the sky.

Table 28: VO Collection Fields

A 9.2. CompositeQuantity (implements Quantity, Object) – a structure used to build complex quantities which have heterogeneous properties (which are of type Quantity).

A.9.3. AtomicQuantity – (implements Quantity) – A single, atomic scientific value.

36


A.9.4. ListQuantity – (implements Quantity) – A List of scientific values. A.9.5. MatrixQuantity – (implements Quantity) – A matrix of values.

A.9.6. Characterization – (implements CompositeQuantity) – a complex object used to summarize important characteristics of an observation. These are organized along the lines of energy, space and time versus a dimensional axis.

A.9.7. Collection – A generalized collection of Objects.

Field(source:field)


Default Value

Description

Id String No N/A The unique id of this collection.

Publication(IQS)

Publication (*publicationsId*)

Yes NULL Description of publications that are related to this collection (v1.1)

Curation(IQS)

Curation (*curationId*)

Yes NULL Description of the curation (history) of this collection. (v1.1)

Objects(IQS)

ListOf<Object> (*objectId*)

No N/A The objects which belong to this collection.

Table 29: VO Collection Fields

A.9.8. UCD – unified content descriptor. An object used to attach semantic content (via an identifying string) to an object.

A.9.9. Coordinates – an object used to specify any location within space and time.

A.9.10. CoordinateSystem – a class used to specify the system in which measurements (e.g. coordinates) are made.

A.9.11. Object – any object which has semantic meaning within the Virtual Observatory.

A.9.12. Person (inherits from Agent) – a class to describe basic information about people. Information may be parsed out of the proposal database or the FITS header in various places, but some IQS

37


remediation will be needed to divide it up appropriately.

Field(source:field)



FirstName(IQS)

String No <emptystring> The first name of the person.

LastName(IQS)

String No <emptystring> The last name of the person.

Affiliation(IQS)

String No <emptystring> The affiliation of the person.

PersonId(IQS)

String No N/A The unique id which identifies this person.

Email(IQS)

String YES NULL The email address of this person.

Table 30: Person Fields

A.9.13. Curation – a class to describe the genesis and history of revisions of a parent class.

Field(source:field)


Default Value

Description

Origin(IQS)

CurationChange(*curationCId*)

No N/A Pointer to the first change” to the collection (e.g. origin/creation information).

Changes(IQS)

<ListOf>CurationChange (*curationCId*)

No N/A

Table 31: Curation Fields

A.9.14. Publication – a class to describe the related and source references of a parent class. Reserved for v1.1 development.

38


Field(source:field)


Default Value

Description

TBD TBD TBD TBD TBD

Table 32: Publication Fields

A.9.15. CurationChange – describes a change made to a collection. Presently fairly simple and is designed to track changes to the membership of the collection (rather than changes to metadata within and about the collection.. although the description field may be used to keep a humanreadable account of these types of changes).

Field(source:field)


Default Value

Description

Agent(IQS)

<ListOf>AgentObject (*agentId*) No N/A Pointer to one or more agents which caused this change to the collection.

Description(IQS)

String No <emptystring>

The humanreadable description of the change.

Date Date No N/A The date upon which the change was made.

PriorObjects <ListOf>Object (*objectId*) Yes NULL Pointer to the list of prior objects held by this collection. If NULL, it is assumed that no change in the state of objects occurred.

Table 33: CurationChange Fields

A.9.16. Quantity (inherits from AstroObject) – structure used to describe scientific quantities. This structure implicitly underlies any quantity in the other fields, e.g. Anything which has an attached unit, datatype and value. Optionally Quantities also allow description of errors and semantic meaning via UCDs.

39


Field(source:field)


Default Value

Description

UCD(IQS)

UCD Yes NULL Semantic value of the quantity.

DataType(IQS)

DataType No String The data type of the quantity. Allowed subtypes not described in this document include 'float', 'string', 'integer'. Subtypes have additional metadata.

Unit (IQS)

String Yes NULL The units of the quantity. System of units is TBD at this point, but SI seems likely choice.

MetaData(IQS)

<ListOf>Quantity Yes NULL Quantities which describe metadata about this quantity. Common quantities include “Accuracy” quantities.

Table 34: Quantity Fields

A9.17. Accuracy (inherits from Quantity) – structure used to record the accuracy of measurements which have errors. This structure implicitly underlies any of the numerical measurements in the other fields described in this dictionary when they have errors. The type of accuracy is designated via a UCD value on the quantity. Accuracy class adds no new fields but is restricted to be of the same type/dimensionality as the host quantity it describes.

A.9.18. Agent – A simple class to provide a means to allow People and Software to act as agents of change within the various aspects of curation.

A.10. NOAOwide Classes (meant to be interfaces)

A.10.1. Spectrum – describes an astronomical spectrum.

A.10.2. ObservedData – describes astronomical data which have been observed.

A.10.3. Table – describes any tabular data.

40


A.10.4. Image – describes any image data.

A.11. Some v1.1 Classes (simply to show we have anticipated/accounted for the following)

A.11.1. MosaicedObjectImageDataProductA.11.2. StackedMosaicedObjectImageDataProductA.11.3. StackedObjectImageDataProductA.11.4. DetectionCatalog A.11.5. Detection A.11.6. PointSourceA.11.7. ExtendedSourceA.11.8. PublicationA.11.9. Pipeline

Appendix B. Query Usecases

SIAP Query must be supported. Refer to VO spec on required fields.

41


Appendix C. R4 model UML component figures

Figure 11. Publication DM. For R4.

42


Figure 12. Detection DM components. For R4.

43

Documents

IQS Data Model R3 Design Specification - noao.edu Data Model R3 Design Specification ... A. “Raw” data products (via DTS) ... MEF FITS format