Making Metadata Work For Your National Climate Data Center Asheville NC February 2006 Lynda Wayne US Federal Geographic Data Committee / GeoMaxim Mike

Making Metadata Work For Your

National Climate Data Center • Asheville NC • February 2006

Lynda Wayne US Federal Geographic Data Committee / GeoMaxim

Mike MoellerNOAA Coastal Services Center / Perot Systems

NCDC Making Metadata Work For You FGDC / NOAA

Objectives

After the workshop, students can:

effectively read and comprehend metadata

make the business case for metadata

discriminate between minimal and quality metadata

develop a metadata template that uses a range of mandatory, conditional, & optional CSDGM elements

select and access metadata software

locate and access online resources


What IS Metadata?

Data ‘reporting’ WHO created the data?

WHAT is the content of the data?

WHEN was it created?

WHERE is it geographically?

HOW was the data developed?

WHY was the data developed?


What IS Metadata?

title

supplemental information

abstract

time period

author

sources

(file) size


What IS Metadata?

entity

attributes


Let’s Make Metadata

Turn to your neighbor and document the following:

Titlename

Theme Keywordswork & interests areas

Supplemental Informationunique interest, event, or quality


Metadata as a Data Component

MetadataTitleOriginatorPurposeAbstractGeographic ExtentTime Period of ContentSources UsedProcess DescriptionPositional AccuracyEntities and Attributes…….

phenomenaphenomena map / image / databasemap / image / database metadatametadata

abstraction abstraction

content context

DATA


DATA

map

(con

tent

)m

etadata

(context)

Metadata: Building Block of theNational Spatial Data Infrastructure

access

share

use

• search• download

• describe• publish

• assess• apply


Metadata: Building Block of theNational Spatial Data Infrastructure

…a set of actions and new ways of accessing, sharing and using geographic data that enables far more comprehensive analysis of data to help decision-makers choose the best course(s) of action.

National Spatial Data Infrastructure

map

(con

tent

)

metad

ata (context)

NSDI

use

share

access


Metadata: Building Block of theYour Own Spatial Data Infrastructure

map

(con

tent

)m

etadata

(context)

My SDI

use

share

access


Metadata the Awful

Geospatial Data Producers perceive metadata as:

time consumingexcessivefor the benefits of others

Geospatial Managers perceive metadata as:

non-productivecostly (time, training, software)

derive classify

collect

planimetric imagery

analysis

alternativecommittee

review

PLAN

charette

metadata?

metadata?

metadata?

metadata?

metadata?

metadata?


Metadata the Magnificent

Metadata can support:data management

project management

If it is:considered a component of the data

created during data development

populated with rich content

derive classify

collect

planimetric imagery

analysis

alternativecommittee

review

PLAN

charette

meta

meta

meta

meta


Data Management via Metadata

DataAccountability

Discovery

& Re-use

Maintenance

& Update

DataLiability


Data Management:Data Maintenance and Update

MaintenanceAre the data current?

• older than?• prior to significant change?

Are the data valid?• newer sources?• better methods?

UpdateContact informationDistribution policies, availability, pricingURLSISO 19115 Topic CategoriesNew derivations


Data Management:Data Discovery and Re-use

Marketing metadata in the past…

If you create metadata, other people can discover your data

If you create metadata, you can find your own data

New and improved message…


Data Management:Data Discovery and Re-use

Data discovery bythemes / attributes

geographic extent

temporal character

methods

contributors

quality

Discoverable data is usable data


Data Management:Data Accountability

Exercise in data comprehension:

What do you know?

What don’t you know?

What should you know?

Are you willing to associate yourself with the metadata record?



Repeatable process:methodsvariablesparameters

Defensible process:public participationdecision-making

GIS-savvy public

Metadata can be crafted to preserve individual accountability in team projects



Return on InvestmentCatalog assessment:

# times occurs as inhouse source# times occurs as external sourcesignificance of source contribution

To facilitate discovery:standardized titling schemespreferred citation w/in Use Constraints


Data Management:Data Liability

Metadata is declaration of:Purpose

originator’s application of the data? ontological pedigree

Use Constraintsappropriate and inappropriate utility of the data

Completenesswhat is missing from the data

Distribution Liabilityexplicit liability of the producer and assumed liability of

the consumer

What to do…

What not to do…


Project Management via Metadata

PROJECTPLANNING

DELIVERABLES

COMMON LANGUAGE

PROJECT MONITORING


Project Management:Project Planning

Project design document:description / intentgeo & temporal extentsource dataattribute requirements

Benefits:expectations outlinedintegrated into processprovide medium to record progress


Project Management:Project Monitoring

Monitor:

status

QA/QC measures

changes

Requires that the metadata be actively maintained and reviewed.

time

mile

ston

es


Project Management:Common Language for Multi-participants

Common reference for:descriptors & parameterskeywords, vocabularies & thesauricontact informationattributesdistribution information

If reviewed regularly by all participants,improved opportunity for coordinating:

source dataanalytical methodsnew information

tem

plat

e


Project Management:Contract Deliverables

Metadata is a data component therefore part of any data deliverable

Clearly specify your metadata requirement via a metadata:

classification/validation schema, e.g. ‘level’ of metadata

template

specification manual

Provide a sample record if possible


Conclusions

If created with utility in mind, metadata can serve multiple roles:

and this is only human utility of metadata…

archivist preserve lineage

accountant preserve investment

developer document process

manager track progress

contributors communicate

consumer locate, assess, access data


Data Management:Reality Check

Discussion Questions:

Which of these data and project management metadata applications are you most likely to use and why?

Are there additional metadata applications that would serve your organization?


Making Metadata Part of the Process

New tools enable us to better integrate metadata creation into the data development process



If metadata were collected throughout the data process…

1. more accurate (no guessing)

2. more details

3. better decision-making.

meta

dat

a



But what can I do to better incorporate metadata into the data development process?


Make the business case:preserve data investments

limit data liability

manage data resources

find new data resources

easier data transfer

more efficient data distribution

step one:build adminstrative support


Emphasize individual benefits:reduce workload..in the long term

field fewer data inquiries

document personal contributions

step two:build technical support


Support your staffinclude in job descriptions & performance measures

provide

staff support

tools

training

step two:build technical support


Organizational templatesidentify pertinent fields

populate fixed fieldsstandardized language

distribution methods

standards used

build source and contact libraries

step three:build templates


step four:distribute the effort

Iden

tific

atio

n In

form

atio

n

• title

• abs

tract

• pur

pose

• tim

e pe

riod

of c

onte

nt

• key

words

• com

plet

enes

s

Attrib

ute

Info

rmat

ion

• attr

ibut

e la

bels/

dom

ains

Data

Qua

lity

Attrib

ute

Info

rmat

ion

• com

pila

tion

accu

racie

s

• com

pila

tion

proc

esse

s

• sou

rce

files

• fie

ld c

heck

acc

urac

ies

• ana

lytica

l pro

cess

es• a

ttrib

ute

valu

es

Data Processing Data Planning Data Analysis

manager technicians analysts


Managers Can WRITE metadata

data planning fieldstitle abstract purposegeographic extent keywordsand many more…

and manage the effortmetadata coordination

metadata enforcement

step five:lead by example


Establish policies:mandate use of standards and templates

develop boilerplate metadata deliverable language for data contractors

require units to publish their metadata

publish metadata SOP to document policies and procedures

step six:policies and procedures


Implementation Issues

What are the roadblocks to metadata implementation within your organization?


Operational Content

To fully realize the operational value of metadata it is important to utilize standardized vocabularies and provide rich content.

The following content recommendations support:internal data accessproject monitoringISO transition geodata.gov participation

…see Handout ‘Metadata Quick Guide’


Operational Content:Internal Data Access

TitleMini abstracts

Minimum: where, what, when

Best Practice: who, why, resolution, filename, source

Supplemental InformationInclude internal referencing information

• project acronyms

• regional/disciplinary references


Operational Content:Internal Data Access

Theme and Place Keywords• internal and external thesauri/vocabularies

Process Contacts• record as part of Process Steps

Distribution: Resource Description• internal data/document references such publication

numbers, PINs or other cryptic references


Operational Content:Project Monitoring

Attribute Accuracy

Positional Accuracy

Process Steps

•Prescribe content (template)•Assign responsibilities•Actively maintain/review


Operational Content:ISO Transition

Theme KeywordTheme Keyword Thesaurus: ISO 19115 Topic Categories

Include one or more Topic Categories:boundaries oceans health transportation economy societyelevation farming locationbiota structure environmentutilitiesCommunication intelligenceMilitarygeoscientificInformation inlandWatersimageryBaseMapsEarthCover planningCadastre

climatologyMeteorologyAtmosphere


Operational Content:geodata.gov

Online Linkagerepeatable fieldused to determine content typemetadata, downloadable data, WMS, ArcIMS

Linkage Type Include

metadata server standard .html

downloadable data ftp:// or .zip, .tgz, .gz, .dxf, .eoo, etc.

WMS ‘getmap’ request w/ layer name, vers., image format, SRS

ArcIMS http://<server>/image/<service_name>


Writing Quality Metadata

Practice what I preach..

Using an actual data set, write a quality:Title:

Abstract:

We’ll use these later when creating metadata.


How Do I Create Metadata?

Steps to Success:

1. Understand the organization and content of the CSDGM

2. Learn to use the CSDGM Workbook (aka The Greenbook)

3. Select a metadata editor tool

4. Apply the CSDGM to your data


CSDGM Overview

The CSDGM documents:

digital geospatial datadigital geospatial data

imagery digital land surveysGIS files geographic databases

MANY fields will not be applicable to your data

Your missionYour mission is to understand the standard well-enough to model it for your own use.


CSDGM Overview

Standard is organized into seven primary sections

There are three ‘supporting’ sections that repeat within the primary sections:

citation• a standardized way to cite the data and source data

time period information• a standardized way to record dates

contacts• a standardized way to document contacts


CSDGM Overview

1. Identification InformationGeneral bibliographic information including:

title, originator, data contact, status, date, time period of content, abstract, purpose, keywords, geographic location

2. Data Quality InformationLineage and data assessments

sources, process methods, accuracy, data processing contact


CSDGM Overview

3. Spatial Data OrganizationData format:

vector, point, raster

4. Spatial Reference InformationCoordinate system parameters:

horizontal / vertical coordinate system, projection, datum


CSDGM Overview

5. Entity and Attribute InformationDatabase design

entities, attributes, domains, description of data values

6. Distribution InformationHow to acquire the data

distribution contact, available formats, online distribution website, costs


CSDGM Overview

7. Metadata Reference InformationGeneral information about the metadata record itself

metadata contact, metadata standard used, metadata creation date, metadata review date


CSDGM Overview

But that wasn’t enough for some people like Ted..

Extensions:added elements to the CSDGM

• Remote Sensing Extension

Profilescontent requirements for specific elements

may included added elements as well• Biological Profile

• Shoreline Profile


Mine the Metadata Exercise

Use your new understanding of the CSDGM to navigate through the sample metadata record and find answers to a set of questions.


CSDGM Workbook

AKA The Greenbook, it provides:background information about the development and purpose of the CSDGMexplanation of the organization of the CSDGMFAQs about specific elementssample values for specific elementsinstructions on creating Profiles and Extensionssample Metadataa glossary of terms

the CSDGM Workbook is your best metadata friend


Metadata Tools

How do I determine the best metadata tool?Does your organization provide a tool that meets specific data management objectives?Does your GIS software provide an internal metadata editor?Do you need a tool that supports one or more CSDGM Profiles or Extensions? Do you need to document data resource beyond those managed within your GIS? Do you need a tool that is easily distributed to partners? What features are most important to you?

auto-capture of information bundling of metadata with data creation and use of templates easy to use interface robust help systems and tutorials


Metadata Tools

What metadata editors are available?Freeware:

tkme/xtmeUSGS, supports all profiles/extensions, well-maintained

MetaVistUSFS, desktop, supports biological profile

MERMAidNOAA CSC, online utility supports biological and shoreline profiles


Metadata Tools

What metadata editors are available?Commercial

SMMS (Intergraph GeoMedia and stand-alone)

auto capture from multiple file types, contact/source libraries, data exploration utilities

ArcCatalogmetadata ‘synchronized’ w/data, ability to enclose

documents, metadata bundled with data export

Blue Angelhighly customizable, supports non-geospatial metadata


Metadata Tools

What other metadata tools are available?Parsing and Validation

Metadata Parser (mp)checks for conformance to CSDGM specified:

• conditionality (are all mandatory elements are populated?) • data formats • element domains

mp batchbatch processor shareware from Intergraph

Chew N Spit (cns)• Rectifies indentation• Inserts capitalization and underbars

FGDC ‘compliant’ metadata must pass mp!


Metadata Tools

What other metadata tools are available?Parsing and Validation

error2htmprovides user-friendly interpretation of mp error report

dbfmetaconverts dbase to CSDGM Entity Attribute elements

mq provides interface for editing large, diverse collections of

metadata


Metadata Tools

Formshardcopy or online


Metadata Validation Tools

mp - Metadata Parser • Checks for CSDGM syntax

– Element names– Mandatory elements– Element content (domains and logical consistency)

• Embeds tags for NSDI Clearinghouse Distribution

cns – Chew ‘N Spit • Rectifies indentation• Inserts capitalization and underbars

FGDC ‘compliant’ metadata must pass mp!


CSDGM Elements


CSDGM Elements


CSDGM Elements


CSDGM Elements

Turn topage 35

in theGreen Book


CSDGM Elements


CSDGM Elements

Calendar DatesYYYYMMDD

Time of DayHHMMSSSS

CoordinatesLat/Lon Decimal Degrees

Network addresses & file namesService://hostname:port/path/filename


CSDGM Production Rules

16

What’s Mandatory? What’s Not?

Mandatory if Applicable - must be provided if the data set exhibits the defined characteristic.

Optional - provided at the discretion of the data set producer.

Mandatory - must be provided.

MeaningData

ElementCompoundElement

10/06/95



24

Using the Graphics to Make Decisions - 1

All elements are colored yellow, so all are mandatory and must be reported.

M M / A O

CompoundElement 1

CompoundElement 1.1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2

10/06/95



26


Compound Element 1 is mandatory.

Compound Element 1.1 is optional. The producer must decide if it wants to provide the element.

If yes, Data Elements 1.1.1 and 1.1.2 are mandatory.

If no, do not report Compound Element 1.1, Data Element 1.1.1 or 1.1.2, and skip to Data Element 1.2.

Data Element 1.2 is mandatory. M M / A O

CompoundElement 1

CompoundElement 1.1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2

10/06/95



28


Compound Element 1 is mandatory.

Compound Element 1.1 is mandatory.

Data Element 1.1.1 is mandatory.

Data Element 1.1.2 is mandatory if applicable. The producer must decide if the data set has the characteristic defined by the data element. If yes, then Data Element 1.1.2 is mandatory. If no, then Data Element 1.1.2 is not reported.

Data Element 1.2 is optional. M M / A O

CompoundElement 1

CompoundElement 1.1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2

10/06/95



30


Compound Element 1 is mandatory if applicable. If not applicable to the data set, do not report any elements. If applicable, it is mandatory and:

Compound Element 1.1 is mandatory.

Data Element 1.1.1 is mandatory if applicable. If not applicable, do not report it. If applicable, it is mandatory.

Data Element 1.1.2 is mandatory.

Data Element 1.2 is optional. M M / A O

CompoundElement 1

CompoundElement 1.1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2

10/06/95




CSDGM

Section One: Identification

Elements you use to ‘shop’ for a data set of interest:

WHO ? title, originator, publication date

WHAT ? abstract, keywords

WHERE ? geographic extent, browse graphic

WHEN ? time period of content

HOW? access/use constraints, status


CSDGM


CSDGM

Section Eight: Citation Information

Traditional elements used to document publications. Used for the data set you are documenting and source data sets.

Originator responsible party

Publication Date date made available

Title what, when, where and more

Presentation Form GIS file, imagery, dbase, etc.

Online Linkages view, download, WMS, etc.


CSDGM

Section Nine: Time Period Information

Fields used to document various dates referenced through out the metadata including:

Identification Information: Data Citation Publication DateData Quality Information: Source Data Citation Publication DateData Quality Information: Source Time Period of ContentData Distribution Information: Data Availability

Options include:

Single DateRange of DatesMultiple Dates


CSDGM

Section Ten: Contact Information

Contact information for individuals referenced through out the metadata record including:

Identification Information: Data Set ContactData Quality Information: Process Step ContactData Distribution Information: Distribution ContactMetadata Information: Metadata Contact


CSDGM


CSDGM

Section Two: Data Quality

How was the data set developed?

• Source Information

• Data processing

What checks were made of the data set?

• Logical Consistency

• Positional Accuracy

• Attribute Accuracy


CSDGM


CSDGM

Section Three: Spatial Data Organization

Point, Line, or Vector?


CSDGM


CSDGM

Section Four: Spatial Reference Information

Coordinate System


CSDGM


CSDGM

Section Five: Entity & Attribute Information

Database Description


CSDGM


CSDGM

Section Six: Distribution Information

How can I get the data set?• Distribution Contact Information

• Available Formats (digital and hardcopy)

• Standard and Custom Order Processes

• Online option

• Offline media

• Distribution Liability

• Fees


CSDGM


CSDGM

Section Seven: Metadata Information

Elements you use to ‘shop’ for a data set of interest:

WHO wrote the metadata record

WHAT metadata standard was used?

WHEN was the metadata created/updated?

Is the metadata available?

metadata access and use constraints


CSDGM


Transition to ISO Metadata

Standardized Metadata

Content Standard for Digital Geospatial Metadata (CSDGM) Vers. 2.0

remains current US standard

International Organisation for Standardization (ISO)

standard completed

US adoption in progress


1. Content

CSDGM similar and new elements

Domains and conditionality

Essential or core metadata

2. Format

UML / XML

Multi-level metadata

What’s new with ISO metadata?


CSDGM Similar ElementssameMetadata Standard Name

new terminology and definitionsCSDGM = Originator

ISO = Responsible Party

w/ Role = originatorcustodianownerpublisher….

ISO Metadata: Content


New (non-CSDGM) ElementsInternational elementsData / Metadata Language

Extended elementsCSDGM = geographic and temporal extent

ISO = geographic, temporal, & vertical extent

Lacking elementsISO Topic Category – 19 standardized subject categories to facilitate data discovery



Domains and ConditionalityFewer mandatory elementsCore contains only 7 mandatory elements

Fixed domains and code lists in place of free text to control vocabulary and improve discoveryResponsible Party - Role and Topic Categories represented as codes

• Role = Originator (006)

• Topic Category = Transportation (018)

More optional elements




ISO Topic Categoriesenable discovery by topic within geodata.gov

enable notification of Data Community Stewards

facilitate translation of CSDGM metadata to ISO formatboundaries oceans health

transportation economy societyelevation farming locationbiota structure environmentutilitiesCommunication intelligenceMilitarygeoscientificInformation inlandWatersimageryBaseMapsEarthCover planningCadastreclimatologyMeteorologyAtmosphere


ISO Core Metadata22 basic metadata elements

comprise essential metadata for all geospatial datadigital data

maps and charts

text

Element conditionalitymandatory

conditional

optional

all national profiles of ISO must include the core



ISO Metadata: Core Elements

Dataset title Spatial representation type

Dataset reference date Reference system

Dataset responsible party Lineage statement

Geographic location On-line resource

Dataset language Metadata file identifier

Dataset character set Metadata standard name

Dataset topic category Metadata standard version

Spatial resolution Metadata language

Abstract Metadata character set

Distribution format Metadata point of contact

Additional extent info (vert / temp) Metadata date stamp

**mandatory


Building the national profile

Upgrade conditionality

Extend codelists

Establish codelists

core elements extensions

ISO Metadata: New Content


ISO Metadata: UML Model


ISO Metadata: 19139 XML Schema


the good news….

XML supports consistent and robust implementation via software tools

• Format and exchange- current implementation - txt, doc, db file, etc.

- exchange format is txt

• Internal metadata harvesting and other automated features

ISO Metadata: Implications


more good news….

Multi-level metadataData Series - wetlands

Dataset - Paducah wetlands

Feature Type - marsh

Feature Instance - Mandy’s Marsh

Attribute Type - salinity

Attribute Instance - actual % water salinity



the not-so-great news….

Standard can no longer be ‘read’ by most geospatial data developers

greater dependency on tools

lack of direct contact difficult for some metadata creators - analogous to moving from command line to GUI



Crosswalks drafted for:CSDGM Vers. 2.0

Dublin Core, others

Conversion software in development:FGDC sponsored effort with Intergraph /SMMS for standalone

ESRI has drafted for ArcCatalog

What about legacy metadata?


Now….1. Continue creation using CSDGM vers.2 !!!

2. Use ISO-poised softwareESRI ArcCatalog

Intergraph / SMMS

MetaD, FGDC/Polytechnic University Catalunya

3. Add Topic Categories as theme keywords with ‘ISO 19115 Category’ as keyword thesaurus

4. Monitor FGDC website for news, materials, and training opportunities

How do I prepare?


Soon…1. Educate and inform your organization that a

migration, and assistance, is in the future

2. Develop a metadata conversion plan using crosswalks and conversion software

3. Adopt ‘CSDGM vers. 3.0’ when finalized

How do I prepare?



Why Metadata?

Utilizing External Data….


Why metadata?

Be a part of the BIG picture

Federal Geographic Data Committee (FGDC)- coordinates national geospatial data development

National Spatial Data Infrastructure (NSDI)- creation and distribution of seamless national geospatial data

Global Spatial Data Infrastructure (GSDI)- creation and distribution of seamless global geospatial data


Why metadata?

NSDI Framework Data- cooperatively-designed data dictionaries for seven key

‘reference’ themes

• Elevation and Bathymetry

• Hydrography

• Geodetic Control

• Cadastral

• Transportation

• Governmental Units

• Orthoimagery


state / local national

Why metadata?

NSDI Clearinghouse Network- international network of metadata distribution ‘nodes’

• FGDC• EROS• ESRI• NRCS• NOAA CSC• ALASKA GDC

internationalNCDC

NGDC

NWS

CSC

query

available metadata records

user portal gateways nodes

GeospatialOne-Stop

Other Metadata ServicesWMS • OARS • WAFS


Why metadata?

NSDI Geospatial One Stop

metadata and links to data data view and webmapping servicesdata acquisition plans - marketplace thematic communities

…..fast tracking the NSDI



National Geospatial Programs Office

FGDC

Geospatial One-stop

National Map

data

access

coordinationnation

al •

sta

te •

loca

l • p

riva

tem

etadata • services • comm

unity

policies • standards • partnerships


Who creates metadata?

OfficiallyFederal organizations producing digital geospatial data

Federally funded projects that produce digital geospatial data

Morally and practicallyAnybody that creates digital data

It’s the right thing to do!

Documents

Making Metadata Work For Your National Climate Data Center Asheville NC February 2006 Lynda Wayne US Federal Geographic Data Committee / GeoMaxim Mike