DDI for the Uninitiated

ACCOLEDS /DLI Training: December 2003

Ernie Boyko Statistics Canada

Chuck Humphrey University of Alberta

Cataloguing Experiences

How many have catalogued using• MARC• Dublin Core

Cataloguing Experiences

Objectives of cataloguing• Inventory control• Location tool• Access • Distribution

Enter DDI

Documentation in a standardized mark-up languageo Data Documentation Initiative (DDI)

http://www.icpsr.umich.edu/DDI/

An Example

American Public Opinion and U.S. Foreign Policy, 1994

http://www.icpsr.umich.edu/DDI/samples/index.html

http://www.icpsr.umich.edu:8080/DDI/SAMPLES/06561.xml

http://datalib.library.ualberta.ca/accoleds/workshops/index.html

XML-DDI Benefits

The display of data documentation through a variety of style sheets;

Input for further processing, such as creating statistical package command files, conducting advanced searches, comparing variables across data files, driving data extraction engines, etc.

Data Documentation

There is a need for comprehensive data documentation that allows easily

o Finding variables • By subject groupings• By keywords, phrases or terms• By response categories (value labels)• Through linkages from the questionnaire

Data Documentation

There is a need for comprehensive data documentation that allows easily

o Tracing variables back to their origins• To a question • To a response category for a multiple

response item• To the variables from which it was computed

for a derived variable.

Data Documentation

There is a need for comprehensive data documentation that allows easily

o Understanding the corrections that must be made because of the sampling methodology

What’s next?

Let’s assume we have <ddi> compliant files … so what’s next?

What are the choices?

General Choices

Feed your own system (input from a structured file)

Look at systems using <ddi> files directly

Wait for SAS, SPSS, etc. to become XML enabled

Wait and see

Projects Using DDI

NESSTAR Health Canada -- DAIS SDA, Berkeley ICPSR’s metadata University of Minnesota US Census Bureau Harvard Virtual Data Center

Global Access, Local Support

Data Producers

Data users

NESSTAR Central Server

Data Observatory WorkbenchData Observatory Workbench

Text– Journal articles– User guides– Methodology

instructions

Tools– Finding and

sorting– Browsing– Analysing– Publishing

Data– Survey– Indicators– Administrative– Geographical

People– Email– Conferences– Experts– Discussion lists

HyperlinksHyperlinks

Data Sharing - The NESSTAR Way (in 3 Steps)Data Sharing - The NESSTAR Way (in 3 Steps)

1. Prepare your data using the Nesstar Publisher

Microdata in SPSS, SAS, Stata,Statistica, ascii or other formats

Table- or aggregated data in Excel,Ascii or other formats

Documentation/metadata in various text-formats, including XML

Data or metadata sitting in relational databases

Import

• Import data and metadata from a variety of formats

• Cut and paste additional metadata from external sources

• Use templates to enforce structure and local ”best practice”

• Organize your variables in groups and sub-groups

• Add local controlled vocabularies or thesauri

• Validate your data/metadata against the DDI and your local ”best practice”

• Output DDI-instances and/or publish to a Nesstar server

2. Publish your data to a Nesstar server

• Publish over the Web or a local area network (LAN)

• Organize your data in folders and sub-folders

• Define the access conditions of your data

• Customize the user-interface to your data

Publish

Data Store

Data Sharing - The NESSTAR Way (in 3 Steps) – (cont’d)

Data Sharing - The NESSTAR Way (in 3 Steps) – (cont’d)

Data Sharing - The NESSTAR Way (in 3 Steps) – (cont’d)

Data Sharing - The NESSTAR Way (in 3 Steps) – (cont’d)

3. Share and explore your data through a variety of interfaces

• Nesstar Explorer – a feature rich data browser (Java application)

• Nesstar light – the standard web-browser interface to Nesstar resources and services

• Choose between a variety of customized interfaces

• Develop your own customized interface or integrate Nesstar services in an existing web-application

Access

Data Store

Demo

URL: http://nesstar1-4.essex.ac.uk/nesstarlight/

Where do we go from here?

Need to start producing <ddi> filesNeed to create incentives for survey

managers to create <ddi> filesNeed to work cooperatively to

convert legacy files

What’s ACCOLEDS’ role?