Images of the Babbage Engine taken from the
Computer History Museum
website
http://www.computerhistory.org
Learning to Make Your Data Work Harder Than
You Do
Do it once, do it right
same data for different tools and users in the most automated way possible, not recreating it
Its about transforming and repurposing the
Three Words:
USE PLANNING DOCUMENTS
In the age of Google, why have fielded data?
•More efficient for both data entry and for systems to search, retrieve and ingest
•Parsed, discretely fielded data can be recombined mechanically for a variety of outputs and uses, including XML
Data flow and useCataloging utility (relational database)
InstitutionalDigital Repository
Web 2.0XMP (in images)
RSS feedsWebsites
tools
UsersXMP (in images)
Flat ExcelPDF
Delivery SystemsARTstor
MDIDCONTENTdm
LUNA Insight etc.
XML
xslt
xslt
xsltx
slt
Data flow and useCataloging utility (relational database)
InstitutionalDigital Repository
Web 2.0XMP (in images)
RSS feedsWebsites
tools
UsersXMP (in images)
Flat ExcelPDF
Delivery SystemsARTstor
MDIDCONTENTdm
LUNA Insight etc.
XML
xslt
xsltxslt
A pithy answer to “why relational?” (for cataloging)
• Message from Jan Eklund to VRA-L, Feb 20, 2008, subject: Re: CONTENTdm and metadata (now posted on VRA web under Resources)– Complexity: “complexity cannot be captured efficiently
in a flat data model because basically you have to leave space in every record to accommodate the most complex object you will ever encounter. This adds up to a lot of wasted space, and wasted space means more money…”
– Consistency: “all the descriptive data about the work is entered once, and every image that shows this work inherits the same information”
Excel sample (“flat file” output)
Notice that each row represents an image file and conflates the work and image records (repeats the information about the work for each image).
Each repeating value (like Artist) must have a column reserved for possible use.
XML sample—more like a flexible accordion—expands as needed
ER Diagrams show related tables
Authority record
Numeric key
All the information about the agent is supplied from this file on the basis of the numeric key
A note field is possible for every Core 4 element
Repeating values are supported for each element (using portals or subforms)
Numeric key
“indexed” value (in this case the sort name)
“display” value done to CCO recommended formatting. Note that the Agent Nationality is supplied automatically here by theLink (numeric key) to the Agent Authority
Creation of an XSLT
• XSLT stands for Extensible Stylesheet Language Transformation. XSLT is XML-based. You can use a stylesheet to take an XML document and turn it into plain text, PDF documents, web pages, or to import fielded data into other applications.
• In this case sample it creates a tab delimited file and specifies the field names in the headers when it is converted into Excel (extra step is to preserve diacritics with Unicode)
VCat• Begun 2004 with goal of being fully
relational, VRA Core 4 and CCO compliant and capable of Core 4 XML output—that goal met
• Reality in 2010—flattened Excel is still the lingua franca. The XML export stylesheet was used as basis to create a flattened Excel export
• So, there are now 2 exports (XSLTs) that can provide XML and Excel
VCat folderVCat folder
You can create as many stylesheets as you like for specific purposes
.xsd=schema; .xsl=stylesheet; .xml=document
Data flow and useCataloging utility (relational database)
InstitutionalDigital Repository
Web 2.0XMP (in images)
RSS feedsWebsites
tools
UsersXMP (in images)
Flat ExcelPDF
Delivery SystemsARTstor
MDIDCONTENTdm
LUNA Insight etc.
XML
xslt
xsltxslt
Sample XML output (small clip)
Data flow and useCataloging utility (relational database)
InstitutionalDigital Repository
Web 2.0XMP (in images)
RSS feedsWebsites
tools
UsersXMP (in images)
Flat ExcelPDF
Delivery SystemsARTstor
MDIDCONTENTdm
LUNA Insight etc.
XML
xslt
xsltxslt
Creation of a mapping document to a standard
• Flattened Core 4
• Flattening repeating fields
Sample Excel output(a small clip)
Data flow and useCataloging utility (relational database)
InstitutionalDigital Repository
Web 2.0XMP (in images)
RSS feedsWebsites
tools
UsersXMP (in images)
Flat ExcelPDF
Delivery SystemsARTstor
MDIDCONTENTdm
LUNA Insight etc.
XML
xsltxsltxslt
Data Dictionary
Creation of a Data Dictionary for each tool
• Data dictionaries help set the display look of the data that the patron sees—this can be customized and where the use of “index” and “display” values of Core 4 are crucial
• They also set the things the patron does not see—under the surface search parameters, like using early and late date (index fields) to do “fuzzy” searching
Display data is like publishing: arranges data attractively for user
Difference in user display and cataloger mode
We are used to seeing this in OPACs
Sample MDID Data Dictionary
Set import, field labels, thumb captions, sorting, searching, keyword searching, DC mapping for cross collection searching, advance search pop-down lists, etc.
VCat ARTstor
VCat-ARTstor Data Dictionary
Concatenate fields; “prepend” global information or labels
Set display order of grouped fields
Clustered (grouped) fields; ability to concatenate information or “preprend” information
Data Dictionary settings in action
Setting thumb captions
In this case, ARTstor has a floating information window; in other tools this would be a place to use the INDEX value of the name (which is the
sort value instead of the display)
Also allows user to change thumb sort
Users use keywords
Same data, different tools and users
• The following 3 slides show the same data prepared for our stock and royalty-free publishing site hosted on SmugMug—the educational data is reduced and compressed into an IPTC-like caption and keywords only and written into the image header. This means it can be seen using Cooliris as well (which is fun).
The Cooliris “wall” of images with captions
In general, you seek to adapt the xslt stylesheet and the data dictionary as needed rather than changing the data that you produce centrally—that should remain consistent to a standard and you should seek the ability to express that in a standard xml schema, as well as any other stylesheets. Hopefully future tools will ingest from the standard schema.
Thinking about how to present grouped or complex objects
• Think about this upfront so that your cataloging can help facilitate groupings—use of data values
• Also think about what needs to be consistently fielded data (including local field structure) to help order and sequence manuscripts and time-based works
• These will require local fields and data dictionary mapping and settings
Pragmatic, phased approaches
• Being able to find and update older records easily and consistently into full Core 4 when it is better supported in tools
• Supplying the data now in some useful form
“Collection” record