Computers in Libraries 2003, Washington, DC Digital Library Collections & Services Landscape & Strategies Roy Tennant California Digital Library

Computers in Libraries 2003, Washington, DC

Digital Library Digital Library Collections & ServicesCollections & Services

Landscape & StrategiesLandscape & Strategies

Roy TennantRoy Tennant

California Digital LibraryCalifornia Digital Library


Questions, Questions, Questions, Questions, QuestionsQuestions

• You will leave with more You will leave with more questions than answersquestions than answers

• If I do my job right, they will be If I do my job right, they will be the the rightright questions questions

• Feel free to ask questions as Feel free to ask questions as we go alongwe go along


““It was the best of times, it It was the best of times, it was the worst of times…”was the worst of times…”

— A Tale of Two Cities, Charles Dickens


The Common The Common PerceptionPerception


The RealityThe Reality• Too many information sources with Too many information sources with

which to copewhich to cope• A lack of human assistance at the time A lack of human assistance at the time

and place neededand place needed• Not enough ways to filter, sort, and Not enough ways to filter, sort, and

otherwise narrow in on what is neededotherwise narrow in on what is needed• Access is limited to what is free, or Access is limited to what is free, or

what has been purchased or rented on what has been purchased or rented on behalf of a clientelebehalf of a clientele

• Many useful resources are only Many useful resources are only available in printavailable in print


The Most Commonly The Most Commonly Proposed SolutionProposed Solution


Digital Library MythsDigital Library Myths

• Having everything in digital form Having everything in digital form will solve our information access will solve our information access problemsproblems

• Soon (or eventually) everything Soon (or eventually) everything will be digitalwill be digital

• Any collection of digital objects Any collection of digital objects can be a digital librarycan be a digital library

• Everyone agrees about what Everyone agrees about what comprises a “digital library” and comprises a “digital library” and how to build one — all we must do how to build one — all we must do is do the work is do the work


A Digital Library Is…A Digital Library Is…• A collection of digital objects and/or A collection of digital objects and/or

information that is:information that is:– SelectedSelected– OrganizedOrganized– Made AccessibleMade Accessible– PreservedPreserved

• A set of services that help you to find A set of services that help you to find and use those objects and informationand use those objects and information

• Often supported by a physical Often supported by a physical collection and collection and alwaysalways by professional by professional staffstaff


OutlineOutline• Themes:Themes:

– Library CatalogsLibrary Catalogs– Digital Library Collections Digital Library Collections – Digital ReferenceDigital Reference– Libraries as PublishersLibraries as Publishers– Cross-Database SearchingCross-Database Searching

• Technologies:Technologies:– XMLXML– MetadataMetadata– InteroperabilityInteroperability


Library CatalogsLibrary Catalogs

• We seem to be unable to We seem to be unable to provide an easy and effective provide an easy and effective information locating toolinformation locating tool

• Keep in mind that only Keep in mind that only librarians like to librarians like to searchsearch, , everyone else likes to everyone else likes to findfind

• We are even failing at things We are even failing at things we have explicitly tried to dowe have explicitly tried to do

• Let’s take a look at the Let’s take a look at the evidence…evidence…






Typical SearchesTypical Searches

• Known ItemKnown Item• ““A Few Good Things”A Few Good Things”• ComprehensiveComprehensive


Typical Searches: Typical Searches: Known ItemKnown Item

• The good: searches can be The good: searches can be limited to a particular field: limited to a particular field: author, title, etc.author, title, etc.

• The bad: limiting to a The bad: limiting to a particular field doesn’t always particular field doesn’t always act the way you expectact the way you expect




Typical Searches:Typical Searches:“A Few Good Things”“A Few Good Things”

• The one type of search we The one type of search we have so far have so far ignored ignored in library in library system designsystem design

• A type of search that we can A type of search that we can do something about do something about todaytoday

• Bring Google-style relevance Bring Google-style relevance to library catalogs (e.g., for to library catalogs (e.g., for union catalogs, sort by number union catalogs, sort by number of holding libraries)of holding libraries)


Typical Searches: Typical Searches: ComprehensiveComprehensive

• Most library catalogs hide many things Most library catalogs hide many things available via regional cooperative or ILLavailable via regional cooperative or ILL

• It is difficult, if not impossible, to search It is difficult, if not impossible, to search all appropriate journal databasesall appropriate journal databases

• Most libraries do not provide good Most libraries do not provide good access to gray literature and web sitesaccess to gray literature and web sites

• Subject headings are often unintuitive, Subject headings are often unintuitive, and catalogs give no guidanceand catalogs give no guidance

• Catalogs give no chapter-level access to Catalogs give no chapter-level access to book contentbook content





The Rescue of PrintThe Rescue of Print

• Many library users want only that Many library users want only that which is convenient (read digital)which is convenient (read digital)

• Print resources are, therefore, Print resources are, therefore, increasingly overlooked (I call increasingly overlooked (I call this the “convenience this the “convenience catastrophe”)catastrophe”)

• We must fight this trend by We must fight this trend by enriching our catalog records enriching our catalog records with tables of contents, indexes, with tables of contents, indexes, book covers, etc. to entice users book covers, etc. to entice users to print booksto print books


Digital Library Digital Library CollectionsCollections

• Applying the digital versions of Applying the digital versions of traditional library activities traditional library activities (selecting, acquiring, etc.) to (selecting, acquiring, etc.) to put library collections onlineput library collections online

• One of the most well-understood One of the most well-understood of digital library activities of digital library activities

• Many examples, resources, Many examples, resources, consultants, and vendors are consultants, and vendors are available to help a library available to help a library digitize their collectionsdigitize their collections



Digital ReferenceDigital Reference

• Putting the human help where it’s Putting the human help where it’s needed — onlineneeded — online

• Software is now available that Software is now available that provides for:provides for:– Queuing of patrons with audible alertsQueuing of patrons with audible alerts– Chat between librarian and userChat between librarian and user– Push web pagesPush web pages– Form sharingForm sharing– HighlightingHighlighting– Saved and/or emailed transcriptsSaved and/or emailed transcripts– StatisticsStatistics







XMLXML

• A method of creating and A method of creating and using tags to identify the using tags to identify the structurestructure and and contentscontents of a of a document — document — not how it should not how it should be displayedbe displayed

• The tags used can be arbitrary The tags used can be arbitrary or can come from a or can come from a specificationspecification

• There are two types: There are two types: well well formedformed and and valid…valid…


Well-Formed XMLWell-Formed XML

• Follows Follows generalgeneral tagging rules: tagging rules:– All tags begin and endAll tags begin and end

• But can be minimized if empty: <br/>But can be minimized if empty: <br/>

– All tags are properly nested:All tags are properly nested:• <author><name>Mark <author><name>Mark

Twain</name></author>Twain</name></author>

– All attribute values are quoted:All attribute values are quoted:• <subject scheme=“<subject scheme=“LCSHLCSH”>Music</subject>”>Music</subject>

• Software can check to make sure a Software can check to make sure a given document follows these basic given document follows these basic rulesrules


Valid XMLValid XML• Uses only specific tags and rules as Uses only specific tags and rules as

codified by one of:codified by one of:– A document type definition (DTD)A document type definition (DTD)– A schema definitionA schema definition

• Only the tags listed by the schema or Only the tags listed by the schema or DTD can be usedDTD can be used

• Software can take a DTD or schema and Software can take a DTD or schema and verify that a given document adheres to verify that a given document adheres to the rulesthe rules

• Editing software can prevent an author Editing software can prevent an author from using anything except allowed tagsfrom using anything except allowed tags


Transforming XMLTransforming XML

• XML Stylesheet Language — XML Stylesheet Language — Transformations (XSLT)Transformations (XSLT)– A markup language and programming A markup language and programming

syntax for processing XML syntax for processing XML – Used to transform XML to another Used to transform XML to another

format (e.g., to HTML for delivery to format (e.g., to HTML for delivery to standard web clients) or from one set standard web clients) or from one set of tags to anotherof tags to another

• An XML An XML parserparser• A method to bring all the pieces A method to bring all the pieces

together if serving to the web (e.g., together if serving to the web (e.g., CGI program, Java servlet, etc.)CGI program, Java servlet, etc.)


MetadataMetadata• Structured information about Structured information about

an object or collection of an object or collection of objectsobjects

• Types:Types:– DescriptiveDescriptive– AdministrativeAdministrative– StructuralStructural– PreservationPreservation


Metadata StandardsMetadata Standards

• Dublin Core:Dublin Core: a set of basic fields a set of basic fields primarily for systems primarily for systems interoperabilityinteroperability

• MODS:MODS: a MARC-like bibliographic a MARC-like bibliographic formatformat

• METS:METS: a structural standard for a structural standard for encapsulating a digital object or encapsulating a digital object or set of digital objects, including set of digital objects, including one or more segments of one or more segments of descriptive and/or administrative descriptive and/or administrative metadatametadata


Libraries as PublishersLibraries as Publishers

• Libraries are increasingly Libraries are increasingly becoming involved with becoming involved with publishing activitiespublishing activities

• University libraries are University libraries are capturing scholarship before it capturing scholarship before it leaves campus, and making it leaves campus, and making it freely available to allfreely available to all

• Two examples:Two examples:– RepositoriesRepositories– Book publishingBook publishing


RepositoriesRepositories• Two flavors:Two flavors:

– InstitutionalInstitutional– Topic or type focusedTopic or type focused

• Characteristics:Characteristics:– Often author-maintained; therefore Often author-maintained; therefore

metadata may be of uneven quality/quantitymetadata may be of uneven quality/quantity– Often compliant with the Open Archives Often compliant with the Open Archives

Initiative harvesting protocolInitiative harvesting protocol

• Benefits:Benefits:– Captures a grey literature not always Captures a grey literature not always

collected by librariescollected by libraries– If OAI-compliant, can be “crawled” and If OAI-compliant, can be “crawled” and

indexedindexed



Dspace screen shotDspace screen shot



Aggregation Portal

SubjectPortal

The OAI ModelThe OAI Model

MIT

UC

Stanford




Book PublishingBook Publishing

• Academic libraries and university Academic libraries and university presses are teaming up:presses are teaming up:– Libraries provide technical expertise, Libraries provide technical expertise,

online access, persistence, professional online access, persistence, professional collection managementcollection management

– University presses provide editing, print University presses provide editing, print publication, imprimatur, marketingpublication, imprimatur, marketing

• Case Study: University of California Case Study: University of California Press and the California Digital Press and the California Digital LibraryLibrary


Bookencodedin XML

Information

Transformation

Web Server

XSLTStylesheet Presentation

XHTML Document

(no displaymarkup)*

HTMLStylesheet

(CSS)

* Dynamic document


OtherMetadata

LibraryCatalog

METSRepository

MODS record

DC record

Structure

RecordCreationProgram

OtherMetadata

LibraryCatalog

METSRepository

Project

Profile

FieldExtractionProgram

MODS record

DC record

Structure

IndexingSoftware


SearchIndex

OtherMetadata

LibraryCatalog

METSRepository

Project

Profile


MODS record

DC record

Structure

IndexingSoftware


SearchIndex

SearchIndex

Imagemetadata

OtherMetadata

LibraryCatalog

METSRepository

Project

Profile


MODS record

DC record

Structure

IndexingSoftware


SearchIndex

Query & Profile

OtherMetadata

LibraryCatalog

METSRepository

Project

Profile


MODS record

DC record

Structure

IndexingSoftware


SearchIndex

XSLT

Resultsin XML

OtherMetadata

LibraryCatalog

METSRepository

Project

Profile


MODS record

DC record

Structure

IndexingSoftware


SearchIndex

XSLT

Resultsin XML

Object request

OtherMetadata

LibraryCatalog

METSRepository

Project

Profile


MODS record

DC record

Structure

IndexingSoftware


SearchIndex

XSLT

Resultsin XML

XSLTMETSrecordin XML

OtherMetadata

LibraryCatalog

METSRepository

Project

Profile


MODS record

DC record

Structure

IndexingSoftware


SearchIndex

XSLT

Resultsin XML

Book segment requested

OtherMetadata

LibraryCatalog

METSRepository

Project

Profile


MODS record

DC record

Structure

IndexingSoftware


SearchIndex

XSLT

Resultsin XML

Segment extracted and transformed to HTML

XSLT


XML & XSLT XML & XSLT DemonstrationDemonstration


Cross-Database Cross-Database SearchingSearching

• Necessary for the user to:Necessary for the user to:– Not have to know which database to Not have to know which database to

searchsearch– Not have to go to each database Not have to go to each database

individually to searchindividually to search– Not have to know the particular Not have to know the particular

commands to search in each databasecommands to search in each database

• An incredibly complex problem An incredibly complex problem that will likely take years to come that will likely take years to come close to solving wellclose to solving well


A Future I Would Like to A Future I Would Like to SeeSee


A Model for Achieving ItA Model for Achieving It

RepositoriesWeb

SearchEngines

LibraryCatalogs

JournalArticle

Databases

DigitalLibrary

Collections

The Integration Engine

The UserInterface

OnlineAssistance

http://searchlight.cdlib.org/cgi-bin/searchlight

Source: ARL Statistics


InteroperabilityInteroperability• The digital library “holy grail”The digital library “holy grail”• Main requirement: widespread Main requirement: widespread

adoption of specific standards adoption of specific standards and protocolsand protocols

• Progress:Progress:– XML as the basic syntaxXML as the basic syntax– OAI provides a harvesting modelOAI provides a harvesting model– METS, MODS, and DC are key METS, MODS, and DC are key

metadata standardsmetadata standards– Technologies such as Web Services Technologies such as Web Services

provide real-time interoperabilityprovide real-time interoperability


Understanding the Understanding the LandscapeLandscape

• We must provide access to more We must provide access to more resources than ever beforeresources than ever before

• Many are digital, some are notMany are digital, some are not• Some are interoperable, many Some are interoperable, many

are notare not• We need to find ways to build We need to find ways to build

unified user services from a unified user services from a disparate collection of resourcesdisparate collection of resources

• Tools and strategies for doing Tools and strategies for doing this are becoming availablethis are becoming available


TrendsTrends

• More and faster change (“change is the More and faster change (“change is the only constant”)only constant”)

• Better control of, and access to, Better control of, and access to, historically difficult to access materials historically difficult to access materials (e.g., working papers, data sets, etc.)(e.g., working papers, data sets, etc.)

• More publication options:More publication options:– Digital repositoriesDigital repositories– Institution-based peer-reviewed publication Institution-based peer-reviewed publication

avenues (journals, books, etc.)avenues (journals, books, etc.)

• A greater diversity of material types:A greater diversity of material types:– Multimedia, data, etc.Multimedia, data, etc.


A Few Technologies A Few Technologies and Trends to Watchand Trends to Watch

• Repository systemsRepository systems• Systems for online peer review and Systems for online peer review and

publicationpublication• New kinds of “cataloging” (e.g., New kinds of “cataloging” (e.g.,

Dublin Core, METS, MODS)Dublin Core, METS, MODS)• XMLXML• Open Archives Initiative Open Archives Initiative • Web servicesWeb services• Cross-database search toolsCross-database search tools


The Right Questions to AskThe Right Questions to Ask

• What do we need to do to serve our What do we need to do to serve our users better?users better?

• How can we build an infrastructure How can we build an infrastructure that can be used for a variety of that can be used for a variety of purposes?purposes?

• How can we better integrate access How can we better integrate access to print and digital material?to print and digital material?

• How can we interoperate with other How can we interoperate with other systems and services?systems and services?

• What should we stop doing so we What should we stop doing so we can do what is more important?can do what is more important?

Documents

Computers in Libraries 2003, Washington, DC Digital Library Collections & Services Landscape & Strategies Roy Tennant California Digital Library