CWSpace Archiving Courseware Websites to DSpace Using a Content Packaging Profile & Web Services © Digital Library Federation Spring Forum, 2006 April

CWSpaceArchiving Courseware Websites to DSpace

Using a Content Packaging Profile

& Web Services

©

Digital Library Federation Spring Forum, 2006

April 11, 2006 — Austin, Texas

William Reilly, Rob Wolfe — MIT Libraries

2006

DLF Spring 2006 MIT's CWSpace 2

Project Overview

Vision: Increase value in Institute publication: OCW– Also: Increase

value in Open Source platform: DSpace

* CLE = Collaborative Learning Environment

Goal: Archive OCW to MIT’s DSpace– Also: Make content

available to CLEs *

CWSpace: MIT iCampus project, nearing completion

Method: Interoperability of systems– Use Content Packaging standard– Use Web Services standard


Perceived Need: inter-operability™

• Obvious observation: Higher Ed. Technologies increasingly seeking integration with Library & Repository holdings

• Not so obvious(?): Libraries seeing their charter to include collection, preservation, dissemination of Teaching & Learning materials (?)


InterOp: What’s Available


InterOp: What’s Achievable 1 of 3


InterOp: Challenges 2 of 3


InterOp: Possible Resolution? 3 of 3


OCW Materials in DSpace = New Ground

Educational Materials Digital Archive

New kind of content: educational, teaching, learning

POLICY - Cost/merit evaluation of archival treatment. Preservation and IP considerations

New emphasis (ideally) on re-use, re-purposing: aggregation, disaggregation

GRANULARITY - Challenge to digital archive; how flexible can it be? (cf. CMS)

New kind of structure, composition: website, compound digital object, rich metadata

COMPLEXITY - Need to package multi-file content, plus metadata

Content PackagingNew needs for system-to-system access: CLEs, Image Tools

BACK-END / FRONT-END - Need for networked access

Web Services


Content Packaging


Content Packaging: Steps Toward a Profile

• Model OCW Content

• Choose Packaging Specification

• Profile That Specification


Content Package

Package Interchange File (PIF)

• Usually a .zip file• Contains a manifest• Manifest references

files (some in, some out of the package)


Object Models

• Mapping one world to another

• Content Package design considerations

• Use, re-use, subsequent disseminationOCW DSpace

SIP DIP

…

…


OCW to DSpace mapping


Which Packaging Spec.?

• Content packaging specifications are generic loose envelopes to carry content, metadata

• Choice of package is largely determined by user community

• Dspace to use METS internally• Higher Education uses IMS

Content Packaging

• METS• IMS-CP• MPEG-21

DIDL• XFDU


METS and IMS-CP


IMS-CP & OCW Content Model


IMS-CP & DSpace Object Model


OCW to IMS-CP to DSpace


Profiling Package Standards

Every Implementation of IMS-CP or METS will:

• Constrain the Standard• Extend the Standard• Interpret the Standard

The sum of these adjustments is the implementation profile

The Standards themselves encourage profiling in a systematic way:

IMS Application Profile Guidelines Technical Manual

http://imsglobal.org/ap/apv1p0/imsap_techv1p0.html

METS Profile Schema and Guidelines

http://www.loc.gov/standards/mets/mets-profiles.html











Application Profiles










Profiled Content Package…

Now with Package defined…

…need to move that across the network from producer to consumer… DSpace

Content Provider


Web Services


DSpace dev: LNI & PKGers


Lightweight Network Interface

0. What Was The Problem?

1. What Is the LNI?

2. DSpace & WebDAV

3. Example of Use: Submit OCW

4. Future Development…


What was the Problem…?Content Providers wanted…• Networked Access: Programmatically access the DSpace object model, but not

running on the DSpace server

Basic (Lightweight Network Interface (LNI))

[ Access, Deposit, Limited Mods. ]

• Submit

• Retrieve Item w. Metadata

• Build own U/I

• Collection Mgt, Map Items

Advanced (possible future LNI dev.)

[ Replace, Withdraw, Delete ]

• Specialist metadata

• Automated metadata

• Quality control checks

• Format migration …

DSpace

Content Provider


What Is the LNI?

The "Lightweight Network Interface" is:

• A remote API to control DSpace

• It's essentially WebDAV: Extensions to HTTP

– A SOAP (with WSDL) equivalent is provided

• It's a mapping: —

– a "complete and comprehensive" view of the DSpace public object

API…

(the "business logic" layer)

– …as mapped to WebDAV semantics


DSpace: Model• DSpace digital repository• Differs from file system• Submit URI not known• Workflow• …


WebDAV: Model, Basics• Each Resource

named by a URL

• Properties are "name value" pairs

• Properties are extensible

• Property values can be found, and modified

DAV = "Distributed Authoring and Versioning"

URL


WebDAV: Model, Fuller• Resources

can be Collections or leaf nodes

• Collections can nest

• Everything can be discovered descending from the DAV server's root '/' resource


DSpace Mapped to WebDAV• WebDAV


DSpace Implemented as WebDAV

1. Properties– WebDAV– Custom for DSpace

2. LNI URLs

3. WebDAV Methods


Properties: WebDAV

• DAV:creationdate• DAV:displayname• DAV:getcontentlength• DAV:getcontenttype• DAV:getlastmodified• DAV:resourcetype• …

http://www.webdav.org/specs/rfc2518.html

Akin to file system properties,with add'l. metadata


Properties: Custom DSpace (1 of 3)

• ALL DSpace Objects:– dspace:type

• "BITSTREAM", "ITEM", "COMMUNITY" etc.

– DAV:resourcetype • "COLLECTION" etc.

– dspace:displayname– dspace:current-user-privilege-set



• Community DSpace Objects– dspace:logo– dspace:short_description– dspace:side_bar_text– dspace:handle– …



• Bitstream DSpace Objects– DAV:getcontentlength = getSize()– dspace:format = getFormat().getID()– dspace:sequence_id = getSequenceID()– …

• Many more DSpace Objects (some 10)– With many more Properties (some 87)

http://wiki.dspace.org/LightweightNetworkInterface


URLs: DSpace LNI• Community

– http://uni.edu/dspace/dav/dso_1721.1$46

• Collection– http://uni.edu/dspace/dav/dso_1721.1$349

• Item– http://uni.edu/dspace/dav/dso_1721.1$5543

• Bitstream– http://uni.edu/dspace/dav/dso_1721.1$5543/bitstream_13

The LNI URLs for DSpace objects are only for use with the LNI.

• They are therefore not to be used:

• as persistent URLs (e.g. bookmark, store for later use)

• as interactive URLs (e.g. browser)


WebDAV Methods in LNI• PROPFIND - very powerful request with multiple functions:

– list names of properties– return property values– recurse through 1 or more levels of a "collection" resource.

• PROPPATCH - changes value of or deletes properties on one resource.• GET - retrieves contents of a resource• PUT - replace or add a new resource to a collection• COPY - map a resource ("Item") to a different Collection.

As noted, equivalent SOAP methods are provided

Not Implemented: Locking, Versioning.


Current DSpace (pre-LNI)Content Provider, currently

• Batch import (& export)

• End user Web U/I

• dspace-admin Web U/I

• SRW, OAI-PMH

DSpace

Batch Importer

OCW Comm.

"Biology (7)"

"Physics (8)"

DSpace Object API

Web U/I Web, butNon Auto-

mated

Non Std.Non Web

Workflow Adm

Item


DSpace with LNIContent Provider using LNI:• LNI SOAP end-point URL

• LNI WebDAV URLs

• dspace-admin (partial)

• SRW, OAI-PMH– (Not replicated in LNI)

• Still available: – Batch import (& export)– End user Web U/I

DSpace

DSpaceLNI.class

OCW Comm.

"Biology (7)"

DSpace Object API

DSpace LNI WebDAV

- Web.- Stds.

- Batch-able.

"Physics (8)"

Item


Submission: OverviewOpenCourseWare Client

Needs 3 Values:

1. LNI SOAP end-point URL http://userid:[email protected]/dspace-lni/lni/DSpaceLNI

2. Handle for OCW Community = hdl:1721.1/125

3. Dept. Name (MIT "Course" Number) for Course to be submitted = "Physics (8)"

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

"Biology (7)"

DSpace Object API

DSpace LNI WebDAV

Lang. -Indep.

Platform -Indep.

"Physics (8)"

Item


Submission: 1. LookupRequest: OCW Comm. HANDLE

LNI lookup() = http://dspace.mit.edu/dspace-lni/dav/lookup/handle/1721.1/125

Response: OCW Comm. LNI URL http://dspace.mit.edu/dav/dso_1721.1$125

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Biology (7)"dso_1721.1$197

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Lookup


Submission: 2. CollectionsRequest: PROPFIND on LNI URL for children DSpace

Collections' "displaynames":<propfind>

<prop>

<DAV:displayname/>

</prop>

</propfind>

Response: MULTISTATUS<multiStatus>

<resource>

<href URI="dso_1721.1$197"/>

<displayname>Biology (7)</displayname>

</resource>

<resource>

<href URI="dso_1721.1$253"/>

<displayname>Physics (8)</displayname> </resource>...

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Biology (7)"dso_1721.1$197

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Iterate…


Submission: 3. Submit (PUT)Request: PUT to DSpace Collection LNI URL,

including specification of Packager Type (IMS-CP):

PUT /dspace/dav/dso_1721.1$253?package=OCW-IMSCP

....package contents in body...

Response: LNI URL for DSpace Item in "Location" Header:

HTTP/1.1 201 OK

Location: http://dspace.mit.edu/dspace/dav/dso_1721.1$836

....other headers....

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

IngestItemdso_1721.1$836


Submission: 4. Item HandleRequest: PROPFIND on DSpace Item LNI URL for the DSpace

Handle:

<propfind>

<prop>

<dspace:handle/> </prop>

</propfind>

Response: Properties Requested

=== PROPERTIES Successfully returned:

dspace:handle = "hdl:1721.1/836"

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Reverse Lookup

Itemdso_1721.1$836

1721.1/836

OCW Content Mgt. System

Stores DSpace Handle

SQL Server Table: hdl:1721.1/836


Submission: SummaryIn this 4-step OpenCourseWare

submission example, we've seen:• SOAP endpoint URL• Dspace Handles• Dspace LNI URLs• LNI Lookup() function• WebDAV Properties• WebDAV Methods• PROPFIND (LNI request)• MULTISTATUS (LNI response)• LNI "Reverse Lookup"• WebDAV Headers (Location;

Success/Fail)

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

"Physics (8)"1721.1/253

DSpace Object API

DSpace LNI WebDAV

Item1721.1/836


Future Development

• Other Use Cases, Needs– Basic, Advanced …

• Other DSpace objects– Bitstream Format Registry– Metadata Registry …

• Other WebDAV methods– PUT (for Replacement)– DELETE– MKCOL …

CWSpaceArchiving Courseware Websites to DSpace

©

Digital Library Federation Spring Forum, 2006

April 11, 2006 — Austin, Texas

William Reilly, Rob Wolfe — MIT Libraries

2006

Questions, & Etc.

http://cwspace.mit.edu/docs/ProjectMgt/Reports/MIT-CWSpace-DLF-Spring2006.ppt

V. 20060410_1510

Documents

CWSpace Archiving Courseware Websites to DSpace Using a Content Packaging Profile & Web Services © Digital Library Federation Spring Forum, 2006 April