(XAM) is - EMC Community Network

1© Copyright 2008 EMC Corporation. All rights reserved.

Introduction to XAM (eXtensible Access Method): A New Industry Standard Access Method for Fixed Content

Mike HorganSenior Technologist, EMC Centera


Agenda

� XAM History and Goals

� XAM Objects

� XAM Connection & Authentication

� Write Content Example

� Error Handling

� XAM Retention Model

� Additional Resources


What is Fixed Content?

The majority of new

digital data is fixed content

The majority of new

digital data is fixed content

Fixed Content is digital information that is in its “final form.”

� Most new digital content is ‘born’ fixed

� Transactional data is archived, becomes fixed

� Federal regulations impose disciplineFor example, content retention, disposition

� Fixed / variable data– 80% / 20% base (est)– 90% / 60% growth rate (est)


What is XAM?

eXtensible Access Method (XAM) isAn industry-standard interface definition (e.g., API)between Providers (storage systems) and Consumers (application and management software) of Fixed Content storage services

� XAM specification is controlled by SNIA Storage Networking Industry Association (SNIA)– Fixed Content Aware Storage Technical Working Group (FCAS TWG)

controls the XAM Architecture specification– XAM Software Development Kit Technical Working Grou p (XAM SDK TWG)

controls the XAM Language Binding (C & Java for 1.0); SDK TWG also develops the reference API library implementations


XAM Goals

Applications Vendors want:

� Annotate data with associated metadata

� Utilize common compliance-oriented data management policies

� Speak the same language to all types of devices

� Manipulate billions if not trillions of “records” Storage Vendors want:

� Application support for their products

� Efficiently store application data and metadata

� Easy migration between vendors/technology

� Manage billions if not trillions of “records”

End Users want:

� Choices between application vendors

� Choices between storage vendors

� Easy migration between vendors/technology

� Compliance, scalability, performance, $/GB, TCO

XAM

The industry will benefit from a standardized access method to fixed content


The XAM Approach

� Location-independent object naming– Each ‘Data Object’ (XSet) stored in XAM is assigned a globally unique name (an

XUID). – These location-independent names allow data to migrate within or between XAM

storage systems without impacting applications accessing the data.

� Rich metadata – XAM allows applications to bundle MIME-typed contextual metadata together with

application data, facilitating easier data interchange among applications and longer “shelf life” for application data.

� Pluggable architecture for storage system support– XAM storage system vendors can plug their systems into the XAM API by creating a

provider for the Vendor Interface Module API.– XAM also provides a standardized set of management disciplines and semantics for

fixed content, such as retention, query, shredding, etc.


XAM Timeline

Q4 2004 – IBM and EMC formulate a joint vision and begin work on a proposal

Early Q3 2005 – v1.0 of XAM Spec available, HP, HDS, Sun endorse XAM, join XAM Team

Q4 2005 – XAM Team donates v1.2 of XAM Spec to SNIA; donation accepted, placed under control of FCAS TWG

Q4 2005 – XAM Team donates v1.2 of XAM Spec to SNIA; donation accepted, placed under control of FCAS TWG

Q2 2007 XAM API V0.6 released for SNIA member technical review, call for IP and public preview

FCAS TWG counts >40 members, working on official API specification

Q1 2007 - SDK TWG formed to develop “gold” standard generally available

SNW Dallas Oct 2007 – first public XAM demonstration with EMC, Sun & HP storage


XAM Software Architecture

Application ProgramISV / Custom

XAM API LibrarySNIA

VIMReference VIM - SNIA

VIMVendor A

VIMVendor B

XAM Toolkit LibrarySNIAXAM API

XAM Toolkit API

XAM API

VIM APIVIM API


XAM Object Model

� XAM defines 3 primary objects– XSet

� Primary storage abstraction in XAM� Stores metadata and data in a collection of fields

– XSystem� Logical container of XSets� Each XSystem instance represents a connection with a particular XAM storage system

– XAM Library� Responsible for discovering and managing all VIMs in the XAM application environment� Provides fields to report on and control global XAM attributes (API Revision Level, API

Logging Level, etc.) � Serves as a factory for XSystem instances via Connect method


XAM Fields

� A field is a data element that can be attached to an XSet, XSystem, or XAM Library object

� All XAM fields have a name and a value, along with the followingstandard attributes

Determines whether this field participates in determining the XUID for the XSet. Changes to a Binding field require that a new XUID begenerated upon Commit.

Bindingboolean

Indicates if the field can be modified by an application. The ReadOnly attribute can only be modified by the XSystem.

ReadOnlyboolean

Length of field data, in bytes. Length is determined by the XSystem and not set by the application.

Lengthinteger

MIME Content-type (see RFC 2046) of the field’s value. Type is US-ASCII encoded and limited to 512 characters.

Typestring

XAM Field Attributes


� Field Names can be arbitrary strings (UTF-8 encoded) up to 512 bytes in length; reverse-DNS naming is encouraged:

com.company1.email.fromHeadercom.company2.caseNumbercom.company3.imaging.pageScans.1

XAM Field Names

System-defined fields always begin with ‘.’; applications may not create fields in this namespace.

.*

SNIA-owned standard namespace.org.snia.*

org.snia.xam.*

XAM storage system vendor-defined fields..vnd.<Reverse DNS>.*

Reserved for standard XAM-defined fields on primary objects.

.xam.*

.xsystem.*

.xset.*

XAM Reserved Field Namespaces


XAM Field Types

� XAM fields always have an associated MIME Content-type– MIME standard defined by RFC 2046

� XAM defines a common set of data types for application use– Referred to as stypes

� Fields with stype MIME types are called Properties– By definition, all fields defined with non-stype MIME types are Streams

Description

Arbitrary byte stream; see RFC 2046.application/octet-stream

Portable Document Format (PDF) data.application/pdf

Common MIME Content-type

Image data in various formats; see RFC 2046, RFC 3302.

image/gif, image/jpeg, image/png, image/tiff

XML data with optional encoding; see RFC 3023.text/xmltext/xml; charset=<charset>

Text data with optional encoding specification; see RFC 3676.

text/plaintext/plain; charset=<charset>


XAM Field Simple Type Definitions

MIME Content-type Length (bytes)Description

Actual lengthISO8601 timestamp, using UTF-8 encoding.

application/vnd.snia.xam.datetimexam_datetime

Actual lengthUTF-8 encoded Unicode string, with a maximum length of 512 bytes.

application/vnd.snia.xam.stringxam_string

Actual lengthBinary XUID.application/vnd.snia.xam.xuidxuid

8IEEE754 double-precision floating point.application/vnd.snia.xam.doublexam_double

8Signed twos complement 64-bit integer.application/vnd.snia.xam.intxam_int

1TRUE or FALSE.application/vnd.snia.xam.booleanxam_boolean

XAM stype


XAM Primary Objects - XSet

Properties- “Simple” types (Boolean, Int, Float, String, DateTime, XUID)

- Type checked/enforced by storage system

- Manipulated via “Property Get/Set” methods

XStreams- Bytestreams, up to 264 bytes

- Type assumed to be a valid MIME-type, but not checked/enforced bystorage system

- Manipulated via Posix-style I/O methods (e.g., open, read, write, close)

XSetXSet Fields

� XSet – Addressable “Unit of Storage” in XAM Model

� To store data in an XSystem, an application must:

– Create XSet instance via XSystem– Create/Populate XSet fields with data– Commit the new XSet to persistent storage, saving resultant XUID


XSet Standard Fields

Presence vs. absence of uncommitted modifications.YesNo.xset.dirty

XUID of the XSet.YesNo.xset.xuid

Most recent time that the XSet was opened or committed.YesNo.xset.time.access

Most recent time that the XSet was modified.YesNo.xset.time.modification

Time that the XUID was assigned to the XSet.YesYes.xset.time.xuid

Time the XSet was created.YesYes.xset.time.creation

DescriptionRead OnlyBindingField Name


XSet Field ‘Binding’ Attribute

XSetXSet Fields

• When a field is marked ‘Binding’ ( ) it means that this field’s value has a direct correspondence with this XSet’s XUID (the field’s value is relevant for the XSet’s identity)

• ‘Non-Binding’ fields may be freely modified within the XSet, just as with traditional read/write storage

• However, on an attempt to modify a ‘Binding’field, the XSystem silently creates a completely new XSet, an identical copy of the original XSet, and assigns it a new XUID; the original XSet must be preserved under the original XUID

• Applications are free to decide which XSet fields are ‘Binding’ at the time they are created


XUID – XSet Unique Identifier

� XUID is the permanent name for an XSet– Assigned by XAM storage system– XUIDs are globally unique

� XUID native format is binary sequence (10 – 80 bytes)– Base64 (RFC 2045) recommended for printable interchange

� XSet’s XUID has a strict relationship with the XSet’s ‘Binding’ fields– If a ‘Binding’ field is modified, a new XSet with a new XUID is created upon commit– The old XSet is preserved as is

data end…opaque dataXUID CRC-16

XUIDlength

reserved (zero)Vendor OIDreserved

(zero)

7978…109876543210

XUID Format


XAM Primary Objects - XSystem

� XSystem is an abstraction in the XAM API representing a logical container of XSets.

– This is distinct from and possibly a subset/superset of the physical XAM Storage System .

� An XSystem instance combines an XSystem with an authenticated connection to one or more XAM storage systems .

– An XSystem instance is equivalent to a XAM session.– In the XAM API, an XSystem instance is created by calling XAM_Connect with a valid

XRI (XSystem Resource Identifier).– XSystem instances are used to create, retrieve, and delete XSets.– XSystem instances also have fields describing the system’s supported capabilities

and management policies.


XRI – XSystem Resource Identifier

� An XRI may look like this:– snia-xam:// [vimname!] xsystemname [,param=value[{,p aram=value}]]

– Conforms to RFC 3987 “Internationalized Resource Identifiers (IRIs)” allowing non-ASCII characters

� xsystemname is vendor-specific– But cannot include ‘/’ or ‘?’ or ‘@’ or ‘!’

� vimname is vendor-specific– But cannot include ‘!’

� Example:

snia-xam://acme_vim!xam.acme.com:8012,audit=yes

Application

XSystem

Connect()

XAM LibraryXAM Library


XAM Primary Objects - XAM Library

� XAM Library Object is a singleton providing the application “entry point”into the XAM API.

� XAM Library Object provides one method:

XAMLibrary_Connect(xam_string inXRI, xsystem_handle* outXSystem)

� Fields on XAM Library Object control generic library behavior, and can pass settings to XSystem instances.

– XAM API Logging controlled via XAM Library fields.– During XAMLibrary_Connect, all XAM Library fields are copied onto new XSystem

instances (including app-defined fields).


XAM Library Standard Fields

Type ReadOnlyBinding

yesnoxam_string.xam.vim.list.<name>

yesnoxam_string.xam.apiLevel

nonoxam_string.xam.log.path

nonoxam_int.xam.log.verbosity

nonoxam_int..xam.log.level

yesnoxam_string.xam.identity

XAM Library Field Name


XAM Write Example - Overview

� Connect to XSystem

� Create New XSet

� Add XSet Metadata

� Write XSet Data

� Commit XSet

� Release Resources

XAM Storage System

XAM Storage System

XAM 1.0 Application Program

XAM API Libraryxam.dll

VIMexample_vim.dll

XAM API

VIM API

XSS Protocol


XAM Write Example – Connect to XSystem

xam_string vXRI = “snia-xam://example_vim!10.1.1.1”;

vStatus = XAMLibrary_Connect(vXRI,&vXSystem);


XAMLibrary_Connect

VIMexample_vim.dll

snia-xam://example_vim!10.1.1.1 XAM Storage System10.1.1.1

XAM Storage System10.1.1.1


XAM Connection Authentication

� Once connected, an XSystem instance must authenticate

� XAM uses SASL Authentication Framework– Simple Authentication and Security Layer (RFC 4422)– PLAIN and ANONYMOUS methods always available– More advanced SASL methods (DIGEST-MD5, SECURID, KERBEROS-V5)

advertised via XSystem property� .xsystem.auth.SASLmechanism.list.<mechanism>

� PLAIN Authentication– “PLAIN\0\0<name>\0<secret>\0”


XAM Write Example – Authenticate


XSystem_Authenticate

VIMexample_vim.dll

“PLAIN\0\0<name>\0<secret>\0” XAM Storage System10.1.1.1


� authDataLength = BuildAuthBuffer(authBuffer,authName, authSecret);

� vStatus = XSystem_Authenticate (vXSystem, authBuffer, authDataLength, &authStream);

� XStream_Close (authStream);


XAM Write Example – Create XSet


XSystem_CreateXSet

VIMexample_vim.dll

XSET_MODE_MODIFY XAM Storage System10.1.1.1


� xset_handle vXSet;

� vStatus = XSystem_CreateXSet(vXSystem, XSET_MODE_MODIFY, &vXSet);


XAM Write Example – Add XSet Metadata


XAMLibrary_Connect

VIMexample_vim.dll

“com.example.archive.invoice_id” XAM Storage System10.1.1.1


� vStatus = XAM_CreateString (vXSet,“com.example.archive.invoice_id”, /* field name */TRUE, /* binding */vInvoiceID); /* value */


XAM Write Example – Write Binary Data

� XAM stores binary data in special fields called Streams

� Streams present POSIX-like write/read interface

xstream_handle vXStream;

vStatus = XAM_CreateXStream(vXSet, “com.example.archive.invoice_image", /* field name */TRUE, /* binding */“image/tiff", /* MIME type */&vXStream);

… Transfer Image Data ….

vStatus = XStream_Close(vXStream); /* completes write*/


XAM Write Example – Writing XStream Data

� Transferring data from a file to an XStream

do {/* Read data from file into buffer. */vReadLength = fread(vDataBuffer, sizeof(char), BUFF ER_SIZE,

vInputFile);if (vReadLength > 0){

vPosition = 0;while (vPosition < vReadLength){

/* Write data from buffer to XStream. */vStatus = XStream_Write(vXStream, &vDataBuffer[vPos ition],

vReadLength - vPosition, &vWriteLength); vPosition += vWriteLength;

}

/* Update accounting info */vTotalWrite += vPosition;vTotalRead += vReadLength;

}} while (vReadLength > 0);


XAM Write Content – Commit XSet


XSet_Commit

VIMexample_vim.dll

xset_handle XAM Storage System10.1.1.1


� vStatus = XSet_Commit (vXSet, &vXUID);

� After Commit, application stores XUID return value permanently, for future XSet retrieval


XAM Write Content – Code Overview

// Connect to XSystemvStatus = XAMLibrary_Connect(vXRI, &vXSystem);

// XSystem AuthenticationvAuthDataLength = BuildAuthBuffer(vAuthBuffer, sAut hMethod, sAuthName, sAuthSecret);vStatus = XSystem_Authenticate(vXSystem, vAuthBuffe r, vAuthDataLength, &vAuthStream);

// Close auth streamvStatus = XStream_Close(vAuthStream);

// Create XSetvStatus = XSystem_CreateXSet(vXSystem, XSET_MODE_MO DIFY, &vXSet);

// Add some identifying metadatavStatus = XAM_CreateInt(vXSet, PROPERTY(format_revi sion), true, 1);vStatus |= XAM_CreateString(vXSet, PROPERTY(invoice _id), true, vInvoiceID);vStatus |= XAM_CreateString(vXSet, PROPERTY(cust_id ), true, vCustomerName));

// Write invoice image data using multiple threads an d streams…vStatus = writeFileData(vXSet, vFilename, &vTotalBy tes, &vSegmentCount);

// Additional metadatavStatus = XAM_CreateInt(vXSet, PROPERTY(image_count ), true, vSegmentCount);

// Commit XSetvStatus = XSet_Commit(vXSet, &vXUID);

// Display resulting XUIDXUIDToString(vXUID, &vXUIDString);printf("Successfully wrote XSet, Base64 XUID:\n %s \n", vXUIDString);


XAM Error Handling

� All XAM C API methods return a status code– Java binding uses exceptions

� Return code of 0 (zero) indicates success

� Generic error codes defined in xam_errors.h; XSystem can return detailed error info via XAM_GetErrorToken()

� Application must check status after every API call

/* Connect to XSystem. */vStatus = XAM_Connect(vConnectString, &vXSystem);printXAMStatus("Connect", vXSystem, vStatus);

if (!vStatus) {/* Create new XSet. */vStatus = XSystem_CreateXSet(vXSystem, XSET_MODE_MOD IFY,&vXSet);printXAMStatus("Create XSet", vXSystem, vStatus);


XAM Retention Model

� Retention is the ability of an XSystem to prevent XSet deletion for an application-specified time period.

� All XAM Retention policies have 3 primary aspects:

– Enabled: Has the associated retention been turned on?

– Duration: How long shall deletion be prevented (in milliseconds)?

– Start Time: When does the retention clock start counting down?

� XAM Base Retention– Start Time == XSet XUID Time

� XAM Event Retention– Start Time is defined at some future date,

triggered by business conditions

� XAM Hold– “Freezes” an XSet until all holds are

released

XSet access is readonly;

delete XSet is not allowed

Hold time criterion is met at time =

last release occurs

XSet

Created

Standard XSet Access

Delete XSet Allowed

Hold(1)

start time

Hold(2)

start time

Release (1) Release (2)


More XAM Areas to Explore

Interesting areas that didn’t fit!� XIterator Object

– Provides field iteration

� XAsync– Asynchronous read/write/copy support

� Advanced XSet Management– Standard Management Policy enuemeration and application APIs – Support for auto-deletion (upon expiration) and selective content shredding

� XAM Query– Nonrelational (but still SQL-like) XSystem Query facility, with optional full-text support

� XAM Jobs Model– Generic API for executing standard (like Query) and device-specific “jobs”

� XAM Security Model– Fine-grained security model for XSet access control

� XAM Import/Export Model– Fully vendor-independent data migration, transparent to applications!


� EMC Developer’s Network (CAS/Centera Department )– http://developer.emc.com/

� EMC’s XAM Application Experts– [email protected]

� SNIA XAM Home– http://www.snia.org/xam/home

� SNIA XAM Initiative– http://www.snia.org/apps/org/workgroup/xam/

� SNIA FCAS TWG (XAM Technical WorkGroup)

– http://www.snia.org/apps/org/workgroup/fcastwg/

� SNIA SDK TWG(XAM Software Developer Kit Technical Work Group)

– http://www.snia.org/apps/org/workgroup/xamsdktwg/index.php

For More Information

Documents

(XAM) is - EMC Community Network