Storage Network Industry Association(SNIA) - Long Term ...web.stanford.edu/.../Day2/HighlevelTrends/RaymondClarke_SNIA.pdf · Raymond A. Clarke Enterprise Storage Consultant, Sun

Raymond A. Clarke

Enterprise Storage Consultant, Sun Microsystems

Board of Directors, SNIA Data Management Forum

Storage Network Industry Association(SNIA) -Long Term Archive & Compliance Storage Initiative(LTACSI)

Perspectives & Activities

Sun Microsystems, Inc. Page 2May 28, 2008

Session Outline

• SNIA Organization> The Role of SNIA Forums > Data Management Forum

>Organization, Mission & Activities>Long Term Archive & Compliance Storage

Initiative(LTACSI)– Goals and Objectives– The 100 Year Archive Requirements Survey– Activities

● LT-SRG● SD-SCDF


3

SNIA Le ga l Notic e

The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced without modificationThe SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee.


www.snia.orgSNIA Corporate www.snia.ca

SNIA Canadawww.SNIA-Europe.com

SNIA Europe

www.snia.org.cnSNIA China

www.snia-India.org

SNIA India

www.snia.org.au

SNIA Australia and New Zealand

www.snia-j.orgSNIA Japan

www.snia-sa.orgSNIA South Asia

SNIA Technology CenterColorado Springs, USA

SNIA Technology CenterTsinghua University

Beijing, China

Over 7000 Active MembersOver 450 Organizations

Storage Network Industry Association


SNIA OrganizationBoard of Directors

TechnicalCouncil

TechnologyInitiatives &

Forums

Education & Certification

3 Forums2 Initiatives

16 TechnicalWork Groups

SNIAStaff

SNIA Forums and Initiatives primarily focus on technology promotion standardization &

education technical marketing activities. Forums do not develop technologies or specifications, though

Initiatives can.

Typical Forum Activities and Responsibilities• Collaboration between end-user professionals, developers, integrators,

professional services, and alliance partners• Identifying and documenting customer requirements• Defining the need for standards, promoting SNIA technical work groups,

and providing market requirements• Maintaining an active, public-facing, market education web-site • Participating in events and technology demonstrations• Publishing white papers, articles and media outreach• Interacting with external associations and trade groups

• Security Forum • IP Storage Forum• Data Mgmt Forum• Storage Mgmt Initiative• XAM Initiative• Green Storage Forum


6

DMF ProgramsData

ManagementForum

ILMInitiative

Long Term Archiveand Compliance Storage Initiative

Data ProtectionInitiative

100 Year ArchiveTask Force

CDP SIG

LiaisonXAM Initiative,

FCAS, LT-DIRP TWG

Professional SvcTask Force

Storage Services

AlliancesMarketingCommittee

AIIM, ITAA, ISACA, NARA .. and more

ARMA

WebCommunications

EventManagement

SpeakersBureau

VTL SIG

ILM Tech Work Group Liaison

D-Dup, Data Integrity,

more

International SNIA Affiliates


DMF

Data Protection Initiative(DPI)

Information Lifecycle Management Initiative

(ILMI)

Long term Archive and Compliance Storage Initiative

(LTACSI)

Defining new approaches and best practices for continuous data

protection, VTL, Data D-Dup, and backup/recovery

Developing, teaching and promoting ILM practices, implementation methods,

and benefits

Addressing challenges in developing, securing, retaining & preserving digital information

over the long term

About the Storage Networking Industry Association (SNIA)• SNIA’s primary goal is to ensure that storage networks become complete and trusted solutions across the IT

community• For additional information about SNIA see www.snia.org

About the SNIA Data Management Forum (DMF)• The DMF is a sub-group of SNIA acting as the worldwide authority on Data Management, Data Protection and

ILM• The DMF is a collaborative storage industry resource available to anyone responsible for the accessibility and

integrity of their organization’s information.

www.snia-dmf.org

About the Data Management Forum(DMF)

http://www.snia.org/



http://www.snia-dmf.org/


Current LTACSI Activities

• SNS > Presentation - Best Practices for Long-Term

Retention of Digital Information • SNW

> SD-SCDF press release> SD-SCDF FAQs

• TWG formation> Please join

• Solving Digital Preservation in the Business Datacenter Article

• Decomposition of Archival System• What is SD-SCDF presentation


Long-Term is Real!More information is being kept long-term than you think – and it is at risk!

53%

15%

13%

3%

2%

4%

4%

1%

2%

4%

0% 10% 20% 30% 40% 50% 60%

Percent of Responses

Permanent

>100 Yr

>50 Yr

>25 Yr

>20 Yr

>10 Yr

> 5 Yr

Life of Product

Life of Company

>10 Yrs beyondProject

Longest Retention Requirement

Longest Requirement68 % over 100 Years83% over 50 Years

Requirements vary by organization type, information type, and compliance rules/risk

Source: 100 Yr Archive Requirements Survey 2007 , N=104


Long-Term Retention Projects

Logical/ Physical Migration and MovementLaunch a TWG to define a “SD-SCDF*”, a self-describing, self-contained data format standard

SD-SCDF provides a standard container allowing all key preservation attributes to be maintained over time and across virtualized repositories

Conduct Market EducationSpeaking, papers, web

Interact with international community working on retention and archive

10

Long Term- Storage Reference Guide

(LT-SRG)


Why do we need a LT-SRG

• Over the last several years electronic information archival and storage as a technology and business requirement has been elevated from the “plumbing” to the attention of the board room.

• Business and regulatory requirements have exasperated storage requirements from terabytes to petabytes faster than anytime in history, and it continues.

• Emerging technologies are entering the fray and solving technical problems while potentially compounding the concerns related to Long Term archival.


On top of that …

• There are no clear owners of the problem domain, everyone is fighting for the ball.

• Vendors in the fray> Storage, Messaging, Archival,

Data Curation, Hygiene/Security, Records Management, Business Intelligence, Legal Response, Search and Indexing, Document Management…

• Industries in the fray> Storage, Compliance, Legal,

Messaging, Records Management, Archivist,Curators, Top Tier Consultancies


Charte r of the LT-SRG C ommitte e

• The LT -SRG ch ar te r is to ad dr ess th e mor e practical im ple mentatio n and d esign ch alle nges for long-term ar ch iva l ut il izing bot h a cu rrent state an d futu re s tate fr am ework.

What’s here, what’s coming and how to best use it and what to be aware of…

The LT-SRG will create 3 major initiatives for 2008; extending the work product produced within the 100 Year Archive to develop a next phase market requirements based on the initial responses received from the 100 Year Archive survey and analysis. Essentially, “the how”.The resulting work products will be consumer friendly guides aimed at end-users (practitioners) and re-usable frameworks for furthering the continuation of the research. All work-product will remain in-line with current industry standards and recommendations. In other words, we are not attempting to initiate new or competing standards.


LT-SRG Initiatives

The LT-SRG will initiate a “living archive system decomposition” an end-to-end archival infrastructure. Like mapping of the human genome. Identifying each component from software to hardware to process, and map each components impact to long-term storage and archival.

The LT-SRG will initiate research and produce guidance on disruptive technologies to provide best practice recommendations on the utilization of disruptive technologies within the framework produced in number one.

The LT-SRG will initiate further research extending the 100 Year Archive survey data to extrapoloate “the how”.


Research and Guidance on Disruptive Technologies

• Candidates for disruptive technology review to be selected and reviewed in context of the LT-SRG charter.

> These include areas such as storage interconnects, caching, and de-duplication technology as higher-level categories for further analysis as to their impact and development of best practices and recommendations.

> Given that each of these technologies will make their way into the archival “eco-system” it is prudent that a decomposed view exist so they can be easily mapped using a structured research methodology to understand their impact on all of the component pieces of the archival infrastructure and their impact alongside of Long Term Archival technology and business requirements.


Living Archive System Decomposition

Create a logical mapping of component pieces into technology and business requirements. A high-level example may be found in “encryption” as a technological component, this component can be de-composed to the various types of encryption utilizedA business mapping would be the impact of encryption on business requirements such as content accessibility, etc. A proposed result of the decomposed may be

1) identification 2) description of current and future state 3) business requirement 4) conflicting requirement identification 5) best practice 6) recommendation.

17


Re searc h and Guidan ce on Dis ruptiv e Te chnolo gies

Candidates for disruptive technology review to be selected and reviewed in context of the LT-SRG charter.

These include areas such as storage interconnects, caching, and deduplication technology as higher-level categories for further analysis as to their impact and development of best practices and recommendations.

Given that each of these technologies will make their way into the archival “eco-system” it is prudent that a decomposed view exist so they can be easily mapped using a structured research methodology to understand their impact on all of the component pieces of the archival infrastructure and their impact alongside of Long Term Archival technology and business requirements.

18

Self-Describing Self-Contained Data Format (SD-SCDF)


What is SD-SCDF

SD-SCDF is a "container" or "wrapper" format which supports a number of different types of coded “content", encoded with any of a variety of applications, together with a metadata wrap-per which describes the content contained within the SD-SCDF fileSD-SCDF is being designed to address a number of problems with data formats over time, and is intended as a platform-agnostic stable standard for future archive content.

SD-SCDF – Information Format and Storage Standard


Why SD-SCDF?

No consensus on what ‘archiving’(particularly for the long term) really means, or what services might be available from an archive

Preserving the information is not the same as preserving the data bits(or vice versa)Will the repository infrastructure understand the information in 10, 50 or 100 years?How can we know when effective archiving is being achieved?

Lack of consensus impedes establishment of standards and commercial support services


Operational Strategy

Data stored before SD-SCDFUtilize export and import or migration processes

When SD-SCDF is adopted widelyEliminate export and importAllow direct transfers of SD-SCDF between:

Storage systemsTransparent to Applications


SD-SCDF Objectives:Enabling Scalable Cost-Effective Migration

Requirements and considerations:Media and platform agnosticUtilize preservation objects as the basic artifacts

Leverage the OAIS Archival Information Package (AIP)

Reduce the risks for data lossCan be physically moved from one storage system to another storage system without conventional copying

Preservation

System (Phase 2)

Move Move

….

Preservation

System (Phase 3)

Preservation

System (Phase 1)


Content Data Object the raw data that is the focus of the preservation.

Representation Informationthe information required to interpret the raw data to its designated community.

Reference• globally unique and persistent identifiers for the content information.

Provenance• the history and the origin of the content information and any changes

that may have taken place since it was originated, and who has had custody of it since it was originated.

Context• documents reason for creation of the content information and

relationship to its environment.Fixity

a demonstration that the particular content information has not been altered in an undocumented manner.

OAIS AIP Logical Structure


What is SD-SCDF and what is XAM

SD-SCDF is a "container" or "wrapper" format which supports a number of different types of coded “content", encoded with any of a variety of applications, together with a metadata wrapper which describes the content contained within the SD-SCDF fileSD-SCDF is being designed to address a number of problems with data formats over time, and is intended as a platform-agnostic stable standard for future archive content.

SD-SCDF – Information format andstorage standard

XAM gives applications a standard interface and metadata to communicate with object storage devicesThis “handshake “ between the application and the storage medium enables interpretation of application metadata with the annotation of storage services metadata whereby policy-based decisions can be instrumented

XAM – Interface standard


XAM Interface Integration

Vendor A Storage Provider

Vendor ACustom VIM

VIM Interface

Vendor CXAM Proxy

Vendor BStorage Provider

Vendor ACustom VIM

Vendor SpecificProtocol

StandardizedWire Protocol

Wire-Protocol Reference VIM

StandardizedWire Protocol

Wire-Protocol Reference VIM

Vendor SpecificProtocol

VIM InterfaceVIM InterfaceVIM Interface

Standardized Reference XAM API Library

XAM Interface

Application A

XAM Interface XAM Interface

Application B Application CXAM gives applications a standard interface and metadata to communicate with object storage devicesThis “handshake “ between the application and the storage medium enables interpretation of application metadata with the annotation of storage services metadata whereby policy-based decisions can be instrumented

XAM Library can include a standard SD-SCDF containerApplications can then write a standard long-term interchangeable format using this library

Content Data Object

Representation Information

XAM – Interface Standard


Potential Benefits of SD-SCDF

Media agnosticTape, disk, future media

Vendor and Platform agnosticSelf-describingSupport self-contained data

Include means to represent internal links and cross references

Support different storage technologies (initially XAM and others in the future)Performance

Need to have good performance even for large data that includes text and binaries Enable parallel reads and writes

InteroperabilityNeed to be able to migrate data between different systems without loss of data Can be interpreted in the future

ExtensibleAdditional information which may be added in the futureVendor specific extensions

Cost (free parsers)Readable by both humans and machines

Ability to do offline inspectionSupport additional functions on the data

compression, encryption, cryptography


“The Archive”

The Ultimate Archive Environment


E m a i lE m a i lO r a c l e

P r i m a r yD a t a b a s e

P r i m a r y D i s k

E m a i l A r c h i v e r

D a t a b a s e A r c h i v e r

S A P

S T R U C T U R E D D A T A U N S T R U C T U R E D D A T A

V i d e oV i d e o I m a g e sI m a g e s

Mixed Workgroup

HoneycombRAIN SAM-FS

(CIS)

5320 NAS(smaller)

Offsite Tape 5320 NAS

SATA / FC Modular Disk

Tape Libraries& Virtual Tape

SAM-FS(CIS)(larger)

NFS

Sun's Archive Solutions Portfolio Will Get you There


Future Activities

• IEEE Archival Storage Life Cycle Workshop > Workshop on Digital Archive

Preservation and Sustainability - White Paper

• Community suggestions and requests


In a nut shell

it just ain’t easy


We n

eed yo

ur h

elphttp://community.snia-dmf.org

http://www.snia.org/forums/dmf

http://community.snia-dmf.org/



Resources

• SNIA Data Management Forumwww.snia.org/forums/dmf

100 Yr Archive Task Force Requirements Survey “Terminology Bridge” White Paper & Glossary

• DMF Communityhttp://community.snia-dmf.orgA networking and collaborative community, working to create “information-centric enterprises”

33


http://community.snia-dmf.org/

Thank You.The Network is the Computer.TM

Raymond [email protected](212) 558-9321

mailto:[email protected]


100 Year A rc hiv e Tas k Forc e

ObjectivesProduce a “best practices for long-term digital information retention” reference model similar to the Sedona project or OAIS

Solve the storage-side of physical and logical migration

Teach ILM-based practices for long-term digital information retention

(ISO 14721:2002)

35


Lo ng -Term Re te ntion P roj ects

Long-Term Retention Reference Model

Requirements (done)

Glossary (done)

Best practices for storage

Define a reference architecture covering migration, security, etc.

Meta-data provided thru XAM

Logical MigrationLaunch a TWG to define “SD-SCDF”, a self-describing, self-contained data format standard

Conduct Market EducationSpeaking, papers, web

Interact with international community working on retention and archive

36

Documents

Storage Network Industry Association(SNIA) - Long Term ...web.stanford.edu/.../Day2/HighlevelTrends/RaymondClarke_SNIA.pdf · Raymond A. Clarke Enterprise Storage Consultant, Sun