Upload
vandan
View
220
Download
0
Embed Size (px)
Citation preview
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 1© 2008 TATA Consultancy Services Limited. All Rights Reserved.
www.storage-developer.org
Open Source Storage Solution - ILM
Udayan Singh
Tata Consultancy Services
September 2008
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 2© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 2
Abstract
Majority of Products on Open Source Storage Technology exist as
individual products, rather than part of well-defined solutions to
Enterprises. Enterprises however look for solutions based on multiple
open source products working together, rather than segmented products.
This is important to utilize the true benefits of open source technology.
The proposal addresses a step for Open Source Products in Storage
Domain, to be integrated as Solutions for Mid-Tier Enterprises. A Proof of
Concept for Information Lifecycle management (ILM) is considered as an
example and other solutions that can be developed based on Open Source
Products are shared.
This is an ongoing work; hence open to feedback / suggestions.
www.storage-developer.org
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 3© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 3
Agenda / Overview
Need for Open Source SolutionsOpen Source Storage ProductsScope and Solution Requirements of IndustryProof of Concept - File based ILMSummary
www.storage-developer.org
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 4© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved.
Need for Open Source Solutions
Open Source Software Adoption
Will accelerate in 2008
Industry shifts as recognized
“Solutionization”- high volume
development of targeted
solutions - esp for SMBsSource: “Worldwide Software Business Strategies 2008 Top 10
Predictions”, IDC
“SMEs generally have limited IT
resources. Surveys confirm that in
the data protection arena, these
companies look for ease of use,
low cost, and then functionality - in
that order.”
Figure 1 Two - Thirds of Enterprises have Some Degree of Interest in Open Source
“How Interested are you in adopting open source software?”
Base - 1017 respondents at North America and European Enterprises (percentages may not total 100 because of rounding)
Source - Enterprise and SMB Software survey, North America and Europe, Q3 2007
44322 Source - Forrester Research, Inc.
www.storage-developer.org
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 5© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 5
Open Source Storage Products
NAS Data Protection
Management Software
OpenFilerfreenas NanoNASNAS-linuxPenguin-IT NAS
BaculaAmanda BackupPCiFolder
AperiStorageIM StorageRevolutionopenQRMSBLI
SAN (IP/FC) Tape Utilities
UNH-iSCSIiSNS
Tape DispatcherOpenTMSTaper
snpingNaigosOpen Storage ToolkitrsyncOpenPegasus
www.storage-developer.org
Have a large number of products but a dearth of proven business solutions
Open Source Applications ECMMailServer
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 6© 2008 TATA Consultancy Services Limited. All Rights Reserved.
Solution Requirements of Industry
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 7© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 7
Solution Requirements of Industry
Following Areas can be possible solutions for Industry that can
be based on Open Source Storage:
Content Management Market - USD 3.8 Billion in 2007 - IDC, 2007
Mail Server Integration
Green Storage
Disaster Recovery
ECM Integration
File Based ILM
Open source projects have led the way demonstrating the value of collaboration – Forrester
New Guidelines for Data Centers
SMEs can take a look at DP Solutions from Open Source –Taneja Group
SMBs need cost optimal Information management solutions
www.storage-developer.org
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 8© 2008 TATA Consultancy Services Limited. All Rights Reserved.
Content Management Solutions
CMS- Alfresco
New Possible Feature post Integration Possible Open Source Component
High Availability usingRAID - 0/1/5/6/10
Openfiler
Tiered Storage Openfiler / OpenSolaris / rsync / Bacula / Amanda
Snapshot support Openfiler
Replication (block based) Openfiler
More powerful backup (Disk to Disk/Disk to Tape)
Bacula/Amanda
Single Management UI New Development or integration with Management UI of NAS Filer
www.storage-developer.org
Are
a of
Wor
k
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 9© 2008 TATA Consultancy Services Limited. All Rights Reserved.
Mail Server
Mail Server- Zimbra Open Source
New Possible Feature post Integration Possible Open Source Component
High Availability usingRAID - 0/1/5/6/10
openfiler
Tiered Storage Open Source NAS/OpenSolaris/rsync/ Bacula/Amanda
Snapshot Support Openfiler
Replication (block based) Openfiler
More powerful backup (Disk to Disk/Disk to Tape)
Bacula/Amanda
Single Management UI New Development or integration with Management UI of NAS Filer
www.storage-developer.org
Are
a of
Wor
k
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 10© 2008 TATA Consultancy Services Limited. All Rights Reserved.
Green Storage
Feature Possible Open Source Component
Server Virtualization XenSource / Vmware
Support for SATA Drives OpenFiler
Tape Storage Bacula / Amanda
Disk Spindown Spindown 0.2.2
Solid State Drives None
Calculating Real time Power usage on Storage System
None
De-Duplication None
www.storage-developer.org
Are
a of
Wor
k
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 11© 2008 TATA Consultancy Services Limited. All Rights Reserved.
Proof of Concept - OSILMS
Aim of PoCSelected Open Source SoftwareArchitecture - Software and Storage SystemsSoftware DevelopedResults
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 12© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 12
Aim of PoC
Objective
To Validate File based ILM Solution, developed based on Open Source
Storage Software / Components
Solution under consideration for Proof-of-Concept (OSILMS - Open
Source Information Lifecycle Solution)
To verify if the Open Source Components work in seamless manner
with each other.
Find gaps in above and develop an application or enhance existing
products to meet the requirement of solution.
www.storage-developer.org
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 13© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 13
Dat
a D
estr
uctio
n
Storage Management / Optimization
Archive / Compliance
Backup / Recovery
Data Store/Reuse
Policy Based Tiering
ILM Overview & Mapping
Information Lifecycle
ECM/BI/Other Applications
Dat
a C
reat
ion
Dat
a C
lass
ifica
tion
OSILM - App
OSILM-Agent
OSILM-Agent
OSI
LM-A
gent
Policy
Management
Operational
Infrastructure
www.storage-developer.org
Bacula
OpenFiler / OpenSolaris
Rsync
Aperi / OpenFiler Mgr
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 14© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 14
Observations
No Support of Appliance’s integration with Storage Management S/W (e.g.
Aperi)
OpenFiler (Own Management Utility)
OpenSolaris (Own Management Utility)
An Application is required (e.g. OSILM-App) that would integrate the
various components in ILM Solution. It should have the functionality:
Delivering Storage Services to Administrator
Need for Plug-ins to be developed where the gap exists.
www.storage-developer.org
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 15© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 15
Aperi Server
Bacula Storage Daemon
OSILMS - PoC Setup
Aperi UIOSLIM App
Linux Fedora Client
OpenSolaris Client
Windows Vista Client
Linux RedHatClient
Windows XP Client
File Server 1 (NAS Head) [Primary Storage] » OpenFiler 2.3
File Server 2 (NAS Head) [Primary Storage] » OpenFiler 2.3
Storage Server [Secondary Storage] » OpenSolaris
rsync
rsync
Bakula Backup / Administration Server
www.storage-developer.org
FC Switch
Storage Array
Tape Storage [Tertiary Storage]
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 16© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 16
OSILMS Design
Managem
ent /UI
Data Agent
Storage Services
DB
Storage Devices
www.storage-developer.org
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 17© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 17
PoC Validation
www.storage-developer.org
Normal Operation
Objective
1. Determine network throughput of PoC setup when
tiering process (rsync) is inactive / not triggered
2. Determine File read-write operations per second
Preparation
Disable tiering policy
Steps
1. Using iometer
1. Start network I/O operations using Iometer
Dynamo - for a 20 minutes duration
2. Record the performance information during the
run
2. Using tiobench Run and record File read-write
performance on NAS Head
Loaded Operation
Objective
Determine network throughput of PoC setup when tiering
process (rsync) is active / triggered
Preparation
1. Modify policy so it will trigger tiering operation 5
minutes into the workload characteristic analysis
2. Pre-load the primary with 12 GB of data that has been
modified to qualify for the tiering operation once rsync
is triggered
Steps
1. Start network I/O operations using Iometer Dynamo -
for a 20 minutes duration
2. Start the Tiering operation using rsync
3. Record the performance information during the run
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 18© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 18
Results
www.storage-developer.org
Disk Performance Test Results - tiobench
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 19© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 19
Results
Operations Per Sec Average Latency Maximum Latency
Total I/Os per Sec
Read I/Osper sec
Write I/Os per sec
Avg. I/O Response time (ms)
Avg. Read Response time (ms)
Avg. Write Response time (ms)
Max. I/O response time (ms)
Max. Read Response time (ms)
Max. Write Response time (ms)
NormalOperation
4121.28741 2062.28642 2059.00099 0.242293 0.370027 0.114355 48.940372 48.940372 38.142611
LoadedOperation
3674.35422 1837.85196 1836.50226 0.271794 0.417208 0.126273 374.959497 374.959497 44.510734
www.storage-developer.org
Performance Test Results - Iometer
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 20© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 20
Summary
SMEs are looking for solutions that are proven. This provides an opportunity for open source storage products in market to work on end-to-end proven solutions (by integrating in a seamless manner with other products) that can be used by end-user companies.
www.storage-developer.org
Storage Developer Conference 2008 © 2008 Insert Copyright Information Here. All Rights Reserved. 21© 2008 TATA Consultancy Services Limited. All Rights Reserved. © 2008 TATA Consultancy Services Limited. All Rights Reserved. 21
Questions ?
Questions ??
www.storage-developer.org
Special Thanks to Nishi Gupta
I would also like to thank Reena Dayal, Amit Shuklaand Shaham Khan of Storage CoE, TCS for making this
demonstration possible