View
213
Download
0
Category
Preview:
Citation preview
This document is solely for the use of Sony Pictures Entertainment (SPE) personnel. No part of it may be circulated, quoted, or reproduced for distribution outside the SPE organization without prior written approval from IBM.
© Copyright IBM Corporation 2009
Sony Pictures Entertainment
Digital Backbone File Management and Infrastructure
Nov 3, 2009
2
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Agenda
DBB recap
Workflow and throughput requirements
Approaches
Virtual File Repository (VFR)
Server and storage infrastructure options
Pricing
3
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
IBM and SPE have been growing the motion picture DBB for more than a year and today we’re discussing HSM
“2012 Project”1
Q2 2008
Finalized Workload4
Q4 2009
• Grew to 500 TB • 2 PB LTO
DBB “2012 Expansion”2
Q2 2009TSM/HSM Sandbox3
Q3 2009
• 380 TB Storage• GPFS• Bladecenter
• Bladecenter• 32 TB raw storage• GPFS TSM/HSM testing
Today’s Discussion
• Validated GPFS-TSM/HSM policy works
• Write speed to tape below expectations
• Backbone storage resource constraint
• File management limitation
• Pipeline workflow flexibility
• Colorworks volume req’mt increased
• Additional TV project ingest
• Colorworks request to write directly to tape accessible by DBB
1. Storage pool for 2012 and no specified throughput requirements.
2. Expanded storage pool to 500TB usable to support future projects TBD. Backbone would grow over 5 years to 24PB of activity. Model for volume of 30TB/day or 350MB/sec for a 24-hour day.
3. Leveraged existing 2012 sandbox to test TSM/HSM integration with GPFS.
4. Full documentation on next 2 pages of this document. 1.8GB/sec throughput.
4
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
In October, SPE finalized its bandwidth requirements for the DBB
5
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
At SPE’s request, IBM presented an alternative workflow to support Colorworks direct write to LTO
6
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
IBM has identified three approaches, and we will focus on VFR with TSM today
Approach Pros Cons
Virtual File Repository (VFR) with TSM
Improves tape speed write performance
Provides pipeline flexibility Leverages GPFS and TSM
VFR is a services offering
TSM/HSM with disk caching Improves tape speed write performance
Standard IBM software Inexpensive at low volume
Scalability limitations of TSM/HSM database
Limitation of one TSM server in GPFS file system
Limited pipeline flexibility – doesn’t support alternate workflow
High Performance Storage Subsystem (HPSS)
High-end data movement services offering
Exceeds SPE’s current scalability requirements
Expensive Implementation resource
constraint Services offering
This document is solely for the use of Sony Pictures Entertainment (SPE) personnel. No part of it may be circulated, quoted, or reproduced for distribution outside the SPE organization without prior written approval from IBM.
© Copyright IBM Corporation 2009
Virtual File Repository and TSM
8
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
VFR in combination with TSM/archive will provide the following services
VFR makes all the files look like they are in a file system– VFR simplifies and virtualizes the TSM interface
VFR aggregates 1000s of files into larger TSM objects
VFR mapfiles store information about all of the individual files that
comprise the larger object
The large objects are archived via standard TSM software
Supports copy and dir/ls
Existing Sony scripts request a set of files for retrieval using the
VFR API
VFR mapfiles are used to locate files on TSM managed tape to
restore them
9
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
The Virtual File Repository (VFR) provides improved performance and functionality
Improved performance
– A group of files are written as a single object on tape
– Larger objects written to tape at a faster rate
– Reduction in load on TSM database Robust design
– The map between tape objects and disk files is stored with the files, both on disk and on tape, and can also be saved independently
Selected file(s) retrieval – saves time and space
– Individual files can be retrieved without retrieving the entire group Enhanced functionality
– Files can be restored to a different location than the saved location, significantly reducing time and space
– Move content directly to tape and bypass the digital backbone storage system – frees ups space on the backbone and reduces bandwidth required
Utilize standard TSM / archive software – provides the movement to tape
10
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Virtual File Repository
SonyScripts
VFR API
TSM
TSM tape
GPFSBackbone
VFR DataMovers
Sony scripts interact with the files through the VFR API, requesting files for applications
VFR moves files between GPFS and TSM.
VFR stores a sequence of files as one TSM object
VFR retrieves single files, partial sequences of files, or an entire sequence of files from TSM
TSMDatabase
SonyApplications
GPFS CW
11
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
It is possible to deploy VFR by January 2010
Phase 1: Enable reading/writing to tape – Store entire directories from disk to tape
– Retrieve entire directories from tape to disk
– Invoke data movers from the command line
– Migration of data to tape via a manually invoked VFR script
Phase 2: VFR API and partial file recall– Retrieve subsets of the directories from tape to
disk
– Invoke data movers from Sony scripts (VFR API)
– Plan for skills transfer to SPE
– Plan for transition support to SPE
2 weeks afterContract Signing and pre-reqs are installed
Requires collaboration with Sony engineers
12
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Workflow database application for scheduling, tracking, tracing, and auditing of data movers
Open Tape support for export and archive. These tapes are self describing without reading the entire tape. They contain an XML index file.
VFR can be expanded to provide additional functionality
13
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
TSM and VFR incremental infrastructure requirements are minimal
2 TSM servers – reuse the existing 3550 M2s
– Add 2 fiber channel HBAs
VFR code will run on existing blades
This document is solely for the use of Sony Pictures Entertainment (SPE) personnel. No part of it may be circulated, quoted, or reproduced for distribution outside the SPE organization without prior written approval from IBM.
© Copyright IBM Corporation 2009
Server and storage infrastructure options
15
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Based on SPE’s revised volumes, IBM modeled the storage requirements
Understand aggregate storage performance requirements
Design a system to meet those requirements
Reuse existing assets
Modeling Objectives
Assumptions: 2 MB block size Varied throughput till one of the system
components began to be stressed
Measures: Internal FC utilization External Host Adapter utilization Hard Drive utilization Processor utilization PCI Bus utilization
Modeling Parameters
This is where most bottlenecks
occurred
16
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
IBM’s modeling produced design recommendations
Loop optimizations DS4800 performs best with disk drawers in even multiples of 4 DS5000 performs best with disk drawers in even multiples of 8
Performance optimizations Increasing quantity and speed of Host Adapters had biggest impact, as would be
expected for a large block size workload SATA disk drives provide sufficient bandwidth if present in sufficient quantity
GPFS performance with mixed subsystems Parallel file workload runs at the speed of the slowest subsystem in the cluster Storage controllers should be balanced
Metadata workloads GPFS metadata and TSM/VFS database should be on separate, high
performance fibre channel subsystems
17
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Moving the metadata to a separate subsystem improves throughput and response time
Coresident vs Offloaded Metadata
0
50
100
150
200
250
300
350
400
450
500
DS48004 x 4 Gb HAs
224 disks (current)Metadata coresident
DS48004 x 4 Gb HAs
224 disks (current)Metadata separate
Th
rou
gh
pu
t in
MB
/sec
0
5
10
15
20
25
30
Res
po
nse
Tim
e in
ms
Coresident metadataLarge block responsetime 35% worse
Coresident metadataThroughput at least 10% worse
Coresident metadataMetadata responsetime 80% worse
18
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
IBM identified four options, all of which leverage the existing assets
1. Keep current disk config, move GPFS metadata onto a separate subsystem 2 x DS4800 with 14 drawers of 1 TB SATA drives
1 x DS5100 with 12 drawers of 1 TB SATA drives
1 x DS3400 with 2 drawers containing 14 x 450 GB 15K FC drives for GPFS metadata
2. Rebalance current disk config, move GPFS metadata onto a separate subsystem 2 x DS4800 with 12 drawers of 1 TB SATA drives (a recommended config)
1 x DS5100 with 16 drawers of 1 TB SATA drives, addnl disk attach feature is added (a recommended config)
1 x DS3400 with 2 drawers containing 14 x 450 GB 15K FC drives for GPFS metadata
3. 2 x DS5100; rebalance all existing drawers; Upgrade SAN with 8 Gb/s blades 2 x DS5100 with 20 drawers of 1 TB SATA drives,
1 DS5100 upgraded w addnl disk attach feature and 8 x 8 Gb/s are added 1 DS5100 net new with addnl disk feature and 8 x 8 Gb/s are added
1 x DS4800 reused with 1 drawer of 16 x 450 GB 15K FC drives for GPFS metadata
4. 2 x DS5300; rebalance all existing drawers; Upgrade SAN with 8 Gb/s blades 2 x DS5300 with 20 drawers of 1 TB SATA drives,
1 DS5300 upgraded from DS5100 w addnl disk attach feature,16 x 8 Gb/s are added; performance upgrade 1 DS5300 net new with addnl disk feature and 8 x 8 Gb/s are added
1 x DS4800 reused with 1 drawer of 16 x 450 GB 15K FC drives for GPFS metadata
19
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
IBM made a number of assumptions in its design
Assumptions Services to reconfigure the disk drawers are included Existing disk subsystems are located in the same location they will
be repurposed Existing SAN Director can be upgraded with additional blades Existing SAN edge switches can continue to be utilized
The workflow provided by SPE does not include the following: Disaster Recovery Local copy (synchronous) services or remote copy (asynchronous) Non-production environments
– Test, Development, Quality Assurance, Etc Process change; head room for growth, etc Workloads different from that modeled Other
Not included No tape upgrades included Cabling and cables not included Remote Support Manager for DS5000 not included
20
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Overall System Performance and PricesDisk Performance Upgrade Options
0
1000
2000
3000
4000
5000
6000
7000
2 x DS4800 14drawers
DS5100 12 drawers
2 x DS4800 14drawers
DS5100 12 drawers
2 x DS4800 12drawers
Upg DS5100 16drawers
Upgraded 2 xDS5100 20 drawers
2 x DS5300 20 drawers
Th
rou
gh
pu
t M
B/s
ec
DS4800 A: 4 x 4 Gb/s HA current
DS4800 B: 4 x 4 Gb/s HA current
DS5100 A: 8 x 4 Gb/s HA current
DS5100 A': 8 x 4 Gb/s HA; addnl disk opt
DS5100 A'': 8 x 8 Gb/s HA; addnl disk opt
DS5100 B'': 8 x 8 Gb/s HA; addnl disk opt
DS5300 A: 16 x 8 Gb/s HA; addnl disk opt
DS5300 B: 16 x 8 Gb/s HA; addnl disk opt
Est. Sony DBB original throughput reqt
Option 1 - only add GPFS metadata ctlr
Option 2 - add metadata ctlr & re-
balance disks
Option 3 - add metadata ctlr; upgr
to DS5100 & 8 Gb/s SAN
Option 4 - add metadata ctlr; upgr
to DS5300 &8 Gb/s SAN
$13,662 $13,662
Base Case
$191,957
$225,256
21
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Overall System Performance and Prices
Options 1 and 2 are not recommended due to potential limited headroom
Disk Performance Upgrade Options
0
1000
2000
3000
4000
5000
6000
7000
2 x DS4800 14drawers
DS5100 12 drawers
2 x DS4800 14drawers
DS5100 12 drawers
2 x DS4800 12drawers
Upg DS5100 16drawers
Upgraded 2 xDS5100 20 drawers
2 x DS5300 20 drawers
Th
rou
gh
pu
t M
B/s
ec
DS4800 A: 4 x 4 Gb/s HA current
DS4800 B: 4 x 4 Gb/s HA current
DS5100 A: 8 x 4 Gb/s HA current
DS5100 A': 8 x 4 Gb/s HA; addnl disk opt
DS5100 A'': 8 x 8 Gb/s HA; addnl disk opt
DS5100 B'': 8 x 8 Gb/s HA; addnl disk opt
DS5300 A: 16 x 8 Gb/s HA; addnl disk opt
DS5300 B: 16 x 8 Gb/s HA; addnl disk opt
Est. Sony DBB original throughput reqt
Option 1 - only add GPFS metadata ctlr
Option 2 - add metadata ctlr & re-
balance disks
Option 3 - add metadata ctlr; upgr
to DS5100 & 8 Gb/s SAN
Option 4 - add metadata ctlr; upgr
to DS5300 &8 Gb/s SAN
$13,662 $13,662
Base Case
$191,957
$225,256
Est. Sony DBB with alternative workflow/
throughput reqt
22
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
In all four options nearly all assets are reused
1. All disks and controllers reused DS3400 added for metadata
2. All disks and controllers reused One existing DS5100 upgraded
DS3400 added for metadata
3. One new DS5100 and upgrade current DS5100 to have 8 x 8Gb/sec Host Adapters and addnl drive attachment (up to 448) All SATA drives and drawers reused
One DS4800 reused, one not reused
4. One new DS5300 and upgrade current DS5100 to DS5300 DS5300s to have 16 x 8 Gb/sec Host Adapters and 16 GB cache
All SATA drives and drawers reused
One DS4800 reused, one DS4800 not reused
1 set of 8 x 4 Gb/s adapters in the DS5100 not reused
23
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
DS4000/5000 Upgrade Options
Upgrades that change the controller only of the DS4000 or DS5000 series are disruptive, but not destructive
– DS5100 can replace a DS4800 in place. This requires an outage but the data remaining on the disk is unchanged
– DS5100 can be upgraded to DS5300 in place. This requires an outage but the data remaining on the disk is unchanged
Reconfiguration of disk drawers is disruptive and destructive– Data must be migrated or backed up and restored
– Migration can be done non-disruptively but requires interim space for the migration or backup
Recommendation is to do controller and SAN upgrade and reconfiguration of disks prior to production
24
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Projected Library Capacity Requirements
Additional Required Frames
SLOTS L53 D53 D53 S54 S54 S54 S54 D53Exisitng Slot Count 2423 287 408 408 1,320 1,320 1,320 1,320 440Required Slot Count 6,701 6,823
10 Feet20 Feet
MONTHS WEEKSTOTAL WEEKS
70TB+35TB PER WEEK
TOTAL TB STORED
18 4.5 81 105 8,505
Assumptions:
• 18 months of stored data at 105 TB of data per week.
• 8.5 PB of storage in month 18.
• The stored data is not compressible.
• The media will fill to 70% of the native capacity.
• LTO4 (560 GB consumed capacity) requires 10,600 slots.
• Current library has 2,423 slots.
• The plan includes moving to LTO5 (1.2 TB consumed capacity) upon its introduction.
• An undetermined amount of LTO4 media will remain in the library.
IBM recommendation:
Add three S54 and one D53 frames (totaling 6,700 slots) upon the introduction of LTO5.
This document is solely for the use of Sony Pictures Entertainment (SPE) personnel. No part of it may be circulated, quoted, or reproduced for distribution outside the SPE organization without prior written approval from IBM.
© Copyright IBM Corporation 2009
Pricing
26
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
TSM and VRF initial costs
TSM SW and installation services included TSM uses existing x3550 servers VFR uses existing servers (Blade Center) Cabling and cables not included Support and maintenance not included
TSM and VFR TSM Services 69,000$ TSM SW 40,000$ VFR services 100,000$
TSM and VFR subtotal 209,000$
27
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Storage Options Pricing Summary
Storage Requirements Option 1 - GPFS metadata ctlr only 13,662$ Option 2 - reconfig drawers 31,912$ Option 3 - 5100s w 8 Gb/s 191,957$ Option 4 - 5300s w 8 Gb/s 225,256$
28
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Our review uncovered licensing and maintenance requirements on the current system
Required ISL license for switches added (current trial license expired) Maintenance on SAN B16 switches added (currently no
maintenance on these switches)
Licensing and Maintenance of Current SystemPurch temp license for ISL 1,260$ Maint on SAN B16 2,100$
Subtotal 3,360$
29
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Optional Considerations
Data movers (x3650) recommended to: – Isolate the data movement requirements of DBB
– Provide additional Backbone ingest and request servicing
Maintenance on the 28 EXP drawers attached to the DS4800s is currently only 9x5. This should be upgraded to 24x7.
The tape library currently only has room for 2-3 months of data at the projected storage rates. Cables and installation will be required. IBM can provide Relocation of disk from Imageworks to Stage 6 not included. IBM can provide BladeCenter SAN infrastructure can be upgraded to 8 Gb/s
Optional DBB ConsiderationsData Mover servers x3650 27,222$ Maint upgrade on EXPs to 24/7 26,891$ Tape Library 120,392$ Cables (est) 5,850$ HS21 HBA upgrade to 8 Gb/s 5,247$ BladeCenter H switch upgrade 4,457$
Optional subtotal 190,059$
30
DBB File Management
Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009
Sony DBB – Recommended Next Steps
• Schedule move of imageworks backbone to the Colorworks location
VFR and TSM:
• Finalize agreement for VFR work
• Phase 1: Begin ASAP
• Phase 2: Can be done in parallel – collaboration with Sony engineers
• Purchase and install TSM software
• “Option B” to install disk cache and TSM/HSM if implementation delays occur
Equipment to support new workload / workflow:
• Select storage option – order / install (some options require more time)
• Other: data movers, maintenance, ISL
Target date for completion: January 1st, 2010 – ready for Spiderman 4
Recommended