View
243
Download
3
Category
Preview:
Citation preview
CSG Storage Survey Results 1
CSG Storage Workshop
CSG Storage Survey Results 2
Workshop Team
• Bruce Vincent (Stanford)• Scotty Logan (Stanford)• Dennis Cromwell (Indiana)• Ron Thielen (U Chicago)• Kitty Bridges (Michigan)• Additional presenters:
– Cory Snavely (Michigan)– Jim Pepin (USC, en route to Clemson)
CSG Storage Survey Results 3
Agenda• 1-1:20 Survey Results (Kitty)• 1:20-2:15 Framing the discussion: definitions,
emerging technologies, policy, architecture (Ron)
• 2:15-2:45 Case Study: Research storage (lots of small files, as in genomics) (Jim)
• 2:45-3 BREAK• 3-3:30 Case study: library archive storage
(Cory)• 3:30-4 Case study: research storage (lots of
big files, as in astronomy, physics) (Dennis)• 4-4:45 Case study: tiered storage architecture,
virtualization (Ron, Scotty)
CSG Storage Survey Results 4
Survey Results
CSG Storage Survey Results 5
Institutions Responding (19)
• Brown• Carnegie Mellon• Columbia• Duke• Georgetown• Harvard• Indiana• MIT• NYU• Penn State
• Princeton• Stanford• University of California• University of Chicago• University of Colorado• University of Michigan• University of Virginia• University of Wisconsin• Virginia Polytechnic
Institute and State University
CSG Storage Survey Results 6
File Service Offered
Distributed file service (AFS, DCE, DFS, WAFS)
42%
Microsoft DFS 11%
CIFS 53%
NFS 47%
WebDAV 47%
Xythos 26%
Other 37%
CSG Storage Survey Results 7
File Service Quotas<100 MB
101-300 MB
301 MB-1 GB
1 GB- 10 GB
>10 GB
Students 4 4 2 8 0
Faculty 2 2 3 8 1
Staff 2 2 3 8 1
Other individuals
2 2 2 6 0
Dept’s 2 2 3 8 2
Courses 2 2 3 5 3
Research groups
2 2 2 5 3
CSG Storage Survey Results 8
How Many Staff Manage It?
Separate groups? Yes: 74% (14) No: 26% (5)
1 to 3 staff 11
4 or 5 staff 6
6 or more 2
CSG Storage Survey Results 9
Staffing
Do you have problems recruiting and retaining skilled storage staff?
Yes= 47.4% (9) No= 52.6% (10)
CSG Storage Survey Results 10
Mitigating these problems?• Adjusted the salary levels for storage staff to reflect the
market.• Hardware and Software Consolidation, automation/tools,
delegation/self-service• By keeping things simple (e.g. no SAN requirements, etc.),
but it is unclear how this will scale.• Standardize on more current mainstream technologies and
move away from technologies with a strong historical legacy but not a significant current relevance such as AFS.
• We're currently using some contractors, and more training for our existing staff, but we're still looking for ideas.
• Take longer to find people who love storage and that is what they want to do as a career.
CSG Storage Survey Results 11
Mitigating these problems?• Staff members with good storage management skills are
just one of the areas where we worry about finding and keeping talent. It is difficult for us to match corporate compensation, but we find that money is not the ultimate decision maker. Training is important and we are diligent about keeping a training budget. We allocate $1400 per FTE in the organization. This is pooled and allocated as appropriate, so it gives us some flexibility to keep staff skills fresh and to build depth in critical areas. We also like to keep work interesting, which is very important to highly motivated technical staff. In addition, we recognize outstanding work with both small monetary bonuses and publicity. Still, we recently lost our best SAN person.
CSG Storage Survey Results 12
Client replication technologies, (MS Offline Files & Folders or Apple's
Portable Home Directories?
Yes=12% (2) No=88% (15)
CSG Storage Survey Results 13
Block storage (11 using, considering)• Considering iSCSI• Don’t provide block storage direct to end-users. We think we
can do it via iSCSI but haven't really done much than ponder the possibility.
• SAN and NAS, as well as DAS• Provide block storage to major campus "services" through EMC
software & the EMC DMX & Clarion storage systems.• Fiber channel primarily iSCSI planned for future• EMC DMX and Clariion• Fibre Channel SAN available to servers hosted in data center• Block storage is only provided in the data center • NAS• NFS, iSCSI and SAN attached.• SAN
CSG Storage Survey Results 14
Storage in Data Center• EMC Clarion and DMX, HP EVA 8000 and 5000, also use CISCO directors
for the SAN• Network Appliance• EMC, NetApps• Almost exclusively EMC; currently researching other vendor's offerings
(SUN, Hitachi and Apple)• IBM, Sun/Hitachi• EqualLogic for iSCSI ACNC for direct attached• Network Appliance, SUN, Apple, Hitachi, IBM• IBM FastT and Shark HP SANStorage• Network Systems: EMC CX700 Storage Array for Databases, FileServices,
Exchange and Backup Storage Brocade Silkworm Switches Storage TEK L180 Tape Library; Administrative Computing: Storage Tek V2X Storage Array for mainframe, AIX Oracle systems Brocade Silkworm Switches Storage Tek Timberwolf Tape Library
• EMC DMX, Clariion HP EVA Equallogic iscsi
CSG Storage Survey Results 15
Storage in Data Center• Cisco MDS 9000 series switches, Brocade Silkworm, EMC Clariion, DMX
and Sun• EMC DMX and Clariion storage arrays, EMC NS704G NAS gateway,
Nexsan ATAbeast and SATAbeast storage arrays, IBM and StorageTek tape libraries, Cisco and Qlogic Fibre Channel directors and switches Emulex and Qlogic Fibre Channel HBAs
• NetApp, Dell/EMC SAN, Sun/Hitachi SAN, StorageTek SAN, IBM SAN• EMC DMX, HDS USP Tagmastore, IBM DS4800, Sun 5320 NAS Gateway• IBM, Sun, and Network Appliances for central services• Primary SAN for Enterprise systems will be an Hitachi Tagmastore,
moving from an IBM Shark. We have a mix of Direct attached storage, SAN and NAS for other areas inlcuding EMC and HP.
• Primarily Sun storage (w/ Brocade switches) for Unix platforms. Windows environments typically have direct attach storage on file servers from Dell.
• IBM mid-range SAN disk, Brocade switches, IBM tape robotics with 3592 drives, NetApp filers, Linux and Windows servers, IBM SAN Volume Controller for virtualization
CSG Storage Survey Results 16
Remote Access
0
2
4
6
8
10
12
native client VPN WebDAV Other
CSG Storage Survey Results 17
Underlying Tech for Service
0
2
4
6
8
10
12
14
16
File Service Databases WebPublication
Other
Direct attached storage
Network attached storage
Storage attached network (SAN)with iSCSI
Storage attached network (SAN)with Fiber channel
CSG Storage Survey Results 18
Specialized Storage for HPC?
Yes= 37% (7) No= 63% (12)
NetAppSort of, we provide Network Appliance filer space to our research clusters.EMC DMX2000 and DMX800 via Cisco SAN FabricBlueArc (and other dedicated NFS servers)IBM SANIBM DS4800
CSG Storage Survey Results 19
If one backend technology for all storage (7 reporting)
• Moving into that direction - virtualized storage - but not there yet. We use the IBM SAN Volume Controller (SVC) on the hDSAN side to virtualize across the EMC and HP storage arrays.
• Network Appliance Filers• Some is on direct attached storage ( a case by case choice by
the user / service) but the vast majority is on EMC storage over a fiber channel SAN
• SAN technology with an EMC Clariion storage array that provides shared access to centralized storage.
• EMC DMX and Clariion FC storage presented via FC McData switches, and CIFS via EMC NS704G NAS gateway. Expect to expand to NFS and CIFS in future.
• HDS Tagmastore USP 1100
CSG Storage Survey Results 20
Backing it up
TSM 12 (was 13)
TSM and replication 3
Legato 2
Atempo 1
Amanda 1
Veritas 1
Disk-based MAID technology 1
CSG Storage Survey Results 21
Meets Needs?
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Unacceptable Poor OK Pretty good Excellent
CSG Storage Survey Results 22
Backing up Mobile Devices?• mobile devices = PDAs - no. mobile devices
= laptop - yes, moving to LiveBackup from Atempo.
• Connected desktop backups• TSM (3)• Support for these devices is provided
through a server that is backed up with our Backup Services. The only supported application is based on MS Exchange and utilizes existing mailboxes already on the servers and being backed up
CSG Storage Survey Results 23
Desktop Backup
0
1
2
3
4
5
6
7
8
9
connected tsm atempo veritas
We are investigating providing desktop and laptop backup and have made a short list of vendors in this space. Included are: Atempo, Asigra and Iron Mountain
CSG Storage Survey Results 24
Tiered Solution (17)• TSM (4)
– All of our (3) tiers of storage use the same core back up technology (TSM) which allows individual customizations of timing and overall strategy
– Multiple tiers of disk storage, but predominantly use TSM clients on the hosts for backup. A few exceptions for our large ERP systems where we clone the volumes storing the database and back them up from a dedicated server to minimize the backup window
– TSM, NSR, IBM's "flashcopy" and NetApp's snapshots– IBM Tivoli Storage Manager policy domains, management
groups, and storage groups allow us to implement numerous combinations of storage tiers and timing strategies for different requirements.
CSG Storage Survey Results 25
Tiered Solutions• Use the traditional Unix methodology of daily, weekly, and
monthly incremental backups.• See detailed slide deck (Walter Wong from CMU)• We haven’t done any tiered data with backups, we do however
utilize different disk solutions based on the service requirements. For Example: Exchange: We use only Fiber Channel 15000 RPM Drives File Servers: We use Raid 5 ATA Drives Backup 2 Disk: We use Raid 3 ATA Drives
• 2 tiered storage based on perf/avail. No backup distinction yet.• Use Veritas Netbackup for large backups that require a small
window to complete because of the technology Vertias provides with support for multiple streams we can complete backups more quickly than we can with TSM.
• All issues we need to consider though!
CSG Storage Survey Results 26
Separate Archive Service• Operate a ADIC StorNext HSM for research data that sort of fits
this mission.• TSM (5)
– We currently do minimal archiving using a function in TSM and store both to tape and to disk.
– We provide archive via IBM Tivoli Storage Manager for servers and services hosted by IT Services.
– Do not provide a separate archive service and currently use the archive component of both TSM and Legato
• We do some end-of-semesters that are archived for 5 yrs.• Minimal archive service for CIS internal use only, using an archive
to tape with Legato Networker. We see archival strategies as a means to increasing usable storage and will be looking into other application and file system archive solutions in the near future. Mainframe environment: monthly archive of unchanged files to tape.
CSG Storage Survey Results 27
Library Archives Service (19)
Library doesn’t provide service = 68.4% (13)
Library alone provides service = 26.3% (5)
Library provides a service in partnership with central IT = 5.3% (1)
CSG Storage Survey Results 28
Considered Outsourcing?• We have a partnership with the Duke Health System. They have a large
SAN for health care data that we leverage for top tier storage.• Twice researched the option and rejected it due both to cost and loss of
local flexibility.• Have looked at outsourcing backup twice. Both times the cost benefit was
not evident. When you assess in detail the breakdown of roles/resp breakdown between customer and provider, there is still a significant resource commitment on the customer side.
• Considered outsourcing our desktop backups. We had issues with lack of support for non-Windows platforms, some support and bandwidth problems, and no way to take a single charge from the vendor and break it out to re-bill our users.
• This topic brings perceived security/privacy concerns to the surface; a cost/benefit analysis will be interesting; outsourcing may fit as a piece of the life cycle management of information; we are just at the early stage of "consideration."
• Considered it but found we could do it cheaper by centralizing all of campus
CSG Storage Survey Results 29
BC/DR Impact• Increased the storage requirements and increased the importance of
the storage strategy. A driving factor in our desire to explore a more centralized enterprise storage architecture.
• Customers are requesting shorter RPO's and RTO's and in some cases are willing to accept longer RTO's if RPO can be improved. Causing us to more seriously assess replication, quasi single instance stores, virtualization etc. Pushed us toward service mirroring, replication – For top-tier storage, the SAN is in a different location than the ATLs. TSM
and offsite tape storage is also used to provide recovery capabilities.– Working on a project with Virginia Tech to leverage our NLR connections
to hold copies of critical data at their location.– We must modernize and update to new software, new hardware, employ
disk-to-disk-to-tape, and use off-site replication of storage arrays.– Have looked to adopt technologies that support replication and failover in
order to support an enhanced DR strategy.– SAN extension and replication combined with both server and storage
virtualization enable us to address BC and DR requirements in ways that were previously unapproachable.
CSG Storage Survey Results 30
BC/DR• Post 911 BCP and DRP awareness has increased dramatically but
unfortunately that awareness has not been accompanied by significant increased funding. We are now faced with almost uncontrolled growth in amount of stored data. Since we actively mirror all institutional data the end result of the explosion in data storage has been the consumption of our entire storage budget in "storing". Our costs have grown too fast for us to be able to mount a detailed and effective BCP.
• Currently in the process of re-evaluating our entire backup and storage architectures. Also driven by the new compliance issues around Electronic Evidence and E-Discovery legislation.
• Current disaster recovery strategy is ambiguous. Nightly copies of our backup tapes are sent to Iron Mountain for storage. In the event of a true loss of datacenter disaster we would look for assistance from our vendors and peer institutes to assist us in our recovery. We have taken under advisement the need to have a well documented and institutionally driven business continuity plan. When we do move forward in developing this there should be many driving factors for storage, including requirements for data redundancy, archival, and data accessibility.
CSG Storage Survey Results 31
Summary of Unsolved Problems
• Funding, funding models, costs• Smart data storage (data de-duplication,
compression, life cycle management)• Multi-platform, with as close to native access as
possible• Replacing current distributed file services (DCE,
DFS)• Virtualization and tiering• More, more, more (and staying ahead of or on par
with demand)
CSG Storage Survey Results 32
Details on Unsolved Problems
• In the process of implementing a broad array of storage, backup and recovery services. Developing the detailed delivery approaches, the funding/business models and the policy/procedure details will require considerable effort in the coming year.
• Data de-duplication technology will make a huge improvement in our ability to adequately backup our data
• Cost control which equates to storage volume control. We are actively researching the acquisition of an automated archiving solution which would necessarily include storage classification tools / software so we can control both what and where (in terms of tiers and/or archiving) we store as well as simply how much. We are also looking in to adding a 4th tier of storage with lower performance and (hopefully) greatly reduced costs
• A good replacement for what we have in DCE/ DFS now.
• Archival – Desktop Backup – Centralized Storage for Research Computing
• True virtualization of file and storage infrastructure.
CSG Storage Survey Results 33
Details on Unsolved Problems
• Universal client access with a reasonable authentication / authorization environment doesn't exist. vendor storage implementations are about 10 years behind commonly accepted ideas in storage/filesystem research and they don't seem to be generally aware of that fact. scalable horizontal growth of storage is just starting to get mainstream and usually is appearing with the iSCSI vendors. backup technology is slow to move to CDP ("continuous data protection") and still rooted in old models that don't fully leverage the cheap storage that is available. transparent tiering of data based on usage with policy is barely a glimmer in commercial products. Selling "ILM" solutions is confusing the market place and solutions for properly tagging data is being confused in or buried by marketing babble data scrubbing of failed disk drives in RAID sets doesn't seem to be in the common thought process yet; similarly, disk level encryption is still in its infancy; tape encryption is getting better but that is still slow. key escrow of all this is also a pain. "disks are cheap; storage is expensive" isn't common knowledge so there is unreasonable pressure to provide "enterprise" class storage at commodity pricing. Compare the cost of a 750GB drive ($0.60 per GB) vs. "cheap RAID" ($1.33 per GB) vs. "enterprise class SAN" ($10+ per GB).... and none of these costs include replication for BC/DR, backups, etc (though it does include RAID and hot-spare overheads).
CSG Storage Survey Results 34
Details on Unsolved Problems
• The need to provide a variety of network file system solutions (cifs,nfs, iscsi) and not a good platform for enabling that type of access easily to the storage infrastructure we have built.
• No single file protocol that works well from on and off campus on Linux, Mac and Windows without extra client software, provides rich access controls and supports our central authentication service. Enterprise scale laptop backup software is still lacking for non-Windows platforms. Key escrow for device and backup encryption is poorly supported by vendors.
• Growth of digitization of information, information life cycle management, encryption, compression
• Petabyte storage solutions for researchers. Trying to find a scalable way to store tens of Petabytes w/o managing multiple storage arrays.
• Adequate funding to keep up with the ever increasing demand within the university community
• Increasing demand for centralized file services from departments. Multi-site redundancy of storage for critical services. An enterprise backup solution that would allow us to provide backup services for department with local storage.
CSG Storage Survey Results 35
Summary• Growth in data is a huge problem - an unfunded
mandate in higher ed– Federal and other requirements for keeping and protecting
data for longer periods– Unmanaged data is becoming a larger problem - we’re just
keeping ever ything because it is too hard to clean things up
• Inefficiency– Not aligning least used data with least expensive solution– Backing up a lot that doesn’t need backup, not backing up
what does
• Cost and funding models• Technology pieces complicated to knit together into a
solution
CSG Storage Survey Results 36
NEXT: Setting the Stage & Case Studies
• Setting the stage– Defining terms, regulations, architecture
• Case studies– Library archive– Large data stores
• Small data files• Large data files
– Tiered storage and virtualization– (Case Study on website)
Recommended