Upload
rygetz
View
68
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Operating in Cluster-Mode
Citation preview
Technical Report
Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Operating in Cluster-Mode Reena Gupta, Sandra Moulton, NetApp
September 2012 | TR-4078
Executive Summary
This technical report provides best practices and recommendations for the deployment and
optimal performance of NetApp® Infinite Volume. Infinite Volume is a new innovation in
scalable NetApp storage systems that allows customers to add storage as they grow without
increasing operational complexity. This document provides functional and operational
guidance for Infinite Volume in the Data ONTAP® 8.1.x architecture operating in Cluster-Mode.
2 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
TABLE OF CONTENTS
1 Introduction ........................................................................................................................................... 3
1.1 Purpose and Scope ........................................................................................................................................3
1.2 Intended Audience ..........................................................................................................................................3
2 Target Workloads and Use Cases ....................................................................................................... 4
3 Features of Infinite Volume ................................................................................................................. 5
4 Infinite Volume Deployment ................................................................................................................ 5
4.1 Supported Configuration .................................................................................................................................5
4.2 Components of an Infinite Volume ..................................................................................................................6
4.3 Aggregate Configuration .................................................................................................................................6
4.4 Cluster Configuration ......................................................................................................................................8
4.5 Vserver Configuration .....................................................................................................................................9
4.6 Configuration of the Infinite Volume ................................................................................................................9
4.7 Mounting the Infinite Volume......................................................................................................................... 10
4.8 Maxfiles and Maxdirsize ................................................................................................................................ 10
4.9 Upgrading and Reverting .............................................................................................................................. 11
5 Infinite Volume Operations ................................................................................................................ 11
5.1 Managing with System Manager 2.1 ............................................................................................................. 11
5.2 Capacity Management .................................................................................................................................. 12
6 Data Protection ................................................................................................................................... 14
6.1 Snapshot Copies ........................................................................................................................................... 14
6.2 Protection for Namespace Constituent ......................................................................................................... 15
6.3 SnapMirror for Infinite Volume ...................................................................................................................... 16
7 Storage Efficiency .............................................................................................................................. 19
References ................................................................................................................................................. 20
Version History ......................................................................................................................................... 20
LIST OF TABLES
Table 1) Target use cases for Infinite Volume. ...............................................................................................................4
LIST OF FIGURES
Figure 1) Infinite Volume overview. ............................................................................................................................... 3
3 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
1 Introduction
NetApp Infinite Volume is a software abstract hosted over Data ONTAP 8.1.x operating in Cluster-Mode
that provides scalable capacity. Infinite Volume is similar to a NetApp FlexVol® volume in Data ONTAP
except that Infinite Volume can scale up to 20PB of storage capacity without any operational complexity
involved with the management of multiple smaller volumes.
Infinite Volume provides a single mount point accessible over the NFS v3 protocol, and it integrates with
NetApp technologies and products such as NetApp Snapshot™
technology, deduplication, compression,
NetApp SnapMirror® technology, and OnCommand
® System Manager. An Infinite Volume is a compound
volume that consists of multiple constituent volumes, namely, a single namespace constituent and
multiple data constituents. It also includes a namespace redirection layer that provides access to data
files in the data constituents because the data files are not directly visible to the client.
Figure 1 illustrates Infinite Volume as a single large container with billions of files stored in multiple data
constituents.
Figure 1) Infinite Volume overview.
1.1 Purpose and Scope
This document describes the best practices and recommendations for NetApp Infinite Volume, introduced
in Data ONTAP 8.1.x operating in Cluster-Mode. This document provides guidance and best practices for
the configuration, installation, and operation of Infinite Volume, but it does not include the detailed
configuration steps. For the configuration procedure, refer to Data ONTAP 8.1.x operating in Cluster-
Mode documentation. This guide is specific to Infinite Volume and does not cover the general best
practices for Data ONTAP operating in Cluster-Mode or FlexVol volumes.
1.2 Intended Audience
This document is for technical audiences such as storage administrators, architects, system engineers,
professional services engineers, and field personnel responsible for architecting and deploying NetApp
Infinite Volume. This guide assumes knowledge of NetApp storage systems, management of Data
ONTAP Cluster-Mode, Data ONTAP replication technologies, and flexible volumes.
4 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
2 Target Workloads and Use Cases
The Infinite Volume was developed to provide a scalable, cost-effective solution for big content
workloads. Specifically, Infinite Volume addresses the requirements for large, unstructured repositories of
primary data that are also known as enterprise content repositories.
Enterprise content repositories can be subdivided into workloads with similar access patterns and data
protection, protocol, and performance requirements. The first version of NetApp Infinite Volume focuses
on a subsegment of enterprise content repositories called fixed content repositories.
A fixed content repository is characterized by input/output (I/O) patterns in which data is written once and
seldom read. Fixed content, by nature, does not change, nor do updates occur. However, because this
data is used for normal business operations, the content must be kept online for fast retrieval, rather than
moved to secondary storage.
Table 1 describes the target use cases for Infinite Volume.
Table 1) Target use cases for Infinite Volume.
Attribute Product Capability Target Workload Requirements
Infinite Volume in Data ONTAP 8.1.x Fixed Content Repository
File types Unstructured data over NFSv3 Reference rich media Web content
Video, images, audio HTML, ASP, PHP, WebApps
Requirement Large-scale single-file system with high availability and storage efficiency
Online content repository for transcoded media
Maximum price/performance ratio
Typical access pattern
Optimized for sequential read or mix of random I/O with moderate performance
Write once, read seldom
Sequential access, primarily read
Write once, read often
Sequential access, mix of read and write
Capacity Up to 20PB volume or up to two billion files
~5PB volume Multi-PB volumes
Throughput Up to 5GB/sec in a cluster <5GB/sec <5GB/sec
Data protection RAID-DP® technology, Snapshot,
SnapMirror, NFS-based tape backup Snapshot, SnapMirror Snapshot, SnapMirror
5 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
3 Features of Infinite Volume
Infinite Volume features can be grouped into the following categories:
Manageability: The Infinite Volume manageability features include:
A single namespace for the cluster
Scalability in capacity and performance
Capacity balancing at ingest
Simplified setup
Data protection: Infinite Volume supports the following methods of data protection:
RAID-DP protection for disks
Snapshot copies at the Infinite Volume level
SnapMirror at the Infinite Volume level
NFS-mounted tape backup
Efficiency: Infinite Volume supports the following NetApp storage efficiency features:
Deduplication
Compression
Thin provisioning
Resiliency: Infinite Volume supports the use of NetApp’s storage resiliency technologies, such as:
Storage failover
Cluster-Mode protection
Nondisruptive operations
Performance acceleration with Flash Cache
For details about Infinite Volume features, refer to TR-4037: Introduction to NetApp Infinite Volume.
4 Infinite Volume Deployment
The Infinite Volume solution has specific requirements and configurations for deployment in a Data
ONTAP 8.1.x Cluster-Mode environment. It’s important to understand the supported configuration and
deployment steps, because an Infinite Volume is targeted for very specific use cases and workloads.
Although Infinite Volume creation may appear very similar to FlexVol volume creation, there are several
differences.
4.1 Supported Configuration
An Infinite Volume is a single, large, scalable file system that supports up to 20PB (petabytes) of storage
or up to two billion files. Infinite Volume hosted over Cluster-Mode supports the following configuration.
FAS/V6280 and FAS/V3270 systems
Up to a maximum of 10 nodes or 5 HA pairs in a cluster
NFSv3 protocol only
Maximum raw capacity of 20PB
A maximum of 2 billion files
Maximum throughput of 5GB/sec for a 10-node cluster
The Infinite Volume configuration in Cluster-Mode requires a dedicated cluster because it supports one
Infinite Volume in a Vserver and one Vserver for the entire cluster, with the exception of the root Vserver.
6 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
4.2 Components of an Infinite Volume
An Infinite Volume includes the following components.
Aggregate. This is a collection of disk drives configured with RAID-DP.
Node. This is the NetApp storage controller participating in Cluster-Mode.
Namespace constituent. This is a type of flexible volume that contains the directory hierarchy and file names. The file names are associated with redirector (pointer) files, each of which refers to a data file in a data constituent. Only one active namespace constituent is allowed in an Infinite Volume.
Data constituent. This is a type of flexible volume that stores data files. Multiple data constituents are allowed in an Infinite Volume; however, a minimum of two data constituents is required.
Vserver with Infinite Volume. This is a virtual server with a special attribute that hosts the Infinite Volume.
NFSv3 protocol. This protocol provides access to the Infinite Volume from an NFS client through a single mount point.
4.3 Aggregate Configuration
On a NetApp FAS system, aggregates can be created using RAID 4 or RAID-DP. NetApp highly
recommends using RAID-DP for Infinite Volume because it offers excellent resiliency and nondisruptive
background firmware updates for the disk drives. For optimal performance and resiliency NetApp
recommends configuring RAID groups using RAID-DP in groups of 20 disks (18 for data and 2 for parity).
For the Infinite Volume in Cluster-Mode, aggregates are dedicated to data constituents; that is, one data
constituent per aggregate.
Aggregates can be created by using System Manager or the Data ONTAP command line interface (CLI).
The following command creates the aggregate through the CLI. A RAID-DP aggregate will be created by
default.
cluster::> aggr create -aggregate <aggregate name> –nodes <nodename> -diskcount <# of disks> -
disktype <SAS|FCAL|SATA|BSAS>
In System Manager 2.1, when creating the aggregates as part of the Create Vserver wizard and the
Vserver with Infinite Volume option is selected, only RAID-DP aggregates are allowed to be created.
When creating the aggregates using the Aggregate Create wizard, both RAID 4 and RAID-DP aggregates
are allowed to be created.
For maximum usable capacity with optimal performance and resiliency, NetApp recommends creating
each aggregate as follows:
FAS6280: 40 3TB disks configured in 2 RAID-DP groups of 20 disks (18 data + 2 parity)
FAS3270: 20 3TB disks configured in 1 RAID-DP group (18 data + 2 parity)
If this configuration is not possible, NetApp recommends creating an aggregate 10% larger than the
desired constituent size up to the maximum FlexVol size supported on the platform. Because an
aggregate is allowed to hold only one constituent and the constituent can’t exceed the maximum
supported FlexVol size, NetApp does not recommend creating an aggregate with the maximum allowable
capacity of 162TB for the FAS6280 and 105TB for the FAS3270. For the FAS6280, the maximum
aggregate size for a constituent should be no larger than 100TB; for the FAS3270 it should be no larger
than 68TB.
To see the size of your aggregates you can look at the following:
Systems Manager, Configure Aggregates portion of wizard, “Available Space” value
Systems Manager Aggregates tab, the “Total Space” for the aggregate
Command line, the value shown for “Size” from the output of the aggr show command
7 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
For maximum aggregate throughput, distribute the aggregates evenly across all the available nodes in
the cluster. For example, if there are 4 nodes in a cluster and 12 aggregates need to be created, you
should create 3 aggregates per node to get maximum throughput from the entire cluster.
If there are aggregates of different sizes in a cluster, System Manager selects the largest available
aggregate for the namespace constituent and then selects the smallest available aggregate to place the
Vserver root volume on.
Aggregates can be either created before creating the Vserver with Infinite Volume or as part of the Create
Vserver wizard in System Manager. By default the System Manager Vserver with Infinite Volume wizard
will create aggregates with a fixed RAID group size of 16. If you want to create a different RAID group
size such as the recommended maximum configuration, which includes 20-drive RAID groups,
aggregates should be created beforehand using the standard Cluster Administration Aggregate Create
tool. After that, the Create Vserver with Infinite Volume wizard should use all the available aggregates to
create the Infinite Volume spread across all these aggregates.
Best Practices
Create aggregates no larger than 100TB on the FAS6280 and 68TB on the FAS3270 to achieve the maximum usable capacity for each individual constituent. Creating aggregates larger than this will result in wasted space. Create aggregates that will be used for the namespace and data constituents as follows (this requires the aggregates to be created using the Cluster Administration Aggregate Create tool instead of having the Create Vserver with Infinite Volume wizard create them):
- FAS6280: 40 3TB disks configured in 2 RAID-DP groups of 20 disks (18 data + 2 parity)
- FAS3270: 20 3TB disks configured in 1 RAID-DP group (18 data + 2 parity)
Make sure the aggregates are created with the RAID type set to RAID-DP.
Design the aggregates taking the namespace aggregate and the namespace data protection (DP) mirror aggregate into consideration, and use the disks accordingly.
Create the namespace DP mirror aggregate only after the creation of the Infinite Volume.
Create an aggregate for the namespace DP mirror that is equal in size to the active namespace aggregate.
A minimum of four aggregates is required in the cluster to create an Infinite Volume; one aggregate is required for the Vserver root volume, one for the namespace constituent, and two for the data constituents.
Keep the Vserver root aggregate to be the smallest possible three-disk RAID-DP aggregate and allow the System Manager Create Vserver wizard to create this aggregate as part of the Configure Aggregate option. You can rename this aggregate as appropriate for your Vserver.
Before creating a Vserver for Infinite Volume, create all aggregates except for the aggregate needed for the namespace DP mirror, or else that aggregate will be consumed during the creation of the Infinite Volume. Verify that the aggregates used for namespace and data constituents are not larger than 100TB on the FAS6280 and 68TB on the FAS3270.
If creating an Infinite Volume on a FAS3270, validate that the total Infinite Volume size the wizard is going to create does not exceed ((62.9TB * (<# aggregates in the Infinite Volume> - 1)) + 10TB). If it does you need to manually decrease the value to no more than (62.9TB * <# data constituents>) – 10TB for a successful Infinite Volume creation.
For better throughput, distribute the aggregates evenly across all the available nodes in the cluster.
Note:
Only one constituent per aggregate is allowed.
Infinite Volume supports only 64-bit aggregates.
The WAFL® file system or RAID performance characteristics of aggregate sizes will impact the overall
system performance.
8 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
4.4 Cluster Configuration
A Data ONTAP cluster is a group of nodes that can function as a single powerful system. Before an
administrator decides to create an Infinite Volume in a Data ONTAP cluster, it’s important to understand
the following requirements for a cluster.
Infinite Volume requires a dedicated Data ONTAP cluster, because there is only one Infinite Volume supported in one cluster.
In Data ONTAP 8.1.x, an Infinite Volume cannot coexist with a FlexVol volume. The only FlexVol volumes that can be present in the cluster are the node root volume and the Vserver root volume. Both of these volumes have their own dedicated aggregates.
An Infinite Volume supports a maximum of 10 nodes in a cluster with 2PB raw capacity per node.
Apart from the node and Vserver root aggregates an Infinite Volume can span a maximum of:
- FAS/V6280 with DS424x: 140 (1 NS + 1 NS Mirror + 138 DCs)
o 28 aggregates per HA pair
- FAS/V6280 with DS4486: 175 (1 NS + 1 NS Mirror + 173 DCs)
o 35 aggregates per HA pair
- FAS/V3270 with either DS424X or DS4486: 235 (1 NS + 1 NS Mirror + 233 DCs)
o 47 aggregates per HA pair
To deploy an Infinite Volume solution, the first step is to bring up a basic Data ONTAP 8.1.x cluster on
either a FAS/V6280 or FAS/V3270 platform. To create a basic cluster, follow the procedure described in
Data ONTAP 8.1.x Cluster-Mode documentation. The cluster should have only the node root aggregates
and volumes created and should not have any regular Vserver or FlexVol volumes existing in the cluster.
Once the cluster is created, you can use System Manager 2.1 to manage this cluster and complete the
Infinite Volume configuration.
Once a cluster has been set up for an Infinite Volume, it will be restricted for use only with that Infinite
Volume. If you want to use this cluster for FlexVol volumes, you must delete the Infinite Volume, remove
the Vserver with Infinite Volume, and create a Vserver with FlexVol and flexible volume for Cluster-Mode.
Best Practices
Dedicate a cluster for Infinite Volume. Deploy the Infinite Volume on a new cluster running Data ONTAP 8.1.x Cluster-Mode, or clean up an existing cluster by removing the Vservers and nonroot FlexVol volumes.
If deploying Infinite Volume on an existing cluster, make sure that the existing Vservers and regular FlexVol volumes are removed. The cluster should essentially be a new cluster, with only node root volumes.
Note:
Only System Manager 2.1 or the Data ONTAP CLI will support the management of a cluster associated with Infinite Volume.
Only one Vserver with Infinite Volume is allowed in a cluster.
Only one Infinite Volume is allowed in a Vserver or in a cluster for Data ONTAP 8.1.x.
9 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
4.5 Vserver Configuration
Infinite Volume resides in a special Vserver in a cluster called “Vserver with Infinite Volume” with a special
attribute only for Infinite Volume. This special Vserver can be created through the Data ONTAP CLI or the
System Manager 2.1 Create Vserver wizard by selecting the “Vserver with Infinite Volume” option. For
configuration details, refer to Data ONTAP 8.1.x Cluster-Mode documentation.
If any existing Vservers are detected during the creation of the Vserver for Infinite Volume, an error message will be displayed and the creation of the Infinite Volume will fail. Once you have a Vserver with Infinite Volume, you can know that it is the only Vserver in the cluster and you will not be able to create nonroot FlexVol volumes in it.
The Create Vserver wizard requires a minimum of three aggregates to create an Infinite Volume: one for
the namespace constituent and two for the data constituents. Additionally, it picks the smallest aggregate
available in the cluster for the Vserver root volume.
Best Practices
Make sure that there are no existing Vservers in the cluster and that the cluster is as good as new, with only node root volumes.
For the Vserver root aggregate, NetApp recommends allowing System Manager to create the smallest possible three-disk RAID-DP aggregate for the Vserver root volume in the Create Vserver wizard.
For better throughput, create multiple data LIFs, at least one for each NIC in the cluster. Reserve some NICs for cluster-peer interfaces.
Note:
Only one Vserver with Infinite Volume is allowed in a cluster.
In Data ONTAP 8.1.x, the Vserver creation function is enabled only through a product variance request (PVR) process. Contact your local sales representative for details about the PVR.
If creating an Infinite Volume on a FAS3270, validate that the total Infinite Volume size the wizard is going to create does not exceed ((62.9TB * (<# aggregates in the Infinite Volume> - 1)) + 10TB). If it does you will need to manually decrease the value to no more than ((62.9TB * (<# aggregates in the Infinite Volume> - 1)) + 10TB).
4.6 Configuration of the Infinite Volume
An Infinite Volume is a single large container accessible over the NFSv3 protocol. It can be created
through the Data ONTAP CLI or the System Manager 2.1 Create Vserver wizard by selecting the “Vserver
with Infinite Volume” option. For configuration details, refer to Data ONTAP 8.1.x Cluster-Mode
documentation.
Before creating an Infinite Volume, it’s important to understand the following requirements.
Infinite Volume supports only the NFSv3 protocol.
Only one Infinite Volume can be created in a Vserver and a cluster.
Only one namespace constituent can be created in an Infinite Volume.
Infinite Volume uses only UNIX®
security.
An Infinite Volume can sometimes be in a “mixed” state. A mixed state is only applicable to Infinite
Volumes and it indicates that one or more constituents of the Infinite Volume are offline. This state can
occur when:
One of the nodes is offline.
One of the aggregates is offline.
One of the constituents is offline.
10 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
4.7 Mounting the Infinite Volume
An Infinite Volume is mounted on an NFSv3 client as a regular volume by using the same mount
command. There is only one path or junction point for Infinite Volume: the junction path of the namespace
constituent. By default the name for this junction path is “/NS” when the volume is created using the CLI,
but it can be changed to any name. An example of a simple mount command is:
# mount Cluster01-lif1:/NS /mnt/bigvol
The Infinite Volume does not support NFS over UDP and only supports NFS over TCP. Most of the UNIX
clients support rsize and wsize (read/write block size) as 64k by default, but on Red Hat Enterprise
Linux® version 5.3-5.6, Infinite Volume should be mounted with rsize and wsize, specified as 64k.
Export Policy
The Vserver with Infinite Volume and the Infinite Volume are configured with multiple default export
policies.
System export policies: The following policies protect the system’s internals.
repos_root_readonly_export_policy: Applicable only for the Vserver root volume.
repos_restricted_export_policy: Applicable only for the data constituents.
Client access export policies: The following policy controls client access to the Infinite Volume.
repos_namespace_export_policy: Applicable only for the namespace constituent.
By default, the client access export policies control a client’s read and write access to files in the Infinite Volume. You can add export rules to the export policies to further control client access. The
storage system creates the repos_root_readonly_export_policy export policy when you
create a Vserver with Infinite Volume and then creates the repos_namespace_export_policy
and repos_restricted_export_policy export policies when you create the Infinite Volume.
An Infinite Volume can use only one export policy at a time. You can add export rules to the
repos_namespace_export_policy to restrict client access as needed.
For more information on the export policies for Infinite Volume, refer to the “File Access and Protocol
Management Guide for Data ONTAP 8.1.x Cluster-Mode.”
Best Practices
Do not modify the repos_root_readonly_export_policy and
repos_restricted_export_policy export policies, because these export policies restrict the
client from accessing the system files.
The name of the junction path for an Infinite Volume can be only a single element, for example, “/NS.” It can’t be two elements, such as “/vol/NS” or “/NS1/NS2.”
4.8 Maxfiles and Maxdirsize
An Infinite Volume is capable of supporting a maximum of two billion files, which is based on the number
of redirector files supported in the namespace constituent. The namespace constituent size can grow up
to a maximum capacity of 10TB to accommodate the two-billion-file count. In Data ONTAP, a single file
can be as large as 16TB in size. The limits imposed on the maximum number of files and the maximum
number of subdirectories for a regular FlexVol volume in Cluster-Mode are also applicable to the Infinite
Volume.
If the namespace constituent size is 10TB or more, adding more disks to the aggregate containing the
namespace does not increase the space in the namespace constituent and the maximum number of files.
It does help performance by leveraging the additional spindles in the aggregates.
11 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
If the namespace constituent size is less than 10TB, adding more disks to the aggregate containing the
namespace (explained later in the document) does not automatically increase the maximum number of
files in the namespace constituent. It has to be manually increased at the Infinite Volume level using the
CLI.
In Data ONTAP 8.1 and higher, there is no limit to the number of subdirectories; a user can continue to
create subdirectories as long as the maximum directory size and the maximum number of files limit are
not exceeded. For more information on this topic and the commands, refer to the “Data ONTAP 8.1.x
Logical Storage Management Guide.”
It is possible that an existing data file cannot be appended to expand the file size even if the file size is
less than 16TB and despite there being 16TB or more free space available in an Infinite Volume. The
reason is that the Infinite Volume free space is the total sum of the free space of all the data constituents.
However, the maximum size to which a file can grow is limited by the free space of the specific data
constituent in which the file resides. It might also be difficult to diagnose this problem, because it's hard to
determine in which data constituent the file is located. NetApp advises contacting NetApp Global Support
for help in such a situation.
4.9 Upgrading and Reverting
The Infinite Volume feature is available only in Data ONTAP 8.1.x Cluster-Mode. An upgrade from
previous versions of Data ONTAP may have existing Vservers and FlexVol volumes. After upgrading, to
create an Infinite Volume, first remove the existing Vservers and nonroot FlexVol volumes (except for
node root volumes). Once the cluster is ready as a new cluster, you can proceed with the new Vserver
and Infinite Volume creation.
Similarly, a downgrade from Data ONTAP 8.1.x Cluster-Mode to a previous version requires the deletion
of the Infinite Volume and the Vserver with the Infinite Volume and the removal of the namespace mirror,
if created.
5 Infinite Volume Operations
5.1 Managing with System Manager 2.1
NetApp System Manager 2.1 provides easy manageability of the Infinite Volume feature. System
Manager 2.1 has the following built-in wizards to manage the Infinite Volume.
Create Vserver. This wizard helps to create a Vserver for the Infinite Volume, to create Infinite Volume, and to enable the Vserver for NFSv3 access with data LIF creation and export policies.
SnapMirror Relationship Create. This wizard helps to create a SnapMirror relationship between a source Infinite Volume and a disaster recovery site so that the source Infinite Volume data can be replicated to a SnapMirror destination volume for disaster recovery purposes.
In addition to these wizards, System Manager provides an interface for the management of cluster,
aggregates, Snapshot copies, storage efficiency, NFS protocol, export policies, expansion of the volume,
space monitoring, and namespace.
Best Practice
Make sure that the cluster is added to System Manager before starting the management of Infinite
Volume or the Vserver.
Note: Follow the best practices for individual functions separately as documented in this guide, even if those functions are managed through System Manager.
12 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
Note:
NetApp recommends using System Manager 2.1 for managing the Vserver and Infinite Volume.
Data ONTAP 8.1.x requires a PVR to get the Vserver enabled for Infinite Volume. Contact your local sales representative for details about the PVR.
System Manager allows manageability at the Infinite Volume level, not at the individual constituent level. If there are specific tasks that need to be performed at the constituent level, contact NetApp Global Support for guidance.
5.2 Capacity Management
Auto Capacity Balancing for File Placement
Infinite Volume provides capacity balancing across the nodes as data is ingested. When new files are
written to an Infinite Volume, they are distributed evenly across data constituents on a round-robin basis,
with preference for data constituents that have the most available space.
In general, incoming files are created in the following manner.
Data constituents with more than 80% used space generally do not receive incoming files.
Data constituents with less than 80% used space receive incoming files in a round-robin fashion.
Data constituents that are empty or nearly empty receive most of the incoming files.
Over time, this method of creating files tends to balance space usage across the data constituents. This
is an automatic background function that helps to avoid problems created by uneven capacity balancing.
If a file cannot be created on a data constituent, Data ONTAP tries to create the file on other data
constituents before returning an error to the client.
When data constituents become unbalanced due to adding new data constituents, Data ONTAP
preferentially places data for new files in empty or underutilized data constituents. This is an automatic
process that does not require intervention from the storage administrator or the user. The automatic
process, however, does not fix the imbalances caused by file size changes or the deletion of existing files.
Best Practices
Infinite Volume capacity utilization and AutoSupport™
events should be carefully monitored.
To avoid hot spots, NetApp recommends adding storage space when the data constituent capacity approaches 60%.
Capacity Addition by Adding Disks to the Cluster
You can use the volume show command to check the status of the Infinite Volume and the space usage
of the volume.
cluster::> volume show -volume iv1
Vserver Volume Aggregate State Type Size Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
vs1 iv1 - online RW 3072TB 205TB 90%
Although NetApp recommends creating aggregates using the recommended configuration documented in
the aggregate configuration section, above, for use with Infinite Volume, it is possible to create smaller
aggregates. In this scenario, it might be possible to add more disks to existing aggregates up to the
maximum size (100TB for FAS6280 and 68TB for FAS3270) to increase the capacity of the Infinite
Volume.
Before adding disks, check the available space of the aggregates to determine which of the aggregates
are running at the same capacity. For example, if there are five aggregates that are 25TB in size running
at 60% used and 10 aggregates that are 50TB in size running at 25% used, you should evenly add more
13 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
disks to those five 25TB aggregates. Remember that the maximum recommended aggregate size for a
FAS6280 is 100TB and for FAS3270 it is 68TB. Space utilization of the aggregates can be checked by
using the aggr show command, disks can be added by using the aggr add-disks command, or
aggregate space can be managed through System Manager.
cluster::> aggr show iviv_aggr*
Aggregate Size Available Used% State #Vols Nodes RAID Status
--------- -------- --------- ----- ------- ------ ---------------- ------------
iv_aggr_001 80TB 72TB 10% online 1 beacon-a raid_dp
iv_aggr_002 80TB 72TB 10% online 1 beacon-b raid_dp
iv_aggr_003 80TB 72TB 10% online 1 beacon-a raid_dp
cluster::> aggr add-disks -aggregate iv_aggr_001 -diskcount 14
cluster::> aggr add-disks -aggregate iv_aggr_002 -diskcount 14
cluster::> aggr add-disks -aggregate iv_aggr_003 -diskcount 14
Best Practices
When increasing the size of aggregates by adding more disks, always add the same number of disks to the undersized aggregates to evenly increase the capacity.
Additional disk drives should be added only to the data constituent aggregates. Adding disks to the namespace constituent will not increase the capacity of the Infinite Volume for storing content.
Adding disks to expand an aggregate beyond 100TB on a FAS6280 or 68TB on a FAS3270 will not result in any additional usable space for the Infinite Volume.
Capacity Addition by Adding Aggregates and/or Nodes to the Cluster
Infinite Volume capacity can also be increased by adding new aggregates with the newly installed disks
or by adding more nodes with storage to the cluster. New aggregates can be created from System
Manager or the CLI by using the aggr create command. Similarly, new nodes can be added to the
existing cluster by launching the System Manager’s advance option for managing a cluster or through the
CLI by using the cluster join command. After adding the new aggregates/nodes to the cluster, make
sure to use volume resize in System Manager or run the volume modify command on the CLI to
expand the volume.
The following example shows the volume modify command to increase the size of the Infinite Volume.
cluster::> volume modify -vserver vs1 -volume iv1 -size +2PB
If a SnapMirror relationship was set up for this Infinite Volume, increase the size of the destination volume
before increasing the source volume or the next SnapMirror update will fail.
Best Practice
If the existing nodes in a cluster are close to the maximum number of aggregates per HA pair limit (35
for FAS6280 with DS4486, 28 for FAS6280 with DS424x, 47 for FAS3270 with either drive shelf),
NetApp recommends adding more nodes to the cluster and creating the aggregates on new nodes
rather than on the existing nodes for better performance.
Expanding the Infinite Volume
To accommodate data growth, the cluster’s storage capacity can be increased by adding disks to existing
aggregates, adding aggregates, or adding more storage nodes to the cluster as described in the previous
section. The Infinite Volume needs to be resized or expanded to accommodate the additional capacity
added to the system in the following scenarios.
When an Infinite Volume is thin provisioned (for example, when a volume much larger than the actual available space from the aggregates is created) and at some point more disks are added to the
14 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
existing aggregates containing data constituents. In this case, there is no need to resize the Infinite Volume unless the actual new capacity of the volume is more than what was provisioned.
When the Infinite Volume is thin provisioned and new aggregates are added to existing nodes, or new nodes are added to the cluster. In this case, you must resize the Infinite Volume to create new data constituents on the new aggregates.
When the Infinite Volume is thin provisioned and the size of the aggregate containing the namespace constituent is greater than 10TB. If new disks are added to the namespace aggregate, there is no need to resize the Infinite Volume. This is because the namespace constituent has the volume guarantee set to "volume” and therefore its size does not increase by the addition of more disks to this aggregate. In this case, adding more disks will result in improved performance for new write and subsequent read requests because of the increased number of spindles in the aggregate.
If the Infinite Volume is thin provisioned and the size of the aggregate containing the namespace constituent is much smaller in size, for example, 5TB, it can still grow to accommodate a 10TB namespace constituent. In that case, you must resize the Infinite Volume to include the increased size of the namespace constituent.
Note: If you resize an Infinite Volume on FAS3270 systems, the maximum size is calculated incorrectly in the tool. Don’t set this value to larger than ((62.9TB * (<# aggregates in the Infinite Volume> - 2)) + 20TB) if you have a namespace mirror and ((62.9TB * (<# aggregates in the Infinite Volume> - 1)) + 10TB) if you don’t have a namespace mirror for a FAS3270 configuration. Any attempt to resize an Infinite Volume above this maximum will fail. The maximum size is calculated correctly for the FAS6280.
Best Practices
Before creating an Infinite Volume, make sure the minimum number of aggregates is available in the cluster and assigned to the Vserver.
Make sure to use the UNIX security type.
Create the Infinite Volume with the “volume guarantee” set to “none” for thin provisioning. This is the default setting when creating the volume through System Manager, but it needs to be set when using the CLI.
Do not attempt to remove any aggregates from the Infinite Volume without first properly planning for moving the data out of these aggregates, since shrinking an Infinite Volume is not supported.
6 Data Protection
6.1 Snapshot Copies
An Infinite Volume supports Snapshot copies at the Infinite Volume level. Snapshot copy names should
not exceed 255 characters. In System Manager, the snapshot create and snapshot restore
operations on Infinite Volume can be performed in the same way as with FlexVol volumes. The Snapshot
copies of Infinite Volume have an additional property called “state.” This property has three values:
Valid. This state indicates that the Snapshot copies of all constituents are intact.
Partial. This state indicates that the Snapshot copies of one or more data constituents are missing.
Invalid. This state indicates that the Snapshot copy of the namespace constituent is missing.
If for some reason the namespace constituent is offline or missing, all the Snapshot copies will start
indicating an invalid state. Once the namespace constituent is recovered and brought online, the
invalid Snapshot copies will return to a valid state. Snapshot copies in the invalid state cannot be
restored. Snapshot copies in the partial state cannot be restored in the admin mode. Contact NetApp
Global Support if it is necessary to restore a partial Snapshot copy.
15 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
If a namespace resiliency (DP) mirror is configured on a cluster, the total number of Snapshot copies
available for an Infinite Volume is 253. This is because the namespace resiliency mirror uses two
Snapshot copies for the data protection SnapMirror relationship.
The Snapshot copies for Infinite Volume cannot be set for autodelete. However, when you manually
delete Snapshot copies of Infinite Volumes, you can run the delete operation in the background.
For details on the configuration of Snapshot copies, refer to the “Data ONTAP 8.1.x Cluster-Mode Data
Protection Guide.”
Best Practices
NetApp does not recommend that a Snapshot policy have a schedule for less than an hour.
It is important to reserve enough disk space for Snapshot copies so that the active file system always has space available to create new files or modify existing ones.
Do not attempt to rename a Snapshot file, because this is not supported.
Do not remove invalid state Snapshot copies without verifying why they became invalid. If the
namespace constituent has recovered correctly and invalid Snapshot copies still appear, they
can be deleted because they are just occupying space.
Infinite Volume must be unmounted before executing the volume snapshot restore
command.
Infinite Volume must be online to be able to perform any Snapshot operation (create, delete,
restore).
6.2 Protection for Namespace Constituent
The namespace constituent in the Infinite Volume keeps the redirector pointer files for every data file on
the data constituents. If the namespace constituent is not available at any point in time, the file operations
on recently accessed files (read, write, get/set attrs) and so on will continue to work (due to the
caching of file handle mappings in memory). Namespace or directory operations (mkdir, ls) and so on
will not work.
To mitigate risk, you can set up an intracluster data protection (DP) mirror for the namespace constituent
on other aggregate(s), preferably on another shelf, and another node in the cluster. This namespace
mirror can be periodically updated based on the recovery point objective (RPO) requirements. A
SnapMirror license is not required for this special mirror.
The process to create a namespace mirror and to recover the data from the namespace constituent
requires diagnostic-mode commands and should be undertaken only with the help of NetApp Global
Support.
Best Practices
Make sure a data protection mirror is created for the namespace constituent, preferably on another shelf or another node in the cluster.
Make sure an aggregate for the namespace DP mirror is available in the cluster, which is ideally the same size as the active namespace aggregate. If it is not easy to keep the same aggregate configuration for the RAID group size, number, and type of disks for the DP mirror, it can be made just large enough to contain the namespace constituent.
Note: If the aggregate for the namespace DP mirror does not have the same configuration as the active namespace aggregate, performance by the new namespace constituent (the former namespace DP mirror) will be degraded.
16 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
6.3 SnapMirror for Infinite Volume
NetApp SnapMirror replication provides easy recovery from a disaster for the Infinite Volume. The entire
Infinite Volume can be replicated to another disaster recovery site using a single SnapMirror relationship
without managing individual data constituent SnapMirror relationships. The SnapMirror configuration and
recovery can be managed through System Manager 2.1.
Note:
A Vserver with Infinite Volume is only allowed to be mirrored to another Vserver with Infinite Volume. This means that both the source and the destination volumes must be Infinite Volumes.
Only a one-to-one SnapMirror relationship is allowed between Infinite Volumes. The secondary Infinite Volume cluster is dedicated to be the only SnapMirror destination for the primary volume.
The SnapMirror destination must be the same system model as the source. In other words if the source Infinite Volume resides on a FAS/V6280, then the SnapMirror destination must also be a FAS/V6280.
Setting Up SnapMirror
The process for setting up the SnapMirror for Infinite Volume and FlexVol volumes is the same.
To set up the disaster recovery cluster for the Infinite Volume, perform the following steps in the order
listed.
1. Make sure SnapMirror is licensed on the source and destination clusters.
2. Create Vserver and aggregates on the destination cluster identical to those on the source cluster in terms of count and sizes. Verify that the SnapMirror destination cluster contains the same system model as the source. In other words, if the source Infinite Volume resides on a FAS/V6280, then the SnapMirror destination must also be a FAS/V6280.
3. Create a cluster-peer relationship between the source cluster and the destination cluster using the intercluster network interfaces. To set up a data protection mirror between cluster peers, refer to the “Data ONTAP 8.1.x Cluster-Mode Data Protection Guide.”
4. While creating the destination Vserver, the Create Vserver wizard will create the destination Infinite Volume (without constituents). Once the SnapMirror relationship is established and initialized, the constituents are created in the destination Infinite Volume.
5. If you are creating the SnapMirror relationships through System Manager, you can create a destination Infinite Volume as part of the SnapMirror Create wizard, if it does not already exist.
6. It is important to create the data protection Infinite Volume on the same system model, with the same size volume as the primary site's volume. Also, create the same number of aggregates of the same
sizes as the primary site. Otherwise, the snapmirror create operation will fail.
7. Use the snapmirror create and snapmirror initialize commands to initialize the DP
Infinite Volume.
SnapMirror Updates and Schedules
The SnapMirror destination Infinite Volumes can be updated on automatic schedules (by adding schedule
parameters during a "create" or "modify" operation), or they can be updated manually. To update a
destination Infinite Volume manually, run the snapmirror update command.
The SnapMirror update schedule depends on the number of incremental changes in the Infinite Volume
that need to be replicated and the bandwidth between the sites. NetApp recommends not scheduling
SnapMirror updates to less than an hour.
17 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
Failing Over to the Secondary Site
In case the source Infinite Volume becomes unavailable (due to a disaster or an availability issue), the
users can be redirected for failover access to the destination Infinite Volume. To do this, perform the
following steps.
1. Issue a snapmirror break command on the destination Infinite Volume.
2. Export the destination Infinite Volume for applications to access data.
3. Restart the applications at the destination cluster to connect to the destination Infinite Volume.
Best Practices
A cluster-peer relationship and an intercluster interface must be created before setting up the data protection SnapMirror relation for Infinite Volume.
An identical cluster configuration including the model of nodes, the Vserver, and an identical set of aggregates (same count and size) should be created on the destination as the source, before creating the SnapMirror relation.
The destination volume must have space equal to or greater than the source volume at all times, even during configuration changes. This is required so that there is enough space in the destination data constituents to hold the mirrored data. A SnapMirror relationship must be established between the source and destination disks and volumes before executing the
snapmirror initialize command.
When adding additional aggregates or nodes, consider adding them in equal numbers (that is, the
aggregate count should match) on both the source and destination sites. Perform the snapmirror
update to initialize the newly added data constituents in the Infinite Volume.
Do not set the SnapMirror update schedule to less than an hourly interval.
To increase the capacity of an Infinite Volume on the source cluster, first expand the capacity on
the destination cluster by the same or a greater amount, and then perform a volume resize
operation. Following this, add capacity on the source cluster and then perform volume resize
followed by a snapmirror update command to initialize the newly added constituents to the
destination volume.
Do not attempt snap delete or snap restore on the mirrored destination Infinite Volume,
because it is a read-only volume and these functions are not allowed on a read-only state of the volume.
Do not attempt the snap delete or snap restore operation on the base Snapshot copy on the
source Infinite Volume or else future updates will not take place. The snapmirror resync
operation might be required in case this operation was performed. If SnapMirror cannot find a common Snapshot copy, resync will also fail.
Recovering the Primary Site from the Secondary Site
After the client data access has been failed over to the destination Infinite Volume, users can continue to
operate in this configuration and make this volume the new primary volume. However, when the original
source cluster is available, you may want to resync the original source Infinite Volume with all the new
changes that have been applied to the current primary (originally the destination) Infinite Volume. This
can be achieved by performing a flip-resync. A flip-resync essentially makes the original source Infinite
Volume a mirror of the current primary volume. This enables higher levels of data protection for the
primary data (which now resides on the destination volume).
18 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
To perform a flip-resync, execute the following steps.
1. Execute the snapmirror create command to create a flip-side relationship from the new primary
to the original primary:
cluster::> snapmirror create -source-path dr-cluster://dr_vs1/dr_iv1 -destination-path
cluster://vs1/iv1 -type DP
2. Resync the original source volume iv1 with the current primary repository dr_ncv1:
dr-cluster::> snapmirror resync -source-path dr-cluster://dr_vs1/dr_iv1 -destination-path
cluster://vs1/iv1 -type DP
Note: If the number of available aggregates is not exactly the same at the destination as at the source, the snapmirror resync, initialize, and update operations will not be successful.
Note: After the recovery operation, if the output of volume snapmirror show on the destination does not display healthy, check if there are “partial” state Snapshot copies at the destination. This means the recovery process was not successful. Check the job progress of the snapmirror resync to determine the cause of the unsuccessful completion.
Continued Mirroring to Source Infinite Volume
For business continuance, an administrator may want to continue operation on the flip-side configuration
for a while, using scheduled updates. To set up a flip-side relationship, perform the following steps.
1. Delete the original relationship from the destination cluster:
dr-cluster::> snapmirror delete -source-path cluster://vs1/iv1
-destination-path dr-cluster://dr_vs1/dr_iv1
2. If a relationship for the flip-side relationship is not already created, create a new relationship on the source cluster to facilitate flip-side mirroring:
cluster::> snapmirror create -source-path dr-cluster://dr_vs1/dr_iv1 -destination-path
cluster://vs1/iv1 -type DP -schedule 5min
3. If a relationship was already created, the schedule can be provided by using a snapmirror modify
command:
cluster::> snapmirror modify -destination-path cluster://vs1/iv1 -schedule hourly
Note: An administrator may choose to operate in the flip-side configuration for an extended period of time and can potentially make this the ongoing mirroring setup.
Reestablishing the Original Relationship After Recovery
When both the clusters are up and running and the recovery points are close to each other, the
administrator may want to reestablish the original relationship. This involves performing the following
steps.
1. Stop the client access on the new primary Infinite Volume (the original destination Infinite Volume).
2. Allow the next update to bring the secondary Infinite Volume to the same recovery point as the primary Infinite Volume. This can also be achieved by explicitly performing a manual update.
3. Break the secondary Infinite Volume:
cluster::> snapmirror break -destination-path cluster://vs1/iv1
4. Create a new relationship on the destination cluster to facilitate mirroring from the original source if it doesn't already exist:
dr-cluster::> snapmirror create -source-path cluster://vs1/iv1 -destination-path dr-
cluster://dr-vs1/dr_iv1 -type DP -schedule hourly
19 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
5. Perform a flip-resync on the current primary Infinite Volume to reestablish the original relationship:
dr-cluster::> snapmirror resync -source-path cluster://vs1/iv1 -destination-path dr-
cluster://dr-vs1/dr_iv1
6. Resume the client access to the original primary Infinite Volume (iv1).
7. Delete the flip-side relationship from the source cluster:
cluster::> snapmirror break -destination-path cluster://vs1/iv1
The preceding procedure will reestablish the original relationship that existed prior to failover. The original
source cluster will start to rehost the primary Infinite Volume. The destination cluster will host the mirror
(secondary) Infinite Volume.
Recovering Failed or Missing Constituents on the Primary Site from a Secondary Site
If any of the constituents (including the namespace constituent) have failed or are missing in the primary
Infinite Volume, you can use the snapmirror resync operation to restore the primary volume. The
resync operation will create new data protection constituents and perform initialization for the missing
constituents. For the remaining constituents, the snapmirror resync operation will perform the
resync operation at the constituent level, thereby avoiding full initialization of the constituent. The resync
operation determines the deleted constituents based on the current Infinite Volume configuration.
Note: An appropriate warning message will be displayed before proceeding with the resync operation.
7 Storage Efficiency
NetApp Infinite Volume supports storage efficiency features such as NetApp deduplication, compression,
and thin provisioning.
Deduplication is performed at the Infinite Volume data constituent level within an aggregate and not
across the Infinite Volume. For example, if the same file was stored twice in the Infinite Volume on two
different data constituents, the blocks between the data constituents will not be deduplicated.
Deduplication will be performed only on duplicate blocks of data within a single data constituent.
Deduplication requires space for metadata in both the data constituent and the aggregate. For details,
refer to TR-3966: NetApp Data Compression and Deduplication Deployment and Implementation Guide.
Similar to deduplication, compression is done at the data constituent level within an aggregate.
Compression works in the same way for the data constituent as it does with FlexVol volumes. Both
deduplication and compression features are enabled at the Infinite Volume level. They are set to off by
default and can be turned on as needed.
NetApp thin provisioning technology is an efficient way to provision storage, because the storage is not
preallocated upfront. The Infinite Volume can be thin provisioned and more disks can be added to the
aggregates in the Vserver as required.
Note: The namespace constituent cannot be thin provisioned, deduplicated, or compressed. The namespace constituent is always volume space guaranteed.
Best Practice
Schedule deduplication to occur during off-peak hours.
20 Best Practices for NetApp Infinite Volume in Data ONTAP 8.1.x Cluster-Mode
References
The following references were used in this report.
TR-3982: “Data ONTAP 8.1 Operating in Cluster-Mode: An Introduction” http://media.netapp.com/documents/tr-3982.pdf
TR-4037: “Introduction to NetApp Infinite Volume” http://media.netapp.com/documents/tr-4037.pdf
“Data ONTAP 8.1 Logical Storage Management Guide”
https://library.netapp.com/ecm/ecm_get_file/ECMP1138167
“Data ONTAP 8.1 Cluster-Mode Data Protection Guide”
https://library.netapp.com/ecm/ecm_get_file/ECMP1136081
Data ONTAP 8.1.1 Cluster-Mode documentation
http://support.netapp.com/documentation/docweb/index.html?productID=61476
TR-3966: “NetApp Data Compression and Deduplication Deployment and Implementation Guide” http://media.netapp.com/documents/tr-3966.pdf
Version History
Version Date Document Version History
Version 1.0 June 2012 Initial release
Version 1.1 September 2012 Updated to include support for FAS/V3270
NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customer’s responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.
© 2012 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, AutoSupport, Data ONTAP, FlexVol, OnCommand, RAID-DP, SnapMirror, Snapshot, and WAFL are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. TR-4078-0612
Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct configurations that are supported by NetApp. Specific results depend on each customer's installation in accordance with published specifications.