85
S2 Information life cycle management (ILM) is a comprehensive approach to managing the flow of an information system's data and associated metadata from creation and initial storage to the time when it becomes obsolete and is deleted. Information Storage And Management PRIYA

Ism Practical File

Embed Size (px)

Citation preview

Page 1: Ism Practical File

S2

Information life cycle management (ILM) is a comprehensive approach to managing the flow of an information system's data and associated metadata from creation and initial storage to the time when it

becomes obsolete and is deleted.

Information Storage And Management

PRIYA

Page 2: Ism Practical File

A

REPORT FILE

ON

“INFORMATION STORAGE & MANAGEMENT”

CS-703

Submitted to

RAJIV GANDHI PROUDYOGIKI VISHWAVIDAYALAYA, BHOPAL

(UNIVERSITY OF TECHNOLOGY OF MADHYA PRADRESH)

IN PARTIAL FULFILLMENT FOR THE REQUIREMENT OF FIFTH SEMESTER

BACHELOR OF ENGINEERING

IN

COMPUTER SCIENCE

(2010-2011)

SUBMITTED TO:- SUBMITTED BY:-

Mr Gyan Shivhare Priya Agrawal

Lecturer CSE Dept. Roll no. 0914CS081081

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

NRIITM GWALIOR (M.P.)

1

Page 3: Ism Practical File

NRI INSTITUTE OF TECHNOLOGY AND MANAGEMENT, GWALIOR

CERTIFICATE

This is to certify that PRIYA AGRAWAL (0914CS081081) student

of Bachelor of Engineering (Computer Science and Engineering) 7 th

semester submitted her report file of CS-703 INFORMATION

STORAGE & MANAGEMENT for the partial fulfilment of the

requirement of 7th semester of Bachelor of Engineering in Computer

Science and Engineering under my Guidance. I wish her success in

the future.

Mr Gyan Shivhare

Lecturer

Dept. of CSE/IT

2

Page 4: Ism Practical File

RECOMMENDATION

It is here by recommended that the Practical file of CS-703

INFORMATION STORAGE & MANAGEMENT submitted by

PRIYA AGRAWAL (0914CS081081) student of Bachelor of

Engineering (Computer Science and Engineering) 7th semester may be

accepted in partial fulfilment of the requirement of 7th semester of

Bachelor of Engineering in Computer Science and Engineering.

Prof. Amit Singh

Reader and Head

Dept. of CSE and IT

3

Page 5: Ism Practical File

INDEX

SNO. Name of experiments Remark

1. Explain ILM with its benefits

2. Evolution of various Network storage technology NAS , SAN.

3. Explain RAID with its complete level and priority level algorithm.

4. Explain intelligent disk subsystem architecture with its virtualization

5. Compare various storage network technologies

6. Explain the concept of virtualization and also storage virtualization with its components.

7. Explain in detail concept of backup and recovery with its methodology

8. Report file on cloud computing used in the major project

4

Page 6: Ism Practical File

1. Explain Information Life Cycle Management with its benefits.

Introduction: Information life cycle management (ILM) is a comprehensive approach to managing the flow of an information system's data and associated metadata from creation and initial storage to the time when it becomes obsolete and is deleted. Unlike earlier approaches to data storage management, ILM involves all aspects of dealing with data, starting with user practices, rather than just automating storage procedures. Also in contrast to older systems, ILM enables more complex criteria for storage management than data age and frequency of access.ILM products automate the processes involved, typically organizing data into separate tiers according to specified policies, and automating data migration from one tier to another based on those criteria. As a rule, newer data, and data that must be accessed more frequently, is stored on faster, but more expensive storage media, while less critical data is stored on cheaper, but slower media. However, the ILM approach recognizes that the importance of any data does not rely solely on its age or how often it's accessed. Users can specify different policies for data that declines in value at different rates or that retains its value throughout its life span. A path management application, either as a component of ILM software or working in conjunction with it, makes it possible to retrieve any data stored by keeping track of where everything is in the storage cycle.

ILM Implementation: The process of developing an ILM strategy includes four activities—classifying, implementing, managing, and organizing:

Classifying data and applications on the basis of business rules and policies to enable differentiated treatment of information.

Implementing policies by using information management tools, starting from the creation of data and ending with its disposal

Managing the environment by using integrated tools to reduce operational complexity

Organizing storage resources in tiers to align the resources with data classes, and storing information in the right type of infrastructure based on the information’s current valueImplementing ILM across an enterprise is an on-going process. Figure illustrates a three-step road map to enterprise-wide ILM.Steps 1 and 2 are aimed at implementing ILM in a limited way across a few enterprise-critical applications. In Step 1, the goal is to implement a storage networking environment. Storage architectures offer varying levels of protection and performance and this acts as a foundation for future policy-based information management in Steps 2 and 3. The value of tiered storage platforms can be exploited by allocating appropriate storage resources to the applications based on the value of the information processed.Step 2 takes ILM to the next level, with detailed application or data classification and linkage of the storage infrastructure to business policies. These classifications and the resultant policies can be automatically executed using tools for one or more applications, resulting in better management and optimal allocation of storage resources.

5

Page 7: Ism Practical File

Step 3 of the implementation is to automate more of the applications or data classification and policy management activities in order to scale to a wider set of enterprise applications.

ILM Framework: The ILM Framework is carefully designed to account for different

enterprise data management requirements. These requirements are categorized into five major

disciplines, but can concurrently span multiple disciplines. These five disciplines (as

illustrated in the diagram) are:

1. Regulatory & Compliance Requirements: These represent demands made on the

organization by a variety of oversight bodies. These include: national and local governments,

industry trade organizations, self-regulating organizations (SRO), and internal operational

standards. Examples of these requirements span from Sarbanes-Oxley (SOX) to internal Six

Sigma mandates.

2. Information Management Requirements: These represent policy and procedural

requirements for the declaration, management, and disposition of data. They are derived

partly from industry best practices and partly from internal procedures. The latter involve

regulatory and compliance requirements as inputs.

3. Business Process Requirements: These requirements represent the essential bridge

between business and IT. They are derived and integrate records' management and regulatory

requirements with IT operational standards that cover infrastructure management and data

protection.

6

Page 8: Ism Practical File

Fig: ILM Framework

4. Tiered Infrastructure Requirements: These requirements translate business value of data

into a tier of the IT infrastructure. By defining these requirements, an organization can ensure

that critical data resides on premium IT infrastructure. At the same time, non-critical data is

moved to less expensive infrastructure. Collectively, this ensures that IT dollars are

effectively applied to support the business.

5. Data Protection Requirements: These requirements define the operational strategy to

safeguard enterprise data against mishaps. Generally speaking, data protection can be broken

down into two categories: 1) business continuity--through the design of redundant systems;

and 2) disaster recovery--through the design of off-site, parallel recovery centres.

ILM Process Model: The ILM Process model, defining the actions that can be taken on

information at any one time, the options available while taking those actions and the path an

individual should follow to ensure the information remains secure throughout its lifetime

from creation to deletion.

7

Page 9: Ism Practical File

ILM Process Model

Creation: A lot of information is created as part of the everyday business process within

organisations. Upon creation, the creator must consider the content of the information they

are creating and make a decision as to whether or not it requires access control. If not, then its

lifecycle can continue without applying the ILM Process. If it is, the Information

Classification, Impact Sensitivity Categorization and Access Control Policy Definition stages

of the process must be performed.

Storage: Information must be stored appropriately to reflect the information protection

requirements defined by the Information Classification and Impact Sensitivity Categorization

stages. Encryption is not mandated but if the information is highly classified or has high

impact sensitivity then confidentiality assurance must be considered. The physical location of

the storage devices and the encryption of information are example considerations.

Information Sharing: Information shared in collaborative, de-perimeterised environments is

subject to a different threat model to information stored locally. In light of this the

Information Classification, Impact Sensitivity Categorization and Access Control Policy

Definition stages of the process must be performed before sharing the information, even if

this has already been performed on creation of the information.

8

Page 10: Ism Practical File

Data Transfer: The information transfer protocol between collaborating parties should take

account of the information protection requirements as defined by the Information

Classification and Impact Categorization stages. Encryption is not mandated but if the

information is highly classified or has high impact sensitivity then confidentiality assurance

must be considered. Endpoint compliance and the encryption of information in transit are

example considerations.

Update: If the information is updated, the updater must consider the content of the

information they are adding/updating and make a decision, using the Information

Classification Scheme and Impact Sensitivity Categorization processes, as to whether or not

the changes present additional and/or modified information protection requirements. Changes

in Access Control Policy and data transfer security are examples of such protection

requirements that may change due to modified information content.

Deletion: Deletion of the information should reflect its classification and impact sensitivity

labels. If the information is labelled as having to be securely destroyed then just placing it in

the system “Trash” is not acceptable.

Information Classification: Information that is shared in de-perimeterised environments must be accurately labelled with information protection requirements according to the sensitivity of the content within the information resource in terms of a risk-based assessment of the business impact of an incident or threat.

The information creator or individual intending to share the information in a collaborative, de-parameterised environment must consider the threat to the business if the information were accessed and/or modified by individuals with an identity outside of a particular domain. This could be the internal organisation, external business community or named specific individuals, details of which must be specified in the Information Classification Scheme for the organisation. Information handling requirements, legality and temporal aspects must also be considered at the Information Classification phase. Secure destroying of information; clear ownership rights; changes in classification after a particular date and time; and corporate governance constraints are examples of the detail that should be evident from the labelling process. If the information requires classification according to the Information Classification Scheme, the information must be correctly labelled.

Impact Sensitivity Categorization: Information must be labelled with an Impact Sensitivity

level based on the measures of Confidentiality, Integrity, Authenticity and Availability

required to adequately protect the information in situ, use and transit. The Jericho Forum

paper titled Impact Sensitivity Categorization proposes a six-level impact sensitivity scale

that represents the impact magnitude should the protection measures not be effectively

deployed. The creator or individual that is intending to share the information in a

collaborative, deperimeterised environment must conduct an Impact Sensitivity analysis to

determine the controls required to maintain information assurance in a de-perimeterised

environment in relation to the Confidentiality, Integrity and Availability requirements of the

information.

9

Page 11: Ism Practical File

Access Control: Authentication and Authorisation should be applied to principles requesting

access to information. Appropriate access control technology should then be used to enforce

the authorisation response. Access Controls should be reflective of and responsive to the

information security requirements defined in the Information Classification and Impact

Sensitivity Categorization stages.

Tiered Architecture: Tiered storage is the assignment of different categories of data to different types of storage media in order to reduce total storage cost. Categories may be based on levels of protection needed, performance requirements, frequency of use, and other considerations. Since assigning data to particular media may be an on-going and complex activity, some vendors provide software for automatically managing the process based on a company-defined policy.

Fig: Tiered Architecture

As an example of tiered storage, tier 1 data (such as mission-critical, recently accessed, or top secret files) might be stored on expensive and high-quality media such as double-parity RAIDs (Redundant arrays of independent disks). Tier 2 data (such as financial, seldom-used, or classified files) might be stored on less expensive media in conventional storage area networks (SANs). As the tier number increased, cheaper media could be used. Thus, tier 3 in a 3-tier system might contain event-driven, rarely used, or unclassified files on recordable compact discs (CD-Rs) or tapes.

IT Tiered Architecture: The IT Tiered Infrastructure is comprised of four layers: Production, Replication, Recovery, and Archive. These four layers are designed to meet specific support requirements for applications residing in a Tier.

10

Page 12: Ism Practical File

At the Production layer, the servers, storage devices, network connectivity, management staff, security, access controls and operational processes are defined, designed and deployed based on the policies and support requirements per application. Subsequent layers may use some or all of these components in its architecture.

The Replication layer provides supporting architecture related to data replication support functions for each tier. It is important to note that each tier in the IT Tiered Infrastructure may not have a need to provide data replication functions to support its applications. The importance of identifying the Replication Layer in the IT Tiered Infrastructure is to provide a holistic end-to-end view of the needed IT architecture.

At the Recovery layer, the focus is on the Data Protection plan with specific details related to the day-to-day backup and restore operations. Each Tier will have a different Data Protection schema with specific technical and operational processes designed to support the recovery of data types within the Tier. In the Tier1 Recovery layer, the architecture may only consist of disk-to-disk backup and restore capabilities, because of SLA, RPO, and RTO requirements, while the Tier3 Recovery layer architecture would consist of backup and restore from tape media. The Tier2 Recovery layer might consist of a combination of disk-to-disk or disk-to-tape capabilities.

The Archive layer is the final layer within this model. It encompasses the functions and architecture needed to support data archiving operations and long term support. Data types within this layer are considered to be critical and useful information, and may be kept past its data retention schedule based upon its usability. This layer may not be needed for a Tier3 application or data type. De-duplication of information is critical at this layer. In the past, data would be written to a tape media and stored for long periods of time as part of an archive strategy. Recent legal penalties and search/hold requirements make storage of data to tape no longer acceptable practice. Backup tapes are not considered searchable media and are very inefficient for this function.

A content management system should be deployed to manage the data types stored at this layer. Application databases have specific functions allowing for the management of data which has been extracted from the source system and stored on an archive target system. These systems require on-going support and management. The level of management and specific support requirements should be identified in the Data Classification model.

Benefits of ILM: Implementing an ILM strategy has the following key benefits that directly address the challenges of information management:

1. Improved utilization by using tiered storage platforms and increased visibility of all enterprise information.

2. Simplified management by integrating process steps and interfaces with individual tools and by increasing automation.

3. A wider range of options for backup, and recovery to balance the need for business continuity.

11

Page 13: Ism Practical File

4. Maintaining compliance by knowing what data needs to be protected for what length of time.

5. Lower Total Cost of Ownership (TCO) by aligning the infrastructure and management costs with information value. As a result, resources are not wasted, and complexity is not introduced by managing low-value data at the expense of high-value data.

2. Evolution of various storage network technologies

DAS: (Ideal for Local Data Sharing Requirements)

Direct-attached storage, or DAS, is the most basic level of storage, in which storage devices are part of the host computer, as with drives, or directly connected to a single server, as with RAID arrays or tape libraries. Network workstations must therefore access the server in order to connect to the storage device. This is in contrast to networked storage such as NAS and

12

Page 14: Ism Practical File

SAN, which are connected to workstations and servers over a network. As the first widely popular storage model, DAS products still comprise a large majority of the installed base of storage systems in today's IT infrastructures.

Although the implementation of networked storage is growing at a faster rate than that of direct-attached storage, it is still a viable option by virtue of being simple to deploy and having a lower initial cost when compared to networked storage. When considering DAS, it is important to know what your data availability requirements are. In order for clients on the network to access the storage device in the DAS model, they must be able to access the server it is connected to. If the server is down or experiencing problems, it will have a direct impact on users' ability to store and access data. In addition to storing and retrieving files, the server also bears the load of processing applications such as e-mail and databases. Network bottlenecks and slowdowns in data availability may occur as server bandwidth is consumed by applications, especially if there is a lot of data being shared from workstation to workstation.

DAS is ideal for localized file sharing in environments with a single server or a few servers - for example, small businesses or departments and workgroups that do not need to share information over long distances or across an enterprise. Small companies traditionally utilize DAS for file serving and e-mail, while larger enterprises may leverage DAS in a mixed storage environment that likely includes NAS and SAN. DAS also offers ease of management and administration in this scenario, since it can be managed using the network operating system of the attached server. However, management complexity can escalate quickly with the addition of new servers, since storage for each server must be administered separately.

From an economical perspective, the initial investment in direct-attached storage is cheaper. This is a great benefit for IT managers faced with shrinking budgets, who can quickly add storage capacity without the planning, expense, and greater complexity involved with networked storage. DAS can also serve as an interim solution for those planning to migrate to

13

Page 15: Ism Practical File

networked storage in the future. For organizations that anticipate rapid data growth, it is important to keep in mind that DAS is limited in its scalability. From both a cost efficiency and administration perspective, networked storage models are much more suited to high scalability requirements.

Organizations that do eventually transition to networked storage can protect their investment in legacy DAS. One option is to place it on the network via bridge devices, which allows current storage resources to be used in a networked infrastructure without incurring the immediate costs of networked storage. Once the transition is made, DAS can still be used locally to store less critical data.

NAS: (File-Level Data Sharing Across the Enterprise)

Networked storage was developed to address the challenges inherent in a server- based infrastructure such as direct-attached storage. Network-attached storage, or NAS, is a special purpose device, comprised of both hard disks and management software, which is 100% dedicated to serving files over a network. As discussed earlier, a server has the dual functions of file sharing and application serving in the DAS model, potentially causing network slowdowns. NAS relieves the server of storage and file serving responsibilities, and provides a lot more flexibility in data access by virtue of being independent.

NAS is an ideal choice for organizations looking for a simple and cost-effective way to achieve fast data access for multiple clients at the file level. Implementers of NAS benefit from performance and productivity gains. First popularized as an entry-level or midrange solution, NAS still has its largest install base in the small to medium sized business sector. Yet the hallmarks of NAS - simplicity and value - are equally applicable for the enterprise market. Smaller companies find NAS to be a plug and play solution that is easy to install, deploy and manage, with or without IT staff at hand. Thanks to advances in disk drive technology, they also benefit from a lower cost of entry.

In recent years, NAS has developed more sophisticated functionality, leading to its growing adoption in enterprise departments and workgroups. It is not uncommon for NAS to go head to head with storage area networks in the purchasing decision, or become part of a NAS/SAN convergence scheme. High reliability features such as RAID and hot swappable drives and components are standard even in lower end NAS systems, while midrange offerings provide enterprise data protection features such as replication and mirroring for business continuance. NAS also makes sense for enterprises looking to consolidate their direct-attached storage resources for better utilization. Since resources cannot be shared beyond a single server in DAS, systems may be using as little as half of their full capacity. With NAS, the utilization rate is high since storage is shared across multiple servers.

14

Page 16: Ism Practical File

The perception of value in enterprise IT infrastructures has also shifted over the years. A business and ROI case must be made to justify technology investments. Considering the downsizing of IT budgets in recent years, this is no easy task. NAS is an attractive investment that provides tremendous value, considering that the main alternatives are adding new servers, which is an expensive proposition, or expanding the capacity of existing servers, a long and arduous process that is usually more trouble than it's worth. NAS systems can provide many terabytes of storage in high density form factors, making efficient use of data centre space. As the volume of digital information continues to grow, organizations with high scalability requirements will find it much more cost-effective to expand upon NAS than DAS. Multiple NAS systems can also be centrally managed, conserving time and resources.

Another important consideration for a medium sized business or large enterprise is heterogeneous data sharing. With DAS, each server is running its own operating platform, so there is no common storage in an environment that may include a mix of Windows, Mac and Linux workstations. NAS systems can integrate into any environment and serve files across all operating platforms. On the network, a NAS system appears like a native file server to each of its different clients. That means that files are saved on the NAS system, as well as retrieved from the NAS system, in their native file formats. NAS is also based on industry standard network protocols such as TCP/IP, FC and CIFS.

SANs: (High Availability for Block-Level Data Transfer)

15

Page 17: Ism Practical File

A storage area network, or SAN, is a dedicated, high performance storage network that transfers data between servers and storage devices, separate from the local area network. With their high degree of sophistication, management complexity and cost, SANs are traditionally implemented for mission-critical applications in the enterprise space. In a SAN infrastructure, storage devices such as NAS, DAS, RAID arrays or tape libraries are connected to servers using Fibre Channel. Fibre Channel is a highly reliable, gigabit interconnect technology that enables simultaneous communication among workstations, mainframes, servers, data storage systems and other peripherals. Without the distance and bandwidth limitations of SCSI, Fibre Channel is ideal for moving large volumes of data across long distances quickly and reliably.

In contrast to DAS or NAS, which is optimized for data sharing at the file level, the strength of SANs lies in its ability to move large blocks of data. This is especially important for bandwidth-intensive applications such as database, imaging and transaction processing. The distributed architecture of a SAN also enables it to offer higher levels of performance and availability than any other storage medium today. By dynamically balancing loads across the network, SANs provide fast data transfer while reducing I/O latency and server workload. The benefit is that large numbers of users can simultaneously access data without creating bottlenecks on the local area network and servers.

SANs are the best way to ensure predictable performance and 24x7 data availability and reliability. The importance of this is obvious for companies that conduct business on the web and require high volume transaction processing. Another example would be contractors that are bound to service-level agreements (SLAs) and must maintain certain performance levels

16

Page 18: Ism Practical File

when delivering IT services. SANs have built in a wide variety of failover and fault tolerance features to ensure maximum uptime. They also offer excellent scalability for large enterprises that anticipate significant growth in information storage requirements. And unlike direct-attached storage, excess capacity in SANs can be pooled, resulting in a very high utilization of resources.

There has been much debate in recent times about choosing SAN or NAS in the purchasing decision, but the truth is that the two technologies can prove quite complementary. Today, SANs are increasingly implemented in conjunction with NAS. With SAN/NAS convergence, companies can consolidate block-level and file-level data on common arrays.

Even with all the benefits of SANs, several factors have slowed their adoption, including cost, management complexity and a lack of standardization. The backbone of a SAN is management software. A large investment is required to design, develop and deploy a SAN, which has limited its market to the enterprise space. A majority of the costs can be attributed to software, considering the complexity that is required to manage such a wide scope of devices. Additionally, a lack of standardization has resulted in interoperability concerns, where products from different hardware and software vendors may not work together as needed. Potential SAN customers are rightfully concerned about investment protection and many may choose to wait until standards become defined.

Content Addressed Storage

Content-addressed storage (CAS) is a method of providing fast access to fixed content (data that is not expected to be updated) by assigning it a permanent place on disk. CAS makes data retrieval straightforward by storing it in such a way that an object cannot be duplicated or modified once it has been stored; thus, its location is unambiguous. The term was coined by EMC Corporation when it released its Centera product in 2002. When an object is stored in CAS, the object is given a name that uniquely identifies it, and that also specifies the storage location. This type of address is called a "content address." It eliminates the need for an centralized index, so it is not necessary to track the location of stored data. Once an object has been stored, it cannot be deleted until the specified retention period has expired. In CAS, data is stored on disk, rather than on tape. This streamlines the process of searching for stored objects. A backup copy of every object is stored to enhance reliability and to minimize the risk of catastrophic data loss. In the event of a hardware failure, the system administrator is notified by a remote monitoring system.

17

Page 19: Ism Practical File

A significant advantage of CAS is the fact that it minimizes the storage space consumed by data backups and archives, preventing what some engineers call a "data tsunami" (the overwhelming build-up of information, much of which is obsolete, redundant, or unnecessary). Another advantage is authentication. Because there is only one copy of an object, verifying its legitimacy is a simple process.

Just a Bunch of Disks (JBOD)

If you have some disks in a system that you decide not to configure into a RAID array, what do you do with them? Traditionally, they are left to act as independent drive volumes within the system, and that's how many people in fact use two, three or more drives in a PC. In some applications, however, it is desirable to be able to use all these disks as if they were one single volume. The proper term for this is spanning; the pseudo-cutesy term for it, clearly chosen to contrast against "redundant array of inexpensive disks", is Just a Bunch of Disks or JBOD.JBOD can be thought of as the opposite of partitioning: while partitioning chops single drives up into smaller logical volumes, JBOD combines drives into larger logical volumes. It provides no fault tolerance, nor does it provide any improvements in performance compared to the independent use of its constituent drives. JBOD doesn't really have a lot to recommend it. It still requires a controller card or software driver, which means that almost any system that can do JBOD can also do RAID 0, and RAID 0 has significant performance advantages over JBOD. Neither provides fault tolerance, so that's a wash. There are only two possible advantages of JBOD over RAID 0:

18

Page 20: Ism Practical File

1. Avoiding Drive Waste: If you have a number of odd-sized drives, JBOD will let you combine them into a single unit without loss of any capacity; a 10 GB drive and 30 GB would combine to make a 40 GB JBOD volume but only a 20 GB RAID 0 array. This may be an issue for those expanding an existing system, though with drives so cheap these days it's a relatively small advantage.

2. Easier Disaster Recovery: If a disk in a RAID 0 volume dies, the data on every disk in the array is essentially destroyed because all the files are striped; if a drive in a JBOD set dies then it may be easier to recover the files on the other drives (but then again, it might not, depending on how the operating system manages the disks.) Considering that you should be doing regular backups regardless and that even under JBOD recovery can be difficult, this too is a minor advantage.

19

Page 21: Ism Practical File

3. Explain RAID with its complete level and parity algorithms.

Introduction: There are many applications, particularly in a business environment, where there are needs beyond what can be fulfilled by a single hard disk, regardless of its size, performance or quality level. Many businesses can't afford to have their systems go down for even an hour in the event of a disk failure; they need large storage subsystems with capacities in the terabytes; and they want to be able to insulate themselves from hardware failures to any extent possible. Some people working with multimedia files need fast data transfer exceeding what current drives can deliver, without spending a fortune on specialty drives. These situations require that the traditional "one hard disk per system" model be set aside and a new system employed. This technique is called Redundant Arrays of Inexpensive Disks or RAID. ("Inexpensive" is sometimes replaced with "Independent", but the former term is the one that was used when the term "RAID" was first coined by the researchers at the University of California at Berkeley, who first investigated the use of multiple-drive arrays in 1987.)

The fundamental principle behind RAID is the use of multiple hard disk drives in an array that behaves in most respects like a single large, fast one. There are a number of ways that this can be done, depending on the needs of the application, but in every case the use of multiple drives allows the resulting storage subsystem to exceed the capacity, data security, and performance of the drives that make up the system, to one extent or another. Originally, RAID was almost exclusively the province of high-end business applications, due to the high cost of the hardware required.

There are three primary reasons that RAID was implemented:

Redundancy: Redundancy is the most important factor in the development of RAID for server environments. This allowed for a form of backup of the data in the storage array in the event of a failure. If one of the drives in the array failed, it could either be swapped out for a new drive without turning the systems off (referred to as hot swappable) or the redundant drive could be used. The method of redundancy depends on which version of RAID is used.

Increased Performance: The increased performance is only found when specific versions of the RAID are used. Performance will also be dependent upon the number of drives used in the array and the controller.

Lower Cost: All managers of IT departments like low costs. When the RAID standards were being developed, cost was also a key issue. The point of a RAID array is to provide the same or greater storage capacity for a system compared to using individual high capacity hard drives. A good example of this can be seen in the price differences between the highest capacity hard drives and lower capacity drives. Three drives of a smaller size could cost less than an individual high-capacity drive but provide more capacity.

20

Page 22: Ism Practical File

Concepts of RAID:

These are the concepts that describe how RAID works, how arrays are set up, and how different RAID levels work to improve reliability and performance.

a. Physical and Logical Arrays and Drives: The fundamental structure of RAID is the array. An array is a collection of drives that is configured, formatted and managed in a particular way. The number of drives in the array, and the way that data is split between them, is what determines the RAID level, the capacity of the array, and its overall performance and data protection characteristics. Deciding what types of arrays to set up, and how to configure them, is the first thing you do when setting up a RAID implementation.

Understanding arrays and drives can get awfully confusing because high-end RAID systems allow for such complexity in how they are arranged. To get a better handle on this, let's look at the "hierarchy" in which data in a RAID system is organized:

1. Physical Drives: The physical, actual hard disks that comprise the array are the "building blocks" of all data storage under RAID.

2. Physical Arrays: One or more physical drives are collected together to form a physical array. Most simple RAID setups use just one physical array, but some complex ones can have two or more physical arrays.

3. Logical Arrays: Logical arrays are formed by splitting or combining physical arrays. Typically, one logical array corresponds to one physical array. However, it is possible to set up a logical array that includes multiple physical arrays (typically used to allow multiple RAID levels). It is also possible to set up two entirely different logical arrays from a single physical, as described here.

4. Logical Drives: One or more logical drives are formed from one logical array (much the same way they would normally be formed from one physical drive in a non-RAID system). These appear to the operating system as if they were regular disk volumes, and are treated accordingly, with the RAID controller managing the array(s) that underlie them.

b. Mirroring: Mirroring is one of the two data redundancy techniques used in RAID (the other being parity). In a RAID system using mirroring, all data in the system is written simultaneously to two hard disks instead of one; thus the "mirror" concept. The principle behind mirroring is that this 100% data redundancy provides full protection against the failure of either of the disks containing the duplicated data. Mirroring setups always require an even number of drives for obvious reasons. The chief advantage of mirroring is that it provides not only complete redundancy of data, but also reasonably fast recovery from a disk failure. Since all the data is on the second drive, it is ready to use

21

Page 23: Ism Practical File

if the first one fails. Mirroring also improves some forms of read performance (though it actually hurts write performance.) The chief disadvantage of RAID 1 is expense: that data duplication means half the space in the RAID is "wasted" so you must buy twice the capacity that you want to end up with in the array. Performance is also not as good as some RAID levels.

Block diagram of a RAID mirroring configuration. The RAID controllerduplicates the same information onto each of two hard disks. Note that

the RAID controller is represented as a "logical black box" since its functionscan be implemented in software, or several different types of hardware

(integrated controller, bus-based add-in card, stand-alone RAID hardware.)

Mirroring is used in RAID 1, as well as multiple-level RAID involving RAID 1 (RAID 01 or RAID 10). It is related in concept to duplexing. Very high-end mirroring solutions even include such fancy technologies as remote mirroring, where data is configured in a RAID 1 array with the pairs split between physical locations to protect against physical disaster.

c. Duplexing: Duplexing is an extension of mirroring that is based on the same principle as that technique. Like in mirroring, all data is duplicated onto two distinct physical hard drives. Duplexing goes one step beyond mirroring, however, in that it also duplicates the hardware that controls the two hard drives (or sets of hard drives). So if you were doing mirroring on two hard disks, they would both be connected to a single host adapter or RAID controller. If you were doing duplexing, one of the drives would be connected to one adapter and the other to a second adapter.

22

Page 24: Ism Practical File

Block diagram of a RAID duplexing configuration. Two controllers are usedto send the same information to two different hard disks. The controllers areoften regular host adapters or disk controllers with the mirroring done by the

system. Contrast this diagram with the one for straight mirroring.

Duplexing is superior to mirroring in terms of availability because it provides the same protection against drive failure that mirroring does, but also protects against the failure of either of the controllers. It also costs more than mirroring because you are duplicating more hardware. Duplexing is one option when implementing RAID 1 (though most people think of RAID 1 as only being mirroring, and certainly, mirroring is much more commonly implemented than duplexing.)

d. Striping: The main performance-limiting issues with disk storage relate to the slow mechanical components that are used for positioning and transferring data. Since a RAID array has many drives in it, an opportunity presents itself to improve performance by using the hardware in all these drives in parallel. For example, if we need to read a large file, instead of pulling it all from a single hard disk, it is much faster to chop it up into pieces, store some of the pieces on each of the drives in an array, and then use all the disks to read back the file when needed. This technique is called striping, after the pattern that might be visible if you could see these "chopped up pieces" on the various drives with a different color used for each file. It is similar in concept to the memory performance-enhancing technique called interleaving.

Striping can be done at the byte level, or in blocks. Byte-level striping means that the file is broken into "byte-sized pieces". The first byte of the file is sent to the first drive, then the second to the second drive, and so on. Sometimes byte-level striping is done as a sector of 512 bytes. Block-level striping means that each file is split into blocks of a certain size and those are distributed to the various drives. The size of the blocks used is also called the stripe size (or block size, or several other names), and can be selected from a variety of choices when the array is set up.

23

Page 25: Ism Practical File

Block diagram of a RAID striping configuration. One controller (which againcan be hardware or software) splits files into blocks or bytes and distributes

them across several hard disks. The block size determines how many "pieces" files will be split into. In this example, the first block of file 1 is sent to disk #1, then the second block to disk #2, etc. When all four disks have one block of file 1, the fifth

block goes back to disk #1, and this continues until the file is completed. Note that file 3 is only on one disk; this means it was smaller than the block size in this case.

Striping is used in the implementation of most of the basic, single RAID levels (and by extension, any multiple RAID levels that use those single RAID levels). However, the actual way striping is set up, and how it is used, varies greatly from level to level. RAID 0 uses block-level striping without parity; RAID 3 and RAID 7 use byte-level striping with parity; and RAID 4, RAID 5 and RAID 6 use block-level striping with parity.

e. Parity: Mirroring is a data redundancy technique used by some RAID levels, in particular RAID level 1, to provide data protection on a RAID array. While mirroring has some advantages and is well-suited for certain RAID implementations, it also has some limitations. It has a high overhead cost, because fully 50% of the drives in the array are reserved for duplicate data; and it doesn't improve performance as much as data striping does for many applications. For this reason, a different way of protecting data is provided as an alternate to mirroring. It involves the use of parity information, which is redundancy information calculated from the actual data values. The principle behind parity is simple: take "N" pieces of data, and from them, compute an extra piece of data. Take the "N+1" pieces of data and store them on "N+1" drives. If you lose any one of the "N+1" pieces of data, you can recreate it from the "N" that remain, regardless of which piece is lost. Parity protection is used with striping, and the "N" pieces of data are typically the blocks or bytes distributed across the drives in the array. The parity information can either be stored on a separate, dedicated drive, or be mixed with the data across all the drives in the array. The parity calculation is typically performed using a logical operation called "exclusive OR" or "XOR". The "OR" logical operator is "true" (1) if either of its operands is true, and false (0) if neither is true. The exclusive OR operator is "true" if and only if one of its operands is true; it differs from "OR" in that if both operands are true, "XOR" is false. This truth table for the two operators will illustrate:

24

Page 26: Ism Practical File

Input Output

#1 #2 "OR" "XOR"

0 0 0 0

0 1 1 1

1 0 1 1

1 1 1 0

If you calculate "A XOR B" and then take that result and do another "XOR B" on it, you get back A, the value you started with. That is to say, "A XOR B XOR B = A". This property is exploited for parity calculation under RAID. If we have four data elements, D1, D2, D3 and D4, we can calculate the parity data, "DP" as "D1 XOR D2 XOR D3 XOR D4". Then, if we know any four of D1, D2, D3, D4 and DP, we can XOR those four together and it will yield the missing element.

Let's take an example, suppose we have the following four bytes of data: D1=10100101, D2=11110000, D3=00111100, and D4=10111001. We can "XOR" them together as follows, one step at a time:

D1 XOR D2 XOR D3 XOR D4= ((D1 XOR D2) XOR D3) XOR D4= ((10100101 XOR 11110000) XOR 00111100) XOR 10111001= (01010101.XOR 00111100) XOR 10111001= 01101001 XOR 10111001= 11010000

So "11010000" becomes the parity byte, DP. Now let's say we store these five values on five hard disks, and hard disk #3, containing value "00111100". We can retrieve the missing byte simply by XOR'ing together the other three original data pieces and the parity byte we calculated earlier, as so:

D1 XOR D2 XOR D4 XOR DP= ((D1 XOR D2) XOR D4) XOR DP= ((10100101 XOR 11110000) XOR 10111001) XOR 11010000= (01010101 XOR 10111001) XOR 11010000= 11101100 XOR 11010000= 00111100

which is D3, the missing value. Compared to mirroring, parity (used with striping) has some advantages and disadvantages.

25

Page 27: Ism Practical File

The most obvious advantage is that parity protects data against any single drive in the array failing without requiring the 50% "waste" of mirroring; only one of the "N+1" drives contains redundancy information. (The overhead of parity is equal to (100/N)% where N is the total number of drives in the array.) Striping with parity also allows you to take advantage of the performance advantages of striping.

The chief disadvantages of striping with parity relate to complexity: all those parity bytes have to be computed--millions of them per second!--and that takes computing power. This means a hardware controller that performs these calculations is required for high performance--if you do software RAID with striping and parity the system CPU will be dragged down doing all these computations. Also, while you can recover from a lost drive under parity, the missing data all has to be rebuilt, which has its own complications; recovering from a lost mirrored drive is comparatively simple.

The entire RAID levels from RAID 3 to RAID 7 use parity; the most popular of these today is RAID 5. RAID 2 uses a concept similar to parity but not exactly the same.

RAID Levels

Single RAID Levels: Single RAID levels are by far the most commonly used since they are simpler and less expensive to implement, and satisfy the needs of most RAID users.

There are eight "regular" RAID levels, which are used to varying degrees in the "real world" today. A few levels, especially RAID 0, RAID 1 and RAID 5, are extremely popular, while a couple are rarely if ever seen in modern systems.

RAID Level 0

Common Name(s): RAID 0. (Note that the term "RAID 0" is sometimes used to mean not only the conventional striping technique described here but also other "non-redundant" ways of setting up disk arrays. Sometimes it is (probably incorrectly) used just to describe a collection of disks that doesn't use redundancy.)

Technique(s) Used: Striping (without parity)

Description: The simplest RAID level, RAID 0 should really be called "AID", since it involves no redundancy. Files are broken into stripes of a size dictated by the user-defined stripe size of the array, and stripes are sent to each disk in the array. Giving up redundancy allows this RAID level the best overall performance characteristics of the single RAID levels, especially for its cost. For this reason, it is becoming increasingly popular by performance-seekers, especially in the lower end of the marketplace.

26

Page 28: Ism Practical File

This illustration shows how files of different sizes are distributed between thedrives on a four-disk, 16 kiB stripe size RAID 0 array. The red file is  4 kiB in

size; the blue is 20 kiB; the green is 100 kiB; and the magenta is 500 kiB.They are shown drawn to scale to illustrate how much space they takeup in relative terms in the array--one vertical pixel represents 1 kiB.

(To see the impact that increasing or decreasing stripe size has on theway the data is stored in the array, see the 4 kiB and 64 kiB stripe sizeversions of this illustration on the page discussing stripe size issues.)

Controller Requirements: Supported by all hardware controllers, both SCSI and IDE/ATA, and also most software RAID solutions.

Hard Disk Requirements: Minimum of two hard disks (some may support one drive, the point of which escapes me); maximum set by controller. Any type may be used, but they should be of identical type and size for best performance and to eliminate "waste".

Array Capacity: (Size of Smallest Drive * Number of Drives).

Storage Efficiency: 100% if identical drives are used.

Fault Tolerance: None. Failure of any drive results in loss of all data, short of specialized data recovery.

Availability: Lowest of any RAID level. Lack of fault tolerance means no rapid recovery from failures. Failure of any drive results in array being lost and immediate downtime until array can be rebuilt and data restored from backup.

Degradation and Rebuilding: Not applicable.

Random Read Performance: Very good; better if using larger stripe sizes if the controller supports independent reads to different disks in the array.

27

Page 29: Ism Practical File

Random Write Performance: Very good; again, best if using a larger stripe size and a controller supporting independent writes.

Sequential Read Performance: Very good to excellent.

Sequential Write Performance: Very good.

Cost: Lowest of all RAID levels.

Special Considerations: Using a RAID 0 array without backing up any changes made to its data at least daily is a loud statement that that data is not important to you.

Recommended Uses: Non-critical data (or data that changes infrequently and is backed up regularly) requiring high speed, particularly write speed, and low cost of implementation. Audio and video streaming and editing; web servers; graphic design; high-end gaming or hobbyist systems; temporary or "scratch" disks on larger machines.

RAID Level 1

Common Name(s): RAID 1; RAID 1 with Duplexing.

Technique(s) Used: Mirroring or Duplexing

Description: RAID 1 is usually implemented as mirroring; a drive has its data duplicated on two different drives using either a hardware RAID controller or software (generally via the operating system). If either drive fails, the other continues to function as a single drive until the failed drive is replaced. Conceptually simple, RAID 1 is popular for those who require fault tolerance and don't need top-notch read performance. A variant of RAID 1 is duplexing, which duplicates the controller card as well as the drive, providing tolerance against failures of either a drive or a controller. It is much less commonly seen than straight mirroring.

Illustration of a pair of mirrored hard disks, showing how thefiles are duplicated on both drives. (The files are the same as

28

Page 30: Ism Practical File

those in the RAID 0 illustration, except that to save space I havereduced the scale here so one vertical pixel represents 2 kiB.)

Controller Requirements: Supported by all hardware controllers, both SCSI and IDE/ATA, and also most software RAID solutions.

Hard Disk Requirements: Exactly two hard disks. Any type may be used but they should ideally be identical.

Array Capacity: Size of Smaller Drive.

Storage Efficiency: 50% if drives of the same size are used, otherwise (Size of Smaller Drive / (Size of Smaller Drive + Size of Larger Drive) )

Fault Tolerance: Very good; duplexing even better.

Availability: Very good. Most RAID controllers, even low-end ones, will support hot sparing and automatic rebuilding of RAID 1 arrays.

Degradation and Rebuilding: Slight degradation of read performance; write performance will actually improve. Rebuilding is relatively fast.

Random Read Performance: Good. Better than a single drive but worse than many other RAID levels.

Random Write Performance: Good. Worse than a single drive, but better than many other RAID levels. :^)

Sequential Read Performance: Fair; about the same as a single drive.

Sequential Write Performance: Good; again, better than many other RAID levels.

Cost: Relatively high due to redundant drives; lowest storage efficiency of the single RAID levels. Duplexing is still more expensive due to redundant controllers. On the other hand, no expensive controller is required, and large consumer-grade drives are rather inexpensive these days, making RAID 1 a viable choice for an individual system.

Special Considerations: RAID 1 array are limited to the size of the drives used in the array. Multiple RAID 1 arrays can be set up if additional storage is required, but RAID 1+0 begins to look more attractive in that circumstance. Performance may be reduced if implemented using software instead of a hardware controller; duplexing may require software RAID and thus may show lower performance than mirroring.

Recommended Uses: Applications requiring high fault tolerance at a low cost, without heavy emphasis on large amounts of storage capacity or top performance. Especially useful in situations where the perception is that having a duplicated set of data is more secure than using parity. For this reason, RAID 1 is popular for accounting and other financial data. It is

29

Page 31: Ism Practical File

also commonly used for small database systems, enterprise servers, and for individual users requiring fault tolerance with a minimum of hassle and cost.

RAID Level 2

Common Name(s): RAID 2.

Technique(s) Used: Bit-level striping with Hamming code ECC.

Description: Level 2 is the "black sheep" of the RAID family, because it is the only RAID level that does not use one or more of the "standard" techniques of mirroring, striping and/or parity. RAID 2 uses something similar to striping with parity, but not the same as what is used by RAID levels 3 to 7. It is implemented by splitting data at the bit level and spreading it over a number of data disks and a number of redundancy disks. The redundant bits are calculated using Hamming codes, a form of error correcting code (ECC). Each time something is to be written to the array these codes are calculated and written alongside the data to dedicated ECC disks; when the data is read back these ECC codes are read as well to confirm that no errors have occurred since the data was written. If a single-bit error occurs, it can be corrected "on the fly". If this sounds similar to the way that ECC is used within hard disks today, that's for a good reason: it's pretty much exactly the same. It's also the same concept used for ECC protection of system memory.

It is expensive and often requires many drives--see below for some surprisingly large numbers. The controller required was complex, specialized and expensive. The performance of RAID 2 is also rather substandard in transactional environments due to the bit-level striping. But most of all, level 2 was obviated by the use of ECC within a hard disk; essentially, much of what RAID 2 provides you now get for "free" within each hard disk, with other RAID levels providing protection above and beyond ECC.

Due to its cost and complexity, level 2 never really "caught on". Therefore, much of the information below is based upon theoretical analysis, not empirical evidence.

Controller Requirements: Specialized controller hardware required.

Hard Disk Requirements: Depends on exact implementation, but a typical setup required 10 data disks and 4 ECC disks for a total of 14, or 32 data disks and 7 ECC disks for a total of 39! The disks were spindle-synchronized to run in tandem.

Array Capacity: Depends on exact implementation but would be rather large if built today using modern drives.

Storage Efficiency: Depends on the number of data and ECC disks; for the 10+4 configuration, about 71%; for the 32+7 setup, about 82%.

Fault Tolerance: Only fair; for all the redundant drives included, you don't get much tolerance: only one drive can fail in this setup and be recoverable "on the fly".

Availability: Very good, due to "on the fly" error correction.

30

Page 32: Ism Practical File

Degradation and Rebuilding: In theory, there would be little degradation due to failure of a single drive.

Random Read Performance: Fair. Bit-level striping makes multiple accesses impossible.

Random Write Performance: Poor, due to bit-level striping and ECC calculation overhead.

Sequential Read Performance: Very good, due to parallelism of many drives.

Sequential Write Performance: Fair to good.

Cost: Very expensive.

Special Considerations: Not used in modern systems.

Recommended Uses: Not used in modern systems.

RAID Level 3

Common Name(s): RAID 3. (Watch out for some companies that say their products implement RAID 3 when they are really RAID 4.)

Technique(s) Used: Byte-level striping with dedicated parity.

Description: Under RAID 3, data is striped across multiple disks at a byte level; the exact number of bytes sent in each stripe varies but is typically under 1024. The parity information is sent to a dedicated parity disk, but the failure of any disk in the array can be tolerated (i.e., the dedicated parity disk doesn't represent a single point of failure in the array.) The dedicated parity disk does generally serve as a performance bottleneck, especially for random writes, because it must be accessed any time anything is sent to the array; this is contrasted to distributed-parity levels such as RAID 5 which improve write performance by using distributed parity (though they still suffer from large overheads on writes, as described here). RAID 3 differs from RAID 4 only in the size of the stripes sent to the various disks.

This illustration shows how files of different sizes are distributedbetween the drives on a four-disk, byte-striped RAID 3 array. As with

31

Page 33: Ism Practical File

the RAID 0 illustration, the red file is 4 kiB in size; the blue is 20 kiB;the green is 100 kiB; and the magenta is 500 kiB, with each verticalpixel representing 1 kiB of space. Notice that  the files are evenly

spread between three drives, with the fourth containing parityinformation (shown in dark gray). Since the blocks are so tiny inRAID 3, the individual boundaries between stripes can't be seen.You may want to compare this illustration to the one for RAID 4.

Controller Requirements: Generally requires a medium-to-high-end hardware RAID card.

Hard Disk Requirements: Minimum of three standard hard disks; maximum set by controller. Should be of identical size and type.

Array Capacity: (Size of Smallest Drive) * (Number of Drives - 1)

Storage Efficiency: If all drives are the same size, ( (Number of Drives - 1) / Number of Drives).

Fault Tolerance: Good (Can tolerate loss of one drive.)

Availability: Very good. Hot sparing and automatic rebuild are usually supported by controllers that implement RAID 3.

Degradation and Rebuilding: Relatively little degrading of performance if a drive fails. Rebuilds can take many hours.

Random Read Performance: Good, but not great, due to byte-level striping.

Random Write Performance: Poor, due to byte-level striping, parity calculation overhead, and the bottleneck of the dedicated parity drive.

Sequential Read Performance: Very good.

Sequential Write Performance: Fair to good.

Cost: Moderate. A hardware controller is usually required, as well as at least three drives.

Special Considerations:  Not as popular as many of the other commonly-implemented RAID levels. For transactional environments, RAID 5 is usually a better choice.

Recommended Uses: Applications working with large files that require high transfer performance with redundancy, especially serving or editing large files: multimedia, publishing and so on. RAID 3 is often used for the same sorts of applications that would typically see the use of RAID 0, where the lack of fault tolerance of RAID 0 makes it unacceptable.

RAID Level 4

32

Page 34: Ism Practical File

Common Name(s): RAID 4 (sometimes called RAID 3 by the confused).

Technique Used: Block-level striping with dedicated parity.

Description: RAID 4 improves performance by striping data across many disks in blocks, and provides fault tolerance through a dedicated parity disk. This makes it in some ways the "middle sibling" in a family of close relatives, RAID levels 3, 4 and 5. It is like RAID 3 except that it uses blocks instead of bytes for striping, and like RAID 5 except that it uses dedicated parity instead of distributed parity. Going from byte to block striping improves random access performance compared to RAID 3, but the dedicated parity disk remains a bottleneck, especially for random write performance. Fault tolerance, format efficiency and many other attributes are the same as for RAID 3 and RAID 5.

This illustration shows how files of different sizes are distributed betweenthe drives on a four-disk RAID 4 array using a 16 kiB stripe size. As with theRAID 0 illustration, the red file is 4 kiB in size; the blue is 20 kiB; the greenis 100 kiB; and the magenta is 500 kiB, with each vertical pixel representing

1 kiB of space. Notice that as with RAID 3, the files are evenly spread betweenthree drives, with the fourth containing parity information (shown in gray).

You may want to contrast this illustration to the one for RAID 3 (which is verysimilar except that the blocks are so tiny you can't see them) and the onefor RAID 5 (which distributes the parity blocks across all four drives.)

Controller Requirements: Generally requires a medium-to-high-end hardware RAID card.

Hard Disk Requirements: Minimum of three standard hard disks; maximum set by controller. Should be of identical size and type.

Array Capacity: (Size of Smallest Drive) * (Number of Drives - 1).

Storage Efficiency: If all drives are the same size, ( (Number of Drives - 1) / Number of Drives).

Fault Tolerance: Good. Can tolerate loss of one drive.

33

Page 35: Ism Practical File

Availability: Very good. Hot sparing and automatic rebuild are usually supported..

Degradation and Rebuilding: Moderate degrading if a drive fails; potentially lengthy rebuilds.

Random Read Performance: Very good.

Random Write Performance: Poor to fair, due to parity calculation overhead and the bottleneck of the dedicated parity drive.

Sequential Read Performance: Good to very good.

Sequential Write Performance: Fair to good.

Cost: Moderate. A hardware controller is usually required, as well as at least three drives.

Special Considerations: Performance will depend to some extent upon the stripe size chosen.

Recommended Uses: Jack of all trades and master of none, RAID 4 is not as commonly used as RAID 3 and RAID 5, because it is in some ways a "compromise" between them that doesn't have a target market as well defined as either of those two levels. It is sometimes used by applications commonly seen using RAID 3 or RAID 5, running the gamut from databases and enterprise planning systems to serving large multimedia files.

RAID Level 5

Common Name(s): RAID 5.

Technique(s) Used: Block-level striping with distributed parity.

Description: One of the most popular RAID levels, RAID 5 stripes both data and parity information across three or more drives. It is similar to RAID 4 except that it exchanges the dedicated parity drive for a distributed parity algorithm, writing data and parity blocks across all the drives in the array. This removes the "bottleneck" that the dedicated parity drive represents, improving write performance slightly and allowing somewhat better parallelism in a multiple-transaction environment, though the overhead necessary in dealing with the parity continues to bog down writes. Fault tolerance is maintained by ensuring that the parity information for any given block of data is placed on a drive separate from those used to store the data itself. The performance of a RAID 5 array can be "adjusted" by trying different stripe sizes until one is found that is well-matched to the application being used.

34

Page 36: Ism Practical File

This illustration shows how files of different sizes are distributedbetween the drives on a four-disk RAID 5 array using a 16 kiB stripe

size. As with the RAID 0 illustration, the red file is 4 kiB in size; the blueis 20 kiB; the green is 100 kiB; and the magenta is 500 kiB, with eachvertical pixel representing 1 kiB of space. Contrast this diagram to theone for RAID 4, which is identical except that the data is only on threedrives and the parity (shown in gray) is exclusively on the fourth.drive.

Controller Requirements: Requires a moderately high-end card for hardware RAID; supported by some operating systems for software RAID, but at a substantial performance penalty.

Hard Disk Requirements: Minimum of three standard hard disks; maximum set by controller. Should be of identical size and type.

Array Capacity: (Size of Smallest Drive) * (Number of Drives - 1).

Storage Efficiency: If all drives are the same size, ( (Number of Drives - 1) / Number of Drives).

Fault Tolerance: Good. Can tolerate loss of one drive.

Availability: Good to very good. Hot sparing and automatic rebuild are usually featured on hardware RAID controllers supporting RAID 5 (software RAID 5 will require down-time).

Degradation and Rebuilding: Due to distributed parity, degradation can be substantial after a failure and during rebuilding.

Random Read Performance: Very good to excellent; generally better for larger stripe sizes. Can be better than RAID 0 since the data is distributed over one additional drive, and the parity information is not required during normal reads.

35

Page 37: Ism Practical File

Random Write Performance: Only fair, due to parity overhead; this is improved over RAID 3 and RAID 4 due to eliminating the dedicated parity drive, but the overhead is still substantial.

Sequential Read Performance: Good to very good; generally better for smaller stripe sizes.

Sequential Write Performance: Fair to good.

Cost: Moderate, but often less than that of RAID 3 or RAID 4 due to its greater popularity, and especially if software RAID is used.

Special Considerations: Due to the amount of parity calculating required, software RAID 5 can seriously slow down a system. Performance will depend to some extent upon the stripe size chosen.

Recommended Uses: RAID 5 is seen by many as the ideal combination of good performance, good fault tolerance and high capacity and storage efficiency. It is best suited for transaction processing and is often used for "general purpose" service, as well as for relational database applications, enterprise resource planning and other business systems. For write-intensive applications, RAID 1 or RAID 1+0 are probably better choices, as the performance of RAID 5 will begin to substantially decrease in a write-heavy environment.

RAID Level 6

Common Name(s): RAID 6. Some companies use the term "RAID 6" to refer to proprietary extensions of RAID 5; these are not discussed here.

Technique(s) Used: Block-level striping with dual distributed parity.

Description: RAID 6 can be thought of as "RAID 5, but more". It stripes blocks of data and parity across an array of drives like RAID 5, except that it calculates two sets of parity information for each parcel of data. The goal of this duplication is solely to improve fault tolerance; RAID 6 can handle the failure of any two drives in the array while other single RAID levels can handle at most one fault. Performance-wise, RAID 6 is generally slightly worse than RAID 5 in terms of writes due to the added overhead of more parity calculations, but may be slightly faster in random reads due to spreading of data over one more disk. As with RAID levels 4 and 5, performance can be adjusted by experimenting with different stripe sizes.

36

Page 38: Ism Practical File

This illustration shows how files of different sizes are distributedbetween the drives on a four-disk RAID 6 array using a 16 kiB stripe

size. As with the RAID 0 illustration, the red file is 4 kiB in size; the blueis 20 kiB; the green is 100 kiB; and the magenta is 500 kiB, with each

vertical pixel representing 1 kiB of space. This diagram is the same as theRAID 5 one, except that you'll notice that there is now twice as muchgray parity information, and as a result, more space taken up on the

four drives to contain the same data than the other levels that use striping.

Controller Requirements: Requires a specialized (usually meaning expensive) hardware controller.

Hard Disk Requirements: Minimum of four hard disks; maximum set by controller. Should be of identical size and type.

Array Capacity: (Size of Smallest Drive) * (Number of Drives - 2).

Storage Efficiency: If all drives are the same size, ( (Number of Drives - 2) / Number of Drives).

Fault Tolerance: Very good to excellent. Can tolerate the simultaneous loss of any two drives in the array.

Availability: Excellent.

Degradation and Rebuilding: Due to the complexity of dual distributed parity, degradation can be substantial after a failure and during rebuilding. Dual redundancy may allow rebuilding to be delayed to avoid performance hit.

37

Page 39: Ism Practical File

Random Read Performance: Very good to excellent; generally better for larger stripe sizes.

Random Write Performance: Poor, due to dual parity overhead and complexity.

Sequential Read Performance: Good to very good; generally better for smaller stripe sizes.

Sequential Write Performance: Fair.

Cost: High.

Special Considerations: Requires special implementation; not widely available.

Recommended Uses: In theory, RAID 6 is ideally suited to the same sorts of applications as RAID 5, but in situations where additional fault tolerance is required. In practice, RAID 6 has never really caught on because few companies are willing to pay for the extra cost to insure against a relatively rare event--it's unusual for two drives to fail simultaneously (unless something happens that takes out the entire array, in which case RAID 6 won't help anyway). On the lower end of the RAID 5 market, the rise of hot swapping and automatic rebuild features for RAID 5 have made RAID 6 even less desirable, since with these advanced features a RAID 5 array can recover from a single drive failure in a matter of hours (where without them, RAID 5 would require downtime for rebuilding, giving RAID 6 a substantial advantage.) On the higher end of the RAID 5 market, RAID 6 usually loses out to multiple RAID solutions such as RAID 10 that provide some degree of multiple-drive fault tolerance while offering improved performance as well.

RAID Level 7

Common Name(s): RAID 7.

Technique(s) Used: Asynchronous, cached striping with dedicated parity.

Description: Unlike the other RAID levels, RAID 7 isn't an open industry standard; it is really a trademarked marketing term of Storage Computer Corporation, used to describe their proprietary RAID design. (I debated giving it a page alongside the other RAID levels, but since it is used in the market, it deserves to be explained; that said, information about it appears to be limited.) RAID 7 is based on concepts used in RAID levels 3 and 4, but greatly enhanced to address some of the limitations of those levels. Of particular note is the inclusion of a great deal of cache arranged into multiple levels, and a specialized real-time processor for managing the array asynchronously. This hardware support especially the cache, allow the array to handle many simultaneous operations, greatly improving performance of all sorts while maintaining fault tolerance. In particular, RAID 7 offers much improved random read and write performance over RAID 3 or RAID 4 because the dependence on the dedicated parity disk is greatly reduced through the added hardware. The increased performance of RAID 7 of course comes at a cost. This is an expensive solution, made and supported by only one company.

Controller Requirements: Requires a specialized, expensive, proprietary controller.

Hard Disk Requirements: Depends on implementation.

38

Page 40: Ism Practical File

Array Capacity: Depends on implementation.

Storage Efficiency: Depends on implementation.

Fault Tolerance: Very good.

Availability: Excellent, due to use of multiple hot spares.

Degradation and Rebuilding: Better than many RAID levels due to hardware support for parity calculation operations and multiple cache levels.

Random Read Performance: Very good to excellent. The extra cache can often supply the results of the read without needing to access the array drives.

Random Write Performance: Very good; substantially better than other single RAID levels doing striping with parity.

Sequential Read Performance: Very good to excellent.

Sequential Write Performance: Very good.

Cost: Very high.

Special Considerations: RAID 7 is a proprietary product of a single company; if it is of interest then you should contact Storage Computer Corporation for more details on the specifics of implementing it. All the caching creates potential vulnerabilities in the event of power failure, making the use of one or more UPS units mandatory.

Recommended Uses: Specialized high-end applications requiring absolutely top performance and willing to live with the limitations of a proprietary, expensive solution. For most users, a multiple RAID level solution like RAID 1+0 will probably yield comparable performance improvements over single RAID levels, at lower cost.

Multiple RAID Levels: Multiple RAID levels are most commonly used to improve performance, and they do this well. Nested RAID levels typically provide better performance characteristics than either of the single RAID levels that comprise them. The most commonly combined level is RAID 0, which is often mixed with redundant RAID levels such as 1, 3 or 5 to provide fault tolerance while exploiting the performance advantages of RAID 0.

Here are the some multiple RAID Levels

RAID X+Y vs. RAID Y+X

RAID Levels 0+1 (01) and 1+0 (10)

RAID Levels 0+3 (03 or 53) and 3+0 (30)

RAID Levels 0+5 (05) and 5+0 (50)

RAID Levels 1+5 (15) and 5+1 (51)

39

Page 41: Ism Practical File

RAID X+Y vs. RAID Y+X

Before looking at the specific multiple RAID levels, I have to explain a few things about the way multiple RAID levels are constructed. A multiple RAID level is generally created by taking a number of disks and dividing them into sets. Within each set a single RAID level is applied to form a number of arrays. Then, the second RAID level is applied to the arrays to create a higher-level array. This is why these are sometimes called nested arrays.

Since there are two levels, there are two ways they can be combined. The choice of which level is applied first and which second has an impact on some important array characteristics. Let's take as an example multiple RAID employing RAID 0 and RAID 1 to create an array of ten disks. Much as we can define 10 to be 2*5 or 5*2, we can create our multiple RAID array two ways:

RAID 0, then RAID 1: Divide the ten disks into two sets of five. Turn each set into a RAID 0 array containing five disks, then mirror the two arrays. (Sometimes called a "mirror of stripes".)

RAID 1, then RAID 0: Divide the ten disks into five sets of two. Turn each set into a RAID 1 array, then stripe across the five mirrored sets. (A "stripe of mirrors").

The standard that most of the industry seems to use is that if RAID level X is applied first and then RAID level Y is applied over top of it, that is RAID "X+Y", also sometimes seen as "RAID XY" or "RAID X/Y". This would mean that alternative number 1 above would be called "RAID 0+1" or "RAID 1+0", and that's in fact the terminology that most companies use. Unfortunately, other companies reverse the terms! They might call the RAID 0 and then RAID 1 technique "RAID 1/0" or "RAID 10". Some designers use the terms "RAID 01" and "RAID 10" interchangeably.

In many respects, there is no difference between them: there is no impact on drive requirements, capacity, storage efficiency, and importantly, not much impact on performance. The big difference comes into play when we look at fault tolerance. Most controllers implement multiple level RAID by forming a "super array" comprised of "sub-arrays" underneath it. In many cases the arrays that comprise the "super array"--often called sets--are considered to be logical "single units", which means that the controller only considers one of these "single units" to either be "up" or "down" as a whole. It will make use of redundancy features within a sub-array, but not between sub-arrays, even if the higher-level array means that drives in different sub-arrays will have the same data. That makes this sound much more complicated than it really is; it's much easier to explain with an example. Let's look at 10 drives and RAID 0+1 vs. RAID 1+0 again:

RAID 0+1: We stripe together drives 1, 2, 3, 4 and 5 into RAID 0 stripe set "A", and drives 6, 7, 8, 9 and 10 into RAID 0 stripe set "B". We then mirror A and B using RAID 1. If one drive fails, say drive #2, then the entire stripe set "A" is lost, because RAID 0 has no redundancy; the RAID 0+1 array continues to chug along because the entire stripe set "B" is still functioning. However, at this point you are reduced to running what is in essence a straight RAID 0 array until drive #2 can be fixed. If in the meantime drive #9 goes down, you lose the entire array.

RAID 1+0: We mirror drives 1 and 2 to form RAID 1 mirror set "A"; 3 and 4 become "B"; 5 and 6 become "C"; 7 and 8 become "D"; and 9 and 10 become "E". We then do a RAID 0

40

Page 42: Ism Practical File

stripe across sets A through E. If drive #2 fails now, only mirror set "A" is affected; it still has drive #1 so it is fine, and the RAID 1+0 array continues functioning. If while drive #2 is being replaced drive #9 fails, the array is fine, because drive #9 is in a different mirror pair from #2. Only two failures in the same mirror set will cause the array to fail, so in theory, five drives can fail--as long as they are all in different sets--and the array would still be fine.

Clearly, RAID 1+0 is more robust than RAID 0+1. Now, if the controller running RAID 0+1 were smart, when drive #2 failed it would continue striping to the other four drives in stripe set "A", and if drive #9 later failed it would "realize" that it could use drive #4 in its stead, since it should have the same data. This functionality would theoretically make RAID 0+1 just as fault-tolerant as RAID 1+0. Unfortunately, most controllers aren't that smart. It pays to ask specific questions about how a multiple RAID array implementation handles multiple drive failures, but in general, a controller won't swap drives between component sub-arrays unless the manufacturer of the controller specifically says it will.

The same impact on fault tolerance applies to rebuilding. Consider again the example above. In RAID 0+1, if drive #2 fails, the data on five hard disks will need to be rebuilt, because the whole stripe set "A" will be wiped out. In RAID 1+0, only drive #2 has to be rebuilt. Again here, the advantage is to RAID 1+0.

RAID Levels 0+1 (01) and 1+0 (10):

Common Name(s): RAID 0+1, 01, 0/1, "mirrored stripes", "mirror of stripes"; RAID 1+0, 10, 1/0, "striped mirrors", "stripe of mirrors". Labels are often used incorrectly; verify the details of the implementation if the distinction between 0+1 and 1+0 is important to you.

Technique(s) Used: Mirroring and striping without parity.

Description: The most popular of the multiple RAID levels, RAID 01 and 10 combine the best features of striping and mirroring to yield large arrays with high performance in most uses and superior fault tolerance. RAID 01 is a mirrored configuration of two striped sets; RAID 10 is a stripe across a number of mirrored sets. RAID 10 and 01 have been increasing dramatically in popularity as hard disks become cheaper and the four-drive minimum is legitimately seen as much less of an obstacle. RAID 10 provides better fault tolerance and rebuild performance than RAID 01. Both array types provide very good to excellent overall performance by combining the speed of RAID 0 with the redundancy of RAID 1 without requiring parity calculations.

41

Page 43: Ism Practical File

This illustration shows how files of different sizes are distributed between thedrives on an eight-disk RAID 0+1 array using a 16 kiB stripe size for the RAID 0portion. As with the RAID 0 illustration, the red file is 4 kiB in size; the blue is

20 kiB; the green is 100 kiB; and the magenta is 500 kiB, with each vertical pixelrepresenting 1 kiB of space. The large, patterned rectangles represent the twoRAID 0 "sub arrays", which are mirrored using RAID 1 to create RAID 0+1.The contents of the striped sets are thus identical. The diagram for RAID 1+0

would be the same except for the groupings: instead of two large boxes dividingthe drives horizontally, there would be four large boxes dividing the drivesvertically into mirrored pairs. These pairs would then be striped together toform level 1+0. Contrast this diagram to the ones for RAID 0 and RAID 1.

Controller Requirements: Almost all hardware controllers will support one or the other of RAID 10 or RAID 01, but often not both. Even low-end cards will support this multiple level, usually RAID 01. High-end cards may support both 01 and 10.

Hard Disk Requirements: An even number of hard disks with a minimum of four; maximum dependent on controller. All drives should be identical.

Array Capacity: (Size of Smallest Drive) * (Number of Drives ) / 2.

42

Page 44: Ism Practical File

Storage Efficiency: If all drives are the same size, 50%.

Fault Tolerance: Very good for RAID 01; excellent for RAID 10.

Availability: Very good for RAID 01; excellent for RAID 10.

Degradation and Rebuilding: Relatively little for RAID 10; can be more substantial for RAID 01.

Random Read Performance: Very good to excellent.

Random Write Performance: Good to very good.

Sequential Read Performance: Very good to excellent.

Sequential Write Performance: Good to very good.

Cost: Relatively high due to large number of drives required and low storage efficiency (50%).

Special Considerations: Low storage efficiency limits potential array capacity.

Recommended Uses: Applications requiring both high performance and reliability and willing to sacrifice capacity to get them. This includes enterprise servers, moderate-sized database systems and the like at the high end, but also individuals using larger IDE/ATA hard disks on the low end. Often used in place of RAID 1 or RAID 5 by those requiring higher performance; may be used instead of RAID 1 for applications requiring more capacity.

43

Page 45: Ism Practical File

4. Explain intelligent disk subsystem with its components.

Intelligent storage systems

Intelligent storage systems can be categorized as:

High-end storage systems Midrange storage systems

High-end Storage Systems

High-end storage systems, referred to as active-active arrays, are generally aimed at large enterprises for centralizing corporate data. These arrays are designed with a large number of controllers and cache memory. An active-active array implies that the host can perform I/Os to its LUNs across any of the available paths.

Midrange Storage System

Midrange storage systems are also referred to as active-passive arrays and they are best suited for small- and medium-sized enterprises. In an active-passive array, a host can perform I/Os to a LUN only through the paths to the owning controller of that LUN. These paths are called active paths. The other paths are passive with respect to this LUN. Midrange storage systems are typically designed with two controllers, each of which contains host interfaces, cache, RAID controllers, and disk drive interfaces.

44

Page 46: Ism Practical File

EMC CLARiiON and Symmetrix:

The EMC CLARiiON storage array is an active-passive array implementation. It is the EMC midrange networked storage offering that delivers enterprise-quality features and functionality. It is ideally suited for applications with predictable workloads that need moderate to high response time and throughput.The EMC Symmetrix networked storage array is an active-active array implementation. Symmetrix is a solution for customers who require an uncompromising level of service, performance, as well as the most advanced business continuity solution to support large and unpredictable application workloads. Symmetrix also provides built-in, advanced-level security features and offers the most efficient use of power and cooling to support enterprise-level data storage requirements.

CLARiiON Storage Array

The CX4 series is the fourth generation CLARiiON CX storage platform. Each generation has added enhancements to performance, availability, and scalability over the previous generation while the high-level architecture remains the same. Figure shows an EMC CLARiiON storage array.

45

Page 47: Ism Practical File

CLARiiON is built with modular building blocks and no single point of failure. CLARiiON CX4 is first midrange storage array that supports flash drives with 30 times more IOPS capability. Features of Clarion

Ultra Flex technology for dual protocol connectivity, online expansion via IO modules, and readiness for future technologies—such as 8 Gb/s Fibre Channel and 10 Gb/s iSCSI.

Scalable up to 960 disks. Supports different types and sizes of drives, and RAID types (0, 1, 1+0, 3, 5, 6) . Supports up to 16 GB of available cache memory per controller (Storage Processor). Enhances availability with non-disruptive upgrade and failover. Ensures data protection through mirrored write cache and cache vaulting. Provides data integrity through disk scrubbing. The background verification process

runs continually and reads all sectors of all the disks. If a block is unreadable, the back-end error handling recovers the bad sectors from parity or mirror data.

Supports storage-based local and remote data replication for backup and disaster recovery through Snap View and Mirror View software.

Clarion CX4 Architecture

The Storage Processor Enclosure (SPE) and the Disk Array Enclosure (DAE) are the key modular building blocks of a CLARiiON. A DAE enclosure contains up to 15 disk drives, two link control cards (LCCs), two power supplies, and two fan modules. An SPE contains two storage processors, each consisting of one CPU module and slots for I/O modules.

46

Page 48: Ism Practical File

Figure shows the CLARiiON CX4 architecture.

The CLARiiON architecture supports fully redundant, hot swappable components. This means that the system can survive with a failed component, which can be replaced without powering down the system. The important components of the CLARiiON storage system include the following: Intelligent storage processor (SP): Intelligent SP is the main component of the CLARiiON architecture. SP are configured in pairs for maximum availability. SP provide both front-end and back-end connectivity to the host and the physical disk, respectively. SP also include memory, most of which is used for cache. Depending on the model, each SP includes one or two CPUs.

CLARiiON Messaging Interface (CMI): The SPs communicate to each other over the CLARiiON Messaging Interface, which transports commands, status information, and data for write cache mirroring between the SPs. CLARiiON uses PCI-Express as the high-speed CMI. The PCI Express architecture delivers high bandwidth per pin, has superior routing characteristics, and provides improved reliability. Standby Power Supply (SPS): In the event of a power failure, the SPS maintains a power supply to the cache for long enough to allow the content to be copied to the vault.

47

Page 49: Ism Practical File

Link Control Card (LCC): The LCC provides services to the drive enclosure, which includes the capability to control enclosure functionalities and monitor environmental status. Each drive enclosure has two LCCs. The other functions performed by LCCs are loop configuration control, failover control, marker LED control, individual disk port control, drive presence detection, and voltage status information. FLARE Storage Operating Environment: FLARE is a special software designed for EMC CLARiiON. Each storage system ships with a complete copy of the FLARE operating system installed on its first four disks. When CLARiiON is powered up, each SP boots and runs the FLARE operating system. FLARE performs resource allocation and other management tasks in the array.

Managing the CLARiiON

CLARiiON supports both command-line interface (CLI) and graphical user interface (GUI) for management. Naviseccli is a CLI-based management tool. Commands can be entered from the connected host system or from a remote server through Telnet/SSH to perform all management functions.Navisphere management software is a GUI-based suite of tools that enables centralized management of CLARiiON storage systems. These tools are used to monitor, configure, and manage CLARiiON storage arrays. The Navisphere management suite includes the following:

Navisphere Manager: A GUI-based tool for centralized storage system management that is used to configure and manage CLARiiON. It is a web-based user interface that helps to securely manage CLARiiON storage systems locally or remotely over the IP connection, using a common browser. Navisphere Manager provides the flexibility to manage single or multiple systems.

Navisphere Analyzer: A performance analysis tool for CLARiiON hardware components.

Navisphere Agent: A host-residing tool that provides a management communication path to the system and enables CLI access.

48

Page 50: Ism Practical File

5. Compare various network storage technologies.

Networks

PointsSAN NAS CAS

Connectivity Fiber channel, ISCSI IP IP

Data access method Block File File, objects

Key requirement Deterministic performance, support

for high transaction

Sharing, collaboration Long term data retention, data integrity

Types of application OLTP, Data warehousing, ERP

File and print server, consolidation, product

design

Archiving reference data store

Typical market segmentation

Large business All Middle and large business

Limitations High cost, difficult to manage, less security

Performance degradation due to

remotely accessing of data

Not efficient for frequently changing

data

49

Page 51: Ism Practical File

6. Explain the concept of virtualization and also storage virtualization with its components?

Introduction: Virtualization is the technique of masking or abstracting physical resources, which simplifies the infrastructure and accommodates the increasing pace of business and technological changes. It increases the utilization and capability of IT resources, such as servers, networks, or storage devices, beyond their physical limits. Virtualization simplifies resource management by pooling and sharing resources for maximum utilization and makes them appear as logical resources with enhanced capabilities.

Virtualization presents in many form: Virtualization has existed in the IT industry for several years and in different forms, including memory virtualization, network virtualization, server virtualization, and storage virtualization.

Memory Virtualization: Virtual memory makes an application appear as if it has its own contiguous logical memory independent of the existing physical memory resources.In a virtual memory implementation, a memory address space is divided into contiguous blocks of fixed-size pages. A process known as paging saves inactive memory pages onto the disk and brings them back to physical memory when required. This enables efficient use of available physical memory among different processes. The space used by VMMs on the disk is known as a swap file. A swap file (also known as page file or swap space) is a portion of the hard disk that functions like physical memory (RAM) to the operating system. The operating system typically moves the least used data into the swap file so that RAM will be available for processes that are more active. Because the space allocated to the swap file is on the hard disk (which is slower than the physical memory), access to this file is slower.

Network Virtualization: Network virtualization creates virtual networks whereby each application sees its own logical network independent of the physical network. A virtual LAN(VLAN) is an example of network virtualization that provides an easy, flexible, and less expensive way to manage networks.

Server Virtualization: Server virtualization enables multiple operating systems and applications to run simultaneously on different virtual machines created on the same physical server (or group of servers). Virtual machines provide a layer of abstraction between the operating system and the underlying hardware. Within a physical server, any number of virtual servers can be established; depending on hardware capabilities. Each virtual server seems like a physical machine to the operating system, although all virtual servers share the same underlying physical hardware in an isolated manner.

50

Page 52: Ism Practical File

Server virtualization addresses the issues that exist in a physical server environment.The virtualization layer, shown in Figure 10-1(b), helps to overcome resource conflicts by isolating applications running on different operating systems on the same machine. In addition, server virtualization can dynamically move the underutilized hardware resources to a location where they are needed most, improving utilization of the underlying hardware resources.

Storage Virtualization: Storage virtualization is the process of presenting a logical view of the physical storage resources to a host. This logical storage appears and behaves as physical storage directly connected to the host. Throughout the evolution of storage technology, some form of storage virtualization has been implemented. Some examples of storage virtualization are host-based volume management, LUN creation, tape storage virtualization, and disk addressing (CHS to LBA). The key benefits of storage virtualization include increased storage utilization, adding or deleting storage without affecting an application’s availability, and non-disruptive data migration (access to files and storage while migrations are in progress). Figure 10-2 illustrates a virtualized storage environment. At the top are four servers, each of which has one virtual volume assigned, which is currently in use by an application. These virtual volumes are mapped to the actual storage in the arrays, as shown at the bottom of the figure. When I/O is sent to a virtual volume, it is redirected through the virtualization at the storage network layer to the mapped physical array.The discussion that follows provides details about the different types of storage virtualization, methods of implementation, the challenges associated with the implementation of storage virtualization, and examples of implementation.

51

Page 53: Ism Practical File

Storage virtualization functionality, including aggregation (pooling), transparent data migration, heterogeneous replication and device emulation, can be implemented at the host server in the data path on a network switch blade or on an appliance, as well as in a storage system. The best location to implement storage virtualization functionality depends in part on your preferences, existing technologies and objectives for deploying storage virtualization.

SNIA Storage Virtualization Taxonomy

The SNIA (Storage Networking Industry Association) storage virtualization taxonomy provides a systematic classification of storage virtualization, with three levels defining what, where, and how storage can be virtualized.

52

Page 54: Ism Practical File

The first level of the storage virtualization taxonomy addresses “what” is created. It specifies the types of virtualization: block virtualization, file virtualization, disk virtualization, tape virtualization, or any other device virtualization.The second level describes “where” the virtualization can take place. This requires a multilevel approach that characterizes virtualization at all three levels of the storage environment: server, storage network, and storage, as shown in Figure 10-4. An effective virtualization strategy distributes the intelligence across all three levels while centralizing the management and control functions.

Data storage functions—such as RAID, caching, checksums, and hardware

Scanning—should remain on the array. Similarly, the host should control

Application- focused areas, such as clustering and application failover, and volume management of raw disks. However, path redirection, path failover, data access, and distribution or load-balancing capabilities should be moved to the switch or the network.

The third level of the storage virtualization taxonomy specifies the network level virtualization methodology, in-band or out-of-band.

53

Page 55: Ism Practical File

7. Explain in detail the concept of backup and recovery with its methodology

Back-Up: A backup is a copy of production data, created and retained for the sole purpose of recovering deleted or corrupted data. With growing business and regulatory demands for data storage, retention, and availability, organizations are faced with the task of backing up an ever-increasing amount of data. This task becomes more challenging as demand for consistent backup and quick restore of data increases throughout the enterprise — which may be spread over multiple sites. Moreover, organizations need to accomplish backup at a lower cost with minimum resources.

Backup are performed to serve three purpose-

Disaster Recovery-

Backups are performed to address disaster recovery needs. The backup copies are used for restoring data at an alternate site when the primary site is incapacitated due to a disaster. Based on RPO and RTO requirements, organizations use various backup strategies for disaster recovery. When a tape-based backup method is used as a disaster strategy, the backup tape media is shipped and stored at an offsite location.

Operational Backup-

Operational backup is a backup of data at a point in time and is used to restore data in the event of data loss or logical corruptions that may occur during routing processing. The majority of restore requests in most organizations fall is this category. For example, it is common for a user to accidentally delete an important e-mail or for a file to become corrupted, which can be restored from operational backup.

Operational backups are created for the active production information by using incremental or differential backup techniques. An example of an operational backup is a backup performed for a production database just before a bulk batch update. This ensures the availability of a clean copy of the production database if the batch update corrupts the production database.

Archival-

Backups are performed to address archival requirements. However CAS has emerged as a primary solution for archives, traditional backups are still used by small and medium enterprises for long-term preservation of transaction records, e-mail messages, and other business records needed for regulatory compliance.

54

Page 56: Ism Practical File

Apart from addressing disaster recovery, archival, and operational requirements, backups serve as a protection against data loss due to physically damage of a storage device, software failure, or virus attacks.

Recovery: RPO and RTO are major consideration when planning a backup strategy. RPO defines the tolerable limit of data loss for a business and specifies the time interval between two backups. That is the RPO determines backup frequency. For example, If application A requires an RPO of one day, it would need the data to be backed up at least once every day.

The retention period for a backup can also be derived from RPO specified for a operational recovery. For example, users of application A may request to restore the application data from its operational backup copy, which was created a month ago. This determines the retention period for the backup. Therefore the RPO for application A can range from one day to one month based on operational recovery needs. However, the organization may choose to retain the backup for a longer period of time because of internal policies or external factors, like regulatory directives.

If short retention periods are specified for backups, it may not be possible to recover all the data needed for the requested recovery point, because some data may be older than the retention period.

Long retention period can be defined for all backups, making it possible to meet my PRO within the defined retention periods. However this needs a large storage space, which translates into higher cost. Thus, it is important to define the retention period based on an analysis of all the restore requests in the past and the allocated budget.

Method of backup: Hot backup and cold backup are the two methods deployed for the backup. They are based on the state of the application when the backup is performed. In a hot backup, the application is up and running, with users accessing their data during the backup process. In a cold backup, the application is not active the backup process.

The backup of online production data is more challenging because data is actively being used and changed. An open file is locked by the operating system and is not copied during the backup process until and the user closes it. The backup application backs up open files by retrying the operation on files that were opened earlier will be closed and a retry will be successful.

The maximum number of retries can be configured depending on the backup application. However, this method is not considered robust because in some environments certain files are always open.

In such situation, the backup application provides open file agents. These agents interact directly with the opening system and enable the creation of consistent copies of file. In some

55

Page 57: Ism Practical File

environments, the use of open file agent is not enough. For example, a database is composed of many files varying sizes, occupying several file system. To ensure a consistent database system backup, all files need to be backed up in the same state. That does not necessarily mean that all files need to be backed up at the same time, but they all must be synchronized so that the database can be restored with consistency.

Hot consistency is used in situation where it is not possible to shut down the database. This is facilitated by database backup agents that can perform a backup while the database is active. The disadvantage of a hot backup is that the agents usually affect overall application performance.

It is not enough to back up only production data for recovery to ensure consistency. Certain attributes and properties attached to a file, such as permission, owner, and other metadata, also need to be backed up. Backup of boot sector and partition layout information is also critical for successful recovery.

A point-in-time (PIT) copy method is deployed in environments where the impact of downtime from a cold backup or the performance resulting from a hot backup is unacceptable. A pointer-based PIT copy consumes only a fraction of the storage space and can be created very quickly. A pointer-based PIT copy is implemented in a disk-based solution whereby a virtual LUN is created and holds pointers to the data stored on the production LUN or save location. In this method of backup, the database is stopped or frozen momentarily while the PIT copy is created. The PIT copy is then mounted on a secondary server and the backup occurs on the primary server.

Backup and Restore Operations: When a backup process is initiated, significant network communication takes place between the different components of a backup infrastructure. The backup server initiates the backup process for different clients based on the backup schedule configured for them. For example, the backup process for a group of clients may be scheduled to start at 3:00 am every day.The backup server coordinates the backup process with all the components in a backup configuration. The backup server maintains the information about backup clients to be contacted and storage nodes to be used in a backup operation. The backup server retrieves the backup-related information from the backup catalogue and, based on this information, instructs the storage node to load the appropriate backup media into the backup devices.Simultaneously, it instructs the backup clients to start scanning the data, package it, and send it over the network to the assigned storage node. The storage node, in turn, sends metadata to the backup server to keep it updated about the media being used in the backup process. The backup server continuously updates the backup catalogue with this information.

56

Page 58: Ism Practical File

Backup operation

After the data is backed up, it can be restored when required. A restore process must be manually initiated. Some backup software has a separate application for restore operations. These restore applications are accessible only to the administrators.

Restore operation

57

Page 59: Ism Practical File

8. Report on cloud computing used in the major project.

Introduction: Cloud computing is a general term for anything that involves delivering hosted services over the Internet. These services are broadly divided into three categories: Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS). The name cloud computing was inspired by the cloud symbol that's often used to represent the Internet in flowcharts and diagrams.

A cloud service has three distinct characteristics that differentiate it from traditional hosting. It is sold on demand, typically by the minute or the hour; it is elastic -- a user can have as much or as little of a service as they want at any given time; and the service is fully managed by the provider (the consumer needs nothing but a personal computer and Internet access). Significant innovations in virtualization and distributed computing, as well as improved access to high-speed Internet and a weak economy, have accelerated interest in cloud computing.

A cloud can be private or public. A public cloud sells services to anyone on the Internet. (Currently, Amazon Web Services is the largest public cloud provider.) A private cloud is a proprietary network or a data centre that supplies hosted services to a limited number of people. When a service provider uses public cloud resources to create their private cloud, the result is called a virtual private cloud. Private or public, the goal of cloud computing is to provide easy, scalable access to computing resources and IT services.

Infrastructure-as-a-Service like Amazon Web Services provides virtual server instance to start, stop, access and configure their virtual servers and storage. In the enterprise, cloud computing allows a company to pay for only as much capacity as is needed, and bring more online as soon as required. Because this pay-for-what-you-use model resembles the way electricity, fuel and water are consumed, it's sometimes referred to as utility computing.

Platform-as-a-service in the cloud is defined as a set of software and product development tools hosted on the provider's infrastructure. Developers create applications on the provider's platform over the Internet. PaaS providers may use APIs, website portals or gateway software installed on the customer's computer. Force.com, (an outgrowth of Salesforce.com) and Google Apps are examples of PaaS. Developers need to know that currently, there are not standards for interoperability or data portability in the cloud. Some providers will not allow software created by their customers to be moved off the provider's platform.

In the software-as-a-service cloud model, the vendor supplies the hardware infrastructure, the software product and interacts with the user through a front-end portal. SaaS is a very broad market. Services can be anything from Web-based email to inventory control and database processing. Because the service provider hosts both the application and the data, the end user is free to use the service from anywhere.

Cloud computing describes computation, software, data access, and storage services whereas details are abstracted from end-users. In Cloud Computing, all the things are put on the clouds which can be accessed through internet. Therefore, reducing the cost and other factor.

58

Page 60: Ism Practical File

CLOUD COMPUTING ARCHITECTURE: It can be categorized according to the three types of delivery models, namely Infrastructure as a service (IaaS), Software as a service (SaaS) and Platform as a service (PaaS).

1) Infrastructure-As-A-Service: Infrastructure as a service delivers a platform virtualization environment as a service. Instead of purchasing servers, software, data center space or network equipment, users can buy those resources according to their need. Ex: Telstra, T-systems.

2) Platform-as-a-Service: Platform as a service provides development environment as a service. You can use this platform to develop your own program and deliver it to the users through Internet and servers. Ex: Amazon EC2, Salesforce.com.

3) Software-as-a-Service: Software-as-a-Service (SaaS) delivers software as a service over the internet, so that the users do not need to install or run down the software onto their systems. Ex: Yahoo! E-mail, Salesforce.com.

CLOUD STORAGE: Typical cloud storage system architecture includes a master control server and several storage servers, as shown in Fig.

59

Page 61: Ism Practical File

One of the biggest challenges for the IT’s to hold the large database. Instead of squandering money to buy hard drives or the other storage devices, they can have the better option in the form of Cloud Storage. They can save their data to the remote locations using cloud storage.

Another fruitful aspect of the cloud storage is that you can access your data from anywhere in the world. So, cloud storage provides flexibility and convenience in case of a collaborative work.

ISSUES: With the benefits, however, come risks. Cloud computing gives rise to a number of practical and legal concerns, including the following:

Outages: Unplanned outages are a reality of any cloud computing solution. The inevitable downtime requires organizations to develop strategies and backup plans for how their business needs will be met during times when their cloud applications, and the data stored and processed by those applications, will be unavailable. A company must consider what sort of redundancies and workarounds are necessary to handle both temporary outages and major breakdowns in its cloud-based environment, and how it will be able to implement those alternatives with assistance from the cloud provider and, if necessary, separate and apart from the cloud provider. Understanding the cloud provider’s disaster recovery and business continuity measures, negotiating strong service level agreements and disaster recovery commitments, and implementing various other stop gap measures, such as off-line software synchronization, will help company weather outages in its cloud computing solutions.

Jurisdiction: A major concern with cloud computing is the difficulty of determining where data will be stored, and, thus, what courts have jurisdiction and what law governs the use and treatment of such data (i.e., local, state, federal, foreign, etc.). Information sent or received by an organization or individual using a cloud computing service could be physically located in the United States or any other country in the world. How will a cloud computing customer address situations where one country’s reporting or discovery obligations conflict with the data privacy laws of another county? How will a cloud computing customer protect its intellectual property rights against infringement or other wrongful activity when its cloud-based applications are hosted in a country that does not recognize certain intellectual property protection measures? These and other potential conflicts between the various jurisdictions involved in a cloud computing solution should be resolved from the outset of the arrangement.

Security and Privacy: Concerns have been repeatedly raised over the ability of law enforcement bodies to access sensitive corporate and personal information stored in cloud computing solutions. With privacy laws varying substantially from country to country, and even state to state in the United States, the potential lack of control over the physical locations of storage and processing creates serious data protection and privacy concerns. To highlight the risk, in some jurisdictions, law enforcement could obtain access to the information stored without the affected party ever knowing. Companies should take note that the concerns over security and privacy are not limited to law enforcement. A company considering a cloud computing arrangement will need to know what steps the cloud provider takes to ensure that a customer’s data is not inadvertently disclosed to another customer who may be sharing the same resources. Organizations need to extensively vet the security and privacy standards of a cloud provider, including asking the following questions: a) What

60

Page 62: Ism Practical File

security commitments are taken and are they sufficient to meet my company’s needs? b) Do the terms of use commit the cloud provider to keeping a user’s data secure, or even private, from other legitimate users of the service? c) Do you have the right to perform audits on the cloud provider’s policies and processes? d) What right does the cloud provider have to change those policies and processes?

Licensing and Contractual Issue: Cloud computing requires agreements that provide for a licensing structure and contract terms that fit the cloud computing model of shared resources and web-based computing. Since pricing in cloud computing is typically based on a pay-as-you-go approach, customers need to ensure adequate means for verifying their fee obligations and controls on fee increases. Service levels are key to ensuring that a customer has the needed level of accessibility to its cloud-based IT environment. A customer should carefully consider how it will transition from one cloud provider to another, or away from a cloud computing environment, and what contractual obligations it would need in place to ensure that such a transition occurs smoothly. In addition, any cloud computing agreement should document a comprehensive understanding of each party’s intellectual property rights in the solution, the information stored, the hosted applications, and all developments that result out of the cloud computing arrangement.

61