Oracle Data Warehouse Sizing with EMC Symmetrix DMX-4 and

Oracle Data Warehouse Sizing with EMC Symmetrix DMX-4 and Dell R900

Best Practices Planning

Abstract

This white paper provides an overview of how to size a data warehouse using an EMC® Symmetrix® DMX-4 4500 when attached to Dell R900 servers. The sizing conforms to the standards defined in the Oracle Optimized Warehouse Initiative (OWI), and adheres to EMC best practices for Oracle data warehouses.

February 2009

Copyright © 2009 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

Part Number h6015

Oracle Data Warehouse Sizing with EMC Symmetrix DMX-4 and Dell R900 Best Practices Planning 2

Table of Contents Executive summary ............................................................................................4 Introduction.........................................................................................................4

Audience ...................................................................................................................................... 5 EMC Symmetrix and Dell R900 overview..........................................................5

EMC Symmetrix DMX-4............................................................................................................... 5 Dell R900 series........................................................................................................................... 6

Oracle Optimized Warehouse Initiative.............................................................6 Symmetrix DMX-4 and Dell R900 server scalable building blocks.............................................. 6

Oracle Optimized Warehouse Reference Configurations ........................................................ 6 Symmetrix DMX-4 and Dell R900 Optimized Warehouse ....................................................... 7 Showing capacity and performance scalability ........................................................................ 7 Building block device allocation................................................................................................ 9

Planning the storage layout ...............................................................................9 Choosing a physical disk drive..................................................................................................... 9 RAID 5 vs. RAID 1 ..................................................................................................................... 10 Metavolume configuration.......................................................................................................... 10

Planning Oracle ASM layout ............................................................................12 ASM striping............................................................................................................................... 12 Sequential or random data layout.............................................................................................. 13 Tuning ASM for a data warehouse sequential workload ........................................................... 13 Partition alignment (Windows and Linux) .................................................................................. 13

Planning the database layout ..........................................................................14 Conclusion ........................................................................................................15 References ........................................................................................................15


Executive summary EMC® Symmetrix® DMX-4 is an enterprise-class storage array that can fit any size business. With focus on scalability, security, data protection, and simplicity of management, Symmetrix DMX-4 provides the best choice for any database deployment, including data warehouses. In a similar way Oracle Database 10g and 11g provide the most sophisticated enterprise-class relational database, with self-tuning features, scalability, and simplicity of management. The Dell R900 multi-core server is a high-performance, rackmount, scalable platform capable of meeting both the intense computational needs and high I/O bandwidth needs of the data warehouse. Together, Oracle Database on Dell servers and a Symmetrix DMX-4 storage array create a perfect solution for the deployment of mission-critical, large-scale data warehouses.

Over the last few years there has been a rapid growth in demand for business intelligence (BI) and data mining capabilities for the business. As a result the data warehouse role has changed significantly, from a complementary reporting tool operated by a small group of experts into a critical decision support tool, where managers and executives use sophisticated, but easy to use, tools to run ad-hoc queries, analyze the business state, identify opportunities, and assess trends. Due to these changes it has became even more important to design the data warehouse storage layout so it can continue to grow and yet sustain the required service level for the current and future user community. In order to plan correctly for the increasing demands on the data warehouses, where data size grows rapidly and in most cases the user community as well, it is important to follow a scalable data layout design that meets not only the current requirements but also has the flexibility to meet future changes and demands.

Symmetrix DMX-4 is especially suited for data warehouse deployments by providing a consolidation platform that can support from 96 to 2,400 disks and up to 256 GB of usable mirrored cache to maximize I/O efficiency. This is important because modern BI applications are throughput driven and require a large number of physical disk drives to balance highly parallelized workloads. Large amounts of cache memory improve performance by offsetting the slower rotational speed of low-cost higher-capacity drives like 1 TB SATA by accessing frequently used data, or sequential data streams, from high-speed DRAM. Symmetrix reduces TCO by providing a single highly available storage array suited to deploy from one to many mission-critical databases and applications. Symmetrix Enginuity™ software provides the ability to protect the data by creating local and remote replication in a matter of seconds – regardless of the database size. With TimeFinder®/Snap technology, up to 128 replicas of the database can be created, using just as much storage space as the database change rate. With consistency groups, multiple related data objects are protected as a unit. If a database restore is ever required, as soon as the Symmetrix restore operation starts, the production devices present the restored data immediately to the host (while data is incrementally restored in the background), saving hours and often moreover alternative restore options for large databases.

As data warehouses become mission critical to ongoing business operations, it is important to plan ahead not only for their performance needs, but also for high availability and business continuity. The choice of Oracle Database on Symmetrix storage provides the best of both worlds. Oracle provides a wealth of features with increased focus on simplicity, scalability, availability, and self-tuning, and Symmetrix provides a wealth of data protection options, security, scalability, and top-of-the-line performance and business continuity features. Symmetrix is the first high-end storage array to offer the enterprise Flash drive (EFD) option together with high-performance Fibre Channel drives and large-capacity SATA drives. This allows customers to place the Oracle data on the right storage tier to optimize price/performance returns. Often in data warehouses, placing index, temp, active partitions, or materialized views can boost the overall database performance many times, while simultaneously drop I/O latency from 6-9 ms on traditional disks down to near 1 ms. The Symmetrix DMX-4 storage array provides all the storage tiers the database may need – from ultra-fast EFDs, to fast 4 gigabits (Gb) per second Fibre Channel hard disk drives, and to slower low-cost and high-capacity SATA drives, with a choice of RAID 1, RAID 5, or RAID 6 protection.

Introduction This white paper provides a detailed overview of the best practices for deploying an Oracle Database 10g and 11g data warehouse on a Symmetrix DMX-4 4500 array, with Dell R900 servers, in a manner that Oracle Data Warehouse Sizing with EMC Symmetrix DMX-4 and Dell R900 Best Practices Planning 4

complies with the Oracle Optimized Warehouse Initiative (OWI) framework, and adheres to EMC best practices for Oracle data warehouses. It also highlights the advantage of a balanced building block approach to data warehouse performance and scalability using Oracle RAC with ASM on a Symmetrix DMX™ scalable storage array.

Since at the core for a successful data warehouse deployment is the need for a correct and flexible storage layout design, this paper focuses on the technical topics and considerations around designing the storage layout of an Oracle data warehouse on Symmetrix for growth, performance, and scalability that complies with the Oracle OWI framework.

Audience The primary audience for the paper is storage administrators, system administrators, database administrators, and EMC field and support personnel as well as others who would need to understand the considerations around deploying an Oracle data warehouse on Symmetrix. The audience is expected to have some familiarity with EMC Symmetrix DMX technologies and Oracle Database.

EMC Symmetrix and Dell R900 overview

EMC Symmetrix DMX-4 The OWI architecture defined in this paper is based on the EMC Symmetrix DMX-4 system, which is the latest generation in the Symmetrix DMX series and extends EMC's leadership in the high-end enterprise storage market. The DMX-4 delivers support for the latest generation of disk drive technologies: 4 Gb/s FC and Flash drives for high performance and SATA II for high capacity. Symmetrix DMX-4 is the first and only high-end storage system that can support all of these latest generations of disk drive technologies. The DMX-4 with Enginuity 5773 has been optimized for maximum performance and tiered storage functional flexibility. Enginuity 5773 provides investment protection that delivers performance with information-centric security advancements via integration with RSA enVision. With the DMX-4 and Enginuity 5773 all replication and security activities are easy to manage with the Symmetrix Management Console. And you get all of these capabilities in the industry's most energy-efficient platform, making your data center as “green” as possible.

DMX-4 4500 Up to 64-port 4 Gb/s FC directors 73, 146 GB Flash drives 146, 300 GB FC drives 1 TB SATA II drives Up to 512 (256 usable) GB global memory

Figure 1. Symmetrix DMX-4 4500 disk array


Dell R900 series The R900 series is Dell’s most powerful line of Intel Xeon servers with outstanding performance characteristics based on a four-socket quad-core system board with up to 256 GB of memory and seven PCI-Express slots.

Figure 2. Dell R900 server

Technical details of the R900 server are available at: http://www.dell.com/content/products/productdetails.aspx/server-poweredge-r900

Oracle Optimized Warehouse Initiative The Oracle OWI enables implementation of optimized data warehousing solutions that combine the high performance and scalability of the most popular database for data warehousing - Oracle Database 10g and 11g - with hardware platforms and storage from industry-leading manufacturers, in this case the EMC Symmetrix DMX-4 4500 disk array and Dell R900 servers. This initiative helps customers take the risk out of designing and deploying Oracle data warehouses that are fast and reliable and can easily scale in performance and capacity. There are two components to the initiative: Reference Configurations and Optimized Warehouse:

Oracle Optimized Warehouse Reference Configurations combine industry-standard components into a selection of data warehouses tailored to suit an individual customer's business and technical requirements. Using extensive field experience and technical knowledge, Oracle and its hardware partners have developed data warehouse reference configurations that support varying capacities, user populations, and workloads.

Oracle Optimized Warehouses are complete data warehouse solutions available from Oracle's hardware partners. They are simple and fast to implement with all the server, storage, networking and software components pre-installed, configured, and optimized ready for you to load and run your data warehouse.

Symmetrix DMX-4 as part of the Oracle OWI includes Reference Configurations with leading server vendors as well as Oracle Optimized Warehouses built with Dell R900 servers using industry-standard components. This paper highlights the considerations and layout best practices used for this solution.

Symmetrix DMX-4 and Dell R900 server scalable building blocks

Oracle Optimized Warehouse Reference Configurations Oracle Optimized Warehouse Reference Configurations are balanced system designs for Oracle-based data warehouse databases deployed to a variety of operating systems and types of platforms. The designs specify the platform model, number of processors, number of nodes, and amount of memory, and also define storage, I/O, and networking configurations that are optimized for predetermined data warehouse workloads and capacities. By leveraging Reference Configurations suited for different usage profiles, customers can select the one that best suits their specific price, performance, and compatibility requirements. The wide variety of platform choice enables you to easily integrate the right configuration into your existing IT infrastructure and utilize Reference Configurations as low-risk, implementation starting points.


http://www.dell.com/content/products/productdetails.aspx/server-poweredge-r900

Symmetrix DMX-4 and Dell R900 Optimized Warehouse As part of Oracle Optimized Warehouse, the DMX-4 4500 system was configured to support two Dell R900 nodes where each had 16 CPU cores and four dual-port PCI-Express HBAs for a total of eight paths per node. EMC PowerPath® was used for path failover and load balancing. The DMX was configured with a total of 384, 300 GB 15k rpm disk drives. This provided 192 physical disks per node to support a workload profile with a high rate of large random I/O1.The disks were configured as RAID 5 (3+1), so each node would have 48 RAID groups. Figure 3 shows the hardware components of the solution.

Figure 3. Oracle Optimized Warehouse for EMC Symmetrix DMX-4 4500 with two Dell R900 servers

Showing capacity and performance scalability To measure the scalability of the configuration, the Oracle calibration tool ORION was used. Figure 4 shows the ability of the DMX-4 4500 building blocks to scale from one to four nodes. The throughput of the system was bound by the four dual-port 4 Gb HBAs per node, however, adding more HBAs meant removing the additional four-port GigE PCI card or making the connectivity to the storage less balanced. The basic building block is therefore defined as two Dell R900 nodes to one DMX-4 4500 with a total of 16 4 Gb channels in two nodes. Going to three or four nodes requires the addition of a second DMX-4 4500 frame, which will double the total system bandwidth available from one DMX-4 4500 frame.

1 Often in data warehouse environments, logically sequential operations turn into random I/O by a high level of concurrency and parallelism. In addition LVM striping can randomize the workload if not tuned for sequential operations.


Figure 4. Aggregate I/O bandwidth expected from 1 to 4 Dell R900 nodes

In addition to the ORION benchmark testing, as part of OWI requirements, an actual test data warehouse was created on the candidate hardware and a set of queries provided by Oracle was run on the system. The test warehouse is based on a retail schema. The workload is driven by a series of scripts and does not require a third-party load generation tool. The schema has 12 tables and is focused around two fact tables, sales and returns, with the remaining 10 tables acting as dimension tables. The two main fact tables are range partitioned by date and sub-partitioned by hash on their join key.

The testing consisted of two scenarios. The first scenario was based on gradually increasing the number of concurrent query drivers executing queries against a single Oracle Optimized Warehouse node. Each driver executes a series of queries where the next query can’t start until the previous completed. When multiple drivers are running they are all running simultaneously; however the order of the queries changes from one driver to another. The goal was to see how many drivers can be sustained simultaneously before the pre-set total time of 300 seconds for all queries to complete was reached.

The second scenario introduced an extra node and doubled the data size. In compliance with a building block model, when a server node was added (doubling the CPU power and HBA ports), we also introduced another set of 192 disks to ASM (using ASM rebalance) and storage front-end connectivity to accommodate the new HBAs. No changes to storage back-end connectivity or memory were made. With the addition of the new node, the system was tested again by gradually increasing the number of concurrent query drivers against the aggregate Data Warehouse – this time queries were run simultaneously from both nodes against the database.

Figure 5 shows the results of the OWI Test Kit benchmark for one-node tests, on the left, and for two nodes on the right. The X axis shows the number of concurrent drivers and the Y axis shows the average response time in seconds for the benchmark. At the top of each graph a red line shows the maximum acceptable response, as defined by Oracle, for the test to complete. The results show that with both one and two nodes active against the DMX-4 4500 the response time remained the same through 32 concurrent drivers per node, for a total of 64 drivers in the two-node tests. These results confirm that the benchmark scales linearly from one node to two nodes.


EMC Symmetrix DMX-4 and Dell R900 Best Practices Planning 9

Figure 5. OWI Test Kit results

Building block device allocation The database was designed based on Oracle and EMC best practices. The design was divided into three categories: storage layout planning, volume manager tuning (ASM in this case), and database basic tuning. Each of these categories is discussed in more details in the following sections. Table 1 shows the device allocation used by ASM for each building block (RAC node). As explained earlier, each time a node is added (more memory, CPU, and connectivity), the storage adds front-end connectivity and disks. Each time more devices are added they are rebalanced into the ASM diskgroups. Why EMC recommends multiple ASM disk groups requires an explanation. The reason for separating redo logs and data into their own disk groups are twofold: first, because it allows using TimeFinder to create disk images of the database that can be restored in seconds – regardless of database size. If a media recovery is used after the restore (apply logs to the database) we don’t want to overwrite the redo logs and therefore they need their own ASM disk group. The second reason for separating redo logs is that it allows more flexibility in performance planning and tracking, and also for using different storage tiers or RAID types between logs and data (for example EMC always recommends RAID 1 for logs). In this case the same 15k rpm physicals were used for all objects for simplicity, where the first hypers were redo logs, followed by data hypers as serial concatenated metavolumes and finally temp or FRA devices. Temp files are separated to their own disk group so if SRDF® is used they don’t have to be replicated (temp files are not required for disaster recovery and can be re-created very quickly). Finally, FRA is always recommended to be a separate disk group and potentially be placed on a slower storage tier.

Table 1. Database building block device allocation

ASM diskgroup Size (GB) Count Total (GB) Metavolume2

+Redo 25 24 600 No

+Data 210 48 10,080 8-way serial concat

+TEMP / +FRA 105 48 5,040 4-way serial concat

Planning the storage layout

Choosing a physical disk drive In general, the capacity of 3.5-inch disks has increased dramatically over the last 10 years while the actual drive speeds have barely doubled. The early drives operated at 7,200 rpm and the fastest drives today operate at 15,000 rpm. While drive interface speeds have gone from 20 Mb/s to 2 Gb/s (200 Mb/s) and 2 When metavolumes were used they were always serial-concatenated across the same RAID group (four physicals) creating a sequential data layout on disk for best throughput and workload predictability. To prevent hot spots ASM was tuned to 16 MB allocation units at 128K stripe depth (Fine-Grain template).

Oracle Data Warehouse Sizing with

recently to 4 Gb/s (400 Mb/s) in the DMX-4, the speed of physical disks is limited by their internal physics, drive seek, and latency time. Today the DMX-4 supports a wide range of disk types each with their own capacity and performance characteristics from 73 GB 15k rpm disks to 1 TB 7200 rpm disks. Clearly, deploying a database on 1 TB disk drives will require fewer disks than deploying the same database on 73 GB drives but there are performance implications to be considered. Take the case where a single 1 TB database is deployed on a single 1 TB drive. In this scenario a full table scan can complete no faster than the single 1 TB disk can process 1 TB of data. Alternatively, if the same table is placed on 14 73 GB drives, we could expect the full table scan of the 1 TB table to complete much faster simply because there are 14 physical disks running in parallel and the 73 GB drives spin at twice the rpm of the 1 TB drives. Choosing a disk must include making a decision on what type of performance is required as well as how much capacity is required. For purely sequential workloads most modern 15k rpm FC disk drives can provide around 40 Mb/s per spindle sustained throughput (regardless of the storage array). In typical mixed query data warehouse environments the 15k rpm disk drives provide the best overall performance at around 15-20 Mb/s per spindle. So, for example, to size for 3000 Mb/s of bandwidth, in a mixed query workload, between 150 and 200 physical disks would be necessary to achieve this performance goal, regardless of any capacity requirement or RAID considerations.

In addition it is important to note that in many ad-hoc query environments with a large number of users, the I/O concurrency at the disk level changes the workload from a pure sequential to what seems like more random. In such cases, the 15k rpm drive will be much quicker to serve the random I/O activity than a slower 7,200 rpm disk. Similar causes for full table scans becoming random I/O when they reach the storage are Logical Volume Manager (LVM) striping, like ASM, query parallelism, or file system fragmentation. While the more sequential the workload and data layout are, the fewer disks are required, in most cases spindle count is based on a mixed workload assumption.

OWI tests were conducted using the 300 GB 15k rpm disk drive. If any particular data warehouse installation does not require this much capacity the customer has the option of utilizing 146 GB or 73 GB 15k rpm disks. All three disk types provide the same performance but provide differing capacities at different costs.

RAID 5 vs. RAID 1 The choice of RAID protection will always come down to the speed, reliability, and higher cost of RAID 1 versus the slightly poorer performance, lower reliability, and lower cost of RAID 5. For many mission-critical transaction systems with the highest requirements for performance and reliability, the choice of RAID 1, despite its higher cost, can be easily justified. However, for data warehouse environments that typically require vastly more capacity than transaction environments and that have I/O profiles that exhibit some sequential I/O and relatively low write ratios, the cost savings of RAID 5 outweigh all other factors. Keep in mind that if the system is being configured to meet a specific performance requirement, then the number of spindles required does not depend on RAID type. The choice or RAID protection will determine the usable capacity but the number of physical disks required is the same for RAID 1 or RAID 5.

For simplicity, all OWI test Symmetrix volumes were RAID 5 protected. Since ASM was used, “external-redundancy” was chosen, leaving the RAID protection to the storage array.

Note that EMC recommends RAID 1 for Oracle redo logs protection in production environments. Since logs are relatively small in size there is not much gain by protecting them with RAID 5. On the other hand RAID 1 provides better availability and performance. Often redo logs can be created on the same physical disks as the data devices (even if those are RAID 5 protected). This allows the logs to benefit from the same number of spindles as the data files on one hand, and on the other, since they are different logical devices, they can be part of their own ASM (or any other LVM) disk group as discussed earlier.

Metavolume configuration The standard Symmetrix metavolume creation algorithm will create a metavolume by taking hypervolumes from different RAID groups located on different physical drives. In Figure 6 the red hypervolumes are grouped from nine separate RAID groups (configured as RAID 5 3+1) to create a single 438 GB metavolume. By spreading all hypervolumes over all RAID groups, all physical disks are “shared” by every


LUN in the array. This type of configuration is known as “shared everything.” For typical transaction systems this provides the best performance (due to their random activity with small I/O size), but this is not the case for data warehouse workloads.

Figure 6. Standard metavolume layout

The alternative to the "shared everything" approach is to isolate a concatenated metavolume to the same set of physical drives that comprise a single RAID group as shown in Figure 7. This is done by selecting hypervolumes that reside on the same physical drives. This “shared nothing” approach more closely resembles how most people conceptualize a physical RAID group. Implementing a "shared nothing" RAID group in a data warehouse environment can provide the most predictable and consistent performance. Please note that the Symmetrix software tool (SymmWin) automates the process of creating metavolumes as described previously by choosing the option to create Serial Concatenated Meta Volumes (available for RAID 5).

In a typical "shared everything" metavolume it is common to choose data striping to cause data to be striped evenly over all the meta members. However, for concatenated "shared nothing" metavolumes, data striping would cause logically sequential I/O to be physically random within the disks of the RAID group For that reason, use a “shared nothing” configuration only when the physical data layout (and also the workload) is mainly sequential. Remember that using this storage layout without combining it with tuning the LVM and database layers accordingly (as discussed in the next sections) will not create sequential data layout and will diminish the benefits of a shared nothing design.

All tests discussed in this paper were performed in a "shared nothing" environment using concatenated serial metavolumes on RAID 5 (3+1).


Figure 7. “Shared nothing” metavolume configuration

Planning Oracle ASM layout

ASM striping ASM uses the concept of disk groups. An ASM disk group is a group of ASM disks, and in Symmetrix terms is a host device that could be either hypers or metavolumes. For example the collection of the redo log devices can become the “+REDO” ASM disk group. ASM allows the data to not be mirrored (ASM External redundancy), mirrored twice (Normal redundancy) or three times (High Redundancy). EMC recommends for Symmetrix to not use ASM mirroring and instead use storage RAID 5, RAID 6, or RAID 1 protection as required.

When an ASM disk group is created, Oracle automatically attaches to it a number of pre-defined templates (more can be added manually). ASM Template defines a striping and protection choice. When External redundancy is used then it is the only protection method available; however the choice remains for striping. ASM has two striping options, Coarse and Fine-Grain, where data files, for example, are attached a DATAFILE template by default, which implies Coarse striping, and where log files are attached an ONLINELOG template by default, which implies Fine-Grain striping. Coarse striping means that the only striping is based on the ASM Allocation Unit, or AU, which is by default 1 MB. For example, if a 10 MB data file is created, ASM will allocate to it 10 x 1 MB AUs, and attempt to place each on a different ASM disk. Fine-Grain striping adds a finer level of striping with a default stripe depth of 128 KB. When Fine-Grain striping is used, ASM will choose eight disks in the group, allocate an AU on each, and stripe the data across them with the stripe depth before repeating the process with another set of eight disks. For example, if a redo log of 10 GB is created, ASM will allocate 8 x 1 MB AUs on eight ASM disks, and stripe the logs such as the first 128 KB is on the first AU, the second 128 KB on the second AU and so forth, wrapping back to the first AU after the eighth. Once all eight AUs have filled, ASM will repeat the same method by choosing another set of eight disks, striping the remaining 2 MB across them as before.

In essence, Fine-Grain striping allows sequences of sequential data layout on each ASM disk to the length of an AU, while Coarse striping spreads the data everywhere and almost always creates random data layout. If only a single data file was created at a time chances are that the layout on the ASM disk members will be sequential since ASM allocates AU requests on a first-come first-serve basis and when it wraps to the first disk, it will place the next AU adjacent to the previous. However, in reality, often multiple data files are created simultaneously, in essence creating a random data layout on disk. Also many disk members in the ASM disk groups will prevent much wrapping and therefore reduce the amount of


sequential data sequences on each disk that belongs to the same data file extent. The default ASM stripe size is 128 KB and the default AU size is 1 MB. Based on this, if we wanted to create sequential data layout for sequential data access, we would be served best by having data files use Fine-Grain striping instead of Coarse, and by increasing the AU from 1 MB to 16 MB (more than 16 MB may create hot spots as queries will exercise a small number of ASM disks for quite a while before progressing to the next set of eight disks).

Sequential or random data layout Why it is important to create a sequential layout for workloads that relies primarily on sequential data access? There are two reasons. First, because as mentioned earlier, each physical disk can sustain a certain sequential throughput that drops to less than half if the workload becomes random. That means more drives are needed to support the throughput of a random workload (and likely more expensive since the faster the drive, the quicker it satisfies random read I/O). The second reason is that the Symmetrix data prefetch algorithm can recognize and track multiple sequential I/O streams per device. Together with the large Symmetrix cache size (up to 256 GB mirrored) Symmetrix prefetch makes sure that data is resident in Symmetrix cache before the next I/O comes to request it. The result is optimized disk access, less wait for disk seek time and head placement, and the data is already in Symmetrix cache ready to serve I/O requests.

Tuning ASM for a data warehouse sequential workload Based on the discussion in the previous sections we’ll say that creating sequential data layout on disk for sequential workloads has many benefits over random layout - both from cost, energy saving, disk access optimization, and I/O efficiency. In order to do it using ASM we have to do two things: first, make sure that data files use an ASM template that implies Fine-Grain striping, and second, increase the AU size from 1 MB to 16 MB.

ASM AU size and stripe size can be modified in Oracle Database 10g system-wide by using init.ora underscore parameters _asm_ausize and _asm_stripesize (see Metalink note: 368055.1). In Oracle Database 11g the AU size can be determined during disk group creation as shown in Figure 8.

SQL> CREATE DISKGROUP DATA EXTERNAL REDUNDANCY DISK '/dev/emcpower[a-z]1', '/dev/emcpower[aa-az]1' ATTRIBUTE 'compatible.rdbms' = '11.1', 'compatible.asm' = '11.1', 'au_size' = '16777216';

Figure 8. Creating an ASM disk group with a 16 MB AU in Oracle Database 11g The ASM template can be determined when the tablespace is created (both in Oracle Database 10g and 11g). An example of creating a tablespace that uses Fine-Grain striping (by borrowing the ONLINELOG template) is shown in Figure 9. SQL> create tablespace data_1 datafile '+DATA/data_1(ONLINELOG)' size 32G reuse extent management local uniform size 16m segment space management auto;

Figure 9. Creating a tablespace with a non-default ASM template

Partition alignment (Windows and Linux) Modern hard disk systems use the logical block address (LBA) to determine a location on disk, where blocks are units of 512 bytes. This is true for both SCSI and IDE disks. However, older disk systems used a different addressing scheme called cylinder, head, and sectors (CHS) to describe the geometry of the drive. Due to IBM PC BIOS legacy, all x86-based computers still support CHS addressing, which on modern drives defaults to a geometry of 63 sectors/track and 255 heads/cylinder.

The first block of all hard disks contains a reserved area called the Master Boot Record (MBR) where the beginning of the system boot information is found. The MBR also contains critical partition table information. When data partitions are added to the disk, the first partition has to start after the MBR and partition table management area in order to avoid overwriting them. By default, partitions created on a disk are aligned on a cylinder boundary, with an exception of the first partition. The first partition after the MBR is actually track aligned, presumably since it was felt that it was too wasteful to just use one block for the


MBR in an otherwise empty cylinder. Due to this legacy issue, the first partition starts at the first track boundary, which falls on block address 63, or 63 * 512 bytes = 32256 bytes, a 31.5 KB offset.

Symmetrix arrays, however, define tracks to be 64 KB in DMX-3 and later, or 32 KB in DMX-2 and earlier. Tracks are used to map the Symmetrix cache memory to data stored on logical volumes. Because of the first partition misalignment on x86 systems relative to the storage array tracks, all data on that partition will continue to be misaligned and that misalignment has been shown to cause performance degradation. This legacy issue affects both Linux, and Windows x86 and x86_64 systems when SCSI BIOS is not enabled. The only exception to this partition misalignment is extended partitions on Windows, where a logical volume in an extended partition defaults to an exact 64 KB boundary alignment.

M B R

Reserved (63 blocks)

Default start for partition 1

New start for partition 1

64KB 64KB

Figure 10. Partition alignment to Symmetrix tracks In order to avoid problems created by misalignment it is necessary to align partitions on Linux using the fdisk or parted commands. On Windows DISKPAR.EXE can be used to align a partition to a 64 KB boundary. An example of creating a partition and aligning it on Linux is shown in Figure 11.

# fdisk /dev/emcpowerfw Command (m for help): n [create a new partition] Command action e extended p primary partition (1-4) p [create a primary partition] Partition number (1-4): 1 First cylinder (1-51281, default 1): [Press ENTER to use default] Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-51281, default 51281): [Press ENTER] Using default value 51281 Command (m for help): x [go into Expert mode] Expert command (m for help): b [move beginning of partition] Partition number (1-4): 1 New beginning of data (63-823829264, default 63): 128 [128 blocks => 64KB offset] Expert command (m for help): p [print new partition table] Disk /dev/emcpowerfw: 255 heads, 63 sectors, 51281 cylinders Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID 1 00 1 1 0 254 63 1023 128 823829137 83 2 00 0 0 0 0 0 0 0 0 00 3 00 0 0 0 0 0 0 0 0 00 4 00 0 0 0 0 0 0 0 0 00 Expert command (m for help): w [write partition table to disk and exit]

Figure 11. Creating and aligning a partition to Linux

Planning the database layout Tuning the Oracle Database for a Decision Support Systems (DSS) type workload is out of the scope of this paper. With each Oracle release the database server continues to become more self-tuning and in the OWI tests there was no attempt to tune the database to the specific queries. The only tuning was to choose a PGA


size and DOP that matched the server capabilities to the workload based on Oracle Automatic Workload Repository (AWR) report recommendations.

However, an important aspect of the database layout is preventing high fragmentation that reduces the benefits of sequential data layout as discussed in the previous sections. To avoid fragmentation, fact tables can use locally managed tables with uniform size extents and Oracle Automatic Segment Space Management (ASSM). Each time a table or partition requires new physical storage space in the tablespace (or any Oracle data segment for that matter) it allocates physical space using units called extents. Data extent size can be defined at multiple levels, for example during tablespace or table creation. If ASM Fine-Grain striping is used the extents would have to fill eight ASM disk members at 128 KB chunks (requiring at least 8 x 128 KB or 1 MB of data) before wrapping again to the first disk. Since the storage array is not aware of ASM striping, only an adjacent I/O request on each ASM disk member will be recognized as sequential I/O and invoke Symmetrix prefetch. Therefore the table extents should be set to a few megabytes in size so they will wrap a few times over the eight ASM disks. For that reason it is recommended that the extent size should match the ASM AU size, which is 16 MB if ASM was tuned as described in the previous section. Note that although the extent can be larger as well, make sure that there is enough data to fill each extent (for example, partition or sub-partition) to avoid unused storage space. In contrast, automatically allocated extent management (rather than uniform size) should be avoided. Automatically allocated extents start with extent size equal to the Oracle block size (commonly 8 KB or 16 KB), which is too small and is likely to create very high fragmentation, preventing sequential data layout.

Conclusion There has been a significant increase in Oracle data warehouse deployments by customers looking to implement their first data warehouse or those looking to increase the size of their data warehouse. Using the Oracle Optimized Warehouse Initiative framework as a guide for the server and storage configuration, customers can simplify the hardware procurement process and provide a tested, balanced hardware platform on which to deploy the data warehouse.

References The following can provide more information:

• EMC TimeFinder and SRDF for Oracle 10g Automatic Storage Management — Best Practices Planning white paper http://www.emc.com/collateral/hardware/white-papers/h2919-emc-timefinder-srdf-oracle-db-10g-auto-stor-mgmt-wp.pdf

• EMC Symmetrix DMX 4 for Oracle 10g and Oracle 11g Data Warehouse Layout — Best Practices Planning white paper http://www.emc.com/collateral/hardware/white-papers/h4005-symmetrix-dmx4-oracle-wp.pdf

• EMC Symmetrix DMX-4 Series page on the EMC website http://www.emc.com/products/series/symmetrix-dmx-4.htm

• Oracle Database 11g Documentation Library http://www.oracle.com/pls/db111/

• Dell PowerEdge R900 page on Dell.com (includes rack server documentation) http://www.dell.com/content/products/productdetails.aspx/server-poweredge-r900


http://www.emc.com/collateral/hardware/white-papers/h2919-emc-timefinder-srdf-oracle-db-10g-auto-stor-mgmt-wp.pdf

http://www.emc.com/collateral/hardware/white-papers/h2919-emc-timefinder-srdf-oracle-db-10g-auto-stor-mgmt-wp.pdf

http://www.emc.com/collateral/hardware/white-papers/h4005-symmetrix-dmx4-oracle-wp.pdf

http://www.emc.com/products/series/symmetrix-dmx-4.htm

http://www.oracle.com/pls/db111/

http://www.dell.com/content/products/productdetails.aspx/server-poweredge-r900

Documents

Oracle Data Warehouse Sizing with EMC Symmetrix DMX-4 and