Jason Arneil Exadata Consultant

Slide 1

Jason ArneilExadata ConsultantExadata Consolidations and Migrations:Getting it Right Up FrontOracle EMEA Database Partner of the Year

Oracle Partner of the year 2010 & 2012 +44 (0) 844 811 [email protected] RedStack Technology CompanyOracle EMEA Database Partner of the Year

In this presentation I WANT to share with you some of my experiences over the past year

In that year Ive done a lot of work with a couple of large retailers in the UK on Exadata

This has invariably led to a lot of work doing capacity planning

1Jason ArneilExadata ConsultantThings you want to think about when considering buying Exadata to size it correctly and things to configure beforeyou go liveOracle EMEA Database Partner of the Year


What I really wanted to call this presentation

Well, I want to give you a birds eye view on an Exadata project

Ive been working on a consolidation project for a large retailer for a large part of the last year

Id really like to share some of the experiences and insights gained over this time

2Jason ArneilExadata ConsultantReal World Exadata Configuration TipsWednesday 10:45 Exchange 6Oracle EMEA Database Partner of the Year


However the configuration part has been moved to a new presentation

I highly recommend you make space for that as there will be 3 presenters

In additon to me, youll have Andy Colvin and Frits Hoogland giving you the benefit of their experience3

jarneil.wordpress.com@jarneilOracle EMEA Database Partner of the Year


My name is Jason Arneil

Been in IT for around 15 years

SA/DBA

Written quite a few blog articles

also on twitter

now an oracle AceCustomers


Work in the Engineered Systems Team at e-dba in the UK

Team is 100% focused on the e-dba Exadata customers

e-dba are on Oracle platinum partner

We have our own Exadata x3-2 and exalytics boxes available for POC

5

Oracle EMEA Database Partner of the Year


Quite often my engagement with the customer is AFTER the PO to purchase an exadata has been signed

Often customers take the approach of here is our budget, how much exadata can I buy

rather than working out how much capacity they need, then seeing the bill.

6


This approach can often lead to, too little exadata

7



very rarely do we end up with too much8



I always recommend having a non-production Exadata

It always makes me slightly nervous when a company does not have any non-production Exadata kit, how on earth can you test any release without it?

thankfully, I mostly work with organisations that do have non-prod kit.

Of course, non-production these days seems to have as high SLAs as actual production kit

Id also advocate having similarly sized kit for non-production. Non-production tends to have more instances than production

If you have a full rack to consolidate onto, its going to be difficult to just use a rack in non-prod.

You can always time-share but this invariably leads to further exadata purchases.9



A lot customers I see are consolidating lots of databases on Exadata

A large number of the databases were NOT on 11gR2. None were on Linux, mostly AIX or windows.

This can cause additional challenges when having to change more moving parts at once

All the applications where 3rd party applications NO code changes were possible,

Effectively this was a LIFT AND SHIFT

but some applications were upgraded to ensure theyd run and be supported on11gR2 again this adds to time and increases complexity10AgendaCapacity PlanningOracle EMEA Database Partner of the Year


on to capacity planning

There are really 3 variables you need to ensure that you have enough capacity to cope with the demand for

DISK

CPU

MEMORY

11Agenda



This is an example from one of our customers

From this data including CPU/MEMORY a recommendation had to be made note the lack of IOPS info

12Agenda



Strongly advise mapping out how the capacity of your exadatas will be carved up

will show an example of this later

You need to get an idea of how much resources your databases are using BEFORE moving to Exadata

Then MAP out how those resources will be distributed on your exadata

13Consolidation



Best not to size your purchase based on storage capacity

certainly seems to be what managers and beancounters worry about the most

14Capacity Planning186TB504TBOracle EMEA Database Partner of the Year


Because your Oracle sales person will probably tell you get this much with a Full rack

High Capacity Drives

Except if you are expecting to have the amount of space to store your database data on

Youll be disappointed

15DATA RECO DATA RECO DBFS SystemPartitionsOracle EMEA Database Partner of the Year


This is how the different disks are carved up in an exadata storage cell.

We have system and non system disks in an Exadata Cell

we can see the system and non system disks are in fact carved up like this

the sizing aligns across ALL disks as does where the different griddisks are created within a disk - it all lines up.

Col ACol BCol DCol Cblock headerblock headerblock headerCU headerCompression UnitHybrid Columnar CompressionOracle EMEA Database Partner of the Year


HCC stores data in a non traditional format

DATA still resides in oracle blocks

Data storage now in logical constructs called Compression Units

Compression Unit has multiple oracle blocks

ROWS no longer stored together

Data organised by column in the Compression Unit

Not true columnar storage any row can be read within 1 compression unit 17Table Size Reduction Retail DataHybrid Columnar CompressionOracle EMEA Database Partner of the Year


This Data was taken from an e-dba retail customer

6TB table size uncompressed

going down to under 1TB with archive highCompression Ratio Retail DataHybrid Columnar CompressionOracle EMEA Database Partner of the Year


We have also calculated the compression ratio

we see this goes from 4 with Query Low

up to 7 times smaller data set with Archive HighCompression Ratio Logging DataHybrid Columnar CompressionOracle EMEA Database Partner of the Year


This is another table from a customer

This time the data was much more compressable

This could compress down nearly 50 times with Archive High

That could be a 50TB table compressed to 1TB

Compression is heavily dependent on the the type of the DATATotal Database Size ReductionHybrid Columnar CompressionOracle EMEA Database Partner of the Year


This data is taken from a range of customers

it looks at the reduction in TOTAL database size, not individual tables

Some things like indexes and redo logs etc are NOT compressed

still shows up to x5 times reduction in DB size

This could be a 50TB database stored in 10TBCapacity PlanningIOPSOracle EMEA Database Partner of the Year


I'm not saying you should not properly size your disks for capacity,

But you should also pay very close attention to the amount of IOPS you want to drive through the Exadata

22Capacity Planning504 TBHigh CapacityHigh PerformanceOracle EMEA Database Partner of the Year


You also have a choice of whether to use High Performance or High Capacity drives

It makes quite a difference in BOTH Capacity and IOPS

A HIGH CAPACITY drive has 5 times the capacity of a High Performance drive

HP drives run at 15K RPM

HC Drives are 7,200 RPM23Capacity PlanningOracle EMEA Database Partner of the Year


24Capacity Planning



25Capacity Planning



26Consolidationwriteback flash cacheOracle EMEA Database Partner of the Year


Write back flash allows writes to go straight to flash

then staged to disk later

Can increase write throughput under some circumstances

It is NOT currently enabled by default

Change the setting on the cell27Capacity Planning

Benefit of Writeback flashOracle EMEA Database Partner of the Year


Testing done using Kevin Clossons SLOB benchmark

Along X axis is the number of writing sessions

Along Y AXIS is number of writes I was able to perform per second

Only difference between the two sets of runs is writeback flash being turned on

Nearly a x3 performance improvement with larger number of writers28Capacity PlanningOracle EMEA Database Partner of the Year


To illustrate how you might require different sizes of Exadata RACK when capacity planning on just disk space

Perhaps you only require the disk space that a Quarter Rack HC gives you

29Capacity PlanningOracle EMEA Database Partner of the Year


But if you require 10,000 IOPS

Then you are going to need RACK

30Capacity PlanningCPUOracle EMEA Database Partner of the Year


For each database being moved you want to gather information on how many CPUs are allocated to the database

This is fairly trivial to do: cpu_count parameter for example, but you may want to check whether they are cpu cores or threads

31Capacity PlanningCurrent Allocated CPUOracle EMEA Database Partner of the Year


For each database being moved you want to gather information on how many CPUs are allocated to the database

This is fairly trivial to do: cpu_count parameter for example, but you may want to check whether they are cpu cores or threads

32Capacity PlanningUtilisationOracle EMEA Database Partner of the Year


Its all well and good having a bunch of CPU allocated for a database, but does the database actually utilise all those CPUs

Some APP owners, organisations just want a like-for-like numbers of cpu

Good consolidation is to give the same amount of cpu resource, not necessarily the same number of cpus33Capacity PlanningCurrent CPU RequirementOracle EMEA Database Partner of the Year


34Capacity PlanningSPECOracle EMEA Database Partner of the Year


Not all cpu's are created equal.

A five year cpu will not be as fast or able to do as much work as a cpu in an new Exadata compute node

The Standard Performance Evaluation Corporation

You can use the spec benchmarks to scale between the different types of cpus

Spec int 2006 essentially gives you a measure of the power of a cpu, you can use this to compare different cpus

If you have a cpu requirement of 6 but your new cpus are 3 times as powerful,

you only need 2 cpus worth of your Exadata compute node cpus35Capacity PlanningOFFLOADINGOracle EMEA Database Partner of the Year


The final thing that can effect your Exadata cpu requirement is offloading

This is the fact that some processing that you would on non Exadata have done on your database server

Is now done on your storage cells

This will reduce the amount of cpu required by your databases when moving to Exadata

But by how much is really very application specific

DW type queries much more likely to benefit from offloading than 36Capacity Planning

Current CPUUtilisationXSPEC Scale FactorXOffloading FactorXRequired Exadata CPUOracle EMEA Database Partner of the Year


37Capacity Planning



38Capacity PlanningMemoryOracle EMEA Database Partner of the Year


Memory has a more 1:1 mapping between your current databases

and when you migrate to exadata39Capacity Planning



40Instance PlacementCPUMEMI/OCPUMEMI/OOracle EMEA Database Partner of the Year



and when you migrate to exadata41Instance PlacementCPUMEMI/OCPUMEMI/OOracle EMEA Database Partner of the Year



and when you migrate to exadata42Instance PlacementCPUMEMI/OCPUMEMI/OMAXMAXMAXOracle EMEA Database Partner of the Year



and when you migrate to exadata43Instance PlacementCPUMEMI/OCPUMEMI/OMAXMAXMAXOracle EMEA Database Partner of the Year



and when you migrate to exadata44ConsolidationOracle EMEA Database Partner of the Year


Almost a clich using a picture like this.

Really does illustrate the feedback mechanism in a long running project

produce an initial capacity plan

migrate some instances

check that where you think you were going to be matches where you actually are

redo the capacity plan

45Consolidation



Going to finish with a couple of tips for running databases in a consolidated environment46Whats NewX4-2Oracle EMEA Database Partner of the Year


Now there *WAS* meant to be an oracle announcement tomorrow on the Exadata X4-2

Invites went out, and then the event was postponed

However, there is a lot of info in the public domain regarding what is coming next in the Exadata world:

Oracle have already released standalone X4-2 servers that form the basis of Exadata hardware

47Whats New24 CORESOracle EMEA Database Partner of the Year


We know the X4-2 servers come with 2 socket and up to 12 core chips

They are Intel Xeon processor E5-2600 v2

That would be a 50% increase in core count from an X3-2

48Whats New512GB RAMOracle EMEA Database Partner of the Year


We know the X4-2 datasheet states that you can shove 512GB RAM into these servers

this is a massive amount of memory over what has been available historically

This is going to further ramp up the amount of consolidation

49Whats NewDouble FlashOracle EMEA Database Partner of the Year


We know the X4-2L servers which the storage cells are based on

Have Sun F80 Flash cards which will double the capacity compared to the cards in X3-2

This will 45TB of flash in a full rack50Whats New11.2.3.3.0Oracle EMEA Database Partner of the Year


This has been discussed at OpenWorld

The BIG feature is compression of data in flash

So even uncompressed data on disk when read into flash will be compressed there

The flash cards themselves will do the compressing so offloading the cpu overhead away from the cpu in the storage cells

This gives you the potential of a lot of data being served from flash.51Q & A


ConsolidationInstance CagingOracle EMEA Database Partner of the Year


When you have multiple databases running on the same node

You really don't want one instance to steal all the resources of the node

Limits cpu usage of the database instance process by throttling them

53Consolidationcpu_countOracle EMEA Database Partner of the Year


54Consolidationresource_manager_planOracle EMEA Database Partner of the Year


55Consolidationresmgr: cpu quantumOracle EMEA Database Partner of the Year


You will see this wait event with instance caging

means that a session is having to wait to get allocated onto CPU

want to monitor how many sessions are waiting if its not too many then there is no worry56ConsolidationV$rsrcmgrmetric_historyOracle EMEA Database Partner of the Year


You can check this view to see how many active sessions there have been

and how many sessions have been waiting to get onto CPU

consistently too many waiting think about increasing cpu_count

consistently less active sessions than cpu_count you could think about decreasing57Capacity PlanningoversubscribeOracle EMEA Database Partner of the Year


You can oversubscribe the number of cpus allocated to instances

This can work if you know its unlikely all instances will be using their maximum allocation at the same time

Can work well with test/dev instances as they are often idle

more reluctant to do this in PRODUCTION

Can work if you know different databases are busy at different times58ConsolidationMemoryOracle EMEA Database Partner of the Year


59ConsolidationHuge PagesOracle EMEA Database Partner of the Year


using Huge Pages on linux is a way to allocate large amounts of memory with less overhead

its useful for consolidation as you are likely to be trying to allocate as much of the memory on a compute node as you can

Huge pages increases the size of the memory page allocated to 2MB on linux

Much bigger than the default so there is less pages to keep track off which frees up memory for useful stuff

Not compatible with AMM

Can only be used to allocate Memory for SGA, so remember to keep some back for PGA60Consolidationvm.nr_hugepagesOracle EMEA Database Partner of the Year


using Huge Pages on linux is a way to allocate large amounts of memory with less overhead

its useful for consolidation as you are likely to be trying to allocate as much of the memory on a compute node as you can

Huge pages increases the size of the memory page allocated to 2MB on linux

Much bigger than the default so there is less pages to keep track off which frees up memory for useful stuff

Not compatible with AMM

Can only be used to allocate Memory for SGA, so remember to keep some back for PGA61Consolidationuse_large_pages=onlyOracle EMEA Database Partner of the Year


You can set this parameter on your instance to ensure it will not startup if

it cannot allocate enough Memory backed by huge pages62ConsolidationIORMOracle EMEA Database Partner of the Year


With multiple databases on an Exadata rack, you dont want one db saturating the I/O leading to poor performance for the rest

You can use Exadata I/O Resource Manager to control I/O allocation

IORM only available on Exadata

It works and is configured at the Storage Cell Level

63ConsolidationDATABASEABCALLOCATIONLIMIT45%30%25%75%60%50%Oracle EMEA Database Partner of the Year


IORM is controlled via setting an iormplan on the storage cell Can Guarantee certain amount of disk bandwidth

Can also set a ceiling in amount of I>O an instance can do

Can get very complex with different levels64ConsolidationobjectiveOracle EMEA Database Partner of the Year


The objective option specified the optimization mode for IORM

Low Latency improves latency by reducing disk utilisation

High Throughput

balanced

AUTO65Consolidationscan listenerservices+Oracle EMEA Database Partner of the Year


Surprising how much trouble this can cause

Ive been using 11gR2 for 3 + years, but its quite new for a lot of people

Have had to explain the use of the SCAN listener many, many times66Consolidation



You dont need to be an Exadata jedi to properly size

and consolidate a whole load of databases onto Exadata

Just make sure you have a capacity plan

and use the tools for consolidating workloads that are available67

Documents

Jason Arneil Exadata Consultant