Oracle RAC Presentation at Oracle Open World

Oracle on Linux at BA

From Evaluation To Implementation

15th November 2007

Imtiaz Mazhary Technical Architect

Paul Marden Project Manager

Oracle RAC – From Evaluation to Implementation

little teaser …SQL> create tablespace IMTEST_TS datafile '+DGTEST' size 10m;create tablespace IMTEST_TS datafile '+DGTEST' size 10m*ERROR at line 1:ORA-01119: error in creating database file '+DGTEST'ORA-17502: ksfdcre:4 Failed to create file +DGTESTORA-15001: diskgroup "DGTEST" does not exist or is not mounted

SQL> create diskgroup DGTEST 2 normal redundancy 3 failgroup DGTEST_BH 4 disk '/dev/raw/raw1001' name DGTEST_BH_DISK1001 5 failgroup DGTEST_CB 6 disk '/dev/raw/raw2001' name DGTEST_CB_DISK2001;

Diskgroup created.


little teaser …SQL> create tablespace IMTEST_TS datafile '_DGTEST' size 10m;

Tablespace created.

SQL> !ls -lart $ORACLE_HOME/dbs | grep DGTEST

-rw-rw---- 1 oracle oinstall 1056768 Sep 4 12:32 _DGTEST

Bug 6391521 RAC DATABASE DATAFILE CREATION ALLOWS SPECIFICATION OF NON-SHARED STORAGE


Oracle on Linux Evaluation

•Proposition•Cost reduction

•Improved resilience

•Simplification

•People•Infrastructure Design & Delivery

•DBA Teams

•Linux Infrastructure

•Oracle RAC Pack, IBM, Red Hat, Scaleabilities


Oracle on Linux Evaluation

•Platform Design Stage•Design Patterns

•Technology Stack

•Areas of Testing

•List of Issues

•Deliverables•BA, Oracle and IBM Validated Configuration


Design Patterns – Database Tier

DB01

Service X

SAN Storage

Basic Pattern

Application X


SAN Storage


Application X

DB10R1

Service X

DB10R2

Service X

RAC Lite Pattern


SAN Storage


DB10R1

Service X

DB10R2

RAC Lite Pattern for Consolidation

Application Y

Application X

Service Y Service X

Service Y

Service Y Service X



Application X

DB20R1

Service X

DB20R4DB20R2

Service X

DB20R3

Service X Service X

Full RAC Pattern

SAN Storage


Technology Stack

IBM SDD v’s RHEL Device

Mapper

3rd Voting Disk on NFS


Areas of Testing

•Operating Environment• RHEL 4.5 – CFS, LVM2,

Device Mapper (and IBM SDD)

• SAN Attachment – QLogic vs RHEL QLogic driver

• RHEL 4.5 Cluster Suite

• BMC Patrol, EMC Legato Networker, IBM MQ Series, Control-M . . . .

•Oracle DB 10g R2 (142 tests)

• Serviceability – build, patching & configuration

• Clusterware resilience – interconnect, OCR & voting disks

• ASM configuration• ASM resilience & recovery• Stretched cluster performance• Client connectivity & failover• Backup & recovery, using EMC

Legato Networker• System Monitoring, using BMC

Patrol • EM Grid Control


`

List of key issues – all fixed in 10.2.0.3.0

CLSFMT error on CW install 5527179.993

Use of NFS for 3rd voting disk

Kernel panic caused by aio_put_req

5473171.993

885818 (RH)

Bug – RHEL libaio driver

Problems adding additional node

5449529.993

rootaddnode.sh using incorrect VIP

ASM block corruption 5556890.993

Bug - Block corruptions when dropping & adding failure groups

ASM & CW crashes/reboots on loss of one storage array

5594045.992

Bug - CSS error when processing stale member data requests

DBVerify Segmentation Fault

5682851.993

Bug – back port issue

Loss of one OCR causes CW errors

5593646.992

Feature - to prevent OCR data loss

OCR synchronisation behaviour 5621599.992

Similar to the above

Failure to replace OCR device 5698198.992

OCR storage too small when replacing the failed OCR disk

"read only filesystem" error 1768384 (RH)

Internal RAID array firmware issue?


Evaluation Project Deliverables

•Tested reusable patterns for use across the airline

•Supplier verification of our infrastructure - Oracle Validated Configuration undertaken by:

•IBM

•Oracle

•Red Hat

•OVC link: http://www.oracle.com/technology/tech/linux/validated-configurations/index.html


Database Consolidation

•Proposition •reduce the size of Oracle DB estate

•cost reduction / simplification

•first RAC Lite and Basic Pattern – learning exercise

•Process •LEAN - Runners / Repeaters

•pre-provisioning

•People - internal, RAC Pack

•IT - criteria, examples, experiences

FTD Batch(lxlhrl214)

OAW Batch(lxlhrl213)

Database Service - RAC Lite Pattern

BH CB

Batch/Feed Service (Linux HA)

Node X ConMan

Batch

Node Y

VIP X

RACDBR1

VIP Y

DGDATA_BHDGFRA_BH

Public Network

RAC Network

Storage Network

Database Consolidation - architecture

DGDATA_CHDGFRA_CH

Database Service Basic Pattern

DB1DBservice_ConMan

DBservice2DBservice3

….

DB2DBservice1DBservice2

….

ConManBatch

RACDBserviceFAST

RACDBserviceB

RACDBserviceC

RACDBserviceD

EXT3

RACDBR2

RACDBserviceC

RACDBserviceD

RACDBserviceFAST

RACDBserviceB

FASTBatch

FASTBatch

RACDBserviceFAST

RACDBserviceB

VIP X


Database Consolidation – Progress so far•Current Production Environment:

•30+ applications consolidated

•3 on RAC Lite

•Capacity provisioned for 20 more

•Experiences•Very few technical issues with infrastructure

•Mainly classic problems associated with migration


Database Consolidation - CriteriaRunner Repeater Stranger

Pattern type Basic or RAC Lite pattern

Basic or RAC Lite Pattern

Full RAC pattern or other

Hardware IBM xSeries

(BA standard model)

IBM xSeries

(BA standard model)

IBM xSeries

(standard BA standard model) or other Intel platform

Operating platform RHEL4 64 bit RHEL4 64 bit RHEL4 64 bit or other

RH release

Database Oracle 10g DB or

Oracle 10g RAC

Oracle 10g DB or

Oracle 10g RAC

Oracle 10g RAC

Data size (storage constraint) 60GB or less Greater than 60GB Any

Memory (SGA + User) 2GB or less 4GB or less Any

Overall server capacity

(cpu + memory constraint)

Less than 25% Less than 50% Any

Concurrent clients (cpu constraint)

10 or less Less than 50 Any

Business transactions Less than 3,600 per hour

Greater than 3,600 per hour

Any

Transaction type OLTP OLTP OLTP or Warehouse / DSS

3rd party supplier contract does preclude database consolidation

Yes Yes or No No


DW to Full RAC Pattern•Proposition

•first Full RAC implementation – learning exercise•important but not key business component

•Process - LEAN -- Strangers

•People - internal, NIIT, RAC Pack, Scaleabilities

•IT•Where are we coming from?•What does the cluster look like?•Gotchas / lessons learnt?


Legacy platform

•Old SUN ES6500, 16*400MHz CPUs, 20GB RAM

•Solaris 8, Oracle 9.2

•1.6 TB Database …. Operational Activity Warehouse & FTD

•300+ batch feed jobs

•Oracle Forms 6!

•MQ Series licence costs for 16 CPUs

•Severe performance problems

•No HA



FICO DCS StaffAdmin Baggage ….. …..

WA21R Database - Full RAC Pattern

VIP1

BH CB

Batch/Feed Service (Linux HA)

Node X


FTD Batch (A)

Node Y


OAW Batch (A)

VIP2

DGDATA_CBDGFRA_CB

DGDATA_BHDGFRA_BH

Public Network

RAC Network

Storage Network

Target Platform

WA21R1

WAPRD_BATCH

WAPRD_APP

FDPRD_APP

WA21R2

FDPRD_APP

WAPRD_APP

WAPRD_BATCH

VIP3

WA21R3

WAPRD_APP

FDPRD_APP

FDPRD_BATCH

VIP4

WA21R4

FDPRD_BATCH

FDPRD_APP

WAPRD_APP


DW to Full RAC Pattern

•Proposition•first Full RAC implementation – learning exercise•important but not key business component

•Process - LEAN -- Strangers

•People - internal, NIIT, RAC Pack, Scaleabilities

•IT•Where are we coming from?•What does the cluster look like?•Gotchas / lessons learnt?


Roadmap

•More consolidation on RAC•100 databases in FY0708

•Key Business Systems move to RAC•Orders Data Store Q1-2008

•ba.com FY0809

•SAP for BA Engineering ?

•Mainframe like availability•More about people than technology

•Fast Recovery – Oracle Data Guard


Contact Details

•Imtiaz Mazhary•Technical Architect, Infrastructure Design

•[email protected]

•Paul Marden•Project Manager, Infrastructure Delivery

•[email protected]