Maximum Availability Architecture(MAA): Oracle E-Business Suite Release 12Lyn Pratt, Richard ExleyCMTS, MAA GroupOracle Server Technologies

Metin YilmazSenior Principal Support Engineer,Oracle Support Services

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

<Insert Picture Here>

Agenda

• Maximum Availability Architecture (MAA)• E-Business Suite MAA• E-Business Suite Disaster Recovery• E-Business Suite MAA Demonstration • New Database Features and E-Business Suite

<Insert Picture Here>

Maximum Availability Architecture (MAA)

Maximum Availability Architecture (MAA)Maximum Availability = Unbreakable Architecture + Best Practices

Oracle's best practices blueprint based on proven Oracle high availability technologies and recommendations

Technology + Configuration + Operational Practices

Applications, Enterprise Manager, Application Server, Collaboration Suite and Database

• Constantly validated and enhanced as new products and features become available

• Focused on reducing unplanned and planned downtime

• Focused on making our customers successful• Papers published to the Oracle Technology Network (OTN)

• http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

Presentation Focus

• This presentation is on Oracle E-Business Suite Release 12 MAA

• Please see the MAA home page for detailed papers covering Oracle E-Business Suite 11i and R12 MAA• http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

<Insert Picture Here>

E-Business Suite MAA

E-Business Suite MAATarget Architecture

Primary Site

OracleRAC and

ASM

Disaster Recovery Site

Oracle Data Guard

OracleDatabase

OracleStandby

Database

Database Tier OracleRAC and

ASM

Application Tier

HA Storage

E-Business Suite MAA Oracle Database MAA

Primary Site Disaster Recovery Site

Real Application Clusters & Clusterware

Fault TolerantServer Scale -Out

Data GuardFully Active Failover Replica

FlashbackCorrect Errors by Moving Back in Time

Automatic Storage Management

Fault TolerantStorage Scale -Out

Recovery Manager & Oracle Secure BackupLow Cost High Performance Data Protection and Archival

Database Servers Database Servers

Storage Storage

E-Business Suite MAATransitioning to RAC and ASM

• Migrate Oracle Applications Release 12 to a Real Application Clusters (RAC) environment running Oracle database server 10g Release 2 with Automatic Storage Management (ASM)• https://metalink.oracle.com/metalink/plsql/ml2_documents.showDoc

ument?p_database_id=NOT&p_id=388577.1

OracleDatabase

OracleRAC and

ASM

OracleDatabase

E-Business Suite MAA Application HA Features

Hardware Load BalancersRedundant Configuration

Application Tier

Database Tier

Multiple Web ServersLoad Balanced

Parallel Concurrent ProcessingFault tolerant batch processing

Database or Application Tier

Multiple Forms ServersLoad Balanced

E-Business Suite MAA Load Balancing Configuration

• Load balancing distributes client requests across multiple application tier nodes, providing additional scalability and fault tolerance

• Implementation requires adding and configuring application tier nodes and configuring the load balancer

• Metalink note 380489.1 describes the application configuration options in detail:

• https://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=380489.1

• Load balancer configuration is vendor specific• Persistence through cookies is required

E-Business Suite MAA Parallel Concurrent Processing

• Parallel concurrent processing allows concurrent managers to execute on multiple server nodes at the same time, providing scalability and fault tolerance

• The feature is detailed in the Oracle Applications System Administrator's Guide, Release 12:• http://download.oracle.com/docs/cd/B34956_01/current/acrobat/120

sacg.pdf

• Metalink note 388577.1 explains additional considerations when using PCP with RAC:• https://metalink.oracle.com/metalink/plsql/ml2_documents.showDoc

ument?p_database_id=NOT&p_id=388577.1

E-Business Suite MAATarget Architecture

Primary Site

OracleRAC and

ASM

Disaster Recovery Site

Oracle Data Guard

OracleDatabase

OracleStandby

Database

Database Tier OracleRAC and

ASM

Application Tier

HA Storage

<Insert Picture Here>

E-Business Suite Disaster Recovery

E-Business Suite Disaster Recovery

• In this section we describe how to setup and operate a E-Business Suite disaster recovery site

• Detailed papers describing this process for 11i and R12 are published on the MAA home page• http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

• The process is based on Rapid Clone; see Metalink note 559518.1 for how to clone an EBS environment:• https://metalink.oracle.com/metalink/plsql/ml2_documents.showDoc

ument?p_database_id=NOT&p_id=559518.1

E-Business Suite Disaster RecoveryCreating the Standby SiteStandby Site Creation Process

Stan

dby

Apps

Nod

es

Stan

dby

Dat

abas

e N

odes

Prim

ary

Apps

N

ode

Prim

ary

D

atab

ase

Nod

es

Establish Standby Hardware and

System Software

Establish Oracle Clusterware and

ASM

Clone Database Software

Prepare New Database Instance

Establish Standby Database

Prepare Apps Software for

DR Database

Backup Database

Clone Apps Software

Establish Standby Hardware and

System Software

Prepare for Data Guard Operations

E-Business Suite Disaster RecoveryCreating the Standby Site

• Establish Standby Hardware and System Software• Install the standby site middle tier and database hardware,

networks, etc.• Think carefully about the performance and HA

requirements - best to mimic production?• Plan also for all required external services

• Install System Software

• Establish Oracle Clusterware• Required for ASM and RAC

• Establish ASM

E-Business Suite Disaster RecoveryPreparation on the Primary

• Prepare for Data Guard operation• Enable archive log mode• Enable forced logging• Create database password files• Configure SQL*NET communication between sites• Set database parameters for Data Guard• Create standby redo logs

E-Business Suite Disaster RecoveryClone and Configure Database Software

• Clone database software and prepare the standby instances• Run the E-Business Suite pre-clone utility on a primary database

node• Copy the database home to the standby database servers• [RAC Only] Run adclonectx.pl on each standby database server

• Edit the context file to adjust the cluster_database_instances, instance_number, instance_thread, and undotablespaceparameter values appropriately

• Run adcfgclone.pl on each standby database server• Configure Data Guard database parameters on standby site• Configure SQL*NET Communication on standby site• [RAC Only] Restart the listeners

E-Business Suite Disaster RecoveryClone and Configure Application Software

• Clone the application tier to standby• Run the pre-clone utility on a primary applications tier node• Copy the software to the standby• Run adcfgclone.pl on each standby middle tier server

• Use atTechStack option and run AutoConfig with “run=INSTE8” option as there is no database connection

• [RAC Only] Edit the context file to point Tools OH TWO_TASK, iAS OH TWO_TASK, and Apps JDBC Connect Alias to the appropriate load balancing services

• (Optional) Establish log and out file synchronization• Use your favorite tool for this, for example rsync

E-Business Suite Disaster RecoveryCreate the Physical Standby Database

• Establish the standby database• Back up the production database including archive logs, and the

production control file for standby• Restore the database to the DR site using (one of) the configured

instance(s)• Start managed recovery• Check out MAA papers for tuning best practices:

• http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_RecoveryBestPractices.pdf

• http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_DataGuardNetworkBestPractices.pdf

• [RAC Only] Register the standby database with Oracle Clusterware

E-Business Suite Disaster Recovery Operational Procedures

• Switchover• Be sure you are up to date with redo apply

• Have procedures in place to also be up to date with file system synchronization

• Shut down the app and all but one database instance on each site

• Prepare the primary database for standby, switch the standby to primary, open, and start other instances

• Remove the applications topology from the database • Run AutoConfig on database tier (twice for RAC) then middle

tiers to configure the application for DR operation• Start the Apps• Convert the old primary database to a Data Guard standby

E-Business Suite Disaster RecoveryOperational Procedures

• Failover (assuming you lost the primary database)• Convert the standby to primary operation, open, and start

other instances• Remove the application topology from the database• Run AutoConfig on database tier (twice for RAC) and then on

middle tier to configure the application for the DR site• Start the Apps• If the primary site becomes available, flash the old primary

database back to before the failure and convert it to a physical standby

• If the primary site is permanently lost, create a new standby environment

E-Business Suite Disaster RecoveryOperational Procedures

• Standby testing using flashback• Cancel media recovery on the standby database• Create a guaranteed restore point on the standby database• Switch the redo log on the primary database• Activate the standby database and configure EBS

• Take care with external interfaces such as Workflow -disable them before starting EBS

• [Perform testing at the standby site]• Flash the database back to the restore point

• Remove file system changes?• Drop the restore point• Convert the database back to a physical standby and resume

standby operation

E-Business Suite Disaster RecoveryAutomating Switchover and Failover

• Configure Data Guard Broker to automate Data Guard operation and the database failover and switchover steps

• Configure Data Guard Fast Start Failover to automatically detect a database failure and initiate failover

• Create a script to automate the application configuration and startup

• Create DB_ROLE_CHANGE trigger that will execute the script on switchover and failover

<Insert Picture Here>

E-Business Suite MAA Demonstration

E-Business Suite MAA DemonstrationHardware Vendor Partners

• Part of MAA means proving and testing our best practices

• We work closely with our hardware vendor partners to check that our joint solutions work well together

E-Business Suite MAA TestingDemo Configuration

• The demonstrations were developed and executed using HP Systems hardware and software and F5 load balancers

• Hardware• F5 BIG-IP Local Traffic Manager v9 - 3400 Series• HP DL145 G2 running HP Load Runner v8.1 for load generation• HP Integrity rx2620 servers for the application tier • HP Integrity rx4640 servers for the database tier • HP StorageWorks EVA4000 for database storage

• Software• HP-UX 11i v3 (11.31) for application and database Servers• HP Serviceguard extensions for RAC (SGeRAC) Version A.11.18• Oracle Enterprise Manager Grid Control was used to monitor the

systems

E-Business Suite MAA DemonstrationLab Configuration

Disaster Recovery Site

HP EVA 4000

F5 BIG-IP LTM 3400

Primary Site

OracleRAC and

ASM

Application Tier

HA Storage

Oracle Data Guard

OracleDatabase

Database Tier

Load RunnerHP DL145 G2

HP EVA 4000

F5 BIG-IP LTM 3400

OracleRAC and

ASM

OracleDatabase

HP RX2620 HP RX2620 HP RX2620

HP RX4640 HP RX4640 HP RX4640 HP RX4640

E-Business Suite MAA TestingCurrent Test Configuration

• Testing for R12 was done with HP Systems hardware and software and F5 load balancers

• Hardware• F5 BIG-IP Local Traffic Manager v9 - 3400 Series• HP DL145 G2 running HP Load Runner v8.1 for load

generation• HP BL 460c servers for the application and database tiers• HP StorageWorks EVA4400 for database storage

• Software• Oracle Enterprise Linux 4.6 for application and database

Servers• Oracle Enterprise Manager Grid Control was used to monitor

the systems

E-Business Suite MAA TestingSecondary Site

HP EVA 4400

F5 BIG-IP LTM 3400

Primary Site

OracleRAC and

ASM

Application Tier

HA Storage

Oracle Data Guard

OracleDatabase

Database Tier

Load RunnerHP DL145 G2

HP EVA 4400

F5 BIG-IP LTM 3400

OracleRAC and

ASM

OracleDatabase

HP BL460c HP BL460c HP BL460c

HP BL460c HP BL460c HP BL460c HP BL460c

HP BL460c

Current Configuration

<Insert Picture Here>

E-Business Suite MAA Demonstration

<Insert Picture Here>

New Features and the

E-Business Suite

New Database Features

• Secure File System (SFS) entering beta• Streams Extended Datatype Support (EDS) being

made available this week

Secure File System (SFS)

• File system in the database• Conceptually like NFS client

• User mode filesystem, no changes to Linux kernel

• Standards-based POSIX compatible• Transparent to file system applications• Mount SFS at a Linux mount point with a normal Unix path

SFS – Potential Opportunities forE-Business Suite

• Concurrent Manager log and out files • Shared NFS mount for all PCP processes• Keep in EBS database. Failover to disaster recovery site will

be at virtually the same point in time as the OLTP transactions

• Application software – shared NFS install for local efficiency, DR consistency• Address single point of failure

Extended Datatype Support (EDS)

• Adds Streams support for:• Object columns with simple or nested objects• Object tables• XMLType (CLOB, Binary, ObjectRelational)• Varrays• Partial Spatial types (SDO_GEOMETRY)

• The EDS package generates workarounds for moving the above data

Extended Datatype Support –Generated Workaround

• Streams configuration• Pre-instantiation setup• Post-instantiation setup• Clean-up

• Workaround to move unsupported data• Log table for source and target databases, capable of holding

all the fields in the original table, using only supported data types

• Trigger on base table on source database, to write inserts, updates, deletes to log table

• DML handler on log table on target database, to replicate original DML on the base table

EDS – Potential Opportunities forE-Business Suite

• Reporting database• … if enough columns are now supported for the customer’s

reporting requirements• Still up to the customer to determine viability / mechanics

• Database-level maintenance is being researched:• Rolling database upgrade• Cross-platform migration

HA Sessions, Labs, Demos From Oracle Development

Mon, Sep 22• 2:30 pm - Database 11g: Next-Gen HA, Moscone South 103

Tue, Sep 23• 9:00 am - Active-Active Data Centers, Moscone South 103• 11:30 am - Sharding with Oracle, Moscone South 302• 11:30 am - HA with Oracle VM, Moscone West 3024• 1:00 pm - Active Data Guard, Moscone South 104

Wed, Sep 24• 9:00 am - Fusion Middleware Grid HA, Marriott Nob Hill AB• 11:30 am - RMAN Best Practices, Moscone South 103• 5:00 pm - Data Guard & Real Application Testing, Moscone 102• 5:00 pm - EM in Secure MAA, Moscone West 2001

Wed, Sep 24 (contd.)• 5:00 pm - E-Business Suite HA, Moscone West 2002/04

Thu, Sep 25• 9:00 am - Oracle Secure Backup, Moscone South 102• 10:30 am - Streams Replication, Moscone South 102• 12:00 pm - Rolling Database Upgrades, Moscone South 103• 1:30 pm - Streams Performance, Moscone South 102• 3:00 pm - Oracle Grid Computing, Moscone South 303• 3:00 pm - E-Business Suite R12 MAA, Moscone West 2007• 3:00 pm - Siebel MAA, Moscone South 308• 3:00 pm - Fusion SOA HA & Scalability, Marriott Salon 14/15

Hands On Labs - Thu, Sep 25• 10:30 - 11:30 am, 12:00 - 1:00 pm - Active Data

Guard, Marriott Golden Gate A3

DEMOgrounds, Mon-Thu• Active Data Guard, Streams, Oracle Secure

Backup, RMAN/Flashback, MAA

For More Information

search.oracle.com

ororacle.com

Maximum availability architecture