Expert Oracle RAC 12c

Syed Jaffar Hussain

Tariq Farooq

Riyaj Shamsudeen

Kai Yu

Expert Oracle RAC 12c

Copyright 2013 by Syed Jaffar Hussain, Tariq Farooq, Riyaj Shamsudeen, and Kai Yu

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publishers location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law.

ISBN-13 (pbk): 978-1-4302-5044-9

ISBN-13 (electronic): 978-1-4302-5045-6

Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

President and Publisher: Paul ManningAcquisitions Editor: Jonathan GennickDevelopmental Editor: Tom WelshTechnical Reviewers: Arup Nanda and Sandesh RaoEditorial Board: Steve Anglin, Mark Beckner, Ewan Buckingham, Gary Cornell, Louise Corrigan, Morgan Ertel,

Jonathan Gennick, Jonathan Hassell, Robert Hutchinson, Michelle Lowman, James Markham, Matthew Moodie, Jeff Olson, Jeffrey Pepper, Douglas Pundick, Ben Renow-Clarke, Dominic Shakeshaft, Gwenan Spearing, Matt Wade, Steve Weiss, Tom Welsh

Coordinating Editor: Katie SullivanCopy Editor: Brendan FrostCompositor: SPi GlobalIndexer: SPi GlobalArtist: SPi GlobalCover Designer: Anna Ishchenko

Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.

For information on translations, please e-mail rights@apress.com, or visit www.apress.com.

Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk SaleseBook Licensing web page at www.apress.com/bulk-sales.

Any source code or other supplementary materials referenced by the author in this text is available to readers at www.apress.com/9781430250449. For detailed information about how to locate your books source code, go to www.apress.com/source-code.

orders-ny@springer-sbm.comwww.springeronline.comrights@apress.comwww.apress.comwww.apress.com/bulk-saleswww.apress.comwww.apress.com/source-code/

iii

Contents at a Glance

About the Authors xvii

About the Technical Reviewers xix

Acknowledgments xxi

Chapter 1: Overview of Oracle RAC 1

Chapter 2: Clusterware Stack Management and Troubleshooting 29

Chapter 3: RAC Operational Practices 69

Chapter 4: New Features in RAC 12c 97

Chapter 5: Storage and ASM Practices 123

Chapter 6: Application Design Issues 165

Chapter 7: Managing and Optimizing a Complex RAC Environment 181

Chapter 8: Backup and Recovery in RAC 217

Chapter 9: Network Practices 243

Chapter 10: RAC Database Optimization 285

Chapter 11: Locks and Deadlocks 321

Chapter 12: Parallel Query in RAC 353

Chapter 13: Clusterware and Database Upgrades 381

Chapter 14: RAC One Node 411

Index 431

v

Contents

About the Authors xvii

About the Technical Reviewers xix

Acknowledgments xxi

Chapter 1: Overview of Oracle RAC 1

High Availability and Scalability 1

What Is High Availability? 2

Database Scalability 3

Oracle RAC 4

Database Clustering Architecture 4

RAC Architecture 5

Hardware Requirements for RAC 6

RAC Components 8

Oracle RAC: Cache Fusion 11

RAC Background Processes 14

Achieving the Benefits of Oracle RAC 16

High AvailabilityAgainst Unplanned Downtime 16

High Availability Against Planned Downtime 19

Oracle RAC One Node to Achieve HA 21

RAC Scalability 21

Consolidating Database Services with Oracle RAC 23

Considerations for Deploying RAC 25

Cost of Ownership 25

High Availability Considerations 26

Contents

vi

Scalability Considerations 26

RAC or Not 27

Summary 28

Chapter 2: Clusterware Stack Management and Troubleshooting 29

Clusterware 12cR1 and Its Components 30

Storage Components of Oracle Clusterware 30

Clusterware Software Stack 31

Clusterware Startup Sequence 33

ASM and Clusterware: Which One is Started First? 35

Clusterware Management 36

Clusterware Management Tools and Utilities 36

Start Up and Stop Clusterware 37

Managing Oracle Clusterware 38

Managing OCR and the Voting Disk 40

Managing CRS Resources 42

Adding and Removing Cluster Nodes 43

Troubleshooting common Clusterware Stack Start-Up Failures 45

Diagnose, Debug, Trace Clusterware and RAC Issues 49

Debugging Clusterware Components and Resources 50

Grid Infrastructure Component Directory Structure 52

Oracle Clusterware Troubleshooting - Tools and Utilities 55

CHM 60

Summary 67

Chapter 3: RAC Operational Practices 69

Workload Management 69

Services70

Service Metrics 71

Load Balancing Goals 73

Runtime Failover 75

Service in Second Network 75

Guidelines for Services 76

Contents

vii

SCAN and SCAN Listeners 76

SCAN Listener in Second Network (12c) 80

Guidelines for SCAN Listeners 81

Global Database Services (12c) 81

Failover in RAC 82

TAF 82

Fast Connection Failover 84

WebLogic Active GridLink 84

Transaction Guard (12c) 85

Application Continuity (12c) 85

Policy-Managed Databases 86

Temporary Tablespaces 86

Massive Data Changes 88

Performance Metrics Collection 88

Parameter File Management 88

Password File Management 89

Managing Databases and Instances 90

Managing VIPs, Listeners 92

Miscellaneous Topics 93

Process Priority 93

Memory Starvation 94

SGA size 94

Filesystem Caching 94

Summary 95

Chapter 4: New Features in RAC 12c 97

Oracle Flex Clusters 98

Oracle Flex Cluster Architecture 98

Scalability and Availability of Flex Clusters 99

Configuring Flex Clusters 101

Contents

viii

Flex ASM Architecture 105

Oracle Flex ASM Architecture 105

Flex ASM and Flex Clusters 107

Configuring Flex ASM 107

ASM Clients and Relocating 109

New ASM Storage Limits 109

Replacing ASM Disk in Disk Group 110

Scrubbing ASM Disk Groups and Files 110

Reading Data Evenly in ASM Disk Group 110

Measure and Tune Rebalance Operation 110

What-If Command Evaluation 111

PDBs on Oracle RAC 112

PDB Architecture Overview 113

PDBs on Oracle RAC 116

12cR1: Miscellaneous New Features for RAC 119

Public Networks for RAC: IPv6 Support Added 120

Global Data Services 120

Online Resource Attribute Modification 120

RAC 12cR1: Policy-Based Management and Administration 120

ASM Disk Group: Shared ASM Password File 120

Valid Node Checking: Restricting Service Registration 120

12cR1: Shared GNS 121

RAC 12cR1: Restricting Service Registration 121

Oracle ASM, ACFS, and ADVM: Improvements and New Features 121

NFS High Availability 121

12cR1: CHM Enhancements 121

Windows: Support for Oracle Home User 121

OUI: Enhancements and Improvements 121

12cR1: Installations/UpgradesRunning Scripts Automatically 122

12cR1: Introducing Application Continuity 122

Transaction Idempotence and Java Transaction Guard 122

Contents

ix

Deprecated and Desupported Features 122

Summary 122

Chapter 5: Storage and ASM Practices 123

Storage Architecture and Configuration for Oracle RAC 124

Storage Architecture and I/O for RAC 125

RAID Configuration 127

Storage Protocols 129

Multipath Device Configuration 132

Set Ownership of the Devices 134

ASM 135

ASM Instance 136

ASM Storage Structure 143

Manage ASM Using SQL Command and V$ASM Views 151

Store OCR and Voting Disk in ASM 151

Choose ASM for OCR and Voting Disk at GI Installation 152

Move OCR and Voting Disk Files to a New ASM Diskgroup 155

ACFS 157

Create ACFS 159

Create ACFS for Oracle RAC Home with ASMCA 161

Summary 164

Chapter 6: Application Design Issues 165

Localized Inserts 165

Excessive TRUNCATE or DROP Statements 168

Sequence Cache 169

Freelists and ASSM 172

Excessive Commits 173

Long Pending Transactions 174

Localized Access 174

Small Table Updates 175

Contents

x

Index Design 176

Inefficient Execution Plan 176

Excessive Parallel Scans 177

Full Table Scans 177

Application Affinity 178

Pipes178

Application Change Deployment 178

Summary 179

Chapter 7: Managing and Optimizing a Complex RAC Environment 181

Shared vs Non-Shared Oracle Homes 182

Server Pools 183

Types of Server Pools 184

System-Defined Server Pools 184

User-Defined Server Pools 184

Creating and Managing Server Pools 185

Planning and Designing RAC Databases 188

Policy-Managed Databases 188

Instance Caging 191

Small- vs Large-Scale Cluster Setups 193

Split-Brain Scenarios and How to Avoid Them 194

Understanding, Debugging, and Preventing Node Evictions 196

Node EvictionsSynopsis and Overview 196

Extended Distance (Stretch) ClustersSynopsis, Overview, and Best Practices 199

Extended Distance (Stretch) Clusters: Setup/Configuration Best Practices 200

Setup and ConfigurationLearning the New Way of Things 201

OUI 201

Oracle Enterprise Manager Cloud Control 12c 204

RAC Installation and SetupConsiderations and Tips for OS Families: Linux, Solaris, and Windows 206

Contents

xi

RAC Database Performance Tuning: A Quick n Easy Approach 208

The 3 As of Performance Tuning 208

Summary 215

Chapter 8: Backup and Recovery in RAC 217

RMAN Synopsis 217

Media Management Layer 218

Online Backup and Recovery Prerequisites 219

Non-RAC vs RAC Database 221

Shared Location for Redo and Archive Logs 221

Snapshot Control File Configuration 222

Multiple Channels Configuration for RAC 223

Parallelism in RAC 226

Instance/Crash Recovery in RAC 226

Real-World Examples 230

Manage RMAN with OEM Cloud Control 12c 233

OCR recovery 239

Summary 241

Chapter 9: Network Practices 243

Types of Network 243

Network Layers 244

Protocols 246

VIPs 250

Subnetting 250

Cluster Interconnect 252

Jumbo Frames 254

Load Balancing and Failover 259

Kernel Parameters261

Network Measurement Tools 262

GC Lost Block Issue 266

Contents

xii

Configuring Network for Oracle RAC and Clusterware 268

Establishing IP Address and Name Resolution 271

Network Specification in Grid Infrastructure Installation 274

Network Configuration in Clusterware 277

Network Failover 283

Second Network 284

Summary 284

Chapter 10: RAC Database Optimization 285

Introduction to Cache Fusion285

Cache Fusion Processing 286

GRD 288

BL Resources and Locks 288

Performance Analysis 292

Analysis of the Receiving Side 292

RAC Wait Events 299

GC Current Block 2-Way/3-Way 299

GC CR Block 2-Way/3-Way 301

GC CR Grant 2-Way/Gc Current Grant 2-Way 302

GC CR Block Busy/GC Current Block Busy 302

GC CR Block Congested/GC Current Block Congested 302

Placeholder Wait Events 303

Sending-Side Analysis 303

Block Types Served 306

GCS Log Flush Sync 307

Defending LMS Process 308

GC Buffer Busy Acquire/Release 308

Unique Indexes 310

Table Blocks 311

DRM 313

Contents

xiii

Overview of DRM Processing 314

DRM Stages 316

GRD Freeze 316

Parameters 317

Changes in 12c 317

DRM and Undo 317

Troubleshooting DRM 318

AWR Reports and ADDM 318

ASH Reports 319

Summary 320

Chapter 11: Locks and Deadlocks 321

Resources and Locks 321

SGA Memory Allocation 323

Resource Types 325

Lock Modes 327

Lock-Related Views 327

Pluggable Databases (12c) 328

Troubleshooting Locking Contention 328

Enqueue Contention 331

TX Enqueue Contention 331

TM Enqueue Contention 334

HW Enqueue Contention 335

DFS Lock Handle 336

SV Resources 337

CI Resources 340

DFS Lock Handle Summary 342

Library Cache Locks/Pins 342

Troubleshooting Library Cache Lock Contention 344

Enqueue Statistics 346

v$wait_chains 346

Contents

xiv

Hanganalyze 347

Deadlocks 348

LMD Trace File Analysis 349

Summary 352

Chapter 12: Parallel Query in RAC 353

Overview 353

PX Execution in RAC 357

Placement of PX Servers 358

Measuring PX Traffic 361

PX and Cache Fusion 363

PEMS 364

Parallelism Features and RAC 364

Debugging PX Execution 375

Index Creation in RAC 376

Parallel DML in RAC 377

Concurrent Union Processing (12c) 378

Partition-Wise Join 379

Summary 380

Chapter 13: Clusterware and Database Upgrades 381

Configuration 381

Pre-Upgrade Checklist 383

Initiating an Oracle Clusterware Upgrade 385

The Importance of the Rootupgradesh Script 396

Post-Upgrade Tasks398

Clusterware Downgrade 399

Database Upgrade 402

Deploying Manual Database Upgrade 403

Post-Database Upgrade Steps 405

Database Upgrade Using the DBUA 406

Contents

xv

DBUA Advantages 409

Database Downgrade 409

Summary 410

Chapter 14: RAC One Node 411

The Big Picture 411

Upgrading to 11202 or Higher 412

Deploying RAC One Node Binaries 412

Deploying a RAC One Node Database 415

Satisfying Prerequisites 415

Initiating DBCAs Creation Process 416

Parameters Specific to RAC One Node 418

Managing RAC One Node Database 419

Verifying Configuration Details 419

Verifying the Online Relocation Status 420

Stop and Start the Database 420

Performing Online Database Relocation 420

Handling Unplanned Node and Cluster Reboots 423

Converting Between RAC One Node and Standard RAC 423

Scaling Up to Standard RAC 424

Scaling Down to RAC One Node 425

Managing RAC One Node with Cloud Control 12c 425

Database Relocation with Cloud Control 12c 425

Third-Party Cold Failover vs RAC One Node 428

Summary 429

Index 431

xvii

About the Authors

Syed Jaffar Hussain has over 20 years of I.T. experience that includes more than 14 years of production Oracle database administration. Oracle has honored him with the prestigious Oracle ACE Director role and named him DBA of the Year for 2011, both for his excellent knowledge and for contributions to the Oracle community. He is an Oracle Certified Master (OCM) for Oracle Database 10ga status granted only after passing extensive challenges in a hands-on environment. He is also an Oracle Database 10g RAC Certified Expert. Syed Jaffar is coauthor of Oracle 11g R1/R2 Real Application Clusters Essentials (Packt Publishing, 2011), and he blogs regularly at http://jaffardba.blogspot.com. He is a frequent speaker, has presented at various Oracle conferences, including Oracle OpenWorld, and has delivered many technical Oracle-oriented webinars. Also, he actively participates in most of the social networking sites: you can follow him on twitter/facebook/linkedin on sjaffarhussain id, and he can be reached at sjaffarhussain@gmail.com.

Tariq Farooq is an Oracle technologist/architect/problem-solver and has been working with various Oracle technologies for 20+ years in very complex environments at some of the worlds largest organizations. He is the founding President of the IOUG Virtualization & Cloud Computing Special Interest Group. He is an active community leader/organizer, speaker, author, forum contributor, and blogger. He is the founder of www.BrainSurface.com, a social networking & IT collaboration site with thousands of signed-up users from the various Oracle communities. Tariq founded, organized, and chaired various conferences, including VirtaThon, the largest online-only conference for the various Oracle domains; the CloudaThon & RACaThon series of conferences; and the first-ever Oracle-centric conference at the Massachusetts Institute of Technology (MIT) in 2011. He was the founder and anchor/show host of the VirtaThon Internet Radio series program. Tariq is an Oracle RAC Certified Expert, holds a total of 14 professional Oracle certifications, and is the author of 100+ articles, whitepapers,

and other publications. Tariq has presented and spoken at almost every major Oracle conference/event all over the world, including Oracle OpenWorld, COLLABORATE, BrainSurface, VirtaThon, IOUG, OOUG, OUGLC, TCOUG, UKOUG, The OTNExpert+ Conference, and others. Tariq has been awarded the Oracle ACE and ACE Director awards from 2010 to 2013.

http://jaffardba.blogspot.comsjaffarhussain@gmail.comwww.BrainSurface.com

About the Authors

xviii

Riyaj Shamsudeen is an industry-recognized RAC expert, database internals specialist, and performance tuning specialist with 20 years of experience in implementing, using, and tuning RAC and Oracle products. He is an Oracle ACE Director and proud member of the OakTable network. Riyaj has coauthored four books about Oracle Database and performance. He is an active blogger (at http://orainternals.wordpress.com), President of OraInternals, and frequent international speaker in major conferences such as UKOUG, HOTSOS, OpenWorld, and RMOUG. Social: www.linkedin.com/in/riyajshamsudeen; @riyajshamsudeen.

Kai Yu is a Senior Principal Engineer and technologist in Dells Oracle Solutions Engineering Lab specializing in Oracle RAC, Oracle Virtualization, and Cloud. With 18+ years of experience working on Oracle technology, he has implemented and managed many large, mission-critical Oracle RAC databases, including those in his tenure with an IT organization having more than 2,000 RAC databases. Kai is a well-known author of technical articles and a frequent international speaker at Oracle conferences in 16 countries such as OpenWorld, Collaborate, UKOUG, EMEA OUG Harmony, Norway OUG, OTN Asia/Pacific tour, and Latin America tours. He has also keynoted at Oracle Architect Day.

Kai has served IOUGs Oracle RAC SIG as president, and in two chair positions. He has served the IOUG collaborate conference committee, as well as managing IOUGs RAC boot camp, HA boot camp, and Enterprise Manager 12c deep dive. He was awarded the Oracle ACE Director title in 2010, given the Oracle ACE Spotlight in 2011 by Oracle Technology Network (OTN), and named the 2011 OAUG Innovator of Year Award by the Oracle Applications User Group (OAUG). In 2012,

Oracle Magazine awarded him the Oracle Excellence Award: Technologist of the Year: Cloud Architect. Kai has been active in sharing his Oracle knowledge on his Oracle blog at http://kyuoracleblog.wordpress.com/.

http://orainternals.wordpress.comwww.linkedin.com/in/riyajshamsudeen@riyajshamsudeenhttp://kyuoracleblog.wordpress.com/

xix

About the Technical Reviewers

Arup Nanda (Email: arup@proligence.com; Twitter: @arupnanda; LinkedIn: linkedin.com/in/arupnanda/) has been an Oracle DBA for the last 18 years, spanning all aspects of the job from modeling to performance tuning. He specializes in RAC, Exadata, and High Availability solutions. He has authored four books on Oracle technology, written 500+ published articles, and presented almost 300 training sessions in 22 countries. He is an Oracle Certified Professional, Oracle ACE Director, member of the OakTable Network, member of the Board of Directors of Exadata SIG, and member of the Editorial Board for SELECT Journal (the IOUG publication). Acknowledging his accomplishments and community involvement, Oracle awarded him DBA of the Year in 2003 and Enterprise Architect of the Year in 2012. He blogs at http://arup.blogspot.com.

Sandesh Rao is a Senior Director running the RAC Assurance Team within RAC Development at Oracle Corporation specializing in performance tuning, high availability, disaster recovery, and architecting cloud-based solutions using the Oracle Stack. With more than 14 years of experience working in the HA space and having worked on several versions of Oracle with different application stacks, he is a recognized expert in RAC, database internals, and solving big datarelated problems. Most of his work involves working with customers in the implementation of projects in financial, retailing, scientific, insurance, biotech, and the tech space. His current position involves running a team that develops best practices for the Oracle Grid Infrastructure 12c including products like RAC (Real Application Clusters) and Storage (ASM, ACFS). You can find more details about him and his work at www.linkedin.com/pub/sandesh-rao/2/956/1b7.

http://arup@proligence.comhttp"//@arupnandahttp://linkedin.com/in/arupnanda/http://arup.blogspot.comwww.linkedin.com/pub/sandesh-rao/2/956/1b7

xxi

Acknowledgments

I would like to dedicate this book to my parents (Mrs. and Mr. Saifulla), my wife Ayesha, my three little champs (Ashfaq, Arfan, and Aahil), and the entire Oracle community.

First and foremost, I would like to thank the Almighty for everything and giving me such a wonderful life. Also, my parents for teaching me all the good things in my life and making me what I am today. I owe a very big thank-you to my wife and kids for sacrificing their family time and letting me concentrate on this assignment, and encouraging me all through this phase. Beyond a doubt, this project wouldnt have been possible without the tremendous moral support and encouragement from my wife, friends, and colleagues. Thank you everyone once again.

I would like to take this opportunity to thank my management (Khalid Al-Qathany, Hussain Mobarak AlKalifah, and Majed Saleh AlShuaibi), my dearest friends (Khusro Khan, Mohammed Farooqui, Naresh Kumar, Shaukat Ali, Ahmed, Mohammed Abdul Kaleem, Mohammed Arifuddin Ghori, Haris Afsar Ahmed, Sachin, Ibrahim Ali, Chand Basha, Mohammed Anees, Asgar Ali, Gaffar Baig, Hakeem, Mohsin), my fellow colleagues, well-wishers, supporters, nears and dears for their immense support and constant encouragement.

I am also very thankful from the bottom of my heart to the official technical reviewers (Sandesh Rao and Arup Nanda) for taking some time out from their busy schedule to review our book and for providing their valuable inputs. I cant conclude the list without mentioning the efforts by the official members of Apress, who put everything into this project. This is for every individual who was involved on this project on behalf of Apress: You people just rock and have been of great help to us during the book-writing journey; looking forward to working with the team in future; thank you everyone. A very special thanks to Jonathan Gennick for having confidence in us and giving us the opportunity to write for Apress Publications.

I would like to extend my sincere thanks to my unofficial technical reviewers, in no particular order, Jeff Smith, Bernhard de Cock Buning, Asad Khan, Yury Veli Kanov, Khusro Khan, Shautkat, Aman Sharma, and Inam Ullah Bukhary, who helped me a lot, in terms of reviewing my part of chapters, providing their inputs, plus correcting my grammar.

I would like to conclude this by thanking my fellow coauthors, Riyaj Shamsudeen, Kai Yu, and Tariq Farooq, for the immense contribution, dedication, and hard work that they have displayed over the past year on this book. The book wouldnt have been feasible without your help/support/assistance/contribution. I really enjoyed working with you guys and am looking forward to other opportunities to work with you again in the coming times.

Syed Jaffar Hussian

I would like to take this opportunity to express my endless gratitude and thanks to the Almighty, my parents (Mr. and Mrs. Abdullah Farooq), my wife (Ambreen), especially my kids (Sumaiya, Hafsa, Fatima, and Muhammad-Talha), and everyone else around me, both at home and at work, for continuing to stand behind me while I spent the better part of a year working on this project.

Authoring a book is a painstakingly lengthy and laborious process; this project would not have been possible without the help and guidance of the Apress team: many, many appreciative thanks to them and the book review team, particularly Jonathan Gennick. And finally, kudos and thanks to my awesome friends and coauthors (Syed, Kai, and Riyaj) for the amazing team effort and making this book happen.

Tariq Farooq

xxii

ACknowledgments

I dedicate this book to you, the readers of the book. I thank my familymy wife Nisha and kids Shibi, Irfan, and Imranfor their great support and encouragement. Especially, I should thank my youngest son, Imran, for his understanding while Dad is busy writing the book (as he fondly quotes the book in the air!). My sincere thanks to Apress staff.

Many thanks to my current and past clients (you know who you are; I am thanking you without naming you) for giving me an opportunity to probe complex issues and learn so much more about RAC; Oracle-L listers and peers for asking me thought-provoking questions; and OakTable members for helping me to go to the next level when I am stuck on issues. My sincere thanks to the ContentConcepts guys (http://contentconcepts.in) for the preliminary editing of my chapters.

Riyaj Shamsudeen

I want to thank all the people who have assisted this book, especially the Apress team of Acquisitions Editor Jonathan Gennick, Developmental Editor Tom Welsh, and Copy Editor Brendan Frost for their efforts, patience, and skills in transforming the technical writing into a finished book. I definitely would like to thank the technical reviewers, Arup Nanda and Sandesh Rao, for their time and effort in checking the technical content of the entire book as well as providing their very valuable comments and advice for improving the quality of this book.

I would like to dedicate this book to my parents, who encouraged me to pursue my education and career in computer technology. I would like to thank my wife Jin and my daughter Jessica for their consistent support and providing me the time to work on this book at night and on weekends.

I would like to thank my management at DellDavid Mar, Ibrahim Fashho, Dr. Reza Rooholaminifor their long-time support and encouragement and my colleagues in the Dell Oracle Solution Engineering team for their assistance and many great discussions on Oracle RAC, Red Hat Linux, and Oracle Linux and storage. I also want to specially thank my mentors in Oracle who really inspired me to work with the Oracle community: Dr. Nadia Bendjedou, a senior director from Oracle, and Erik Peterson, General Manager of Oracle Mexico Development Center. Last but certainly not least I would like to thank my coauthors Syed, Tariq, and Riyaj for the amazing teamwork. In particular, I want to thank Syed for his great leadership on the team and his confidence and encouragement.

Kai Yu

Contents at a GlanceContentsAbout the AuthorsAbout the Technical ReviewersAcknowledgments