PB OracleEnterpriseManager10gGridControlHandbookOracle Enterprise Manager 10g Grid Control HandbookWernerDeGruyterMatthewHartDanielNguyenNewYork Chicago SanFrancisco Lisbon London Madrid MexicoCity Milan NewDelhi SanJuan Seoul Singapore Sydney TorontoCopyright 2010 by The McGraw-Hill Companies, Inc. (Publisher). All rights reserved. Except as permitted under the United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher.ISBN: 978-0-07-163423-6MHID: 0-07-163423-1 The material in this eBook also appears in the print version of this title: ISBN: 978-0-07-163422-9, MHID: 0-07-163422-3.All trademarks are trademarks of their respective owners. Rather than put a trademark symbol after every occurrence of a trade-marked name, we use names in an editorial fashion only, and to the benet of the trademark owner, with no intention of infringe-ment of the trademark. Where such designations appear in this book, they have been printed with initial caps.McGraw-Hill eBooks are available at special quantity discounts to use as premiums and sales promotions, or for use in corporate training programs. To contact a representative please e-mail us at bulksales@mcgraw-hill.com.Information has been obtained by Publisher from sources believed to be reliable. However, because of the possibility of human or mechanical error by our sources, Publisher, or others, Publisher does not guarantee to the accuracy, adequacy, or completeness of any information included in this work and is not responsible for any errors or omissions or the results obtained from the use of such information.Oracle Corporation does not make any representations or warranties as to the accuracy, adequacy, or completeness of any informa-tion contained in this Work, and is not responsible for any errors or omissions.TERMS OF USEThis is a copyrighted work and The McGraw-Hill Companies, Inc. (McGrawHill) and its licensors reserve all rights in and to the work. Use of this work is subject to these terms. Except as permitted under the Copyright Act of 1976 and the right to store and retrieve one copy of the work, you may not decompile, disassemble, reverse engineer, reproduce, modify, create derivative works based upon, transmit, distribute, disseminate, sell, publish or sublicense the work or any part of it without McGraw-Hills prior consent. You may use the work for your own noncommercial and personal use; any other use of the work is strictly prohibited. Your right to use the work may be terminated if you fail to comply with these terms.THE WORK IS PROVIDED AS IS. McGRAW-HILL AND ITS LICENSORS MAKE NO GUARANTEES OR WARRANTIES AS TO THE ACCURACY, ADEQUACY OR COMPLETENESS OF OR RESULTS TO BE OBTAINED FROM USING THE WORK, INCLUDING ANY INFORMATION THAT CAN BE ACCESSED THROUGH THE WORK VIA HYPERLINK OR OTHERWISE, AND EXPRESSLY DISCLAIM ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIM-ITED TO IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. McGraw-Hill and its licensors do not warrant or guarantee that the functions contained in the work will meet your requirements or that its opera-tion will be uninterrupted or error free. Neither McGraw-Hill nor its licensors shall be liable to you or anyone else for any inaccu-racy, error or omission, regardless of cause, in the work or for any damages resulting therefrom. McGraw-Hill has no responsibility for the content of any information accessed through the work. Under no circumstances shall McGraw-Hill and/or its licensors be liable for any indirect, incidental, special, punitive, consequential or similar damages that result from the use of or inability to use the work, even if any of them has been advised of the possibility of such damages. This limitation of liability shall apply to any claim or cause whatsoever whether such claim or cause arises in contract, tort or otherwise. up-to-date inforuation on 0racle atabase, 0racle Application Server, Web developuent, enterprise grid couputing, database technology, and business trends Third-party news and announceuents Technical articles on 0racle and partner products, technologies, and operating environuents evelopuent and aduinistration tips Feal-world custouer storiesIf there are other Oracle users at your location who would like to receive their own subscription to Oracle Magazine, please photo-copy this form and pass it along.Three easy ways to subscribe:Webvisit our Web site at oracle.com/oraclemagazine You'll find a subscription foru there, plus uuch uoreFaxCouplete the questionnaire on the back of this card and fax the questionnaire side only to +1.847.763.9638MailCouplete the questionnaire on the back of this card and uail it to P.O. Box 1263, Skokie, IL 60076-8263123FREE SUBSCRIPTIONGET YOURTO ORACLE MAGAZINEOracle Magazine is essential gear for todays information technology professionals. Stay informed and increase your productivity with every issue of Oracle Magazine. Inside each free bimonthly issue youll get: Copyright 2008, Oracle and/or its afliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its afliates. Other names may be trademarks of their respective owners. Oracle TIGHT / Oracle Enterprise Manager 10g Grid Control Handbook / Gruyter, Hart & Nguyen / 163422-3 WHAT IS THE PRIMARY BUSINESS ACTI VI TY OF YOUR FIRM AT THIS LOCATION? (check one onl y)o 01 Aerospace and Defense Manufacturingo 02 Application Service Providero 03 Automotive Manufacturingo 04 Chemicalso 05 Media and Entertainmento 06 Construction/Engineeringo 07 Consumer Sector/Consumer Packaged Goodso 08 Educationo 09 Financial Services/Insuranceo 10 Health Careo 11 High Technology Manufacturing, OEMo 12 Industrial Manufacturingo 13 Independent Software Vendoro 14 Life Sciences (biotech, pharmaceuticals)o 15 Natural Resourceso 16 Oil and Gaso 17 Professional Serviceso 18 Public Sector (government)o 19 Researcho 20 Retail/Wholesale/Distributiono 21 Systems Integrator, VAR/VADo 22 Telecommunicationso 23 Travel and Transportationo 24 Utilities (electric, gas, sanitation, water)o 98 Other Business and Services _________WHICH OF THE FOLLOWING BEST DESCRIBES YOUR PRIMARY JOB FUNCTION? (check one onl y)CORPORATE MANAGEMENT/STAFFo 01 Executive Management (President, Chair, CEO, CFO, Owner, Partner, Principal)o 02 Finance/Administrative Management (VP/Director/ Manager/Controller, Purchasing, Administration)o 03 Sales/Marketing Management (VP/Director/Manager)o 04 Computer Systems/Operations Management (CIO/VP/Director/Manager MIS/IS/IT, Ops)IS/IT STAFFo 05 Application Development/Programming Managemento 06 Application Development/Programming Staffo 07 Consultingo 08 DBA/Systems Administratoro 09 Education/Trainingo 10 Technical Support Director/Managero 11 Other Technical Management/Staffo 98 Other WHAT IS YOUR CURRENT PRIMARY OPERATING PLATFORM (check all t hat appl y)o 01 Digital Equipment Corp UNIX/VAX/VMSo 02 HP UNIXo 03 IBM AIXo 04 IBM UNIXo 05 Linux (Red Hat)o 06 Linux (SUSE)o 07 Linux (Oracle Enterprise)o 08 Linux (other)o 09 Macintosho 10 MVSo 11 Netwareo 12 Network Computingo 13 SCO UNIXo 14 Sun Solaris/SunOSo 15 Windowso 16 Other UNIXo 98 Other99 o None of the AboveDO YOU EVALUATE, SPECIFY, RECOMMEND, OR AUTHORI ZE THE PURCHASE OF ANY OF THE FOLLOWING? (check all that appl y)o 01 Har dwar eo 02 Business Appli cat ions (ERP, CRM, et c.)o 03 Appli cat ion Development Toolso 04 Dat abase Pr oduct so 05 Int er net or Int r anet Pr oduct so 06 Ot her Sof t war eo 07 Middlewar e Pr oduct s99 o None of t he AboveIN YOUR JOB, DO YOU USE OR PLAN TO PUR-CHASE ANY OF THE FOLLOWING PRODUCTS? (check all t hat appl y)SOFTWAREo 01 CAD/CAE/CAMo 02 Collaboration Softwareo 03 Communicationso 04 Database Managemento 05 File Managemento 06 Financeo 07 Javao 08 Multimedia Authoringo 09 Networkingo 10 Programmingo 11 Project Managemento 12 Scientific and Engineeringo 13 Systems Managemento 14 Workflow HARDWAREo 15 Macintosho 16 Mainframeo 17 Massively Parallel Processingo 18 Minicomputero 19 Intel x86(32)o 20 Intel x86(64)o 21 Network Computero 22 Symmetric Multiprocessingo 23 Workstation ServicesSERVI CESo 24 Consultingo 25 Education/Trainingo 26 Maintenanceo 27 Online Database o 28 Supporto 29 Technology-Based Trainingo 30 Other 99 o None of t he AboveWHAT IS YOUR COMPANYS SI ZE? (check one onl y)o 01 Mor e t han 25,000 Employeeso 02 10,001 t o 25,000 Employeeso 03 5,001 t o 10,000 Employeeso 04 1,001 t o 5,000 Employeeso 05 101 t o 1,000 Employeeso 06 Fewer t han 100 EmployeesDURING THE NEXT 12 MONTHS, HOW MUCH DO YOU ANTICIPATE YOUR ORGANI ZATION WILL SPEND ON COMPUTER HARDWARE, SOFTWARE, PERIPHERALS, AND SERVICES FOR YOUR LOCATION? (check one onl y)o 01 Less than $10,000o 02 $10,000 to $49,999o 03 $50,000 to $99,999o 04 $100,000 to $499,999o 05 $500,000 to $999,999o 06 $1,000,000 and OverWHAT IS YOUR COMPANYS YEARLY SALES REVENUE? (check one onl y) o 01 $500, 000, 000 and aboveo 02 $100, 000, 000 to $500, 000, 000o 03 $50, 000, 000 to $100, 000, 000o 04 $5, 000, 000 to $50, 000, 000o 05 $1, 000, 000 to $5, 000, 000WHAT LANGUAGES AND FRAMEWORKS DO YOU USE? (check all t hat appl y)o 01 Ajax o 13 Python o 02 C o 14 Ruby/Railso 03 C++ o 15 Springo 04 C# o 16 Strutso 05 Hibernate o 17 SQLo 06 J++/J# o 18 Visual Basico 07 Java o 98 Othero 08 JSPo 09 .NETo 10 Perlo 11 PHPo 12 PL/SQLWHAT ORACLE PRODUCTS ARE IN USE AT YOUR SI TE? (check all t hat appl y)ORACLE DATABASEo 01 Or acle Dat abase 11go 02 Or acle Dat abase 10go 03 Or acle9i Dat abaseo 04 Or acle Embedded Dat abase (Or acle Li t e, Times Ten, Ber keley DB) o 05 Ot her Or acle Dat abase ReleaseORACLE FUSION MIDDLEWAREo 06 Or acle Appli cat ion Ser ver o 07 Or acle Por t alo 08 Or acle Ent er pr ise Managero 09 Or acle BPEL Pr ocess Managero 10 Or acle Ident i t y Managemento 11 Or acle SOA Sui t eo 12 Or acle Dat a HubsORACLE DEVELOPMENT TOOLS o 13 Or acle JDevelopero 14 Or acle Formso 15 Or acle Repor t so 16 Or acle Designero 17 Or acle Discover ero 18 Or acle BI Beanso 19 Or acle War ehouse Buildero 20 Or acle WebCent ero 21 Or acle Appli cat ion Expr essORACLE APPLI CATIONS o 22 Or acle E-Business Sui t eo 23 PeopleSof t Ent er pr iseo 24 JD Edwar ds Ent er pr iseOneo 25 JD Edwar ds Wor ldo 26 Or acle Fusiono 27 Hyper iono 28 Siebel CRMORACLE SERVI CESo 28 Or acle E-Business Sui t e On Demando 29 Or acle Technology On Demando 30 Siebel CRM On Demando 31 Or acle Consul t ingo 32 Or acle Educat iono 33 Or acle Suppor to 98 Ot her99 o None of t he AboveYOU MUST ANSWER ALL 10 QUESTIONS BELOW.12345678908014004s i g n a t u r e ( r e q u i r e d ) d a t exFrom time to time, Oracle Publishing allows our partners exclusive access to our e-mail addresses for special promo-tions and announcements. To be included in this program, please check this circle. If you do not wish to be included, you will only receive notices about your subscription via e-mail.Oracle Publishing allows sharing of our postal mailing list with selected third parties. If you prefer your mailing address not to be included in this program, please check this circle.If at any time you would like to be removed from either mailing list, please contact Customer Service at +1.847.763.9635 or send an e-mail to oracle@halldata.com. If you opt in to the sharing of information, Oracle may also provide you with e-mail related to Oracle products, services, and events. If you want to completely unsubscribe from any e-mail communication from Oracle, please send an e-mail to: unsubscribe@oracle-mail.com with the following in the subject line: REMOVE [your e-mail address]. For complete information on Oracle Publishings privacy practices, please visit oracle.com/html/privacy/htmln a m e t i t l ec o m p a n y e - m a i l a d d r e s ss t r e e t / p . o . b o xc i t y / s t a t e / z i p o r p o s t a l c o d e t e l e p h o n ec o u n t r y f a xWant your own FREE subscription?Yes, please send me a FREE subscription Oracle Magazine. No.Would you like to receive your free subscription in digital format instead of print if it becomes available? Yes NoTo receive a free subscription to Oracle Magazine, you must fill out the entire card, sign it, and date it (incomplete cards cannot be processed or acknowledged). You can also fax your application to +1.847.763.9638. Or subscribe at our Web site at oracle.com/oraclemagazine10Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Written by and dedicated to the people who are in the heat of the debugging and diagnostics battles. May it help guide you in your quests for the truthWerner De GruyterTo the young and willing, the future authors of all things good.Daniel NguyenThis book is dedicated to the, um, fascinating team that has sprung up around me, elevated my game, and become the enablers for my success: Eric (and Mr. Dobbs), Carl, Daniel, Ashok, Mike, Asim, and Oscar.Matthew HartOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3About the AuthorsWerner De GruyterstartedhiscareerwithOracleinBelgium,inthesupportorganizationbackin1996,rightaroundthetimeEnterpriseManager1.0wasreleased.Afterfouryearsthere,WernermovedtotheUnitedStatesin2000,wherehejoinedtheSystemManagementProductsdevelopmentorganization,withthemissiontohelpdevelopmenteliminatethebarrierstoadoptionforEnterpriseManager,basedonthefeedbackprovidedbycustomers.Inthattime,hesseenthewholemetamorphosisfromEnterpriseManager(theJAVA/C++basedproduct)toGridControl(theweb-basedproduct).HehasworkedwitheveryversionofEnterpriseManagerandGridControl,andhelpeddiagnoseandtroubleshootissuesraisedbycustomersonallthesereleases.Matthew HartisthecoauthoroffiveotherOraclebooks,includingtheupcomingOracle Database 11g RMAN Backup and RecoveryfromOraclePress.HehasbeenworkingwithHighAvailabilitytechnologiesinOraclesinceversion7.3,anddatabasemanagementtechnologiesformostofthatperiod.HecurrentlyworksandlivesinKansasCity,oratleastatitsairportanyway.Daniel NguyenholdsanengineeringdegreefromtheUniversityofColoradoandhasovertenyearsofexperienceinthesoftwareindustry.Hehasadeep-seatedinterestintechnologiesthatchangethespaceinwhichtheycompete.WhileemployedatOracle,Danieltookgreatstridestobecomeprincipalacrossthetechnologystackwhereheaddressedbugs,diagnostics,andescalationsforOracleSupportServices.DanielrecentlytransitionedintothecommunicationsspacewherehehasbeengainfullyemployedbyAvaya,aleadingproviderofcommunicationtechnology,andservesasatransformationleadforvariousserviceabilityinitiatives.About the ContributorFarouk AbushabanisaSeniorPrincipalSupportEngineerattheOracleRockyMountainSupportCenterinColorado.FaroukisafoundingmemberoftheCenterofExcellenceteam,andistheGlobalTechnicalLeadforEnterpriseManagerHighAvailabilitytechnologydeployments.Afterspendingover20yearssupportingcustomerinformationsystems,andcompletinghisMastersdegreeinManagementofInformationTechnologieswithemphasisonProjectManagementandSecurity,hewasinvitedtocontributetothecontentofthisbook.ThetwochaptersonRACClusterManagementandDataGuardManagementwerewrittenbyFarouk.Hewasalsoatechnicalreviewerforseveralchaptersofthebook.FaroukstartedhiscareerasahardwarefieldengineeraftercompletinghisBachelorsdegreeinElectricalEngineeringattheUniversityofColoradoinColoradoSprings.FaroukandhisfamilyresideinthePikesPeakregionofColorado.About the Technical EditorsRaj AggarwalisaprincipaldevelopmentleadatOracleIndiaandworksintheframeworkteamofEnterpriseManager.HehasdesignedanddevelopedsomeofthecoreframeworkservicesofEnterpriseManager.Heworksonperformance,scalability,andhighavailabilityofEnterpriseManagerGridControl.Sohail ZamuradiscurrentlyaSeniorTechnicalSupportEngineerforOracleEnterpriseManager.HeholdsaMastersdegreeincomputersciencewithamajorindatabases.SuhailholdsOraclecertificationsontheOracle9idatabase,OracleDatabase10g,andOracleDatabase10gRealApplicationClusters(RAC).PriortojoiningOracle,Suhailwasanindependentconsultant.Hehasexperienceinday-to-daycoreDBAduties,RAC9i/10g,OracleASM,OracleDataGuardimplementations,performancetuningatthedatabaseandapplicationlevels,anddesigningRMANimplementationsandbackupstrategies.vOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Contents at a GlancePART IInstalling and Deploying Grid Control 1 What Is Grid Control? Enterprise Manager Concepts 3 2 Preparing and Installing the Environment 23GridControlWorkshopsPreparingtheRepository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31AddinganAdditionalOMStoanExistingEnvironment . . . . . . . . . . . . . . . . . . 36EnablingtheAdvancedNetworkingOption . . . . . . . . . . . . . . . . . . . . . . . . . . . 46AddingaThird-PartySSLCertificateforConsoleAccess . . . . . . . . . . . . . . . . . . 47InstallandConfigureEMCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59ImplementingaGridControlProofofConcept . . . . . . . . . . . . . . . . . . . . . . . . . 65 3 Grid Control and Maximum Availability 81GridControlWorkshopSettingUpanSLBforGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4 Deploying Agents into the Enterprise 109GridControlWorkshopAddingSupportforOtherOperatingSystems . . . . . . . . . . . . . . . . . . . . . . . . . . 117 5 Grid Control: Sizing the Environment and Other Best Practices 143GridControlWorkshopsAddinganAdditionalOMStoanExistingEnvironment . . . . . . . . . . . . . . . . . . 146CalculatingtheNumberofServersforanEnvironment . . . . . . . . . . . . . . . . . . . 150CalculatingtheNumberofTargetsanEnvironmentCanHandle . . . . . . . . . . . 152EstimatingRepositorySizeforMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 6 Maintaining and Managing the EMGC Infrastructure 163GridControlWorkshopReportforAlltheBacklogIndicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179vOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 vi OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 ContentsataGlanceviivi OracleEnterpriseManager10gGridControlHandbookPART IIGrid Control Common Tasks and Functions 7 Principles of Target Monitoring 243 8 Managing EM Users and Audits 257GridControlWorkshopsConfiguringEnterpriseUserSecurity(EUS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264ConfiguringSingleSign-On(SSO) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 9 Metrics and Notifications 271GridControlWorkshopsChangingtheCollectionScheduleandUploadIntervalfortheMetricCPUUtilization(%) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276EditCollectionSettingsforCPUUtilization(%)Metric . . . . . . . . . . . . . . . . . . . 279SettingUpanOSCommandforNotifications . . . . . . . . . . . . . . . . . . . . . . . . . . 296SettingUpaPL/SQLCommandforNotifications . . . . . . . . . . . . . . . . . . . . . . . 299 10 Jobs and Task Automation 305GridControlWorkshopsCreatingJobsandNotificationsforaManagedHost . . . . . . . . . . . . . . . . . . . . . 307SelectingOut-of-BoxJobsinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311SettingCredentialsfortheJobSystemtoWorkwithEnterpriseManagerinMicrosoftWindowsEnvironments . . . . . . . . . . . . . . . 311LeveragingtheJobSystemtoRunOSCommandsforaSpecificHost . . . . . . . . 313ModifyingJobPurgePoliciesoftheJobSystem . . . . . . . . . . . . . . . . . . . . . . . . 315CreatingandSchedulingaMultitaskJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317QueryingtheJobLibraryforAllStoredJobs . . . . . . . . . . . . . . . . . . . . . . . . . . . 320CreatingandSavingaJobtotheJobLibrary . . . . . . . . . . . . . . . . . . . . . . . . . . . 321ExecutingJobsUsingtheEnterpriseManagerCommand-LineInterface . . . . . . 322MonitoringtheResultsofaMultitaskJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325RetrievetheResultsfromaJobUsingSQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327RemovingJobsStuckinaStatusofSUSPEND . . . . . . . . . . . . . . . . . . . . . . . . . . 329 11 Reporting and Dashboards 333GridControlWorkshopCreateaDiagnosticsReportonYourManagementAgentsHealth . . . . . . . . . . 338PART IIIEnterprise Manager Power Users Guide 12 Host Management 351GridControlWorkshopsEvaluatingaSampleHost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354EvaluatingHostConfigurationHistory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356SearchingHostConfigurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359ComparingConfigurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360MonitoringPolicyViolationsandHostComplianceScores . . . . . . . . . . . . . . . 363Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3vi OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3ContentsataGlanceviivi OracleEnterpriseManager10gGridControlHandbookMonitoringLogFilesasPartofaLinuxSecurityAdministrationPractice . . . . . . 366TrackingStorageResources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369MonitoringFilesandDirectories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371IssuingCorrectiveActionsWhenThresholdsAreViolated . . . . . . . . . . . . . . . . 374 13 Systems, Services, and Groups 381GridControlWorkshopsCreatingaSysteminGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384CreatingaGenericService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391CreatingaGroup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396CreatingaGridControlSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400SettingUptheGridControlService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 14 Database Management in Grid Control 411GridControlWorkshopsConfigureBackupSettingsinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431ConfigureRecoverySettingsinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . 432CreateanRMANScriptJobinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . 435PerformDatabaseRecoveryfromGridControl . . . . . . . . . . . . . . . . . . . . . . . . . 437RunaSQLScriptAgainstMultipleDatabases . . . . . . . . . . . . . . . . . . . . . . . . . . 447 15 RAC and Cluster Management 453GridControlWorkshopInstallingtheAgentonClusterNodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455 16 Data Guard Management 473GridcontrolWorkshopsCreatingaPhysicalStandby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477ConfiguringFast-StartFailover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484ConfigureaPhysicalStandbyforRAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486 17 Application Monitoring 489GridcontrolWorkshopsCreatingaBeacon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493AddingSSLCertificatesforBeaconPlayback . . . . . . . . . . . . . . . . . . . . . . . . . . 496ModelingaServiceUsinganOut-of-BoxWebApplication:EnterpriseManagerWebSite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498AddAdditionalServiceTestsUsingtheWebTransactionRecorder . . . . . . . . . 505ChangeSystemConfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510AddPerformanceMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511AddUsageMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512SettingUpBeaconWatchLists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513 18 Troubleshooting and Diagnostics 517GridControlWorkshopSettingUptheAgenttoRunJobsandCommand . . . . . . . . . . . . . . . . . . . . . . . 545Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 viii OracleEnterpriseManager10gGridControlHandbookPART IVAppendixes A Configuration Files 573 B Configuration Parameters 577 C Log Files 597 D Environment Variables 605 E Standard Repository Views 611 F Command-Line Tool Options 623 G Acronyms 631 Glossary 639 Index 645ixOracle TIGHT / Oracle Enterprise Manager 10gviii Oracle Enterprise Manager 10g Grid Control HandbookContentsAcknowledgments xixIntroduction xxiiiParT IInstalling and Deploying Grid Control 1 What Is Grid Control? Enterprise Manager Concepts 3Enterprise Management Software 4The Various Management Tools 6DB Control 6AS Control 7Grid Control Concepts 7Grid Control vs Enterprise Manager 7Acronym vs Idiom 8Metrics vs Targets 8Administrator vs User 9Grid Control Components 10Grid Control Console 11The Grid Control Agent 11The Management Server (OMS) 13The Grid Control Repository 13The Flow of Information 14Agent and OMS Communication 14OMS-to-Repository Communication 16User (Web-Client) to OMS Communication 19TCP Ports Used by Enterprise Manager 20The Different Types of Targets 21Summary 22ixOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 x OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 Contentsxix OracleEnterpriseManager10gGridControlHandbook 2 PreparingandInstallingtheEnvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23PreparingfortheInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24PhasedRollout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25TopologyoftheManagementFramework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26TopologyoftheManagedTargets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26PreparingforFutureGrowth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27NamingConventionsforPhysicalHosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28UsingaDifferentHostnamefortheInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29PreinstallationChecksfortheInstallerandtheDatabase . . . . . . . . . . . . . . . . . 30ModifyingtheResponseFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33HardwareRequirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33InstallingGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34EnterpriseManager10gGridControlUsingaNewDatabase . . . . . . . . . . . . . . 34EnterpriseManager10gGridControlUsinganExistingDatabase . . . . . . . . . . 35AdditionalManagementServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36AdditionalManagementAgent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37PostconfigurationontheRepositoryServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37PostconfigurationontheOMSServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38LoaderSetupandSharedFileSystemLoader . . . . . . . . . . . . . . . . . . . . . . . . . . 38Out-Of-BandNotifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39LogandTraceFilesandRotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41OracleHTTPServerConfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45SMTPServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45SecuringtheEnvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45SettingUpAdvancedNetworkingOption(ANO) . . . . . . . . . . . . . . . . . . . . . . . 46ConfiguringHTTPServertoUseSSLforConsoleUI . . . . . . . . . . . . . . . . . . . . . 46EMKEYOperations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48LockingandUnlockingtheOMSServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48SecuringtheDatabaseandtheApplicationServer . . . . . . . . . . . . . . . . . . . . . . 50UsingFirewalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51PostconfigurationforPatchingandProvisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54SoftwareLibrary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54ProxyServerSettings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55MyOracleSupportSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56EMCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57InstallingEMCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58SettingUpEMCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59FilesUsedbyEMCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60SynchronizingEMCLISoftware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61CommandOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61SettingUpaTestEnvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62InstallationandDeploymentBestPractices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62CommonMistakes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62BestPractices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3x OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Contentsxix OracleEnterpriseManager10gGridControlHandbook 3 GridControlandMaximumAvailability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81WhyHighAvailability? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82HA:EliminatingtheSinglePointofFailure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84MakingGridControlDisasterRecoveryReady . . . . . . . . . . . . . . . . . . . . . . . . 95BackupandRecovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99StrengtheningtheExtendedInfrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106SummaryEMHAintheRealWorld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4 DeployingAgentsintotheEnterprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109PreparingtheServerMachine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110HostnamesandHostnameResolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110UserAccounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110TimeZoneandOSClock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111AvailableDiskSpace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112DifferentTypesofInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113StandardOUIInteractiveInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113SilentOUIInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113AgentDownload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114AgentPush(akaUI-BasedPush) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114SharedAgent(akaState-DeployedAgent/NFSAgent) . . . . . . . . . . . . . . . . . . . . 115UsingtheAgentDownloadInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116TheDefaultInstalledSoftware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116CustomizingtheAgentDownloadScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119Usingastaticports.iniFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122Command-LineOptionsfortheAgentDownloadScript . . . . . . . . . . . . . . . . . . 122UsingtheAgentPushInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124ThePrerequisitesforPushInstalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124InstallationSteps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124UsingtheSharedAgentInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125ThePrerequisitesforSharedAgentInstalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125InstallationSteps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125PostinstallationandConfigurationSteps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126EnablingOOBMessages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127ConfiguringtheMonitoringSpace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128LoggingandTracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128TimeoutConfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130TheSNMPSubagent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131SecuringtheAgent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132RegistrationPasswords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132SecuringandUnsecuringtheAgent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134AdvancedNetworkingOptionandDatabaseMonitoring . . . . . . . . . . . . . . . . . 134ConfiguringtheAgentforPAMAuthentication . . . . . . . . . . . . . . . . . . . . . . . . . 134ConfiguringtheAgentforLargeSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135IncomingConnections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135ThreadsandProcesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 xii OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 Contentsxiiixii OracleEnterpriseManager10gGridControlHandbookWatchdogMonitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137OracleDatabaseConnections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138PropertyTimeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139OMSServerRequestTimeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 5 GridControl:SizingtheEnvironmentandOtherBestPractices . . . . . . . . . . . 143JustHowManyOMSServersDoINeed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144AddinganAdditionalOMSServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145XMLLoaders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147MetricDataRollup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153AgentHeartbeating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154JobHandling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155NotificationHandling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157HowBigDoestheDatabaseNeedtoBe? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158InstallingandDeployingGridControl:CommonMistakes . . . . . . . . . . . . . . . . . . . . . . 161Summary:BestPractices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 6 MaintainingandManagingtheEMGCInfrastructure . . . . . . . . . . . . . . . . . . . 163MonitortheMonitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164TheOMSandRepositoryTarget . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164WhatIsMTM? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166Health-Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176CleaningUpFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183DefiningTimeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185ManagingtheCoreInfrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187XMLLoaders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187GridControlJobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193NotificationDispatching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198AgentHeartbeating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201RepositoryMetricCalculations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203DataRollup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206DataPurging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207KeepingUpwithChangesintheDataCenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211InfrastructureChanges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211ServerMachineOperations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217PatchingandUpgradingtheInfrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224KeepingTrackofChanges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231AdministratorManagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238TheCommonMistakes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238BestPractices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3xii OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Contentsxiiixii OracleEnterpriseManager10gGridControlHandbookPART IIGrid Control Common Tasks and Functions 7 PrinciplesofTargetMonitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243MonitoringandManagingaTarget . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244SettingUpaTargetforAdministrationandMonitoring . . . . . . . . . . . . . . . . . . . 245CredentialInformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245MonitoringProperties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249TargetProperties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250NavigatingtheUIConsoleforTargetMonitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251LocatingtheTargetintheConsole . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251FindingtheAgentforaTarget . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252TargetStateandAvailability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253AvailabilityStates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253AvailabilityGrouping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 8 ManagingEMUsersandAudits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257DefiningAdministrators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258GridControlPrivileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259JobPrivileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259ReportPrivileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259TargetPrivileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260TemplatePrivileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260SystemPrivileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260CreatingNewUsers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261ModifyingaUser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262DeletingaUser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262DefiningRolesandAssigningPrivileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263CreatingNewRoles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263DeletingaRole . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263SetUpCentralizedAuthentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263UserManagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266EnablingAuditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266AuditReporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267PurgingtheAuditData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 9 MetricsandNotifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271MaximizingthePowerofGridControlMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272ReviewingMetricsintheGCEnvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272MetricThresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273SettingtheMetricThresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 xiv OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 Contentsxvxiv OracleEnterpriseManager10gGridControlHandbookManagingAlerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282User-DefinedMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287PoliciesinEMGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289SuspendMonitoringwithBlackouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291CorrectiveActions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293GettingtheMost(orShouldWeSayLeast)OutofNotifications . . . . . . . . . . . . . . . . . . 293UnderstandingandUtilizingMethodsEffectively . . . . . . . . . . . . . . . . . . . . . . . 294NotificationRules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301WhatAreSchedules? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 10 JobsandTaskAutomation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306AnatomyofaJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306ManagedTargetstoExecuteJobsAgainst . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307FurtherClassificationofJobs,JobExecutions,andJobRuns . . . . . . . . . . . . . . . 310TheJobSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310UsingandDefiningJobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316JobLibrary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320ANoteonEnterpriseManagerCommand-LineInterfaceandJobs . . . . . . . . . . 321JobsandGroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322Analyzing,Diagnosing,andTroubleshootingJobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324SearchingJobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327MaintainingtheJobSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 11 ReportingandDashboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334CommonUsesforReports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334HowtoGetaReport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335TheAnatomyofaReport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336CreatingaNewReport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337Dashboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343CustomizingDashboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344WritingYourOwnDashboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345AnEnd-to-EndViewofDeployedApplications:TopologyViewer . . . . . . . . . . 346Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347PART IIIEnterprise Manager Power Users Guide 12 HostManagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352ProvisioningHosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352ViewingHostConfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3xiv OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Contentsxvxiv OracleEnterpriseManager10gGridControlHandbookWhatDoYouReallyKnowaboutYourHosts? . . . . . . . . . . . . . . . . . . . . . . . . . 356Oracle,WhatAreMyAgentsMonitoring? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366CreatingandEnforcingStandardsforHostManagement . . . . . . . . . . . . . . . . . . 369Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 13 Systems,Services,andGroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382BuildingaSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383BestPracticesforUsingSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389WebApplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390BuildingaService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390BestPracticesforUsingServices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395RedundancyGroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396BuildingaGroup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396BestPracticesforUsingGroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396DefiningtheEMSystemandService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 14 DatabaseManagementinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411EnterpriseManagerDatabaseControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412ConfiguringDatabasesinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416DatabaseDiscovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416PreferredCredentialsforDatabaseActivities . . . . . . . . . . . . . . . . . . . . . . . . . . 416DatabaseAdministrationinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419DatabaseMonitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419DatabaseAdministration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419TheDatabaseHomePage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429BackupandRecovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430DatabaseBackupsfromGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432PerformingRecoveryinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436BackupManagementandReporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438TheServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438SchemaMonitoringandManagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443DataMovement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445GridControlandManagingtheProductionDatabaseEcosystem . . . . . . . . . . . . . . . . . 446DatabasesandConsolidatedTargets(Groups,Systems,Services) . . . . . . . . . . . 447DatabaseChangeManagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 15 RACandClusterManagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453ManagingRACClustersinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454ConfiguretheClusterComponentsinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 xvi OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 Contentsxviixvi OracleEnterpriseManager10gGridControlHandbookNavigatingThroughtheClusterResources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463Cluster-ManagedDatabaseServices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466ListenerInformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466ASMMonitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472 16 DataGuardManagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473TheHighAvailabilityConsole . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474DataGuardOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475CreatingaPhysicalStandbyDatabaseinGridControl . . . . . . . . . . . . . . . . . . . . . . . . . 476DataGuardManagementOperations:AnOverview . . . . . . . . . . . . . . . . . . . . 477DataGuardMonitoringandMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485ConfiguringaPhysicalStandbyforaRACDatabase . . . . . . . . . . . . . . . . . . . . . 486Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488 17 ApplicationMonitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489TheComplexityofOnlineOperations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490ServicingApplications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492JustifyingServiceAvailability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492WorkingwithBeacons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492WheretoPutBeacons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493SSLMonitoringwithBeacons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496ExposingMonitoringConfigurationTasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505SettingUpWatchLists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513RootCauseAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515DualityExplained . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516 18 TroubleshootingandDiagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517TroubleshootingtheInstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518LogandTraceFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518HowtoDebugOUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519HowtoDebugOPatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521LoggingandTracingtheInfrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523LoggingandTracingtheAgent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523LoggingandTracingtheManagementServer . . . . . . . . . . . . . . . . . . . . . . . . . . 527LoggingandTracingtheRepository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531DiagnosingAgentProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533AgentHang/Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533TargetDiscoveryDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535TargetMonitoringDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536AgentMetricDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537AgentUploadDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542AgentJobDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545TimeZoneDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546DumpingtheAgentState . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3xvi OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Contentsxviixvi OracleEnterpriseManager10gGridControlHandbookDiagnosingOMSProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549JavaDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549OMSTargetDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551XMLLoaderDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554EMJobDebugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556RepositoryValidation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557UsingEMDIAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559WhatIsEMDIAG? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559InstallingEMDIAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561UsingVerifyCommands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563DumpingReports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566ReportingonEMDIAGFindings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569PART IVAppendixes A ConfigurationFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574OMS:ManagementServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576 B ConfigurationParameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577Agent:Propertiesintheemd.propertiesFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578OMS:Propertiesintheemoms.propertiesFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586 C LogFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598AgentLogandTraceFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598OracleNetLogFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599OMS:ManagementServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599OracleHTTPServerLogFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599ASControlAgentLogFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 600OC4JApplications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 600OMSServer(OC4J_EM)LogandTraceFiles . . . . . . . . . . . . . . . . . . . . . . . . . . 601AgentPushInstalls(OC4J_EMPROV)LogandTraceFiles . . . . . . . . . . . . . . . . 601DCMLogFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602OPMNLogFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603WebcacheServerLogFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603Install,Configure,andPatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603Installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604ConfigurationTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604OPatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604 D EnvironmentVariables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605EnvironmentVariablesUsedbyEMCTLontheAgent . . . . . . . . . . . . . . . . . . . . . . . . . . 606EnvironmentVariablesUsedbyEMCTLontheOracleManagementServer(OMS) . . . 608Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 xviii OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 E StandardRepositoryViews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611 F Command-LineToolOptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623EMCTL:Agent-SideCommands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624EMCTL:OMSServer-SideCommands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627EMCTL:ASControlCommands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 630 G Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 639 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645xixOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3xviii OracleEnterpriseManager10gOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Acknowledgmentswishwehadscience-fictiontypetechnologies(already)todumpsomeonesbrainonpaper.Oh,andelectronicpaperwouldbefine,tooThisprojectstartedwiththatexactthoughtinmind:tobringthepracticalknowledgegainedfromhelpingcustomersalltheseyearstopaper,andpresentitinaneasilyconsumablewayforeveryonetoreadandenjoy.Dumpingabrainandwritingdownallthetidbits,tips,andtricksturnedouttoyieldquiteapileofinformation,considerablylargerthanIoriginallythoughtitwouldbe.Thisledtosomeinterestingdebatesonjusthowhighthelimitintheskyreallywasintermsofthesizeandscopeofthisbook.Thefactthatthebookisnotbiggerandthickerthanitalreadyis,isthedirectresultofmycoauthors,andthepublishersandeditorsatMcGraw-Hill,constantlyremindingmethattherewere limits to adhere to,andthatwehadtocapthebooksomewhere.Theresultisrighthere:Acomprehensiveguide,designedtohelpyouthroughtheimplementationandusageofGridControlfromtheinfrastructurestandpoint.Thebraindumpisnotcomplete,butthisistheinformationessentialforthestartofthejourney,thatinformationyoujustneedtostartthequest.Timeforasequel,anyone?WernerDeGruyterIxixOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 xx OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 Acknowledgmentsxxixx OracleEnterpriseManager10gGridControlHandbookThisbooknearlybrokeme.Thefactthatitdidnotdosomustbeattributedfirstandforemosttomycoauthors,DanielNguyenandWernerDeGruyter,andthelast-minutecontributorFaroukAbushaban.IoweadeepdebtofgratitudetoeachofthesemenforpickingupwhereIleftoff,fillinginholes,takingoncoordinationduties,andotherwiseensuringthatthisbookactuallymadeitoutthedoor.Thevalueofthisbookisprimarilytheirs,anditsflawssoglaringlymine.Imustalsothanktheever-patient(untilitstimetonotbepatient)editorLisaMcClainandacquisitionscoordinatorMeghanRiley,whowatchedinhorrorasIwentfrombeingthepredictable,on-timeauthortobeingthatauthor.Iam,truly,deeplysorry,asthisisactuallyworsethanbeingundependablefromtheget-go.Nevertheless,yourdoggedattitudeandfinalultimatumshaveleadtoaheftytomethatwecanallbeproudof.Almostliterallyminutesafterwesignedthecontractforthisbook,mylifewasturnedupsidedownbymassivechangesinmyprofessionallifeandduties,andIfoundmyselfworkingharderthanIeverthoughtpossible,andtestingtheouterlimitsofmyendurance.Throughthis,Ihadtheirrepressiblesupportofmyever-presentmentorandfriend,MartinIngram.Withoutit,Iwouldhavecrumbled.Atthesametime,Iwasintroducedtoanewmentor,whoquicklyinsertedhimselfasavoiceinsidemyhead,becomingafilterformyday-to-dayactivities.AbigthanksmustgotoMr.Birtwellforthatsupport,forbelievinginme,andforhisrelentlessdriveforexcellence.MyteamatAvaya,boththeoldoneandthenewone,mustbethankedformusclingthroughadifficultyearfilledwithfrustrationandloss,butalsosuccessandvindication.ManyofthemdidntknowIwasalsotryingtowriteabookatthesametimeaseverythingelse,andtofindthisbookontheshelvesmayserveasanahamomentforthem,lookingbackatmyshortfuseoncertaindayswhenonlyanall-nighterwouldgetmetodone.Finally,Imustthankmywifeandkids,foronceagainputtingupwithme,notonlywiththeridiculoustravelscheduleIkept,butthelonghoursforbothmydayjobandthisbook,mynightjob.Itwasalongyear,andwithouttheirsupportandloveIwouldhavesuccumbedtoexhaustionmonthsago.Finally,thankyou,thereader,forwantingthisbookandreadingit.Ihopeyoufinditasusefulandrelevantasweintended.MatthewHartIhavebeenchasingtimeaslongasIcaretoremember.Asanathlete,itwasallaboutstreamliningtechniquestoshavemillisecondswitheachlap.Collegetaughtyouthatthemixtureofmidnightoil,caffeine,andmusicaround150BPMwerethenecessaryingredientstocookupadiploma.Asthereaderofthisbook,youtoo,concurthattimeisanexpensivecommodityandtherewardsofourprofessionstemfromourabilitytoseekmorewithless.Itherebythankyou;thereader,forwantingtoarmyourselfwiththismaterialandmakeuseofittocementafuturesuccessstory.IamsureIspeakonbehalfofalltheauthorswhenIsayweareextremelygratefultoMeghanRiley,LisaMcClain,andtherestoftheMcGraw-Hillstaff.Notonlydotheysourcegoodtalent,buttheyalsohave(justenough)patiencetoallowtheirauthorsthetime,space,andemotionalrangeofmotiontolookbackwhiletheylookedforward.Withoutthiskindofpublishingexpertise,youcanbettherewouldbemanymorethingsthatwerenotwriteaswesatdowntoright.Idbeafooltofuelthemisconceptionthatanyachievementrestssquarelyonmysmarts,ambition,orhustle.Theoutliersofanysuccessareinlargepartduetoaselectedcaliberofindividualsatthemostopportunemomentsofmylife:Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3xx OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Acknowledgmentsxxixx OracleEnterpriseManager10gGridControlHandbookMaggieWells:ThankyoufortakingthatgambleandliftingthegatestotheprofessionthatInowcallmyown.Manyexecutivesattachbatteriestothosetheymentor.Ihaveaself-generating,hydrogen-poweredfluxsupercapacitor.MartinIngram:Foroveradecade,Ihaveremainedgratefulforyourunwaveringsupport,presence,andconsideration.Throughyou,Ihavecometotermswiththehaphazardnatureoflife,whichalltoooftenproducesprogressinadvanceofunderstanding.MattHart:Itwasjustamatteroftimebeforeweworkedcloselywitheachotherandneedlesstosaywell,needlesstosay.Iftherewereascienceofsuperheroes,youcouldbeanundercoverone.HmmmMattHartakaMattHart:knownforhisextraordinaryskillsintransforminganyplanetoftheapestoplanetoftheapps.LisaTamura:Thanksforputtingupwithmypersonalandprofessionalgrowingpains.Iambuildingupallthegoodkarmaleftinthisworldsomyfirstbornwillonlybehalfasdifficult.Thetroublewithdoingthingsrightthefirsttimeisthatnobodyappreciateshowdifficultiteverwas.IfthiscompanyeverneededanAvayaSelectServicesdivision,itwouldbetheteamofindividualsIhaveworkedwiththesepastthreeyearstheycanbecountedontomusclethroughinitiatives,justlikethat800-poundgorilla.Beingaproudandcomplexindividual,Ifacemostofmystrugglesalone.Butsometimesyouneedhelppullingoutfromthelowestpointsofyourlife.Thisistomycircle:despitemymissingmanyanoccasion;Icanalwayscountonthemtoconjureupanarmytocovermyback(andballs).Suchrelationshipsarebeyondrareandpricelessinthetruestsenseoftheword.Finally,thisistomyfuturewifeandthecheerleaderofmydreams.Beforeyou,IwastaughttoneverwantanythingthatIcouldntstandtoloseDanielNguyenThis page intentionally left blank Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3IntroductiontsaGrid!ItsaCloud!Itsa.wildtangledmess!Letsjustlookitrightintheeyeandcallitout.WeintheITindustrieshavebeenbarragedwiththebuzzwordsoverandoveragainsincewerolledintothenewmillennium:thetransitiontoutilitycomputing,orgridcomputing,orcloudcomputing(dependingonthesource/exactmonth/positionofthestarsinthesky).Wevedecidedthatwevehitfatigueinallofthis,andtheresnothingwecandoaboutthelanguageandjockeyingforpositionthataccompaniesallofthePR.But,underneathitall,arealtransitionhasbeentakingplace,basedonrealpressuresintheglobaleconomytoleveragetechnologyforbusinessadvantage,whilesuppressingcostsandincreasingreliability.Thisevolutioncanbecharacterizedbythefulfillmentofthepromiseofdistributedcomputing(awholeroomofcommoditizedcomputers,insteadofonemassiveCray);themoveawayfromdesktopapplicationstoperformday-to-daytasks,andthemovetothesupremacyofthebrowser-as-everything-you-will-ever-need;andtheriseofstandards-based,openinterfacesbetweencomputingelements.Thatsjustforstarters,butitsleadingtoinnovationsthathelpourcompaniesgetmorefromtheirdata,morefromtheirnetworks,andultimatelymoreproductivityandinnovationfromtheirpeople.So,peoplecanlookatitandbranditGrid,orUtility,andsoon,butthoseofuswhovebeeninvolvedinitforthepastdecadeunderstandthefundamentaltruth:thecurrenttechnologylandscapeisonebigwildmess.IxxiiiOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 xxiv OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3 Introductionxxvxxiv OracleEnterpriseManager10gGridControlHandbookAndmanagingthehairytangledmessofmoderncomputinginfrastructureelementsisfurthercomplicatedbypressurescausedbythefactthatwearespreadallovertheglobenow,andarebeingaskedtodomoreandmorewithlessmanuallaboravailable.Theresmoretodo,andlesstodoitwith.Thesepressureshaveleftmostofuswithapileoftoolsthatweusetomonitor,manage,andadministerallthedistinctcomponentsthatmakeupourborderlessdatacenters.Butafewyearsago,Oracledecidedtotakeastepback,lookatthebigecosystemmess,andstoptunnelinginonspecifictechnologycomponentstryingtosolvetheseproblems.Instead,Oraclestartedfromscratch,andbuiltatoolthattriedtosolvetheoverallecosystemmanagementproblem,utilizingthepatternsofmanagementthatexistacrossallcomponentshostservers,operatingsystems,databases,applicationservers,theapplicationsthemselves,thenetworkroutingcomponents,loadbalancerseverything(thekitchensinkisontheroadmap).Atthesametime,theywantedtoleveragetheelementsofmoderncomputingthattheywerealsotryingtomanage:distributedcomputing,web-basedapplications,andstandard,openinterfaces.TheresultofthisworkisOracleEnterpriseManagerGridControl.GridControlhasbeenaroundforafewyearsnow,andchancesare,ifyouarereadingthis,youvehadsomeexposuretoit,andeitherreallywanttoleverageandmakegooduseofit,oryouarealreadystaringdownaGridControlimplementationandyoucouldusesomehelp.IfyouarealreadyworkingwithGridControleveryday,helpinguserswiththeworkflow,optimizingtheperformanceandthroughputordebuggingissues,youwillstartseeingpatternsandrepeatingissuespoppingupwitheverynewdeployment.Thesepatternsandrepeatingissuesoftenleadtorealissuesdowntheline,resultingineitherdowntime,orlimitationsintherolloutofGridControl,preventingfurthergrowthoftheecosystemthatEMismanaging.The Complete Grid Control HandbookTheideadrivingthisbookistogiveadministratorsofGridControlthetipsandtrickscreatedforthecommonrepeatingissuesandfrequentmistakespeoplemake.Mostoftheseissuescanbeeasilyavoidedifpeoplearemadeawareofthemupfront,orbyjustpointingouttherightsetofchecksandteststoperform.Andthisiswheretheinformationinthisbookcomesintothepicture.Inthisbook,wecovertheentirelifecycleoftheGridControlproduct,fromtheverybeginning,withthepreparationoftheinstallofGridControl,tothemanagingofthedatacenterinfrastructure,rightdowntothedailychecksanadministratorshouldbeawareoftokeepaneyeontheproduct.Thisbookismorethanjustanimplementationguide;wepackeditasfullofreal-worldexperience,uglytruths,head-scratchers,andgotchasaswereasonablycould.WewantyoutotakefulladvantageofGridControl,andensurethatyoucanuseittomanagetheentireecosystem.Gettingitinstalledisjustthebeginning.EverystepintheGridControllifecycleiscovered,givingyouthebackgroundinformationrequiredtosucceedandflourishwithsomereal-lifeexercisesandworkshopstogiveanyaspiringEMguruthechancetogethisorherhandsdirty.Using This Book EffectivelyLikealltechnicalmanualsworththeirweight,thisbookismeanttobereadable,covertocover,asawaytofamiliarizeyourselfwithGridControlanditsroleincompleteecosystemmanagement.Thetopicsareapproachedinaformatthatallowseachcomplexsubjecttobuildonpreviouschapters,slowlyworkingforwardfromprinciples,toimplementation,toconfiguration,tousage,andfinallytodiagnosingandtroubleshooting.Thestructureofthisbookisdividedintothreeparts,mimickingthestagesanyGridControlsitewillgothrough.Oracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3xxiv OracleEnterpriseManager10gGridControlHandbookOracle TIGHT /OracleEnterpriseManager10gGridControlHandbook/Gruyter,Hart&Nguyen/163422-3Introductionxxvxxiv OracleEnterpriseManager10gGridControlHandbookThefirstistheinstallationoftheEMinfrastructureitself.Thiscrucialstepistheonewithwhichmostproblemsandbottleneckscanbeavoided,iftherightchoicesaremadefortherolloutofthemainEMapplicationandthemonitoringAgents.ThesecondstageinvolvestheinnerworkingsofGridControl,tosetuptherightkindofmonitoringandmaintenanceforboththeEMandtheITinfrastructurethatisbeingmonitored.AndlastbutnotleastisthemanagingoftheITinfrastructureitself:OnceGridControlisrolledoutandoperational,theideaofthetoolistostartmonitoringtheenterprisewithallofitsdifferentapplicationsandmachines.WeconcludePartIIIwithdetailedtroubleshootinganddiagnostics,asthereareenoughmovingpartsthatknowinghowtoisolatefaultsandrectifythemwillconsumemuchofyourtime.WehavealsoincludedaseriesofappendixesthatprovidefastreferencepointsforGridControlconfigurationfiles,parameters,thedifferentlogfiles,environmentvariables,andrepositoryviewsthatareavailable.Thereisalsoacommand-linesyntaxguideandnomodernbookiscompletewithoutitanacronymguideshouldyoufindyourselfscratchingyourhead.Grid Control WorkshopsNoteveryonereadsabookcovertocover.Weknowthis.Sometimesthatsnotthehighercallingofagoodtechnicalbook.Agoodbooklivesnexttothecomputer,withpagesdog-eared,sectionshighlighted,andlittleyellowpost-itshangingofftheside.Thisbookismeanttobeareferenceguideinadditiontoaconceptualexplanation.Wevepackedthisthingwithusefultechniquesandtimesavingpracticesthatyoucanimplementnow,evenifyourealittlespottyonthearchitecture.Sometimesyoujustneedtoknowhow to do it,right?DuringatoughGridControlimplementation,orwhenyouarehuntingforawaytodiscoveradatabase,theresnothinglikeagoodcookbookapproachtogetyouthroughanexerciseandgetyoutowhereyouareconcentratingonyourbusinessagain.So,tohelpwiththehighlightinganddog-earingofpages,weutilizedtheGridControlWorkshopsectionsofthebook.Wheneverweprovideusefulcodeforperformingaspecificoperation,oraseriesofstepstocompleteacertainproject,wemarkitinalargerfontandbox.Whenyouseethisbox,youknowthefollowingpageswillbefilledwiththeactualstepsyouneedtofollowtogetyourjobdonefast.ThinkofGridControlWorkshopsasrecipes,providingtheingredientsandthemixinginstructionsforaquickandeasymeal.Tomakeyourlifeeveneasier,wevecompiledaseparateContentslistingforeveryGridControlWorkshopinthisbook,withitsdescriptivetitleandthepagenumber.YoullfindthisGridControlWorkshopreferenceaspartoftheContentsataGlanceatthebeginningofthebook.UsingtheContentsataGlance,youcanskipdirectlytotheoneyouneedandgetrighttowork.Thatway,ifyoufindourproseboringandconceptsoverblown,youcanstillgetlotsofspecificusefromthisbook.Again,weencourageyoutoreadthebookchapterforchapter.Nothingcanreplaceaconceptualunderstandingofaproduct,especiallywhenthatproductisprotectingyourmostvaluableasset:thedatabase.So,goodluckandhappyhunting.WevedoneourbesttogiveyouthebreadthanddepthrequiredtobeonsolidfootinggoingintoanewGridControlimplementation,ortotakefulladvantageofanexistingone.Getyourhighlighterout,startdog-earingthepages,andgetreadytobeatthebigtangledmessofgridcomputingintosubmission.This page intentionally left blank PartIInstalling and Deploying Grid ControlThis page intentionally left blank ChaPter1What Is Grid Control? Enterprise Manager ConceptsOracle TIGHT / Oracle Enterprise Manager 10g Grid Control Handbook / Gruyter, Hart & Nguyen / 163422-3 4 Oracle Enterprise Manager 10g Grid Control HandbookOracle TIGHT / Oracle Enterprise Manager 10g Grid Control Handbook / Gruyter, Hart & Nguyen / 163422-3 Chapter 1: What Is Grid Control? Enterprise Manager Concepts 5Iknow it will sound like a clich, but you are at the beginning of an important journey to help guide you through planning, deploying, and using a product that has an impact on your entire IT organization. And as with all other big undertakings, planning is a critical phase.This chapter is important because it deals with and explains the very basics of the software called Enterprise Manager. I hope that, in reading this chapter, the one notion you will gather is that Enterprise Manager is more than just some application reporting data from a bunch of machines. There are tips and tricks throughout this book to help you use this product more effectively. But the most important thing I hope people will learn from this book is that Enterprise Manager is indeed focused on the Enterprise, and can have a positive impact on the entire enterprise, in all of its aspects, and all of its diverse branches.Enterprise Management SoftwareIn 2003, Oracle took an unprecedented direction over the landscape of IT and proclaimedwith uncanny foresightthat its software, most notably Enterprise Manager, would be used to address key challenges that many companies would soon come to face. By that time, the dust over Internet technologies had settled, and IT professionals began to find themselves struggling for an end-to-end management solution, as the concept of computing as a commodity begun to catch on.To address the challenges of companies whose business needs change faster than their IT departments can adapt, Oracle unveiled a suite of business software: Oracle Application Server 10g, Oracle Database 10g, and Oracle Enterprise Manager 10g Grid Control. The g in 10g made an explicit call to address grid computing, a deployment topology that integrates all IT resourcesstorage, servers, databases, application servers, and network peripheralsto provide database services to applications on demand.Oracle technology is typically arranged as a multitier architecture (commonly referred to as n-tier), in which the presentation, the application processing, and the data management are logically separate processes. It is worth noting that the concepts of layer and tier are often used interchangeably, when in fact there is one fairly common distinction between the two: a layer represents a logical structuring mechanism for the elements that make up a software solution, while a tier represents the physical structuring mechanism that make up a system infrastructure. With that in mind, a typical deployment using Oracle technology would be divided as follows: The presentation layer, represented as Tier-1, is the entry point for all application client connections seeking data. These requests originate from client machines or handheld devices, and communicate with the other layers by rendering the results from these requests.The application-processing layer, represented as Tier-2, consists of Java Enterprise Edition (J2EE) application servers and HTTP web servers, which Oracle provides in Oracle Application Server 10g. This tier is considered the glue to the other tiers and controls most of the applications functionality by performing detailed processing. If necessary, the communication to the database back end happens by means of various protocols and specifications such as Java Database Connectivity (JDBC) and Oracle Database Connectivity (ODBC).Oracle TIGHT / Oracle Enterprise Manager 10g Grid Control Handbook / Gruyter, Hart & Nguyen / 163422-3 Oracle TIGHT / Oracle Enterprise Manager 10g Grid Control Handbook / Gruyter, Hart & Nguyen / 163422-3 Chapter 1: What Is Grid Control? Enterprise Manager Concepts 5The data management layer, represented as Tier-3, consists of databases and database storage subsystems. In the grid, databases are utilizing Oracles Real Application Clusters (RAC) technology, which allows databases to run packages and custom applications unchanged across a set of clustered servers. The storage subsystem is usually a collection of low-cost disk devices that, when utilized against a solution such as Oracles Automatic Storage Management (ASM), can easily partition and distribute data storage throughout the disk array.Enterprise Manager 10g Grid Control weaves itself into the mix by effectively monitoring these tiers, and is even architected in the same manner; that is, components of Grid Control are built on Oracle technology that is even capable of monitoring itself. Because of this, no company needs to be fully immersed in grid technology to take advantage of what Grid Control offers. This is a cursory overview of the subject and is by no means complete, but it does place Enterprise Manager Grid Control in the correct contextthis is the solution to gravitate toward if you seek to effectively monitor and leverage assets of a data center.The monitoring of these data centers can be done from locations all around the world, making this suite of products a truly enterprise-class application. As Figure 1-1 illustrates, Grid Control can be used on local company networks, on subnets protected by firewalls, and even with targets on the Internet (outside the company network).InternetGrid ControlReportsHTML ConsoleAgentAgentAgentFIGurE 1-1. The Grid Control architectureOracle TIGHT / Oracle Enterprise Manager 10g Grid Control Handbook / Gruyter, Hart & Nguyen / 163422-3 6 Oracle Enterprise Manager 10g Grid Control HandbookOracle TIGHT / Oracle Enterprise Manager 10g Grid Control Handbook / Gruyter, Hart & Nguyen / 163422-3 Chapter 1: What Is Grid Control? Enterprise Manager Concepts 7Administrators, who do not have to be in the same location as the management infrastructure, can use the information provided in Grid Control to manage and maintain their applications. With lights-out management capabilities, they can be alerted about outages and potential problems, giving Grid Control both proactive and reactive monitoring capabilities. Out-of-box, Grid Control supports monitoring of over 200 different types of targets and applications. On top of that, the infrastructure can be extended with additional monitoring plug-ins to support those types of business-critical targets that Grid Control is not aware of by default. This might sound like a lot, or even overkill. But the variety is required to even be considered a manageability solution in todays diverse IT environments. The days when a data center was nothing more than a few database machines put together in a single location are long gone.Recall if you