Spotlight on OracleGetting Started Guide7.6 2010 Quest Software, Inc. ALL RIGHTS RESERVED. This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's personal use without the written permission of Quest Software, Inc. The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document. If you have any questions regarding your potential use of this material, please contact: Quest Software World Headquarters LEGAL Dept 5 Polaris Way Aliso Viejo, CA 92656 email: legal@quest.com Refer to our Web site (www.quest.com) for regional and international office information. Patents Protected by U.S. Patents #7,784,027 and 6,901,582. Additional patents pending. Trademarks Quest, Quest Software, the Quest Software logo, Spotlight and Toad are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. For a complete list of Quest Softwares trademarks, please see http://www.quest.com/legal/trademark-information.aspx. Other trademarks and registered trademarks are property of their respective owners. Spotlight on Oracle 7.6 Getting Started Guide Updated October 2010 Table of Contents Introduction ......................................................................................................................................................... 5Welcome to Spotlight on Oracle ........................................................................................................................ 7An overview of the Oracle architecture .................................................................................................................... 7An overview of Spotlight on Oracle .......................................................................................................................... 9Features of Spotlight .............................................................................................................................................. 11Getting started withSpotlight on Oracle ....................................................................................................... 15Installing Spotlight .................................................................................................................................................. 15Starting Spotlight .................................................................................................................................................... 18Creating a new Spotlight user ................................................................................................................................ 19Creating an Oracle database connection ............................................................................................................... 21Connecting to an Oracle database ......................................................................................................................... 26Using Toad to launch Spotlight .............................................................................................................................. 27Installing and using DTrace within Spotlight .......................................................................................................... 29Using Spotlight on Oracle ................................................................................................................................ 31The Spotlight on Oracle home page ...................................................................................................................... 31Spotlight on Oracle alarms ..................................................................................................................................... 39Spotlight on Oracle drilldowns ................................................................................................................................ 48Spotlight menus ...................................................................................................................................................... 52Using Spotlight to tune your database ................................................................................................................... 52Using Spotlight to predict database performance .................................................................................................. 53Spotlight on Oracle RAC .................................................................................................................................. 59What is Oracle RAC? ............................................................................................................................................. 59 4Spotlight on Oracle Getting Started Guide Installing and authorizing Spotlight on Oracle RAC ............................................................................................... 61Starting Spotlight on Oracle RAC ........................................................................................................................... 61Creating a new Spotlight user ................................................................................................................................ 62Creating a connection to an Oracle RAC database ............................................................................................... 62Connecting to an Oracle RAC database ................................................................................................................ 66Using Spotlight on Oracle RAC .............................................................................................................................. 67Troubleshooting ................................................................................................................................................ 77Product authorization errors ................................................................................................................................... 77Connection problems .............................................................................................................................................. 78Drilldown problems ................................................................................................................................................. 79High Spotlight load on database server.................................................................................................................. 81Appendix: Contact Quest ................................................................................................................................. 83Contact Quest Support ........................................................................................................................................... 83Contact Spotlight support ....................................................................................................................................... 83Contact Quest Software ......................................................................................................................................... 84About Quest Software ............................................................................................................................................ 84Index ................................................................................................................................................................... 85 1Introduction This guide provides the basic information you need to start using Spotlight on Oracle to diagnose the performance of your Oracle databases and clusters. You will read about: How Spotlight displays information about your Oracle databases. How to install and start Spotlight on Oracle. How to create a Spotlight user for your database. How to make a Spotlight on Oracle connection to a database. How to use that Spotlight connection. How to make and use a Spotlight connection to an Oracle RAC cluster. How to resolve some common Spotlight problems. How to contact Quest Software for further assistance. 2Welcome to Spotlight on Oracle Spotlighton Oracle is Quest Softwares powerful diagnostic and problem-resolution tool for Oracle databases. Its unique user interface provides you with an intuitive visual representation of the activity on your database and operating system. This chapter explains the main features of Spotlight on Oracle. An overview of the Oracle architecture This section explains how data flows into and out of your databases. This architecture has been used to design the main Spotlight window. Basic components of an Oracle server The following diagram illustrates some of the basic components of an Oracle server at the memory, database and disk layers. The numbers indicate the order of the flow of information. 8Spotlight on Oracle Getting Started Guide The numbered labels in the Oracle architecture diagram correspond to the following activities: 1 The client program (for example, SQL*PLUS, Oracle Power Objects, or some other tool) sends a SELECT statement to the server process. 2 The server process looks in the shared pool for a matching SQL statement. If none is found, the server process parses the SQL and inserts the SQL statement into the shared pool. 3 The server process looks in the buffer cache for the data blocks required. If found, the data block must be moved on to the most recently used end of the Least Recently Used (LRU) list. 4 If the block cannot be found in the buffer cache the server process must fetch it from the disk file. This requires a disk I/O. 5 The server process returns the rows retrieved to the client process. This may involve some network or communications delay. 6 When the client issues the UPDATE statement the process of parsing the SQL and retrieving the rows to be updated must occur. The update statement then changes the relevant blocks in shared memory and updates entries in the rollback segment buffers. 7 The update statement also makes an entry in the redo log buffer that records the transaction details. Chapter 29 Welcome to Spotlight on Oracle 8 The database-writer background process copies modified blocks from the buffer cache to the database files. The Oracle session performing the update does not have to wait for this to occur. 9 When the COMMIT statement is issued the log writer process must copy the contents of the redo log buffer to the redo log file. The COMMIT statement does not return control to the Oracle session issuing the commit until this write is complete. 10If running in ARCHIVELOG mode, the archiver process copies full redo logs to the archive destination. A redo log is not eligible for re-use until it has been archived. 11At regular intervals, or when a redo log switch occurs, Oracle performs a checkpoint. A checkpoint requires all modified blocks in the buffer cache to be written to disk. A redo log file cannot be re-used until the checkpoint completes. Recommended reading The best information to help you tune and configure Oracle databases can be found online. Use as your first point of reference Web sites such as: The Quest Software Oracle Solutions Web site. The Oracle home page. An overview of Spotlight on Oracle Home page The Spotlight on Oracle home page provides a detailed view of the Oracle database as a whole.The home page helps you locate system bottlenecks quickly. Related server statistics are grouped together on panels that are connected by a series of graphical flows and icons. Spotlight updates these flows in real time so that you can see how quickly data is moving through the system. The icons change color as Spotlight alarms are raised, upgraded, downgraded and canceled. 10Spotlight on Oracle Getting Started Guide The Spotlight on Oracle home page below has six main panels that reflect the Oracle architecture (see page 7).The panels are: The Sessions panel. The Host panel. The Server Processes panel. The SGA panel. The Background Processes panel. The Disk Storage panel. For more information on the panels in the Spotlight home page, and on using them to diagnose a specified Oracle database, see Panels on page 32. Chapter 211 Welcome to Spotlight on Oracle Alarms More detailed information on situations where the diagnosed Oracle database exceeds acceptable performance thresholds can be viewed in the alarms that Spotlight raises. For more information on Spotlight alarms, see Spotlight on Oracle alarms on page 39. Drilldowns More detailed information on the performance of the Oracle database is available in the charts and tables of the various drilldown pages that can be accessed from the Spotlight home page. For more information on Spotlight drilldowns, and on using them to view the detailed performance of a specified Oracle database, see Spotlight on Oracle drilldowns on page 47. Features of Spotlight Spotlight is designed for ease of installation and use, and offers a range of tools for problem diagnosis. Features common to Spotlight applications Features that are common to most Spotlight applications include: Easy installation. The display of real-time data. Comprehensive information in charts and tables. Visible, audible and configurable alarms. Context-sensitive help. Calibration. Recording and playback of historical data. 12Spotlight on Oracle Getting Started Guide Common Spotlight technology. Spotlight on Oracle can run within the same Spotlight console as several other Quest Spotlight products. Features specific to Spotlight on Oracle Spotlight features that are specific to Spotlight on Oracle include: The ability to add new Spotlight on Oracle users, and to convert the accounts of existing Oracle users so they can use Spotlight on Oracle. Access to a number of other Quest applications for viewing detailed information on the specified Oracle database. For details see Spotlight Menus on page 52, or refer to the online help under Using Spotlight on Oracle to access detailed Oracle information. Features specific to Spotlight on Oracle RAC Spotlight features that are specific to Spotlight on Oracle RAC include: The ability to verify connection details for Oracle RAC instances. The ability to open Spotlight on Oracle from Spotlight on Oracle RAC. Predictive Diagnostics Predictive Diagnostics allows Spotlight to collect and analyze performance metrics for: SQL statements as they are executed on the database instance. System bottlenecks as they affect database throughput and response time. Database resources as they reach their full capacity and affect database throughput and concurrency. When enough performance data is collected, Spotlight may be able to predict the future performance for individual SQL statements on that database, for potential bottlenecks in the system, and for database CPU, memory, and disk I/O resources. Note:When you start using Predictive Diagnostics, Spotlight will take about 10 days to collect enough data to make valid predictions on the future performance of the database. Chapter 213 Welcome to Spotlight on Oracle For more information on Predictive Diagnostics, see Using Spotlight to predict database performance on page 53. DTrace analysis (Solaris 10 and later) DTrace is a dynamic analysis tool for Sun Microsystems operating systems. It was first introduced in the Solaris 10 operating system, and can be used ONLY on Sun platforms from Solaris 10 onwards. Spotlight on Oracle can use DTrace to capture details of the interaction between Oracle and the operating system when Oracle uses the O/S services provided by Solaris. To use DTrace within Spotlight, you need to install and configure a number of DTrace scripts on the server under investigation, and then enable the DTrace functionality within Spotlight itself. For detailed instructions on how to do this, see Installing and Using DTrace within Spotlight on page 27. 3Getting started withSpotlight on Oracle This chapter describes what you need to do to start using Spotlight on Oracle. It explains: How to install Spotlight. How to start Spotlight. How to set up a Spotlight user. How to create a Spotlight on Oracle connection. How to connect to an Oracle database. Installing Spotlight To install Spotlight on Oracle, complete the following steps. Close all other applications before loading the CD into your PC or (if you have downloaded Spotlight from the Quest Web site) executing the downloaded Quest installer. You can cancel the installation at any time by pressing the Esc key or by clicking the Cancel button. Before installing and running Spotlight Before you install and run Spotlight, you should ensure that the Spotlight installer and desktop user have the appropriate client permissions on the Spotlight client machine. 16Spotlight on Oracle Getting Started Guide Note:The Spotlight application can be installed only on a supported Windows system where the current user has Administrator privileges. At installation During installation, Spotlight requires write access to the following folders: The Spotlight installation folder (by default, Program Files\Quest Software\Spotlight) Program Files\Common Files\Quest Shared Documents and Settings\All Users\Application Data\Quest Software\Spotlight Documents and Settings\\Application Data\Quest Software\Spotlight and to the following Windows registry keys and sub-keys: HKEY_LOCAL_MACHINE\SOFTWARE\Quest Software\Products HKEY_LOCAL_MACHINE\SOFTWARE\Quest Software\Spotlight HKEY_CLASSES_ROOT In many cases, the Spotlight installer will run at elevated privileges in order to perform the required actions. These privileges can specifically be allowed or disallowed by system administrators by means of standard Microsoft Installer (MSI) policy settings. When running Spotlight During operation, Spotlight requires write access to the following file locations under the Documents and Settings folder: \Application Data\Quest Software\Shared \Application Data\Quest Software\Spotlight You should also verify that you can run the required Oracle client on the Spotlight machine. When authenticating a Spotlight license When you install a new Spotlight application or update license details on an existing application, Spotlight requires: Administrator access to the Windows registry Chapter 317 Getting Started with Spotlight on Oracle Write access to the Console\Licenses sub-folder of the Spotlight installation folder. To install Spotlight on Oracle Quest Spotlight installers can be delivered on CD or as downloaded executable files. If installing Spotlight from a CD: 1 Insert the Quest Installer CD into your CD-ROM drive. Depending on your system's configuration, the installation program may start automatically. If this happens, follow the instructions on the screen to complete the installation. 2 If the installation program does not start, select Run from the Windows Start menu and type the name of the installation package. For Spotlight delivered as part of the Toad DBA Suite, type d:\TOAD DBA Suite for Oracle vvv.exe (where d: is the CD-ROM identifier, and vvv is the version), and click OK. For Spotlight delivered as a standalone installer type d:\SpotlightonOracle_750.exe (where d: is the CD-ROM identifier) and click OK. 3 If you have downloaded the Quest Spotlight installer from the Quest Web site, execute the downloaded installer. 4 Follow the instructions displayed on the screen. By default, Spotlight on Oracle is installed into the following directory, though you can choose a different installation path. C:\Program Files\Quest Software\Spotlight Note:If the database to be diagnosed is running on Windows, and you want to be able to view operating system metricsin Spotlight, you must have permission to read the registry of the database machine. For more information, see Connection problems on page 78. Multiple installations of Spotlight on Oracle Although many Spotlight applications can run within a single Spotlight installation, the standalone Spotlight installer will allow you to install and separately upgrade multiple copies of Spotlight on your computer. 18Spotlight on Oracle Getting Started Guide MSI installation of Spotlight on Oracle MSI installers are available for Spotlight on Oracle. These installers can be run silently, and produce an installation that can be captured and re-used via automatic deployment techniques. Contact Quest Software for more information on Spotlight MSI installers. To authorize Spotlight on Oracle Spotlight comes with a time-limited trial license for Spotlight on Oracle. Contact your Quest Software representative to obtain the necessary registered keys to ensure Spotlight remains available when the trial key expires. After the trial licenses have expired, Spotlight on Oracle reverts to a preview mode, which limits access to the Spotlight on Oracle home page, with one connection only at a time. To view or modify the Spotlight license after you have installed Spotlight on Oracle: 1 Select About Spotlight from the Help menu. (The Installed Products tab contains license details for all installed Spotlight products.) 2 Select View / change product license under any Spotlight on Oracle entry in the list of installed products. For more information, see Product Authorization Errors on page 77. To uninstall Spotlight on Oracle To uninstall Spotlight on Oracle, use the Add/Remove Programs option in the Windows Control Panel in the Start menu that appears when you click the Start button. (For Windows 7 and Windows Vista operating systems Programs and Features has replaced the Add/Remove Programs option.) Starting Spotlight In order to successfully connect to a database, the following steps must be completed: 1 Start the Spotlight application. Chapter 319 Getting Started with Spotlight on Oracle 2 Create a new Spotlight user, if required. 3 Create a Spotlight database connection. 4 Connect to the database. To start Spotlight on Oracle To start Spotlight on Oracle, click the Spotlight desktop icon or 1 On the Windows task bar, click Start. 2 From the Programs menu, chooseQuest Software | Spotlight | Spotlight.Creating a new Spotlight user Spotlight on Oracle users require special privileges, views, and tables so they can diagnose your Oracle databases. To grant these privileges, Spotlight invokes the Oracle User Wizard to create and update the required Spotlight users. Use the wizard to: Add new users to the database that is to be diagnosed. Convert the accounts of existing Oracle users so they can use Spotlight on Oracle. Reconfigure existing Spotlight on Oracle users to work with new versions of Spotlight on Oracle (when available). To create a Spotlight user 1 Start Spotlight on Oracle. 2 From the File menu choose Oracle User Wizard... 3 Click Next. 4 Enter the following details: Enter the SQL*Net connection string Enter the database instance on which you wish to create the user. 20Spotlight on Oracle Getting Started Guide Enter the user name of a DBA account Enter a DBA user name for that database instance (so that the Spotlight user and the relevant objects can be created). The user name requires SYSDBA access. Enter the passwordEnter the DBA user password. 5 Click Next to continue. 6 Click Create a new user and enter the following details: User name Enter a name for the new user. PasswordEnter a password for the new user. Confirm passwordConfirm the password. Select any tableSelect this to enable the user to generate explain plans. Alter systemSelect this to enable the user to trace and terminate sessions. 7 Click Next to continue. 8 Select tablespaces to use for Spotlight tablesThe default tablespace is USERS, if available. Temporary segmentsThe default temporary tablespace is TEMP, if available. 9 Click Next to continue. 10You can choose whether to employ Predictive Diagnostics on this database instance. To use Predictive Diagnostics: Select Create/Upgrade Predictive Diagnostics schema, and then click Next. Choose whether to use an existing Predictive Diagnostics schema (recommended) or install a new schema, and then click Next. Chapter 321 Getting Started with Spotlight on Oracle Choose whether to install a set of sample data (which you can use to experiment with the Predictive Diagnostics feature), and then click Next. Choose the settings that Spotlight will use to schedule the Oracle jobs that collect data on SQL performance and impending bottlenecks, and then click Next. 11Click Next when the user setup is complete. 12Click Finish to complete the task. Creating an Oracle database connection Note:Before creating the connection, ensure that you have created a Spotlight user (see page 19) that has the special privileges required to diagnose your Oracle databases. To create a database connection 1 From the File menu choose Connect... 2 Ensure Spotlight on Oracle is selected on the Connections menu. 3 Double-click the Add new connection icon. 4 A Properties window opens for the new connection. The Details page of the Properties window has several sections: Oracle connection details. ASM connection details. This section is available only for Oracle 10g systems or later. Server connection details. ESX connection details. StealthCollect connection details. This section appears only if you already have installed a Quest application (such as Performance Analysis) that supports StealthCollect. 22Spotlight on Oracle Getting Started Guide 5 Enter the following information in the Details page of the Properties window: Connection StringSelect the name of the database that you wish to connect to. By default this will be the name of the connection. Oracle UsernameEnter the Spotlight user name. Oracle PasswordEnter the Spotlight user password. Oracle HomeSelect an appropriate Oracle Home a location where Oracle client software is installed. Choose a home that contains a version of the Oracle client that is compatible with the server under diagnosis. 6 (Available only in Oracle 10g and later) Select Monitor ASM if you want to monitor the performance of Oracle's Automated Storage Management (ASM) technology. ASM is built into the Oracle database kernel; it simplifies the creation and space management in Oracle databases. Enter the following details: ASM Connection StringEnter the connection string required to link to the ASM database instance that is used for storage management for the Oracle database under investigation. The TNSNAMES entry for the ASM instance must include the option (UR = A) for the Spotlight connection to work properly. Chapter 323 Getting Started with Spotlight on Oracle ASM PasswordEnter the user password that Spotlight uses when logging on to the ASM database instance.There is NO ASM Username box for the ASM connection, as the ASM instance contains only one user SYS that can provide access to ASM-specific information. SYS access to an ASM instance does NOT create a security issue for the Oracle database but still you should limit access to the SYS account to trusted individuals. Because the SYS password is stored in encrypted format, non-privileged users can still access Spotlight on Oracle. To do so: Use a pre-configured or shared Spotlight on Oracle connection, OR Use a Spotlight connection without ASM access. Spotlight will display Oracle-related data, but the ASM drilldown will contain no data. 7 Select Monitor OS if you want to monitor the operating system on the database server. Enter the following details: OS ServerEnter the name of the server that is to be monitored. Use a period . if you wish to monitor the local database. OS TypeSelect the type of operating system to monitor. The supported types are: Unix(REXEC) Unix(SSH) Unix(SSH using Public/Private key) Windows If you select one of the Unix systems, make sure that the relevant SSH or REXEC daemon is running on the database server, and is configured to receive remote connections. 24Spotlight on Oracle Getting Started Guide SSH Port (Enabled only when you choose to connect to the database server via SSH.) Enter the port number that Spotlight will use for its SSH (secure shell) connection to the Oracle database. The default value is 22. OS UsernameType the username you use when you log onto the database server. If the username and password if required are not completed, you will not be able to view the operating system performance statistics for the database server. (Windows) If your login details have been assigned remote access to the registry of the database server, you do not need to complete the OS Username and OS Password fields.(If your login details have NOT been set up on the database server, you must enter a username and password that has access to the registry of the server.) SSH Key Type (Enabled only when you connect to the database server via Unix(SSH using Public/Pri vate key). Choose the type of key to use when making the Spotlight connection. Public-key encryption is supported under SSH2 only. The current options are RSA and DSA. For more information, see Public/Pri vate keypairs in the online help. SSH Private Key Filename (See SSH Key Type above.) Type the location of the file that contains the private key for the Spotlight connection, or click the "..." button to locate it. SSH Passphrase (See SSH Key Type above.) Type the passphrase used to decrypt the private key. OS Password (Enabled when you connect to the database server via Unix(REXEC), Unix(SSH), or Windows). Enter the user password (if required). Chapter 325 Getting Started with Spotlight on Oracle 8 Select the option Connect to OS Spotlight when starting this connection to open the relevant operating-system (OS) Spotlight automatically when you open your Spotlight on Oracle connection.9 Click the Monitor ESX option if you want to monitor the virtual machine that hosts the database. This option is enabled when Monitor OS is checked. Enter the following details. ESX Web ServiceType or select the URL of the ESX Web Service that manages the virtual machine. ESX UsernameType the username required to connect to the ESX Web Service. Note: The username specified must have read only permissions for both the ESX virtual machine and the ESX host that the virtual machine resides on, or may reside on. ESX PasswordType the password required to connect the specified user to the ESX Web Service. Virtual Machine NameClick ... to select the virtual machine. 10Click the Use StealthCollect option if you wish to use StealthCollect to collect SQL statements and performance data from the target database.NoteStealthCollect is installed with Performance Analysis. The Use StealthCollect option is available while StealthCollect is installed. StealthCollect is Quest technology designed to collect SQL statements and performance indicators from Oracle databases and host operating systems without imposing overheads on the target database itself. Enter the following details: SC Port No.Enter the port number where Spotlight can access StealthCollect data. If required, click the button to enter username and password details for StealthCollect. (The default StealthCollect username and password for a new connection profile is quest.) SC Oracle SIDEnter the StealthCollect Oracle SID for the target database. 26Spotlight on Oracle Getting Started Guide 11Click Test to check that Spotlight is able to connect successfully with the details provided. Note For connections to virtual machines, testing the connection validates that Spotlight can connect to the Web Service using the specified username and password combination, and that the virtual machine exists. It does not check permissions, that the virtual machine is turned on, or that it is the same machine as the one specified in the Server Connection Details section. 12Click the Save password box to save all the password details that have been entered. 13Click OK to finish. A connection icon with the name you specified is created. Connecting to an Oracle database Notes: Before connecting to an Oracle database, ensure that you have: Created a Spotlight user (see page 19) that has the special privileges required to diagnose your Oracle database. Created a connection to the Oracle database that you want to diagnose (see page 21). To connect to an Oracle database 1 From the File menu choose Connect... 2 Double-click the connection icon that represents the connection that you want to open. Note:If you encounter problems when connecting, see Connection problems on page 78 for more information. Once you have connected Spotlight requires some additional information once you have connected. Chapter 327 Getting Started with Spotlight on Oracle You may need to set the initialization parameter timed_statistics. If you have a trial version of Spotlight, you may need to obtain a full license key to continue the full functionality of Spotlight after the initial trial period. Setting the initialization parameter In order for Spotlight to accurately monitor the database the timed_statistics parameter must be set to true in the init.ora file. If it is not set to true for that user, a message will be displayed that gives you the option to have Spotlight set the parameter for you. To set the initialization parameter to true In the Warning message click Set timed statistics now. Note:Spotlight can set the parameter only if you have granted the alter system privilege to the Spotlight user. Obtaining a full license key When installed under a trial key, Spotlight runs for a limited time with full functionality. You must register the product before the end of the trial period in order to maintain that full functionality.To do so, contact your Quest Software representative for information on obtaining a full license key. For further information see Product authorization errors on page 77. Using Toad to launch Spotlight Note: This section is relevant for systems with the following software installed: Toad DBA Suite for Oracle with Spotlight on Oracle. To connect to an Oracle database from within Toad 1 Click the Spotlight button where it appears in Toad. 2 Spotlight on Oracle opens at the Oracle database connection associated with this Toad for Oracle connection. 28Spotlight on Oracle Getting Started Guide The first time you open this Oracle database connection from within Toad the Details page opens with the Oracle database and username fields filled in as per the Toad connection. Security requires that you re-enter your Oracle password. Notes: Further information on the Details page: The default connection name is Toad followed by the name of the Oracle database. You may choose to replace this with any connection name of your choice. Note that a limitation of the integration of Toad with Spotlight on Oracle is that this will be a new Spotlight connection, ignoring all other pre-existing Spotlight connections. The Oracle username you used in Toad is filled in by default; however you may choose a different Oracle username to connect to Spotlight. For example, in an environment with multiple database administrators there may be a communal username for Spotlight connections. Simply replace the filled in Oracle username with the appropriate Oracle username and password details. You can enter additional parameters to monitor ASM, the operating system and for virtualization. See page 21, from the Details page of the Properties window, step 4. Notes: Now Spotlight on Oracle is open: The Spotlight Home Page or Drilldown will open relevant to the location in Toad from where the Spotlight button was clicked. You will be prompted to create a new spotlight user if this Oracle username has not yet been created in Spotlight. See page 19. This Spotlight connection can be found in the Spotlight Connection Manager. It can be managed the same as any other Spotlight connection. For example, you can close the connection, edit the connection details and re-open the connection in the Spotlight Connection Manager. Your pre-existing connections to the Spotlight on Oracle database When you click on the Spotlight button in Toad to connect to the Oracle database, a new Spotlight connection to the Oracle database is created, regardless of any pre-existing Spotlight connections to the Oracle database. To avoid the annoyance of setting up Spotlight connections in Spotlight that will be set up identical Toad connections later, you may choose to create all connections through Toad if Toad integration is desired. Chapter 329 Getting Started with Spotlight on Oracle Installing and using DTrace within Spotlight DTrace is a dynamic analysis tool for Sun Microsystems operating systems from Solaris 10 onwards. You can enable and use Spotlight's DTrace drilldown to capture details about the interaction between Oracle and Solaris that would otherwise be unavailable. To use DTrace within Spotlight, you need to: 1 Install and configure a number of DTrace scripts on the server under investigation. 2 Enable DTrace functionality within Spotlight. 3 Use the DTrace drilldown to view the DTrace analysis. To install and configure DTrace scripts 1 Locate the SoO_DTrace_scripts.tar archive package in the Plug-ins\SoO sub-folder of the Spotlight installation folder.2 Unpack the contents of the archive to a directory on the database server. Ensure that the unpacked files have execute permission. 3 In the .profile of the account that will be used to run O/S commands from Spotlight, define a QUEST_SOO_HOME variable that points to that directory on the server.4 Provide that user with DTrace authority by adding the following line to /etc/user_attr: : : : : def aul t pr i v=basi c, dt r ace_pr oc, dt r ace_ker nelTo enable DTrace within Spotlight 1 From the View | Options menu, choose Spotlight on Oracle. 2 Click the DTrace button. 3 Select Show Solaris DTrace drilldowns. 4 Click OK. The DTrace icon will appear on the Spotlight toolbar, and a DTrace option will appear on the ViewGo To menu. 30Spotlight on Oracle Getting Started Guide To view the pages in the DTrace drilldown On the Spotlight toolbar, click the Solaris DTrace button, or From the View | Go To menu, choose DTrace. 4Using Spotlight on Oracle When you use Spotlight on Oracle to diagnose an Oracle database, you can: View the status of the connected database in the panels and dataflows of the Spotlight home page. View the alarms that Spotlight has raised for the connected database. View the detailed performance of the connected database in the various Spotlight drilldowns. Use Spotlight menus and shortcuts to access a number of other Quest applications for viewing detailed information the specified Oracle database. The Spotlight on Oracle home page The Spotlight home page shows an overview of the activity in an Oracle database. The display is based on the Oracle architecture diagram shown in An overview of the Oracle architecture on page 7. 32Spotlight on Oracle Getting Started Guide Panels A panel is a visual feature that groups related information and components on the Spotlight home page. Components change color as alarms are raised for the metrics they contain.There are six main panels on the home page for Spotlight on Oracle. They are: The Sessions panel. This indicates the status of the Oracle communications between client and server. The Host panel. This indicates CPU usage, Virtualization Overhead and free physical RAM information. The Server Processes panel. This shows the status of Oracle server processes. These processes perform database activities on behalf of end users, and mediate database connections. Chapter 433 Using Spotlight on Oracle The SGA panel. This shows details of specific memory areas within the system global area (SGA). The SGA is an area of shared or common process memory that is used to cache frequently used data, SQL statements, procedures and other structures. The Background Processes panel. This displays the following key Oracle background processes: Database Writer (DBWR), Redo Log Writer (LGWR), Recovery Writer (RVWR) and Archiver (ARCH). It also contains a status indicator for Spotlight's Predictive Diagnostics capability (if available). The Disk Storage panel. This represents Oracle database and log files on disk. The panels, and the information they display, are described in more detail below. For a full discussion of the panels, see the Spotlight online help. Note:To view a tool tip for a specific component in a panel, hover the mouse pointer over the component. Sessions panel The Sessions panel indicates the status of the Oracle communications between client and server. Information displayed here includes: The time (in milliseconds) that it takes for the query select user from dual to be submitted, executed and returned. The total number of users connected to the database. The number of users who are currently active. The sum of DB time over all sessions, divided by elapsed time. Note: The Average Active Sessions gauge is not available for Oracle 9i. The flows between the Sessions panel and the Server Processes panel represent the rate of data transfer in both directions across the SQL*NET interface. 34Spotlight on Oracle Getting Started Guide Host panel The Host panel shows metrics for the operating system. The information displayed here includes: The total amount of CPU being used. The number of tasks waiting for CPU resources. The percentage of CPU that is unavailable to the virtual machine (if applicable). The available physical RAM. The amount of memory that has been reclaimed from the virtual machine by ESX (if applicable). Chapter 435 Using Spotlight on Oracle Server Processes panel The Server Processes panel indicates the status of Oracle server processes that perform database activities on behalf of end users, and mediate database connections. Information displayed here includes: Statistics for the PGA memory allocated by server processes. Dedicated server processes that perform work on behalf of a single client process. Shared or multi-threaded servers (MTS) that perform work on behalf of more than one client process. MTS dispatchers that coordinate the allocation of shared servers to client tasks. Server processes that support parallel execution of queries. The current length of the Parallel Execution Queue.Note: The Parallel Exec Queue gauge applies only to Oracle 11gR2. J ob queue server processes that run PL/SQL commands submitted to the Oracle job queue via the DBMS_JOB package. The state of the EM agent connection between Oracle Enterprise Manager (if installed) and the database. The number of server processes that are currently active. The flow to the Server Processes panel from the Disk Storage panel represents the rate of direct physical reads from disk. The flows between the Server Processes panel and the SGA panel represent the rates of: Blocks reads from the SGA by all server processes. Block changes in the SGA by all server processes. Redo buffer entries made by all server processes. SQL parse requests per second by all server processes. SQL execution requests per second by all server processes. 36Spotlight on Oracle Getting Started Guide SGA panel The SGA panel shows details of specific memory areas within the system global area (SGA). The SGA is an area of shared or common process memory that is used to cache frequently used data, SQL statements, procedures and other structures. Information displayed here includes: The size of the buffer cache and its components. These hold data that is accessed frequently. The current size of the SGA. The size of the J ava Pool, which caches class definitions, J ava methods and J ava objects. The size of the Large Pool allocation heap, which is used in MTS for session memory. It can be used by parallel execution and backup processes. The size of the redo buffer. This contains redo entries that are eventually written to the redo log. The size of the Shared Pool, which caches SQL statements, PL/SQL programs, object definitions and session memory for MTS sessions. The size and activity of the result cache (Oracle 11g or later). Whether the database is in manual or auto memory management mode (Oracle 10g or later). The flow between the SGA panel and the Disk Storage panel represents the rate of block reads from disk. The flows between the SGA panel and the Background Processes panel represent the rates of: Block writes by the Database writer (DBWR) process. I/O writes by the Recovery writer (DBWR) process. Redo log block writes by the Redo Log Writer (LGWR) process. Chapter 437 Using Spotlight on Oracle Background Processes panel The Background Processes panel displays the following key Oracle background processes: Database Writer (DBWRn). The Recovery Writer (RVWRn). The Log Writer (LGWRn). The Archiver (ARCHn). Note: The n value in each process icon indicates the number of processes there are of that type. The flows between the Background Processes panel and the Disk Storage panel represent: The rate at which modified blocks are written from the SGA to disk by the DBWR processes. The rate of I/O write operations to the flash recovery area by the RVWR processes. The rate at which redo log entries are written to the redo log files by the LGWR processes. The rate at which the archiver copies from redo log files to archived logs. The Background Processes panel also contains a status indicator for Spotlight's Predictive Diagnostics capability (if installed). This feature helps Spotlight to use collected data to predict database performance. 38Spotlight on Oracle Getting Started Guide Disk Storage panel The Disk Storage panel represents Oracle database and log files on disk, including: Oracle database files. Size of the flash cache. Oracle flashback recovery files. Oracle redo logs. The archive log destinations. Chapter 439 Using Spotlight on Oracle Spotlight on Oracle alarms Spotlight alerts you to problems with your database by issuing an alarm. When an alarm is triggered, Spotlight performs one or more of the following actions: Changes in color or intensity for relevant components. Audible warnings. E-mail notifications. With Spotlight alarms you can: View details of the metric(s) that caused the alarm in a relevant drilldown page. (For more information on drilldowns, see Spotlight on Oracle drilldowns on page 41.) View the help associated with each alarm. View details of the alarm itself in the Alarm Log drilldown. Change the actions that follow the raising of alarms. Enable, disable, or snooze alarms. Filter alarms. Change the thresholds and severities that define alarms. Change alarm notifications. Save alarm details. Note:Detailed information on all these actions can be found in the Spotlight online help. The following are brief descriptions of the alarms specifically associated with Spotlight on Oracle. For possible solutions for the problems indicated by these alarms, or for information on how to diagnose problems further, see Spotlight on Oracle alarms in the online help. Archive Critical Failure alarm An Archive Critical Failure alarm indicates that the number of working archive destinations for the database instance is about to fall below the minimum number needed to archive Oracle redo logs successfully. When you see this alarm, you should make space in the archive log destination by purging or by backing up to an offline medium any archived files that are not immediately needed. 40Spotlight on Oracle Getting Started Guide Archive Destination Failed alarm Spotlight raises an Archive Destination Failed alarm when at least one of the archive destinations specified for redo log files has become full. See Archive Destination Failure alarm below. Archive Destination Failure alarm An Archive Destination Failure alarm when at least one of the archive destinations specified for redo log files is about to become full. When you see this alarm, you should make space in the archive log destination by purging it, or by backing it up to an offline medium. Array Fetch Size alarm When more than one row is retrieved from an Oracle database in a single SELECT (an array fetch), the amount of network traffic associated with SELECT operations is radically reduced. The Array Fetch Size alarm is raised when the array fetch size for the database may be too low. ASM Balance alarm This alarm indicates that there is unbalanced access to one or more of the disks in the ASM (Automatic Storage Management) disk group that has been allocated to the Oracle instance. ASM Service Time alarm This alarm indicates that the average time taken to perform I/O on the ASM instance has exceeded a specified threshold value. (Service time represents the response time of a disk plus any wait time experienced during disk operations.) ASM Status alarm When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The ASM Status alarm indicates that Spotlight is re-establishing its connection to the ASM diskgroup for the Oracle instance. Average Time to Sync a Redo Log Entry alarm The Average Time to Sync a Redo Log Entry alarm (formerly the Average Redo Write Time alarm) is activated when the time taken to write redo log entries exceeds a threshold. Unlike most other Oracle write I/Os, Oracle sessions must wait for redo log writes to complete before they can continue processing. Chapter 441 Using Spotlight on Oracle Buffer Busy Wait alarm The Buffer Busy Wait alarm occurs when a session cannot access a block because it is in use by another session. The two most common causes are insufficient free lists for a table or insufficient rollback segments. Cache Buffer Chains Latch Wait alarm The Cache Buffer Chains Latch Wait alarm occurs because contention for the cache buffer chains latch exceeds a threshold. Cache Buffer LRU Chains Latch Wait alarm The Cache Buffer LRU Chains Latch Wait alarm occurs because contention for the cache buffer chains latch exceeds a threshold. Continued Fetch Rate alarm Oracle sometimes needs to migrate a row in a database from its original block to a new block, or needs to chain an updated row across multiple blocks. These migrated and chained rows cause additional I/O, and the Continued Fetch alarm occurs when the number of these "continued row" fetches exceeds a threshold. CPU Busy alarm The CPU Busy alarm occurs when the total CPU utilization of the system exceeds a threshold. CPU Virtualization Overhead alarm The CPU Virtualization Overhead alarm is activated when Virtualization Overhead is greater than a threshold and CPU Usage is 95% or higher. Datafile Read Time alarm The Datafile Read Time alarm occurs if the average time for a random datafile read exceeds a given threshold. Dispatchers Busy alarm The Dispatchers Busy alarm occurs when all or most dispatcher processes are busy. In an MTS environment, multiple sessions share a smaller number of shared server processes. The servers are usually allocated by a smaller number of dispatcher processes. 42Spotlight on Oracle Getting Started Guide EM Agent Not Connected alarm Oracle Enterprise Manager (where installed) manages Oracle components and administers Oracle databases. OEM communicates with the databases it administers via a set of EM agent processes. The EM Agent alarm indicates that OEM is installed and active, but not connected to the database. Estimated Shared Pool Size alarm This alarm occurs when the Shared Pool is set too small. A possible cause is the use of literals rather than bind variables in SQL statements. You can also increase the size of the Shared Pool. Excessive RBS Activities alarm The Excessive RBS Activities alarm occurs when there are high rollback extent/shrink/wrap activities. Flashback Buffer Wait Percentage alarm This alarm indicates that the percentage of available time spent waiting for the Recovery Writer (RVWR) to write data to the flashback logs has exceeded a specified threshold. Flash Cache Insert Skip: DBWR Overloaded alarm The Flash Cache Insert Skip: DBWR Overloaded alarm is raised when the ratio of insert skips from the DBWR being overloaded exceeds a threshold. Flash Cache Read Latency The Flash Cache Read Latency alarm is raised when the time taken to read a single bock from flash cache exceeds a threshold. This may be caused by a low performance flash device. Flash Cache Write Complete Waits The Flash Cache Write Complete Waits alarm is raised when the ratio of 'write complete waits: flash cache' to DB time exceeds a threshold. Flashback Buffer Wait alarm The Flashback Buffer Wait alarm indicates that the percentage of available time spent waiting for the Recovery Writer (RVWR) to write data to the flashback logs has exceeded a specified threshold. Chapter 443 Using Spotlight on Oracle Flashback Space Usage alarm This alarm is raised when the amount of used space in the Flashback Recovery Area exceeds a threshold. Free Buffer Waits alarm The Free Buffer Waits alarm occurs when free buffer waits (as a proportion of total waits) exceeds a threshold.In Memory Undo Latch Wait alarm The in memory undo latch is associated with Oracle's relatively new in memory undo (IMU) structures. These structures hold in memory information formerly maintained in rollback (undo) segments. Instance Failed alarm Spotlight raises an Instance Failed alarm when a specified instance has failed because of an Archiver error. See Archive Critical Failure alarm above. Insufficient Flashback Logs alarm This alarm fires when the flashback logs in the flashback area do not cover the amount of time specified in the parameter DB_FLASHBACK_RETENTION_TARGET. Job Processes Busy alarm The Job Processes Busy alarm occurs when all or most job queue processes are busy. Large Pool Full alarm This alarm fires when the Large Pool allocation heap is full. The Large Pool is used in MTS for session memory, and can be used by parallel execution and backup processes. Latch Free Waits alarm If a session needs to acquire a latch that is held by another session, a latch free wait may occur. The presence of latch free waits may indicate a bottleneck within the SGA. (The specific action depends on the latch.) 44Spotlight on Oracle Getting Started Guide Library Cache Latches alarms These alarm fire when the time spent waiting on library cache latches or library cache pin latches exceeds a threshold. Library cache latches (or mutexes in 10g R2 and above) protect the cached SQL statements and object definitions held in the library cache within the Shared Pool. Library Cache Miss Rate alarm The Library Cache Miss Rate alarm occurs when the library cache hit ratio falls below a given threshold. Lock Wait alarm The Lock Wait alarm occurs when the proportion of time sessions spend waiting for locks exceeds a threshold. Log Buffer Space Wait alarm The Log Buffer Space Wait alarm occurs if the amount of time spent waiting for space in the redo log buffer exceeds a threshold. Log Switch Time alarm The Log Switch Time alarm occurs when the amount of time spent waiting for log switch events exceeds a threshold. Low Free Physical RAM alarm The Low Free Physical RAM alarm occurs when the servers available RAM is low. Maximum Queued Time alarm The Maximum Queued Time alarm becomes active when a session has been in the parallel execution queue for longer than the specified threshold. Note: Applies only to Oracle 11gR2. Multi-threaded Server alarm The Multi-threaded Server alarm occurs when all or most multi-threaded server (MTS) or dispatcher processes are busy. Chapter 445 Using Spotlight on Oracle Non-reclaimable Flashback Space alarm This alarm is raised when the amount of non-reclaimable used space in the Recovery Area (flashback area) exceeds a threshold. Non-reclaimable files include archived redo logs, which must be backed up before being deleted. Oracle Memory on Disk alarm The Oracle Memory on Disk alarm is activated when the amount of guest physical memory mapped to machine memory is less than the current combined size of Oracle's SGA and PGA. Oracle Parameters for Predictive Diagnostics alarm The Predictive Diagnostics feature cannot predict the behavior of Oracle databases without the access to Oracle data that is granted by several Oracle parameters. Oracle Status alarm When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The Oracle Status alarm indicates that Spotlight is re-establishing its connection to the Oracle instance. Parallel Query Server alarms The Parallel Query Server alarm occurs when all or most parallel servers are busy. Parallel server processes perform work to resolve SQL statements that are being executed in parallel. If all parallel server processes are busy, parallel SQL may be downgraded, or even executed serially, thus increasing its execution time. Spotlight can raise these Parallel Query Server alarms: Parallel Query Servers Busy, Parallel Queries Downgraded, and Parallel Queries Serialized. Parse Ratio alarm The Parse Ratio alarm is raised when the ratio of parse calls to execute calls exceeds a threshold. NoteThis alarm is initially disabled; see Total Parse Wait Time Ratio alarm. Predictive Diagnostics alarms A Predictive Diagnostics alarm (SQL Collector or Bottleneck Collector alarm) is raised when the appropriate collection of predictive data has not been scheduled. 46Spotlight on Oracle Getting Started Guide Predictive Diagnostics Custom SQL Error alarm This alarm indicates that the customized SQL query use to represent database concurrency within Predictive Diagnostics has returned an error. Redo Allocation and Redo Copy Latch Wait alarms The Redo Allocation and Redo Copy Latch Wait alarms are raised when there is contention for the redo allocation latch and redo copy latch. Result Cache Find/Create Ratio alarm When Oracle executes SQL queries, the result cache stores the result sets of specified queries in shared memory for later use. The Find/Create Ratio alarm occurs when the rate of find operations on the result cache drops below the rate of create operations, and indicates that the contents of the result cache are under-used. Result Cache Latch Sleep Ratio alarm Spotlight raises the Result Cache Latch Sleep Ratio alarm when the rate of latch sleeps for the Result Cache latch indicates that there is contention on the Result Cache latch, where multiple processes are competing to use the latch and are causing a significant degradation in response time. Row Cache objects Latch Wait The row cache objects latch protects the contents of the row cache; the section of the shared pool to cache dictionary information. Sequential Read Time alarm This alarm fires when the average time that Oracle takes to read a single block of data typically an index-based lookup is significantly higher than the average time recorded during calibration. Sessions Queued alarm The Sessions Queued alarm becomes active when the number of sessions in the parallel execution queue exceeds a threshold. Note Applies only to Oracle 11gR2. Chapter 447 Using Spotlight on Oracle Shared Pool Latch Wait alarm Shared pool latches control access to the shared pool memory map. Shared Pool Lock Percentage alarm This alarm is raised when one user session holds a lock on object(s) in the Shared Pool for a prolonged period, and causes other sessions to wait on the same resource. This "blocking lock" happens rarely. When it does occur, use a SQL query to find the affected sessions. Shared Server Converted into Dedicated Server alarm Spotlight raises an alarm when many shared servers are converted, leaving too few shared servers to meet the demand. Converted shared servers will have a status of WAIT(RECEIVE) in v$shared_server. Temporary IO Wait Ratio alarm Spotlight raises the Temporary IO Wait Ratio alarm when the time spent in temporary IO waits is too high as a percentage of the time spent performing user calls on the Oracle database. Total Parse Wait Time Ratio alarm Spotlight raises this alarm when the amount of time spent parsing SQL statements as a percentage of the active database time exceeds a level that may cause latch contention on the shared pool and library cache latches. Note:The Total Parse Wait Time Ratio alarm is available for Oracle 10.2 and later. When monitoring earlier Oracle systems, consider reactivating the (disabled) Parse Ratio alarm. Unarchived Logs alarm The Unarchived Logs alarm occurs when the number of unarchived redo logs exceeds a threshold. If all online logs need archiving, database update activity can be suspended while the redo log writer waits for the next log to be archived. Unix Status alarm When a Spotlight connection is lost, Spotlight automatically attempts to re-establish the connection. Where the Oracle database is hosted on a Unix server, this alarm indicates that Spotlight is re-establishing its connection to the Unix server. 48Spotlight on Oracle Getting Started Guide WMI Status alarm When a Spotlight connection is lost, Spotlight automatically attempts to re-establish the connection. Where the Oracle database is hosted on a Windows server, this alarm indicates that Spotlight is re-establishing its WMI connection to the Windows server. Write Complete Wait alarm Spotlight raises the a Write Complete Wait alarm when write complete waits (as a proportion of total waits) exceed a threshold. Write complete waits occur when a session tries to modify a block that is currently being written to disk by the database writer process. Spotlight on Oracle drilldowns Spotlight on Oracle makes it possible for you to view your system at different levels of detail. At its highest level, which is the Spotlight home page, Spotlight displays a visual representation of the status of the major components in the current system. The detail at this level is designed to help you locate and identify bottlenecks quickly. When you have isolated a problem, you can display a drilldown page, whose charts and tables provide a detailed breakdown of the underlying statistics. To open a Spotlight drilldown To access a specified drilldown within Spotlight: Click the relevant drilldown button on the Spotlight toolbar OR Choose the relevant drilldown from the View | Go To menu OR Right-click on a component on the Spotlight home page, and choose an item from the shortcut menu's Show Details sub-menu OR Click on a component on the Spotlight home pageOR Use a keyboard shortcut. Chapter 449 Using Spotlight on Oracle List of drilldowns in Spotlight on Oracle Spotlight on Oracle contains the following drilldowns. DrilldownDescription SessionsThe Sessions drilldown displays information about the users connected to the Oracle database. The pages and sub-pages in the drilldown display information on: All users connected to the Oracle database. Session details for those users, including waits, SQL statements executed, locks, resource usage, and Oracle statistics. Parallel queries executed for all users connected to the Oracle database. Oracle jobs currently scheduled for the database. SQL & Application Workload The SQL & Application Workload drilldown allows you to select, sort and display SQL statements executed according to specified criteria. The drilldown includes: The Top SQL page, which displays the SQL statements that consume the most system resources for the current connection. The Array Fetch page, which displays information on array processing for the Oracle database under investigation. The Parse Activity page, which displays information on SQL statements associated with high parse rates (Oracle 10.2 and later). The Sort Activity page, which displays information on SQL statements associated with high sort activity (Oracle 10.2 and later). The Analyze Trace page (formerly a separate drilldown), which displays the contents of trace files that store information on how Oracle processes the execution of SQL statements, and on the resources it uses to do so (Oracle 10.2 and later). The Result Cache page (Oracle 11g and later) 50Spotlight on Oracle Getting Started Guide DrilldownDescription shows information about the result cache, which stores the result sets of specified queries so they can be re-used. ActivityThe Activity drilldown displays the activity on the database. The pages and sub-pages in the drilldown display information on waits, waiting events, blocking locks, lock activity, latches, query servers, transactions, rollbacks, and buffer busy waits. I/OThe I/O drilldown displays the input and output activity for the database. The pages and sub-pages in the drilldown display information on I/O by category, session, SQL, tablespaces, datafiles, segments and row chaining. Configuration & Memory The Configuration and Memory drilldown shows the memory utilization for the Shared Pool and the SGA, as well as shared pool and buffer cache hit ratios, Oracle alert log and Oracle parameters. Operating SystemThe Operating System drilldown provides information about the performance, processes, and storage on the database machine under investigation.Disk StorageThe Disk Storage drilldown displays information about online redo logs, tablespace utilization, archive destinations, and Flashback recovery. TuningYou can use the Tuning drilldown to view information about Oracle memory resources in the Oracle SGA and PGA (System and Program Global Areas), to tune memory resources where feasible, and to view and tune latch activity for systems with multiple CPUs. ASM (If available) If your system uses Automatic Storage Management (ASM) to manage disk storage, you can use the ASM drilldown within Spotlight on Oracle to diagnose ASM performance. ASM is a storage management solution from Oracle Corporation that automatically reorganizes data based on statistics collected from usage and user access patterns over time.DTraceDTrace is a dynamic analysis tool that can be used on Chapter 451 Using Spotlight on Oracle DrilldownDescription (If available)Sun platforms from Solaris 10 onwards. Spotlight on Oracle uses DTrace to capture details of the interaction between Oracle and the operating system when Oracle uses the O/S services provided by Solaris.For detailed instructions on displaying and using the DTrace drilldown, see Installing and using DTrace within Spotlight on page 27 Predictive Diagnostics (If available) Use the Predictive Diagnostics home page to identify: SQL statements ("degrading SQL") whose performance may not scale adequately in the future as data volumes and SQL execution rates increase. Waiting events ("bottlenecks") that may in the future affect database throughput and response time. Database resources (CPU, memory, and disk I/O) whose limitations may in the future affect database performance. When you start using Predictive Diagnostics, Spotlight will take about 10 days to collect enough data to make valid predictions on the future performance of the database. For more information Predictive Diagnostics, refer to the online help under Spotlight on Oracle (Professional)Predictive Diagnostics. VirtualizationUse the Virtual Machine page to see detailed information about the CPU and memory use of a virtual machine. Alarm LogThe pages in the Alarm Log drilldown display information on the alarms associated with the current Spotlight connection.You can filter the Alarm Log to display only the alarms that meet specified criteria. 52Spotlight on Oracle Getting Started Guide Spotlight menus The Spotlight menus are common to all editions of Spotlight and are explained in the online help under Spotlight Basics. The exceptions to this within Spotlight on Oracle are: The File menu, whose Oracle User Wizard option provides the ability to add new Spotlight on Oracle users, and to convert the accounts of existing Oracle users so they can use Spotlight on Oracle. The Tools menu, which contains links to tools that may help you solve the performance issues that Spotlight highlights. Tools menu options Explain Plan:The Explain Plan allows you to determine the execution plan Oracle applies to a particular SQL statement. Spotlight allows you to view graphical representations of the execution plan for SQL statements being executed by a user or those identified by the Top SQL drilldown. SQL Optimizer:SQL Optimizer (if installed) provides context-sensitive tuning advice for SQL statements based on the Oracle execution plan and the database structure. Note:You must have DBA privileges for the database you are diagnosing. Space Manager:Space Manager (if installed) provides a comprehensive solution for space management and reorganization. Space Manager offers preventive maintenance, problem detection and resolution and capacity planning across any number of databases. You can also launch Space Manager by right-clicking on the Disk Storage icon on the main page or by right-clicking the tables in the Disk Storage drilldown Note:You must have DBA privileges for the database you are diagnosing. Using Spotlight to tune your database Spotlight supports an approach to Oracle performance tuning that could be described as tuning by bottleneck. Spotlight alerts you if any component of the Oracle architecture is forming a bottleneck. Additionally, Spotlight provides tools that allow you to identify and rectify any inefficiency in your databases configuration. Chapter 453 Using Spotlight on Oracle In general, you tune your Oracle instance by reiteratively identifying bottlenecks, contention and critical resources, and by using Spotlight's online advice to remove the bottleneck or contention to improve the performance of the resource. The following is a summary of steps that you can undertake to tune Oracle using Spotlight. Use the Spotlight home page (see page 31) to alert you to any obvious bottlenecks. If Spotlight detects that some component of Oracle constitutes a performance bottleneck, the corresponding component generates a visual or auditory alarm. The exact appearance of the alarm depends on how you have configured alarm severities. When an alarm is current you can press F1 to display help for the alarm. Clicking the object (in the main window) displays the relevant drilldown. Use the Activity drilldown (see page 53) to display overall efficiency and resource usage. In particular, the Event waits chart shows the amount of time database sessions are spending waiting for various resources, as well as the amount of CPU being utilized. Tuning efforts are most likely to succeed if they are concentrated on the resources being used most heavily. The Wait activity topic contains a discussion of the meaning and implications of various wait events. Other pages help you examine detailed activity. For instance, the Top Sessions drilldown shows the users who contribute most to database activity. You can display various details for these sessions, including the SQL statements, locks and resource usage. For further information on tuning your database, see the online help. Using Spotlight to predict database performance Use Predictive Diagnostics to predict the future performance of a database (and identify how to improve that performance) by: Identifying SQL statements whose performance may not scale adequately as data volumes and SQL execution rates increase. Identifying bottlenecks that may in the future affect database throughput and response time. 54Spotlight on Oracle Getting Started Guide Predicting when database resources such as CPU, memory and disk I/O are expected to reach their full capacity, and recommending strategies to enable the database to handle high loads at those times. Access these features through the Predictive Diagnostics home page. For information on the storage requirements of Predictive Diagnostics, see Storage requirements for Predictive Diagnostics. The page contains several sections What To Do Next, Find Impending Problems, Performance Modeling, and Configuration and a set of images and hyperlinks that indicate a basic workflow through the predictive process. How to use Predictive Diagnostics 1 Open the Predictive Diagnostics home page in one of several ways. On the toolbarClick Predictive Diagnostics. Chapter 455 Using Spotlight on Oracle Keyboard shortcutPress CTRL+G. From the View | Go To menu Select Predictive Diagnostics. From the Spotlight on Oracle home page Click Predictive Diagnostics on the Background Processes panel, and in the help window click Show me... Predictive Diagnostics. 2 Click a hyperlink or image in the Predictive Diagnostics home page to open the corresponding diagnostic page. Although you can use any feature in Predictive Diagnostics at any time, we recommend the following procedure for first-time users. Follow the suggestions in What To Do Next. The quality of Predictive Diagnostics predictions depends on the quality of the data on which they are based. What To Do Next tracks the current status of the data and indicates when that data is incomplete or outdated. Wait until enough real data is available. This usually occurs about 10 days after you begin monitoring the database. Before that time, become familiar with how Predictive Diagnostics works by using the supplied sample data. Improve how the database is used now. Employ the Identify Degrading SQL and Identify Database Bottlenecks features to optimize data processing on the host system. Note: Any changes you make MAY cause the data you collected earlier to become outdated. If so, you may need to collect new data for another 10 days. 56Spotlight on Oracle Getting Started Guide Use the Configuration pages to identify the resources available to the database. Set parameters for the CPU, memory and disk I/O resources available to the database.In particular, use Peak Processing Periods to identify WHEN database resources are in greatest demand. This will improve the precision of your predictions. The default settings we provide for these parameters may not be appropriate for your database. Use the Performance Modeling pages. Predict when and why database resources will run out, what will cause performance to suffer, and what you need to do to counter those problems. Use Predictive Diagnostics with other Spotlight features When Predictive Diagnostics identifies a SQL statement that may cause problems in the future, you can: Open the Top SQL drilldown to view further details about the SQL statement. Open Explain Plan to view further information about the statement's execution plan. Open SQL Optimizer to view context-sensitive tuning advice for the SQL statement based on the Oracle execution plan and the structure of the database. When Predictive Diagnostics identifies a bottleneck that may cause problems in the future, you can: Open the Waiting Events page in the Activity drilldown to view further details about the current behavior of potential bottlenecks. When Predictive Diagnostics identifies a database resource whose limitations may cause problems in the future, you can: Open the I/O, Configuration & Memory, and Tuning drilldowns to view further details about the current behavior of CPU, memory, and disk I/O resources. For detailed information on Predictive Diagnostics, see the Spotlight online help under Spotlight on Oracle (Professional)Predictive Diagnostics. Chapter 457 Using Spotlight on Oracle Storage requirements for Predictive Diagnostics The disk storage requirements for Predictive Diagnostics schemas comprise: For sample dataThe Predictive Diagnostics sample data requires approximately 70 MB of storage space. For SQL statements A fixed overhead of approximately 10 MB per schema. An additional requirement of approximately 1 KB per snapshot for each SQL statement. The number of individual snapshots collected depend on the application workload and size of the shared pool. The storage requirements also depend on the collection interval for predictive snapshots, and the length of time that each snapshot is retained. Example If there is a fixed overhead of 10 MB, and there are 500 eligible SQL statements per snapshot, and snapshots are collected twice a day, and each snapshot is kept for one year, the disk storage requirement for SQL statements is: 10MB + (500 x .001MB x 2 x 365) = 375 MB For impending bottlenecks Disk storage requirements depend on how frequently data is collected, and the length of time that snapshots are retained. Examples If data is collected every 10 minutes and retained for six months, the disk storage requirement for impending bottlenecks is approximately 200 MB. If data is collected every 30 minutes and retained for six months, the disk storage requirement for impending bottlenecks is approximately 65 MB. If data is collected every 60 minutes and retained for six months, the disk storage requirement for impending bottlenecks is approximately 35 MB. 58Spotlight on Oracle Getting Started Guide For performance modeling Disk storage requirements depend on how frequently data is collected, and the length of time that performance data is retained. Example If data is collected every 10 minutes and retained for one year, the disk storage requirement for performance modeling is approximately 130 MB. NoteParameters set in Predictive Diagnostics Options and the SQL Statement Trend Analysis wizard also affect the storage requirements for Predictive Diagnostics data. See the online help more for more information. 5Spotlight on Oracle RAC Spotlight on Oracle RAC is an optional add-on to Spotlight on Oracle. Spotlight on Oracle RAC is installed automatically on your system via the Spotlight on Oracle or TOAD DBA Suite installer. Note:The detailed procedures for installing and authorizing Spotlight are covered in Chapter 3, Getting Started with Spotlight on Oracle. Use Spotlight on Oracle RAC to diagnose Oracle databases on multiple host machines under the Real Application Clusters (RAC) architecture. When monitoring the behavior of an Oracle RAC cluster, a database administrator needs to know: 1 How is each of the instances in my cluster performing? 2 How is the database as a whole performing? 3 Is my database experiencing cluster-related contention? To further investigate the behavior of individual database instances within the cluster, the database administrator can use Spotlight on Oracle RAC to launch Spotlight on Oracle on any particular instance within a cluster. For more information, see Viewing Oracle RAC instances with Spotlight on Oracle on page 75. What is Oracle RAC? Real Application Clusters (RAC) is a clustering technology introduced by Oracle in release 9i. It supersedes Oracles previous clustering technology Oracle Parallel Server (OPS). Oracle RAC is a shared disk clustering solution that is significantly different from the architectures offered by Microsoft SQL Server and IBM DB2. Each node in an 60Spotlight on Oracle Getting Started Guide Oracle RAC cluster has equal access to all of the database data through a shared disk subsystem. Data is neither partitioned to specific nodes, nor replicated across nodes. A high-speed network interconnect allows each node to keep its in-memory view of data consistent.An Oracle RAC configuration has the following architectural characteristics:1 Many instances of Oracle run across several nodes. 2 Many instances share a single physical copy of a shared Oracle database. 3 All instances have common data and control files. 4 Every instance has individual redo logs and undo segments. 5 Every instance can simultaneously execute transactions against the same database. The following diagram illustrates the basic components of an Oracle RAC cluster. A user should find an Oracle RAC database operationally identical to a database hosted on a single server. From top to bottom, the important features represented are: Chapter 561 Spotlight on Oracle RAC Cluster interconnect:A high-speed, high-bandwidth communication facility that connects the nodes in an Oracle RAC cluster. Multiple nodes/instances:Oracle instances run on the nodes (host machines) in the Oracle RAC cluster. Each instance comprises an Oracle System Global Area (SGA) plus the corresponding Oracle background processes that retain and process Oracle database requests. Shared disk subsystem:Database files in Oracle RAC systems are stored on multiple disks that are shared by all the nodes in the cluster, and all nodes must be able read and write to those disks. Installing and authorizing Spotlight on Oracle RAC Spotlight on Oracle RAC is installed automatically on your system via the Spotlight on Oracle or TOAD DBA Suite installer. The detailed procedures for installing and authorizing Spotlight are covered in Chapter 3, Getting Started with Spotlight on Oracle. Note:Before installing Spotlight, you should review the normal installation requirements for the products, which include permissions required when installing and running Spotlight, and those required to authorize the Spotlight license. Starting Spotlight on Oracle RAC As with the standard version of Spotlight on Oracle, users of Spotlight on Oracle RAC must complete the following steps in order to connect successfully to a database: 1 Start the Spotlight application. 2 Create a new Spotlight user, if required. 3 Create a Spotlight database connection. 4 Connect to the database. 62Spotlight on Oracle Getting Started Guide The procedure for starting Spotlight on Oracle RAC is the same as that for the standard version of Spotlight on Oracle. This is covered in Chapter 3, Getting Started with Spotlight on Oracle. Creating a new Spotlight user As with the standard version of Spotlight on Oracle, users of Spotlight on Oracle RAC require special privileges, views, and tables so they can diagnose your Oracle RAC databases. These privileges are the same as those required by Spotlight on Oracle, and so Spotlight invokes the Oracle User Wizard to create and update the required Spotlight users. The detailed procedure is covered in Chapter 3, Getting Started with Spotlight on Oracle. Creating a connection to an Oracle RAC database The procedure for creating a connection in Spotlight on Oracle RAC is similar (but not identical) to the equivalent procedure in the standard version of Spotlight on Oracle. It is described in detail below. Note:Before creating the connection, ensure that you have created a Spotlight user (see page 19) that has the special privileges required to diagnose your Oracle RAC databases. To create a database connection 1 From the File menu choose Connect... 2 Select Spotlight on Oracle RAC from the Connections menu. 3 Double-click the Add new connection icon. 4 A Properties window opens for the new connection. The Details page of the Properties window has several sections: Chapter 563 Spotlight on Oracle RAC Connection StringEnter the name (the SQL*Net alias) of the Oracle RAC database that you wish to connect to. The database name is defined within SQL*Net configuration utilities. While this can be the name of any instance in the Oracle RAC cluster, we recommend using the name of the load-balanced instance the instance whose TNSNAMES entry includes the option (LOAD_BALANCE = yes) Oracle UsernameEnter the user name that Spotlight will use when logging on to the Oracle RAC database. Oracle PasswordEnter the user password that Spotlight will use when logging on to the Oracle RAC database. Oracle Home From the Oracle Home list, select an appropriate Oracle Home a location where Oracle software is installed.Choose a home that contains a version of the Oracle client that is compatible with the server under diagnosis. 5 (Available only in Oracle 10g and later) Select Monitor ASM if you want to monitor the performance of Oracle's Automated Storage Management (ASM) technology. ASM is built into the Oracle database kernel; it simplifies the creation and space management in Oracle RAC databases. Enter the following details: ASM Connection String Enter the connection string required to link to the ASM database instance used for storage management for the Oracle RAC database under investigation. The TNSNAMES entry for the ASM instance must include the option (UR = A) for the Spotlight connection to work properly. ASM PasswordEnter the user password that Spotlight uses when logging on to the ASM database instance. There is NO ASM Username box for the ASM connection, as the ASM instance contains only 64Spotlight on Oracle Getting Started Guide one user SYS that can provide access to ASM-specific information. SYS access to an ASM instance does NOT create a security issue for the Oracle RAC database but still you should limit access to the SYS account to trusted individuals. Because the SYS password is stored in encrypted format, non-privileged users can still access Spotlight on Oracle RAC. To do so: Use a pre-configured or shared Spotlight on Oracle connection. Use a Spotlight connection without ASM access. Spotlight will display Oracle-related data, but the ASM drilldown will contain no data. 6 Select Monitor OS if you want to monitor the operating system. Enter the following details: OS TypeSelect the type of operating system to monitor. The supported types are: Unix(REXEC) Unix(SSH) Unix(SSH using Public/Private key), orWindows If you select one of the Unix systems, make sure that the relevant SSH or REXEC daemon is running on each node of the database cluster, and is configured to receive remote connections. SSH Port (Enabled only when you choose to connect to the database server via SSH.) Enter the port number that Spotlight will use for its SSH (secure shell) connection to the Oracle database. The default value is 22. OS UsernameType the username you use when you log onto the database server. Spotlight will connect successfully to all nodes in the cluster only if every node employs the SAME username and password. If the username and password if required are not completed, you will not be able to view the operating system Chapter 565 Spotlight on Oracle RAC performance statistics for the database server. (Windows) If your login details have been assigned remote access to the registry of the database server, you do not need to complete the OS Username and OS Password fields.(If your login details have NOT been set up on the database server, you must enter a username and password that has access to the registry of the server.) SSH Key Type (Enabled only when you connect to the database server via Unix(SSH using Public/Pri vate key). Choose the type of key to use when making the Spotlight connection. Public-key encryption is supported under SSH2 only The current options are RSA and DSA. For more information, see Public/Pri vate keypairs in the online help. SSH Private Key Filename (See SSH Key Type above.) Type the location of the file that contains the private key for the Spotlight connection, or click the "..." button to locate it. SSH Passphrase (See SSH Key Type above.) Type the passphrase used to decrypt the private key OS Password (Enabled when you connect to the database server via Unix(REXEC), Unix(SSH), or Windows). Enter the user password (if required). 7 Select the Launch Spotlight on Oracle option to open a Spotlight on Oracle connection to the Oracle RAC database whenever you open the Spotlight on Oracle RAC connection. In most cases, this should not be necessary. 8 Select the Display Connections Verification Details option to display the Connection Details Verification window when you connect to a specified Oracle RAC cluster. The window allows you to specify and test connections to ALL instances in the cluster. For more information, see To verify connection details on page 66. 9 Click the Save password details box to save all the password details that have been entered. 10Click OK to finish. A connection icon with the name you specified is created. 66Spotlight on Oracle Getting Started Guide Connecting to an Oracle RAC database Use this procedure to connect to an Oracle database running on an Oracle RAC cluster. Notes: Before connecting to an Oracle database, ensure that you have: Created a Spotlight user (see page 19) that has the special privileges required to diagnose your Oracle RAC databases. Created a connection to the Oracle RAC databases that you want to diagnose (see page 62). To connect to an Oracle RAC database Note:When installed under its own trial key, Spotlight runs for a limited time with full functionality. You must register the product befor