CKE-How to Resolve Root Certificate Expiry Issue for Enterprise Manager-Database Control_10.2.0.4

How to resolve Root Certificate Expiry Issue for Enterprise Manager - Database Control (10.2.0.4)

Zaffer Khan, CISA

Blog: http://oracleendeavor.blogspot.com/ Email: [email protected]

1

14th June, 2011

PURPOSE

This paper demonstrates how to resolve the Oracle Enterprise Manager – Database Control

configuration errors in Oracle Database versions 10.2.0.4 or 10.2.0.5, arising due to the Root Certificate

Expiry issue since 31st December, 2010.

CASE STUDY

We use Oracle Enterprise Manager – Database Control for our 10.2.0.4 Production Box. All was fine

until mid of February ‘11, when we noticed some stale information on Enterprise Manager – Database

Control. Despite carrying out the respective activities and the scheduled runs (like ADDM, Segment

Advisor, etc.), for some reason the latest collection/information was not reflected on the Enterprise

Manager – Database Control Dashboard. All it showed was couple of days’ old data.

By the end of March, we decided to drop the Enterprise Manager and create it afresh, so as to resolve

the inconsistent data reflection issue. To our surprise, whenever we tried to setup Enterprise Manager

(manually or using DBCA), it would fail to create the repository with exceptions/errors in the

“emca.log” file. To simulate this issue and find the root cause of the problem, we tried the clean setup

of Enterprise Manager – Database Control on two different test databases, which subsequently

resulted in similar repository creation errors, as follows:

Apr 6, 2011 12:37:11 PM oracle.sysman.emcp.EMReposConfig invoke SEVERE: Error creating the repository Apr 6, 2011 12:37:11 PM oracle.sysman.emcp.EMReposConfig invoke INFO: Refer to the log file at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\DBTEST\emca_repos_create_<date>.log for more details. Apr 6, 2011 12:37:11 PM oracle.sysman.emcp.EMConfig perform SEVERE: Error creating the repository Refer to the log file at C:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\DBTEST\emca_2011-04-06_12-36-45-PM.log for more details. Apr 6, 2011 12:37:11 PM oracle.sysman.emcp.EMConfig perform CONFIG: Stack Trace: oracle.sysman.emcp.exception.EMConfigException: Error creating the repository at oracle.sysman.emcp.EMReposConfig.invoke(EMReposConfig.java:204) at oracle.sysman.emcp.EMReposConfig.invoke(EMReposConfig.java:134) at oracle.sysman.emcp.EMConfig.perform(EMConfig.java:171) at oracle.sysman.emcp.EMConfigAssistant.invokeEMCA(EMConfigAssistant.java:486) at oracle.sysman.emcp.EMConfigAssistant.performConfiguration(EMConfigAssistant.java:1142) at oracle.sysman.emcp.EMConfigAssistant.statusMain(EMConfigAssistant.java:470) at oracle.sysman.emcp.EMConfigAssistant.main(EMConfigAssistant.java:419)


Zaffer Khan, CISA


2

Upon investigation, we found out that the Root Certificate from Certification Authority, which is used

to secure communications via the Secure Socket Layer (SSL) protocol, has expired on 31st December,

2010 for Oracle Database versions 10.2.0.4 and 10.2.0.5. And, if anyone who installs or tries to secure

Enterprise Manager – Database Control on or after 31st December, he is likely to face configuration

errors, just as we did.

In a nutshell, this is what My Oracle Support (MOS) had to say on the subject:

“ATTENTION! After 31-Dec-2010, creating/recreating/securing 10.2.0.4/10.2.0.5 EM DB Control will fail

due to the expiration of the Certificate Authority. More informations in:

NOTE 1217493.1 ATTENTION: Patch Required If You Plan To Configure Enterprise Manager Database

Control With Oracle Database 10.2.0.4 Or 10.2.0.5 On Or After 31-Dec-2010

NOTE 1222603.1 Recovering From Database Control Configuration Errors Due to CA Expiry on Oracle

Database 10.2.0.4 or 10.2.0.5”

One needs to apply Patch 8350262 to the Oracle Home of 10.2.0.4 or 10.2.0.5 Databases, before

configuring the Enterprise Manager – Database Control. No database downtime is necessary to apply

the patch.

Also, the MOS Note says that the existing Database Control configurations are not impacted by the

Root Certificate Expiry issue. Likewise, we encountered the Enterprise Manager configuration issues

only after attempting the re-install of Enterprise Manager – Database Control. Had we not noticed the

stale information on our Enterprise Manager Dashboard, we would not have attempted the Enterprise

Manager reinstall and would not have known about the Root Certificate Expiry issue either.

Why the accurate information was not reflected on the Enterprise Manager Dashboard, could have

been related to the Root Certificate Expiry issue or could have been not. We didn’t investigate on this

to conclude on it. But one thing we can say is that we haven’t noticed the stale information issue after

having patched the production box with Patch 8350262 till date.

Here, I will demonstrate how we applied the Patch 8350262 to our Single-Instance, Test Oracle

Database to resolve the Enterprise Manager configuration issues. The Operating System used was

Microsoft Windows 2003 and the Oracle Database Product used was 10g Release 2 (10.2.0.4).


Zaffer Khan, CISA


3

GETTING STARTED

The two My Oracle Support Notes, namely 1222603.1 and 1217493.1, entail detailed explanation and

resolution for successful Enterprise Manager – Database Control configuration for both Single Instance

as well as RAC Databases.

In circumstances, where the Enterprise Manager – Database Control re-install or re-creation has not

been attempted yet, you can stop the Oracle Database Console and relevant services and directly

apply the patch. Once the patch is successful, you can start the all the stopped services.

In our case, as we had already tried to re-install the Enterprise Manager – Database Control and had

encountered the configuration errors, hence we had to clean the existing Enterprise Manager –

Database Control installation before we could attempt the patch.

Before we apply the patch, we need to ensure that:

1. We have downloaded and extracted the Patch 8350262.

2. We have downloaded the supporting version of OPatch utility for 10.2.0.4 Oracle Home.

3. We have removed any trails of failed Enterprise Manager Installation, if applicable.

NOTE: OPatch Version 10.2.0.5.1 is the version used to patch both 10.2.0.4 as well as 10.2.0.5 databases.

The relevant OPatch utility for 10.2.0.4 Oracle Home and the Patch 8350262 can be downloaded using

My Oracle Support website (http://support.oracle.com/).

Once the necessary files are downloaded, the OPatch utility needs to be extracted to a reference

directory. So, we created a new directory “OPatch” in our Oracle Home, extracted all the files in it.

C:\oracle\product\10.2.0\db_1\OPatch>opatch version Invoking OPatch 10.2.0.5.1 OPatch Version: 10.2.0.5.1 OPatch succeeded.

Then, we need to set the Oracle Home and set the path for “OPatch” such that the executables appears

in the system PATH.

C:\>set path=C:\oracle\product\10.2.0\db_1\OPatch C:\>set oracle_home=C:\oracle\product\10.2.0\db_1


Zaffer Khan, CISA


4

Next, we need to verify the OUI Inventory. If any errors are observed in this step, then we should contact Oracle Support for its resolution. Make sure this step is successful before attempting the patch. NOTE: Ensure that OUI Inventory verification is successful before attempting the patch.

C:\>opatch lsinventory Invoking OPatch 10.2.0.5.1 Oracle Interim Patch Installer version 10.2.0.5.1 Copyright (c) 2010, Oracle Corporation. All rights reserved. Oracle Home : C:\oracle\product\10.2.0\db_1 Central Inventory : C:\Program Files\Oracle\Inventory from : n/a OPatch version : 10.2.0.5.1 OUI version : 10.2.0.4.0 OUI location : C:\oracle\product\10.2.0\db_1\oui Log file location : C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\opatch2011-04-06_11-40-08AM.log Patch history file: C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\opatch_history.txt Lsinventory Output file location : C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\lsinv\lsinventory2011-04-06_11-40-08AM.txt -------------------------------------------------------------------------------- Installed Top-level Products (3): Oracle Database 10g 10.2.0.1.0 Oracle Database 10g Products 10.2.0.1.0 Oracle Database 10g Release 2 Patch Set 3 10.2.0.4.0 There are 3 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home.

-------------------------------------------------------------------------------- OPatch succeeded.

Next, we need to create a directory to retain the Patch 8350262 files in. Accordingly, we created a new

directory “8350262” under “OPatch” directory and then extracted the patch files from the archive file.

C:\oracle\product\10.2.0\db_1\OPatch> C:\oracle\product\10.2.0\db_1\OPatch>cd 8350262 C:\oracle\product\10.2.0\db_1\OPatch\8350262>

Before you begin to apply the patch, ensure that you have stopped the Database Console service. The

database and listener services do not need to be stopped, but ensure that all java processes running

from Oracle Home are stopped.


Zaffer Khan, CISA


5

Using “opatch apply” command, you can now initiate the patch application process. After performing

the prerequisite checks, the OPatch utility will prompt you to enter your My Oracle Support (MOS)

username and password for receiving any future MOS Security updates. You may ignore this or set it

up, as per your comfort.

Choosing not to receive the MOS Security updates will not hinder the patch application process in any

way. To ignore the MOS Setup, just leave the password blank or supply “NONE” when prompted for

MOS configuration and Proxy information.

Before applying the Patch 8350262, the OPatch utility will backup the necessary files for any possible

rollbacks.

NOTE: We have changed the MOS ID to a dummy ID ([email protected]) for demonstration purpose.

C:\oracle\product\10.2.0\db_1\OPatch\8350262>opatch apply Invoking OPatch 10.2.0.5.1 Oracle Interim Patch Installer version 10.2.0.5.1 Copyright (c) 2010, Oracle Corporation. All rights reserved. Oracle Home : C:\oracle\product\10.2.0\db_1 Central Inventory : C:\Program Files\Oracle\Inventory from : n/a OPatch version : 10.2.0.5.1 OUI version : 10.2.0.4.0 OUI location : C:\oracle\product\10.2.0\db_1\oui Log file location : C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\opatch2011-04-06_11-49-09AM.log Patch history file: C:\oracle\product\10.2.0\db_1\cfgtoollogs\opatch\opatch_history.txt ApplySession applying interim patch '8350262' to OH 'C:\oracle\product\10.2.0\db_1' Running prerequisite checks... Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name: [email protected] Provide your My Oracle Support password to receive security updates via your My Oracle Support account. Password (optional): Unable to establish a network connection to Oracle. If your systems require a proxy server for outbound Internet connections, enter the proxy server details in this format: [<proxy-user>@]<proxy-host>[:<proxy-port>] If you want to remain uninformed of critical security issues in your configuration, enter NONE Proxy specification: NONE


Zaffer Khan, CISA


6

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only. Backing up files and inventory (not for auto-rollback) for the Oracle Home Backing up files affected by the patch '8350262' for restore. This might take a while... Backing up files affected by the patch '8350262' for rollback. This might take a while... Patching component oracle.sysman.agent.core, 10.2.0.4.0a... Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emCORE.jar" with "\sysman\jlib\emCORE.jar\oracle\sysman\eml\sec\fsc\FSWalletUtil.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emCORE.jar" with "\sysman\jlib\emCORE.jar\oracle\sysman\eml\sec\rep\RepWalletUtil.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emCORE.jar" with "\sysman\jlib\emCORE.jar\oracle\sysman\eml\sec\util\RootCert.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emCORE.jar" with "\sysman\jlib\emCORE.jar\oracle\sysman\eml\sec\util\SecConstants.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emd_java.jar" with "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\fsc\FSWalletUtil.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emd_java.jar" with "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\rep\RepWalletUtil.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emd_java.jar" with "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\util\RootCert.class" Updating jar file "C:\oracle\product\10.2.0\db_1\sysman\jlib\emd_java.jar" with "\sysman\jlib\emd_java.jar\oracle\sysman\eml\sec\util\SecConstants.class" ApplySession adding interim patch '8350262' to inventory Verifying the update... Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data. Files check OK: Files from Patch ID 8350262 are present in Oracle Home. OPatch succeeded. C:\oracle\product\10.2.0\db_1\OPatch\8350262>

Once the patch is applied, the OPatch utility verifies the affected files and the OUI Inventory, before

confirming that the patch process is a success.

In the situation where the Enterprise Manager services were stopped simply to apply the patch, we

can now start the Oracle Database Console and related services, and start using the Enterprise

Manager – Database Control Dashboard.

For those of us who would want to install, secure or re-create the Enterprise Manager – Database

Control, then these can now be carried out smoothly.

Documents

CKE-How to Resolve Root Certificate Expiry Issue for Enterprise Manager-Database Control_10.2.0.4