www.SageLogix.com

Paper #536

Oracle Log MinerToo Clever For Words

Tim Gorman(tim@sagelogix.com)

PrincipalSageLogix, Inc.

IOUG-A “Live 2003”IOUG-A “Live 2003”

www.SageLogix.com

AgendaAgenda

Basic overview of Log MinerWhat is it?

How to use it?

Why use it?

Oracle9i New FeaturesEnhancements to support Log Miner as a major component of Data Guard’s Logical Standby Database and Streams feature

www.SageLogix.com

What is Log MinerWhat is Log Miner

Log Miner is a mechanism for examining redo log files (online or archived) from any Oracle8, Oracle8i, or Oracle9i database

The database which generated the logs does not have to be the database used to examine them

Oracle server processes perform the I/O on redo log files in a Log Miner session

Session initiated by identifying the log files to be read

Data is retrieved on demand by SQL queries within the session on the view V$LOGMNR_CONTENTS

Easy and safeCan examine logs in another database altogether

Only need to generate snapshots of database data dictionary periodically

www.SageLogix.com

Uses for Log MinerUses for Log Miner

Debugging or auditing DML or DDL actions performed within a specified time periodRecovering dropped tables

by finding the exact SCN of the DROP commandto allow recovery of a CLONE database the precise SCN-1, instead of an approximate time of day

Recovering deleted or updated databy finding the relevant REDO_SQL statement(s) and running the associated UNDO_SQL command(s)

Database Replication componentOracle9i Data Guard Logical Standby DatabaseOracle9i Streams

www.SageLogix.com

Using Log MinerUsing Log Miner

PL/SQL packages supplied with the RDBMS to query information from (online or archived) redo log files

Oracle data dictionary must have previously been exported

using the DBMS_LOGMNR_D.BUILD procedure

contents must be valid from when redo logs were generated

exporting allows point-in-time snapshots of data dictionary

Package DBMS_LOGMNR_D supplied with Oracle8i or Oracle9i

Can be installed on lower version databases, such as Oracle8

Filename: “%OH%/rdbms/admin/dbmslmd.sql”

www.SageLogix.com

Using Log MinerUsing Log MinerPackage DBMS_LOGMNR_D

BUILD

New parameter OPTIONS

STORE_IN_FLAT_FILE

Also requires parameters DICTIONARY_FILENAME and DICTIONARY_LOCATION

STORE_IN_REDO_LOGS (Oracle9i)

Progress of BUILD execution now visible (via DBMS_OUTPUT) when SET SERVEROUTPUT ON enabled in SQL*Plus

SET_TABLESPACE procedure (Oracle9i)

By default, internal tables used by Log Miner reside in the SYSTEM tablespace

SET_TABLESPACE changes to the specified tablespace and moves the tables (if already present somewhere else)

www.SageLogix.com

Using Log MinerUsing Log Miner

SET_TABLESPACE should be a one-time only occurrence

Moving Log Miner tables from SYSTEM to TOOLS tablespace, for example

BUILD should be used regularly to ensure that any future Log Miner sessions will have accurate data dictionary mapping information available

Using Log Miner while viewing data without data dictionary translations is miserable!

All schemas are named “UNKNOWN”

All tables are named “OBJ #”

All columns are named “COL #”

www.SageLogix.com

Using Log MinerUsing Log Miner

Creating a Log Miner sessionRedo log files must first be added to a list

DBMS_LOGMNR.ADD_LOGFILE(file-name, options)

options include:

NEW (clear list and add new file)

ADDFILE (add file to existing list)

REMOVEFILE (remove file from existing list)

www.SageLogix.com

Using Log MinerUsing Log Miner

Then, the Log Miner session must be startedto populate the V$LOGMNR_CONTENTS view with information from the files in the list

Procedure DBMS_LOGMNR.START_LOGMNR

Use parameters start-SCN/stop-SCN or start-time/stop-time to restrict to certain redo records

default: no restrictions, use the full list of files

The redo trail contains only numeric IDs for database objects (not symbolic names) so data dictionary info is necessary to translate to human-readable form:

Online data dictionary

Flat-file text extract of data dictionary

Redo stream contains data dictionary extracts (Oracle9i only on both source and mining side)

www.SageLogix.com

Using Log MinerUsing Log MinerOPTIONS parameter in START_LOGMNR procedure:

Oracle8i:USE_COLMAPSKIP_CORRUPTION

Oracle9i:SKIP_CORRUPTIONPRETTY_SQLDICT_FROM_ONLINE_CATALOGCOMMITTED_DATA_ONLYDDL_DICT_TRACKINGDICT_FROM_REDO_LOGSNO_SQL_DELIMITERCONTINUOUS_MINE

Procedure END_LOGMNRFinishes a Log Miner session

www.SageLogix.com

Using Log MinerUsing Log MinerUSE_COLMAP functionality in Oracle8i

Requires use of logmnr.opt fileMust be located in the same location (directory) as the data dictionary flat-fileFormat of column-mapping entries:colmap = schema table (1, column

[, 2, column [, …]] )maps specified columns to five place-holder columns in the end of the V$LOGMNR_CONTENTS view

PHn_NAME VARCHAR2(32)PHn_REDO VARCHAR2(4000)PHn_UNDO VARCHAR2(4000)

select * from v$logmnr_contentswhere seg_owner = ‘SCOTT’ and seg_name = ‘EMP’and ph1_name = ‘SAL’ and ph1_redo = ‘100000’and ph1_undo = ‘75000’;

www.SageLogix.com

Using Log MinerUsing Log Miner

Oracle9i does it differently, better…Function MINE_VALUE return VARCHAR2

Parameter SQL_REDO_UNDOFlag: constants REDO_VALUE or UNDO_VALUE

Parameter COLUMN_NAMEFully-qualified column name

Returns string with column valueDates always in format DD-MON-YYYY HH24:MI:SS.SS

Returns NULL if column not present or column has NULL value

How to tell the difference, when NULL is returned?

Function COLUMN_PRESENT return NUMBERParameter SQL_REDO_UNDOParameter COLUMN_NAMEReturns 1 if column is present, 0 if not present

www.SageLogix.com

Using Log MinerUsing Log Miner

So, what would this Oracle9i query return?

select dbms_logmnr.mine_value(redo_value,

'SCOTT.EMP.SAL')

from v$logmnr_contents

where dbms_logmnr.mine_value(redo_value,

'SCOTT.EMP.SAL') = ‘0’

or (dbms_logmnr.mine_value(redo_value,

'SCOTT.EMP.SAL') IS NULL

and dbms_logmnr.column_present(redo_value,

'SCOTT.EMP.SAL') = 1)

www.SageLogix.com

Using Log MinerUsing Log MinerV$LOGMNR_CONTENTS view

SCN NUMBER(15) system change numberTIMESTAMP DATE timestamp of redo vectorTHREAD# NUMBER redo log thread numberLOG_ID NUMBER redo log sequence #XIDUSN NUMBER XID (transaction ID) rollback

segment #XIDSLOT NUMBER XID transaction table slotXIDSQN NUMBER XID sequence # of slotRBASQN NUMBER RBA (redo byte address) log

seq #RBABLK NUMBER RBA block within fileRBABYTE NUMBER RBA byte offset within blockUBAFIL NUMBER UBA (undo byte address) file #UBABLK NUMBER UBA block within fileUBAREC NUMBER UBA record within blockUBASQN NUMBER UBA sequence of blockABS_FILE# NUMBER absolute file#REL_FILE# NUMBER relative (to tablespace) file#

www.SageLogix.com

Using Log MinerUsing Log MinerV$LOGMNR_CONTENTS view (cont’d)

DATA_BLK# NUMBER data block address (block #)

DATA_OBJ# NUMBER data object#DATA_DOBJ# NUMBER data block data object#SEG_OWNER VARCHAR2(30) segment ownerSEG_NAME VARCHAR2(81) segment nameSEG_TYPE NUMBER type of segmentSEG_TYPE_NAME VARCHAR2(32) name of type of segmentTABLESPACE_NAME VARCHAR2(30) segment’s tablespaceROW_ID VARCHAR2(18) row IDSESSION# NUMBER session IDSERIAL# NUMBER serial# of sessionUSER_NAME VARCHAR2(30) Oracle account nameSESSION_INFO VARCHAR2(4000) additional infoROLLBACK NUMBER 0=commit, 1=rollbackOPERATION VARCHAR2(30) SQL command typeSQL_REDO VARCHAR2(4000) SQL statementSQL_UNDO VARCHAR2(4000) “reverse” SQL stmtINFO VARCHAR2(32) informational msg

www.SageLogix.com

Enhanced data analysis

Enhanced data analysis

In Oracle9i, LogMiner has been enhanced to provide comprehensive Log Analysis for (almost) all types of data:

Index-organized tables

Clustered tables

Chained and migrated rows

LOBs and LONGs

direct-path loaded data

scalar object types

All DDL commands

Still missing:Collection object types (i.e. VARRAYs and NESTED TABLES)

www.SageLogix.com

Enhanced data analysis

Enhanced data analysis

To support redo logfile-based applications (such as logical standby databases), PK values and/or before-images may need to be added to the redo stream

Database supplemental loggingMinimal

Allows Log Miner to group REDO logs for individual DML statementsALTER DATABASE ADD SUPPLEMENTAL LOG

DATAPrimary key logging

Allows identification of rows logically rather than using ROWIDSRequirement for using Logical Standby database:

ALTER DATABASE ADD SUPPLEMENTAL DATA (PRIMARY KEY, UNIQUE KEY) COLUMNS;

www.SageLogix.com

Enhanced data analysis

Enhanced data analysis

Supplement logging (cont’d)Table supplemental logging

Logs column before-image values

An application might require that the before-image of the entire row be logged

Not just the columns being changed

www.SageLogix.com

New V$ viewsNew V$ views

V$LOGMNR_DICTIONARYShows info about dictionary being used (flat-file or online)

V$LOGMNR_LOGSShows info about log files being analyzed

V$LOGMNR_LOGFILENot documented! Shows info about logfiles being analyzed

V$LOGMNR_PARAMETERSShows options chosen for current Log Miner session

Parameter values passed to START_LOGMNR

V$LOGMNR_SESSIONNot documented! Information about each Log Miner session, including SESSION_ID for joining back to the V$SESSION view.Can be queried from outside an active Log Miner session.

www.SageLogix.com

New V$ viewsNew V$ viewsV$LOGMNR_PROCESS

Not documented! Information about the Oracle server process underlying the current Log Miner session.Cannot be queried from outside an active Log Miner session.

V$LOGMNR_TRANSACTIONNot documented! Information about the transaction listed within the current Log Miner session.Cannot be queried from outside an active Log Miner session

V$LOGMNR_REGIONNot documented! Information about internal memory structures used within the current Log Miner session.Cannot be queried from outside an active Log Miner session.

www.SageLogix.com

New V$ viewsNew V$ views

V$LOGMNR_CALLBACKNot documented! Information about internal memory structures used within the current Log Miner session.

Cannot be queried from outside an active Log Miner session.

V$LOGMNR_STATSNot documented! Usage statistics about each Log Miner session, intended to be joined to V$LOGMNR_SESSION and/or V$SESSION.

www.SageLogix.com

DocumentationDocumentation

Oracle8i/9i Server Administration manualChapter on “Using LogMiner to Analyze Online and Archived Redo Logs”

Oracle8i/9i Supplied Packages Reference manual

Chapters on “DBMS_LOGMNR” and “DBMS_LOGMNR_D”

Oracle9i Server Administration manualSection on “Database Supplemental Logging” and “Table Supplemental Logging”

Oracle9i Data Warehousing Guide manualChapter 15 on “Change Data Capture” (or “Streams”)

Oracle9i Supplied Packages Reference manualChapters on “DBMS_LOGMNR_CDC_PUBLISH” and “DBMS_LOGMNR_CDC_SUBSCRIBE”

www.SageLogix.com

DocumentationDocumentation

Metalink note #148616.1 – Oracle9i Log Miner New Features

Good overview with examples

PL/SQL source files located in $ORACLE_HOME/rdbms/admin

File dbmslm.sql

Package header source for DBMS_LOGMNR

File dbmslmd.sql

Package header source for DBMS_LOGMNR_D

File dbmslms.sql

Package header source for DBMS_LOGMNR_SESSION

More streamlined repackaging of DBMS_LOGMNR

Bug #2137007 filed to add some documentation this to standard doc set – still not visible (Feb 2003)

www.SageLogix.com

Q&AQ&APaper #536

Slides downloadable fromhttp://www.EvDBT.com/papers.htm

And http://www.SageLogix.com

Tim Gorman