Upload
tess98
View
937
Download
6
Tags:
Embed Size (px)
Citation preview
Rick WeaverBMC Software
Audit Your SOX off, and other uses of a DB2 Log Tool
© Copyright 04/13/23 BMC Software, Inc 2
Overview
Mostly generic look at Log Tools- Basics- Examples
Auditing Recovery Data Migration
BMC Log Master for DB2 used for examples
High-Speed Apply Engine benefits
© Copyright 04/13/23 BMC Software, Inc 3
Log Tool Basics
Read the log
Provide context- Associate Unit of Recovery info to DM activity- Combine multiple records into single entities- Manage committed versus rolled back activity
Allow for robust selection criteria
Allow for “ONGOING” processes
Produce outputs- SQL, DDL, LOAD, REPORTS, reusable inputs
© Copyright 04/13/23 BMC Software, Inc 4
Log Tool Basics
Externalization of data- Completion of partially logged updates
Potentially costly process Avoid with DATA CAPTURE CHANGES
- Decompression- Invoke Edit and Field Proc Decode functions- Decode DB2 Internal Formats
Numerics, Date, Time, and Timestamps- Normalize Log Records to the current Version- Serialize XML
© Copyright 04/13/23 BMC Software, Inc 5
Log Tool Basics
Resources Used- BSDS – Boot Strap Datasets- ARCHIVE and ACTIVE Log Datasets- DB2 Catalog- EDM Pool- TABLESPACE VSAM Datasets- IMAGE COPY Datasets- Reusable inputs for reprocessing, SQL application, etc.
© Copyright 04/13/23 BMC Software, Inc 6
Mining the DB2 Log Data – BMC Log Master
Batch Log Scan
Logical Log
Repository
Reports
Online Interface
DMLDDL
LoadFile
Report Writer
SQL Generator
DDL Generator
Load Generator
ArchiveLogs
ActiveLogs
Member BSDS
ArchiveLogs
ActiveLogs
Member BSDS
ArchiveLogs
ActiveLogs
Member BSDS
SQL ProcessorHigh Speed Apply
Load UtilityDB2
© Copyright 04/13/23 BMC Software, Inc 7
Audit Your SOX OffApproach(es)
Ongoing process storing data for later reprocessing- Logical Log – Merged into Dailies, Weeklies…
Report Generation- Audit, Detail, Summary, Catalog Activity- User Control over content
Verbose Detailed output- SQL or DDL
LOAD or LLOG file output to post process
© Copyright 04/13/23 BMC Software, Inc 8
Audit Your SOX OffExamples
Power user (SYSADM, DBADM) activity
Track Authorization changes
Audit all schema changing activities
Look for changes to sensitive columns and tables
Detect Dynamic SQL activity
Report on Utility Execution
Ad Hoc research for cause of data corruption
© Copyright 04/13/23 BMC Software, Inc 9
Audit ExamplePower user activity
Need to reconcile power user activity to change control
Detail Report produced on a daily basis for listed AUTHIDs and sensitive DATABASEs
Report ORDER-ed BY - CORRELATION ID (Jobname, TSO ID, etc.)- UNIT OF RECOVERY
Each activity has to be reconciled to a Change Control Ticket
© Copyright 04/13/23 BMC Software, Inc 11
ONGOING versus FOR LIMIT
ONGOING – TO point resolution- TO CURRENT- TO DATE(*) TIME(-00.30.00)
OR LIMIT- 5 LOG FILES- 7 DAYS- 24.00.00
© Copyright 04/13/23 BMC Software, Inc 12
Audit ExampleTrack Authorization Changes
Need to track all Authorization activity
Catalog Activity Report produced daily
Filter was for all GRANT and REVOKE activity
The Catalog Activity Report is summary level
Another option – VERBOSE DDL
© Copyright 04/13/23 BMC Software, Inc 14
Audit ExampleSchema changing activities
Report on ALL DDL activity in the system
Method- Generate Logical Log on an ONGOING basis- Merge into Weeklies, Monthlies- Generate Reports from Monthly Logical Log
© Copyright 04/13/23 BMC Software, Inc 17
Audit ExampleChanges to sensitive columns or tables
Need was to track the life cycle of a check (cheque)
Customer wrote ISPF front end for Auditors- Auditor would enter
Account Check number Range for activity
- ISPF Application would generate and submit a logscan job- Report could be viewed after job execution
© Copyright 04/13/23 BMC Software, Inc 18
Audit ExampleDynamic SQL activity
WHERE
PLAN NAME IN (
'DSNTEP2',
'DSNTIAUL',
'DSNTIAD',
'DISTSERV',
'DSNESPCS')
OR
(PLAN NAME = 'ADMPROD'
AND CORRELATION ID <> 'ALLOWJOB')
OR
PLAN NAME LIKE 'QMF%'
OR
PLAN NAME LIKE 'CDB%'
OR
PLAN NAME LIKE 'ACT%'
© Copyright 04/13/23 BMC Software, Inc 19
Audit ExampleReport on Utility Execution
Basically, an extract of SYSIBM.SYSCOPY
Can exclude non-utility ICTYPEs- ‘A’ – Alter- ‘C’ – Create
Approach- Generate a LOAD CSV file to post process- Transformed ICTYPE to meaningful value
© Copyright 04/13/23 BMC Software, Inc 21
Recovery Capabilities
Transaction Level Recovery- Avoid RECOVER outages and FIX IT programs- UNDO the bad- REDO the good after a PIT- Back out Integrity Checking
Discover and generate QUIET POINTs
DROP RECOVERY
Test System Regression
Customized Post Processing
© Copyright 04/13/23 BMC Software, Inc 22
Recovery CapabilitiesHigh-speed Apply Engine
SQL or LLOG input- LLOG processing almost always faster- Avoids SQL generation and parsing
Multi-threaded
Control over object distribution
Robust conflict resolution
Restartable
Used as the method for Migration as well as Recovery
© Copyright 04/13/23 BMC Software, Inc 23
UNDO - Take Away Only the Bad Data
BMC Log Master for DB2 can apply UNDO SQL to get rid of bad transactions.
Business Value – ZERO downtime for transaction level recovery!
Good Transaction 1 Good Transaction 2Bad Transaction UNDO Bad Transactions
GenerateUNDO SQL
ApplyUNDO SQL
© Copyright 04/13/23 BMC Software, Inc 29
RecoveryBack out Integrity Report
You choose to do an UNDO…
But, has anything happened to the row between the UNDO range and current
BACKOUT INTEGRITY
© Copyright 04/13/23 BMC Software, Inc 32
Recovery ExampleQUIET POINT Analysis
Avoid QUIESCEs
Customer historically doing a QUIESCE during a ‘low’ processing time every night
Still received 100-200 application time outs
Now use Log Processing to find and manifest QUIESCEs
Finding points with no open Units of Recovery when RECOVER is necessary versus QUIESCE
© Copyright 04/13/23 BMC Software, Inc 35
Automated Drop Recovery – Reduce Risk
Recreates dropped objects
Process is initiated from the online interface
Scans DB2 Log Records
• UNDO DDL to recreate the dropped object
• Syntax for recovery and object ID translation
• DB2 commands to rebind application plans that were invalidated when the object was dropped
• Drop Recovery Report
Generates JCL and outputs to automate Drop Recovery
Post recovery SQL and Rebind
Drive Recovery Technology using copy and log from Dropped Object.
OBID TranslationApplies log to point of DROP
DB2 Subsystem
RECOVER PLUS TechnologyLog Master Technology
© Copyright 04/13/23 BMC Software, Inc 37
Recovery ExampleSystem Regression
Test System Regression- Back out the test cycle versus recover to PIT- Set LOGMARK at beginning of test- Set LOGMARK at end of test- Generate LLOG from Begin to End LOGMARKS- Execute High-speed Apply Engine to UNDO LLOG
Production too…
© Copyright 04/13/23 BMC Software, Inc 38
Recovery ExamplePost Processing
Client has need to reverse changes up to 90 days
Many updates have occurred since
Solution- Produce LLOG or LOAD output for target data- Post Process to reverse changes while preserving current data
© Copyright 04/13/23 BMC Software, Inc 39
Migration
Data Warehouse
Test System Synchronization
To other Platforms- High-Speed Apply Engine- LOAD CSV or SDF (all character) Formats
© Copyright 04/13/23 BMC Software, Inc 40
DB2 Data Migration
REPOSITORYMigrated RBA range
In-flight URIDs
LogicalLog(+1)
LogicalLog(+1)
LogicalLog(+1)
(inflight URID 1988)
RBA 2000 RBA 3000 RBA 4000RBA 1000
Migrated 1000 - 2000less inflight URID 1988
Migrated 2000 - 3000plus inflight URID 1988
DB2 LOG
Input to LOAD utilityor Apply SQL process
Migrated 3000 - 4000
Don’t replicate entire files, just migrate the changes!!!
Log MasterBATCH PGM
Log MasterBATCH PGM
Log MasterBATCH PGM
© Copyright 04/13/23 BMC Software, Inc 41
Migration ExampleTo a Teradata Decision Support System
Used SQL to port- Post processed the SQL to change the comma delimiter
Wanted to limit size of any given extract- Used the OR LIMIT # LOG FILES
Selection criteria- List of 52 Tables to include and 18 batch jobs to exclude
> 800 Partitions – all compressed
Volume ~24 Billion transactions a year
© Copyright 04/13/23 BMC Software, Inc 43
Migration ExampleSynchronizing a Test System
Needed to keep Regression Test system in sync with production to prove changes before implementation (462 Tables)
Use LLOG as the capture format
Use High-speed Apply Engine to process the LLOG
Has a process in place to “Refresh” the test system after major production DDL changes or utility windows
RESETs the ONGOING capture after refresh
© Copyright 04/13/23 BMC Software, Inc 47
UDCLUnload – Drop – Create – Load
Reduce Object Restructuring outage window- Standard Practice
Put object into Read only Unload object being restructured STOP all access to object Drop object Create object with appropriate changes
– Repartitioning– COLUMN changes (metrics or location)
Load Table REBIND plans START access to object
- Outage lasts the entire length of the UNLOAD/LOAD window
© Copyright 04/13/23 BMC Software, Inc 48
Traditional ‘UDCL’ Method
ApplicationAvailable
Pre ImageCopy
Unload Data
Drop Tablespace
Create Tablespace
LoadData
Post Image Copy
RunstatsBind/Rebind
Read Only Stop all Objects - OUTAGE Limited RO Activity
Install New Programs
© Copyright 04/13/23 BMC Software, Inc 49
UDCLUnload – Drop – Create – Load
Reduce Object Restructuring outage window- Log Master / Apply Plus Alternative
Create new object with appropriate changes Unload old object Load new object Run job to Capture changes between Unload and Current (Apply too)
– (Reiterravally, until ready to switch over) Put old object into read only Do final Capture / Apply (reconcile) STOP objects and RENAME
– Old object to temporary name– New object to old name
REBIND plans START and carry on
- Outage only lasts from the point of final capture through rename and follow on actions
© Copyright 04/13/23 BMC Software, Inc 50
SHRLEVEL CHANGE TRANSFORM
Prep Activities - Current tables, Data, Online and Batch Unaffected
Create OCCFrom
TS_Orig, Set
Log Mark(0)
Execute Log Scan from TB_Orig
From Log Mark(0) to Current Time
Creating Apply SQL
TransformTS_New,Rebuild IX_New,
Inline Copy, Runstats
Create new Structures (TS_New,
TB_New, IX_New)
DB2 Log
Set New LogMark(+1)
Apply SQLFiles
UpdateTB_New
StartTS_Orig
Read Only
STOP allRename
TB/IX_Orig toTB/IX_Backup
RenameTB/IX_New to
TB/IX_Orig
Start TS_New,TB_Orig, IX_Orig
Application Unavailable
Rebind
FinalLog Update
TB_New
Application RO
Application Available
Update all TS/IX jobs
& Utils,DROP TS_Orig
© Copyright 04/13/23 BMC Software, Inc 51
Wrap up
Auditing
Recovery
Migration
Alternative uses of migration
Additional ideas to share?
Questions?
© Copyright 04/13/23 BMC Software, Inc 52
Ken McDonald
BMC Software
Audit Your SOX Off,and other uses of a DB2 Log Tool