Using Optim and IMS - IMS UG April 2013 Bloomington

© 2013 IBM Corporation

IBM InfoSphere Optim Technical Overview

Tom Love – Optim Senior IT Client Technical Professional

April 2013

© 2012 IBM Corporation2 Discovering the Value of Optim for Test Data Management, Data Privacy and Data Archiving

Information Management

Value Proposition

Enterprise Data Management

Production Databases Test and Development Databases

• Segregate Data & Move to

Archive

• Deploy Tiered Storage

Strategies

• Retain Data According to

Value

• Simplify Infrastructure

ManageApplication

Data Growth

• Decommission Redundant or

Obsolete Apps

• Gain Control of Application

Portfolio

• Retain Access to Legacy

Data

• Retire Apps and Repurpose

IT Assets

• Migrate Apps from High to

Low Cost Platforms

• Preserve Historical Data

Enable Portfolio

Optimization

• Protect PII Data

• Apply Single Data

Masking Solution

• Use Range of Masking

Techniques

• Maintain Referential

Integrity

• Maintain Contextual

Look and Feel

Ensure Data

Privacy

• Right size Test Apps

• Repeatable Process

• Quickly Deploy New Apps

• Future proof Apps

Speed Application Deployment


Information ManagementInformation Management

3

Supporting Enterprise Environments

Single, scalable, interoperable data management solution provides a central point to deploy policies

to extract, store, port, and protect application data records from creation to deletion

Data Growth Application Retirement Data Privacy Test Data Management

InfoSphere Discovery

OPTIM™

DATACOM



Check Test

Results

Create/ Modify

Application

Create Test

Environment

Inspect and Add

Data for Testing

Refresh Test DataMove to

Production

Test

Fail Pass

A Typical Software Development Lifecycle


Product Overview : Optim Test Database Management

Subsetting

Subsetting

Discovery



Optim™ Data Privacy Solution

Production

Contextual,

Application- Aware,

Persistent Data

Masking

Contextual,

Application- Aware,

Persistent Data

Masking

EBS / Oracle

Custom /

Sybase

Siebel / DB2

Test

EBS / Oracle

Custom /

Sybase

Siebel / DB2

� Substitute confidential information with fictionalized data

� Deploy multiple masking algorithms

� Provide consistency across environments and iterations

� Enable off-shore testing

� Protect private data in non-production environments



Current

Production

Historical

Selective Restore

Restored

Archives

Reporting

Data

Historical

DataReference

Data

Archive

� Complete Business Object provides historical reference snapshot of business activity

� Storage device independence enables ILM

� Immutable file format enables data retention compliance

IBM® Optim™ Data Growth Solution: Archiving

Universal Access to Application Data

Application Application XML ODBC / JDBC

IBM MashUPIBM MashUP


Application XML ODBC / JDBC



Optim z/OS Implementation



� Optim Directory

– Supplements information stored in the database

– Maintains product definitions and tracks processing

– Stores database connection information (DB Aliases)

– Stores user-defined relationships

RelationshipsOptimDIRECTORY

Tables

Stored in Database- Catalog- System Tables- Data Dictionary

ReferentialIntegrityRules

AccessDefinitions

DB Aliases

Maps

The Optim Directory



Database Alias

Establishing the Database Connection

� Optim view of a database connection– High-level qualifier for database object names

• DBalias.creatorid.objectname– Enables cross-Database access– Saved in Optim Directory

Relationships

DB Aliases

Column Maps

Table Maps

ACCTS DB

CUSTINFO DB

SALES DB

OptimDIRECTORY

Tables



OPTIM Concepts: Architecture

Storage Profile

DB Source/

Destination

Server Name• Server address or name •DB Alias• Connectivity via DB Client softwareWork Directory• Server File SystemStorage Profile• Storage and retention policy

Server Name• Server address or name •DB Alias• Connectivity via DB Client softwareWork Directory• Server File SystemStorage Profile• Storage and retention policy

TemporaryData…

Server Name

WorkDirectory

Archive

Architecture

Offline Storage.( Tivoli TSM )

Enterprise Storage( IBM DR550 )

Document Storage

( Tivoli )

Optim Directory



OPTIM Concepts: Architecture with Extended Data Sources

DB Source/

Destination

Server Name

Architecture

Teradata DatabaseServer

z/OS IMS or VSAMDatacom DB

IDMSAny other Structured d

datasource

IBM Federation Server

Optim Directory



� DB Relationships are automatically derived from database RI rules

� Application Specified Relationships

• Can be defined individually to Optim

• Can be imported into Optim from DDL

� Shared by all Optim components

OPTIM

RelationshipsOptimDIRECTORY

Tables

ReferentialIntegrityRules

AccessDefinitions

DB Aliases

Maps

Stored in Database- Catalog- System Tables- Data Dictionary

A Word About Relationships...



Extended Relationships

SE00582SE012

SC00234SC005

NW00564NW012

NC00123(NC)003

MANAGER_ID

Char (7)

SALESMAN_ID

Char (5)

Sales Table District Table

Example 1

Using Substr Function

Parent Table Sales

Substr(SALESMAN_ID,1,2)

Child Table District

DISTRICT_CD

00582NW

00234SE

00564SC

00123NC

MANAGER_NO

Char (5)

DISTRICT_CD

Char (2)

Example 2

Using Concat Function

Parent Table Sales

MANAGER_ID

Child Table District

DISTRICT_CD || MANAGER_NO



Extended Relationships

M35

M18

F56

F45

SEX

Char (1)

AGE

Integer

250

145

135

132

Rate

Numeric (5,0)

Age

Integer

Sales Table Female_Rates Table

Parent Table Child Table

Sales Male_Rates

Sex “M”

Age Age

Sales Female_Rates

Sex “F”

Age Age 250

145

135

318

Rate

Numeric (5,0)

Age

Integer

Male_Rates TableExample 3 Data Driven Relationships



Our Unique Capability: Complete Business Object

Example:General Ledger



Extract - Federated Data Support

Retek / Oracle Other apps / any

DBMS

Custom Inventory

Management / DB2

Complete Business

Object Captures

End to End

Business Process


IBM Optim TDM



OPTIM TDM Relational Extract Facility

� Creating and maintaining test data bases

� Migrating data

� The data and/or the object metadata can be extracted

ExtractFile

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

CUSTOMERS

ORDERS

DETAILS

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

TESTDB

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

QADB

PRODDB

LOAD

EXTRACT

INSERT/UPDATE

LoadFiles

Create -- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

New_DB

ADTM

CM



Optim’s Relational Compare Facility

� Single-table or multi-table compare

� Creates compare file and/or compare Report of results

� For application testing, QA, and to verify database contents

� Enhances productivity by finding unexpected changes in the data

SOURCE 1

SOURCE 2

COMPAREPROCESS

Optim

COMPAREFILE

Optim

Compare

REPORT

........................................................................................................................

Interactive Browse

Verify Test

Results


IBM Optim Data Privacy



During Extract Process

Or

Standalone Convert Process

Or

During Insert/Load Process

Transform or Replace sensitive data using

� Standard mapping rules: Literals, Special Registers,

Expressions, Default Values,Look-up tables

� Complex mapping rules: User exits

De-Identify test data

Production Data

Extract and

Convert

Masked Test Data



ExtractFile

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

Transform / mask sensitive data

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

TESTDB

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

QADB

LOAD

INSERT/UPDATE

LoadFiles

Data Privacy in Application Testing

Extract a relationally intact subset from production database(s)

• Extract data and/or object definitions

• Define a new set of test tables

• Apply masking during population process

• Extract file may be reused but contains un-Masked data

• Good process for testing masks

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

CUSTOMERS

ORDERS

DETAILS

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

CUSTOMERS

ORDERS

DETAILS

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

NewDB

Create

LegacyFiles

BKORDER

SelectionCriteria



ExtractFile

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

TESTDB

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

QADB

LOAD

INSERT/UPDATE

LoadFiles




• Extract data and/or object definitions in pre-masked file

• Use pre-masked Extract file to create new set of tables

• Convert Pre-masked extract file data into second masked extract file

•Share masked extract file to be reused for population step

• Good practice for testing masks using COMPARE

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

CUSTOMERS

ORDERS

DETAILS

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

CUSTOMERS

ORDERS

DETAILS

Masked

ExtractFile

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

NewDB

Create

LegacyFiles

BKORDER

SelectionCriteria



ExtractFile

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----


-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

TESTDB

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

QADB

LOAD

INSERT/UPDATE

LoadFiles



• Most Secure Approach

• Extract data only

• Convert during extract

•Extract file already contains masked data

•Can be shared with testers to reuse

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

CUSTOMERS

ORDERS

DETAILS

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

CUSTOMERS

ORDERS

DETAILSLegacy

Files

BKORDER

SelectionCriteria



ExtractFile

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----


-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

TESTDB

-- ---- ---- ---- ------- ----CUST

-- ---- ---- ---- ------- ----ORD

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETL

QADB

LOAD

INSERT/UPDATE

LoadFiles



• Most Secure Approach

• Extract data only

• Convert during extract

•Extract file already contains masked data

•Can be shared with testers to reuse

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

CUSTOMERS

ORDERS

DETAILS

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

CUSTOMERS

ORDERS

DETAILS

Only Users authorized to see Private data



Consistent Masking across the Enterprise

Masked fields

are consistent

Data is masked

132009824

157342266

SS#s

132009824

157342266

SS#s

DB2

323457245

134235489

SSN#s

323457245

134235489

SSN#s

Client Billing Application



Example 2Example 2Example 1Example 1

PersNbr FstNEvtOwn LstNEvtOwn

27645 Elliot Flynn

27645 Elliot Flynn

Event TableEvent Table

PersNbr FstNEvtOwn LstNEvtOwn

10002 Pablo Picasso

10002 Pablo Picasso

Event TableEvent Table

Personal Info TablePersonal Info Table

PersNbr FirstName LastName

08054 Alice Bennett

19101 Carl Davis

27645 Elliot Flynn

Personal Info TablePersonal Info Table

PersNbr FirstName LastName

10000 Jeanne Renoir

10001 Claude Monet

10002 Pablo Picasso

Referential integrity is maintained with key propagation

Patient InformationPatient InformationPatient Information

Patient No. SSN

Name

Address

City State Zip

Patient No. SSN

Name

Address

City State Zip

112233 123-45-6789

Amanda Winters

40 Bayberry Drive

Elgin IL 60123

123456 333-22-4444

Erica Schafer

12 Murray Court

Austin TX 78704

Data is masked with contextually correct data to preserve integrity of test data

Data Privacy – Key Propagation



Cust ID Item # Order Date

27645 80-2382 20 June 2004

27645 86-4538 10 October 2005

Customers Table

Orders Table

Cust ID Name Street

08054 Alice Bennett 2 Park Blvd

19101 Carl Davis 258 Main

27645 Elliot Flynn 96 Avenue

Original Data

Cust ID Item # Order Date

27645 80-2382 20 June 2004

27645 86-4538 10 October 2005

Customers Table

Orders Table

Cust ID Name Street

10000 Auguste Renoir Mars23

10001 Claude Monet Venus24

10002 Pablo Picasso Saturn25

Without Propagation

Now these

are Orphans!

Why Propagation is Important



Masking Functions

� Column Map � Map unlike column names

� Transform/mask sensitive data

� Datatype conversions

� Column-level semantic date aging

� Literals

� Registers

� Calculations

� Default values

� Substring

� Exits

� Currency conversion

�Social Security

�Credit Card

�Email

�Hash Lookup

�Lookup

�Random Lookup

�NAME tables

�ADDRESS table

�Shuffle

�String manipulation

� PROP…

�…


Optim Archive

Managing Data Growth Retirement and Compliance .



Current

Production

Historical

Selective Restore

Restored

Archives

Reporting

Data

Historical

DataReference

Data

Archive

� Complete Business Object provides historical reference snapshot of business activity

� Storage device independence enables ILM

� Immutable file format enables data retention compliance

IBM® Optim™ Data Growth Solution: Archiving


Application Application XML ODBC / JDBC

IBM MashUP


Application XML ODBC / JDBC



Current Data

0-2 yrs.

Production

Database

Information Lifecycle

Offline Archive

7+ yrs.

Off-line Retention Platform

CD,Tape,Optical, WORM

HP StorageWorks™,

NetApp NearStore® SnapLock™,

IBM Total Storage® solutions

(including the DR550)

EMC Centera™.

Archive File Management

Compressed

Archive File

Archive

Reporting/Query Toolsusing

ODMCognos, Discoverer,

SQLPLUS, Etc.

Non DBMS

Retention Platform

ATA File Server

Centera

DR550

Etc.

On/Near-Line Archive

5-6 yrs.

Active Historical On-Line

3-4 yrs.

Resto

re

Archive

Database

Native Access

Optional



Define the data to be deleted

� Archive all data

� Delete orders and details after they are safely archived

� Preserve semantic intelligence

CUSTOMERS

ORDERS

DETAILS

RETAIN

DELETE



Create the archive

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

CUSTOMERS

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

ORDERS

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETAILS

ProductionDatabase

ArchiveFile

DELETE

Optim Directory( Tables)



Applications accessing the Archive Files

Optim DirectoryTables

Archive

Library

RESEARCH/BROWSE

Use the OPTIM Archive ODM Option

�Direct Access within Your Application using standard SQL

�Defines data-sources for any ODBC or JDBC application

�Joins between multiple data-sources

�archive files and database tables

End-UserQuery / Reporting

ODM



Archive Collections

Collection A

Archive 1,2

Collection A

Archive 1,2

Collection B

Archive 1,2,3,4

Collection B

Archive 1,2,3,4

Archive 1Archive 1

CustomerCustomer

CustomerCustomer

Order(Y01)

Order(Y01)

Detail(Y01)

Detail(Y01)

CustomerCustomer

Order(Y01, Y02)

Order(Y01, Y02)

Detail(Y01, Y02)

Detail(Y01, Y02)

Credit CardCredit Card

ODBC/

JDBC

ODBC/

JDBC

Archive 2Archive 2

Order(Y2001)

Order(Y2001)

Detail(Y2001)

Detail(Y2001)

Archive 3Archive 3

Order(Y2002)

Order(Y2002)

Detail(Y2002)

Detail(Y2002)

Archive 4Archive 4

Credit CardCredit Card

PhysicalLogical

Tech OverviewTech Overview



Archive Archive

Archive Archive

Index Index

Index Index

• Indexes are used to quickly locate specific column values in an archive file.

• Indexes greatly increase the performance when searching large archives

• Indexes provide quick access to data during an ediscovery, audit or other types of request

• Without indexes access to archive data is extremely slow because the entire file(s), which can be very large, need to be searched for the specific data that is required.

Archive Collections

= Virtual Database

Archive Indexes Tech OverviewTech Overview



Restoring Archived Data

Repository

ArchiveFiles

RESEARCH/

BROWSE

Production/Staging Database

Metadata

Mapping

RESTORE

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

CUSTOMERS

-- -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ------ -- ------ -- --------- ----

ORDERS

-- ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ------ ---- ---- ---- ------- ----

DETAILS

Data to Restore


Optim Archive

Portfolio Optimization/Decommissioning



Portfolio Optimization/Decommissioning

� Remove a system from active service but…

� Retain access to its business-critical data

– Retire an application

– Consolidate redundant systems into single “enterprise

standard”

– Migrate portfolio to lower-cost platform

– Consolidate and eliminate unsupported databases and

versions



ArchiveArchive

Optim™ value for Application Retirement

Oracle ERP

DB2 Order Entry

Legacy

WMS

Oracle ERPOracle ERP

DB2 Order EntryDB2 Order Entry

Legacy

WMS

Legacy

WMS

Complete Business Object

Year 2006

Year 2005

Year 2004

Year ….

….….

<XML><Name>John</Name><Zip>08540</Zip>

</XML>

<XML><Name>John</Name><Zip>08540</Zip>

</XML>

AccessAccessExpire

Reta

in

Store

DecommissionDecommission

OPTIM™ Solution



OPTIM™ Technical Advantages

� Archives are immutable–Data + Metadata +

related documents

� Enables compression–Smaller footprint–Simplifies operations,

backup, restore

� Device independent–Direct access–SAN, NAS, CAS, WORM–Retrieve & restore to any

enterprise class database

� Database independent–“Future proofed”

� Supports HSM–As the data ages it can move

to less expensive storage and still be accessed

� Designed with Data Aging policies in mind

–Retention–Expiration

� Part of consistent Optim strategy for all databases and apps

–Decommissioning–Archiving–TDM

OPTIM™ Solution



Access to Archived dataOpen Data Manager (ODM)

� Provides access to data in Archive Files / Archive Collections for applications that use ODBC and JDBC APIs

Optim Archive Files

ODM Primary Server

(Optim Directory & defined

ODM Data Sources)

JDBC or ODBC

accessQuery /

ReportingTools

MS-Excel, MS-Access,

Crystal Reports, etc., that

Use ODBC/JDBC



Archive File Collections

� An Archive Collection is a list of Archive Files that can be logically unioned together as a single data source for Open Data Manager (ODM) access.

� The Archive File Collection Editor is used to create an Archive File Collection

� Tables with matching creator IDs and names in separate Archive Files will be unioned

� Using the Archive File Collection to access data in multiple Archive Files:

� Even if all files do not include a specific table

� Even if all files do not contain a specific column

� Even if the attributes of data in a column vary from file to file

� ODM processes files in the order they are listed in the Archive File

� Collection Editor



Technical Features and Benefits of ODM

� ODM features and benefits include:

–Access archived data, with the ability to join tables, group data, perform unions, etc. under full SQL 92 via

ODBC/JDBC

–Output data in archive files as XML –Retain archive security for archive files and rows/columns.

–Leverage ODM to access archived DB2 data—converts Optim z/OS archive files to Optim (open system) archive

files.

–Perform queries and reports that confederate production and archived data as though data was never relocated

(supports connectors to a data source other than the

archive file).

–Use standard reporting and query tools like Excel, Crystal

reports and SQL utilities against archived data.



• Application access

− User requests data with application function

• Consolidated view

Application-Based Access to Archives

Application

OLTP

Archive

Archive



Self-Help Access to Archives

� Open & independent

� No training of end users or audit staff

� Leverage existing tools and skills

� OLTP not required

� Excel

� Access

� Report Writers –

• Cognos,

• Business Objects,

• Discoverer

� Any SQL-based tool set

� Browser

� Java ApplicationArchive

Archive



Application-Independent Access to Archives

� Application not required– Enables decommissioning of obsolete or redundant applications and

versions– No reliance on application vendor

� OLTP not required– Offline access available– Archive access does not impede production processing

XML, SQL,

ODBC / JDBC

Delivered Browser

Indexed & Secured

Company

Contact

Activity

Archive



Non Production EnvironmentsSubset & Mask

Production EnvironmentsArchive

� Single, scalable, interoperable EDM solution provides a central point to deploy policies to extract, store, port, and protect application

data records from creation to deletion

NAS SAN ATA CAS Optical Tape

AIX Linux z/OS Windows XP/2000 Solaris HP/UX

DB2 LUW DB2 z/OS DB2 iSeries Informix Sybase SQL Server IMS VSAM Adabas Oracle More ….

Optim ™

Data Growth, Data Privacy, Test Data Management, Application Upgrades, Application Retirement

Enterprise Architecture





Using Discovery + Optim for Data Privacy



The Problem: Securing Information

� Locate: Confidential information is dispersed throughout the organization– Hard to locate and easily transmitted, copied and printed

� Monitor: Severe financial, business and regulatory penalties for loss of sensitive and regulated data

– Business challenges - Loss of reputation and customer confidence– Regulatory challenges - GLBA, SOX, HIPAA

� Protect: Difficult to set policies without the capability to monitor and enforce– Lack of automated enforcement mechanisms



Sensitive Data Discovery: Where is all the Sensitive Data? How Long Will it Take to Find It?

DiscoverSensitiveData

Poorly Documented Applications:

… ? …



Sensitive Data Discovery: Structured Data

• Finding Sensitive Data Elements (SDE) in each system can take days• Whole and partial SDE’s can be found in hundreds of tables and fields

R o w M e m b e r S S # A g e P h o n e S e x

1 5 9 5 8 4 6 2 2 6 1 2 3 - 4 5 - 6 7 8 9 1 5 ( 1 2 3 ) 4 5 6 - 7 8 9 0 M

2 5 6 7 4 7 2 5 9 6 1 3 8 - 2 7 - 1 6 0 4 8 ( 1 3 8 ) 2 7 1 - 6 0 3 7 F

3 5 4 0 4 5 0 0 9 1 1 5 4 - 8 6 - 4 1 9 6 2 2 ( 1 5 4 ) 8 6 4 - 1 9 6 1 M

4 5 1 4 7 1 4 3 7 2 1 7 3 - 4 4 - 7 9 0 0 5 5 ( 1 7 3 ) 4 4 7 - 8 9 9 6 F

5 4 9 0 2 0 4 1 6 4 1 9 4 - 2 6 - 1 6 4 8 4 ( 1 9 4 ) 2 6 1 - 6 4 7 6 F

6 4 6 6 8 6 1 1 0 9 2 1 7 - 5 7 - 3 0 4 6 6 6 ( 2 1 7 ) 5 7 3 - 0 4 5 3 M

9 8 7 ,6 2 3 4 4 4 6 2 9 6 2 8 2 4 3 - 6 8 - 1 8 1 2 2 5 ( 2 4 3 ) 6 8 1 - 8 1 0 7 F

9 8 7 ,6 2 4 4 2 3 4 5 6 7 8 9 2 7 2 - 9 2 - 3 6 2 9 8 7 ( 2 7 2 ) 9 2 3 - 6 2 8 0 M

K n o w n S e n s i t iv e D a t aSensitive Data Repository



Why is Sensitive Data Discovery in Structured Data Difficult?

Patient Result Test

3802468 N 53

4182715 N 53

4600986 N 32

5061085 N 53

5567193 N 72

6123913 Y 47

6736304 N 34

7409934 N 34

8150928 N 47

8966020 N 34

System A Table 15

Sensitive Relationship Discovery � Sensitive data can’t be found just by a simple data scan.

– Must connect tables and lookup tables

– Hidden within larger fields (substring)

– Hidden across fields (concatenations)

– Represented differently (lookup tables and case statements)

� “Corporate memory” is poor

– Documentation is incomplete

– SME’s and Data Analysts are only knowledgeable of one or

two systems

� Hundreds of tables and millions of rows:

– Complex

– Difficult to verify

� Data quality problems make discovery even more difficult

Test Name

53 Streptococcus pyogenes

72 Pregnancy

32 Alzheimer Disease

47 Hemorrhoids

34 Dermatamycoses

System Z Table 25

Number Name

4600986 AlexFulltheim

8150928 BarneySolo

6736304 BillAlexander

3802468 BobSmith

5567193 EileenKratchman

7409934 FredSimpson

6123913 GregLougainis

5061085 JamieSlattery

4182715 JimJohnson

8966020 MartinAston

System A Table 1

Number Name

3544600986 AlexFulltheim

5728150928 BarneySolo

3786736304 BillAlexander

6783802468 BobSmith

4035567193 EileenKratchman

8037409934 FredSimpson

4306123913 George Brett

9525061085 JamieSlattery

4594182715 JimJohnson

1288966020 MartinAston

System A Table 1



InfoSphere Basic Discovery For Sensitive Data

� Analyze up to 20 data sources

simultaneously

� Discover sensitive data by comparing

known sensitive data with data in a wide

variety of systems at the push of a

button

� Identified sensitive data elements

(SDEs) are exported to Optim for

masking



Transformation Analyzer Summary for Hidden Sensitive Data

� Automates discovery of complex business rules between data sources

– Finds sensitive data hidden within longer fields (e.g. SSN hidden in a 46 digit routing number)

– Finds sensitive data that has been divided up across multiple columns (e.g. SSN divided into three separate columns)

– Finds sensitive data that has been transformed (i.e. items converted into codes)



Automated Sensitive Data Advantages

� Better ROI – faster time to deployment at lower cost – 5-10x time savings for business object discovery– 5-10x time savings for identifying sensitive information

Automated Sensitive Data

Discovery

Customer Success

Optim Deployment

Technology

Using Optim and IMS - IMS UG April 2013 Bloomington