Stan Muse, System z Client Architect

IBM Global Markets, Financial Services Sector

smuse@us.ibm.com

770-380-2468

Enterprise Data Architecture and IBM Data Virtualization Manager:Data-as-a-Service for the Enterprise

December 7, 2018

IBM STG

Enterprise Data Architecture

Problem Statement

2

For decades clients have been copying data from systems of recordfor analysis and for providing applications services

• ETL processes are complex and consume vast amounts of resources• And still the business units do not have the data they really need

• Data duplication is very costly: Storage, Staff, Software Licenses• COOs and CFOs can not quantify these costs in terms of TCO

• Costly platforms, like Teradata, Exadata, and others consume IT growth budget• This workload could be as big as production online or batch systems

• Data latency is a huge problem for many applications, like fraud detection & trading• Day or week old data is not useful, losses continue to grow

• Data security is compromised with thousands of data feeds to who knows where• Who needs to hack a mainframe? Lack of data security results in expensive data breaches

• Enterprise data architecture has become very complex or unmanageable• Server sprawl is a common problem with runaway costs

• Distributed-oriented CIOs use this to accomplish their get-off-the-mainframe strategy• First copy the data, then rewrite the applications that use it

• Clients are struggling with how to provide mainframe Data as a Service (DBaaS)• This is key to the success of Watson, Hybrid Cloud, and many other new strategic solutions

• Result: Enterprise Data Management and data access has become IT’s costliest and most visible problem• Business growth is stifled

IBM STG

Enterprise Data Architecture

Client Justification for Data Virtualization

3

Clients need less complexity, lower costs, and better security:

• Lower Costs: Mainframe budgets and staff are cut as data and new development go off platformto thousands of distributed servers

• Reduce Complexity: Simplify data architecture, make the mainframe easier to accessand understand by keeping data in place

• Better Data Security: Clients need better data governance & security, with less cost & complexity

• Ability to Monitor Data Usage: most data breaches from the mainframe are from the inside

• Ability to create a chargeback system for data usage for the digital age

• Better Data Governance: Data is growing exponentially, and most clients are not ready

• A Central Data Server for the digital data distribution age

• Avoid long mainframe production systems change control cycle, be more agile for delivering new strategic solutions like machine learning

IBM STG

Enterprise Data Architecture

4

For architectural orientation, we begin with the familiar three layer client-server model as originally defined by Gartner Group in the 1980’s.

PRESENTATION

APPLICATION

DATA

Agenda: Discussion Orientation

IBM System Technology Group

IBM System Technology Group

Gartner Group showed how these 3 layers could be split between tiers, and defined several client-server distributed computing models.

Client

Presentation

Application

Data

Presentation

Application

Data

Presentation

Application

Presentation

Application

Data

Centralized

Computing Distributed

Presentation

Distributed

Application

Distributed

DatabaseDistributed Computing

Application

Data

Application

Data Data

Application

Data

Server

PathIncreasing complexity, support staff, cost

Presentation

(Green Screen) (Thin Client) (Fat Client)

Presentation

Presentation

IBM System Technology Group

Believing that they can reduce IT costs, some large enterprises moving to a Hyper-distributed modelhave dramatically increased cost and complexity, while reducing reliability and availability

IBM STG

Enterprise Data Architecture

8

The Enterprise Data Architecture conceptual model is made up of 7 classifications of data stores, showing the flow of data between them.

Enterprise Data Architecture - data store classifications:

1. Operational

2. Informational

3. Analytic

4. Shared

5. Systemic

6. External

7. Dark

Each of these data store classifications has its own unique requirements for:

• Usage

• Performance

• Recovery

• Security

• Etc.

We will define and discuss each of these in detail…

IBM STG

Enterprise Data Architecture

9

The high level Enterprise Data Architecture with data circulatory system

IBM STG

Enterprise Data Architecture

10

Data VirtualizationManager

The high level Enterprise Data Architecture with data circulatory system

Systems of Record

IBM STG

Enterprise Data Architecture

11

Enterprise Data Architecture data store classifications review:

1. Operational: On-Line Transaction processing and batch systems

2. Informational: Data Warehouse environment including Content & Records management

3. Analytic: Data Search and Analysis & Numerically intensive planning systems

4. Shared: Results of analytic analysis for operational systems usage

5. Systemic: data environment descriptions, parameters, and security authorizations

6. External: Data purchased and imported form external agencies

7. Dark: Archives, backups, image copies of all other data stores

Note: The term “Data Lake” is often used to include any subset of these

A quick review of the Enterprise Data Architecture 7 classifications of data stores:

IBM STG

Enterprise Data Architecture

Some Data Virtualization Use Cases

12

Data Virtualization can replace and simplify current or enable new data uses:

• Separation of distributed ad-hoc query workload from production for resources, tuning, availability

• Everyone accessing and using the same, current (system of record) data - Single version of the truth

• Easier to grant/provide access than existing production Online & Batch LPARs, easier change control for new solutions

• Metadata & defined interfaces, SQL and APIs to enterprise data and micro-services for easier access

• Ability to monitor usage patterns & stop access for intrusion detection, denial of service from distributed applications, cloud

• Access to ad-hoc query tools, JAVA, that are not available on production Online & Batch LPARs

• Provide SQL access to IMS, VSAM, Flat File, SMF, Unstructured and Streaming data with data federation

• Data Exploration for test data management, and best source for ETL when required, or Agile development tool

• Enable New technologies:• Hybrid Cloud access• Machine Learning, Cognitive Computing, Watson• Data federation from IMS DM, VSAM, Flat File, SMF, unstructured data• ISV Industry Solutions data access

• Better data security, eliminate the need for ETL & sending data everywhere• Keep data warehouse, data lakes on the mainframe• Provide a data obfuscation layer for better security or testing• z/OS DB2 Views with DVM can expose (select & project) only data needed from all data sources

Infrastructure simplificationPromotes overallbetter data usage

IBM STG

Enterprise Data Architecture

zEIS LPAR Conceptual Model: A central access point or data bus for all data flowing from the mainframe

Production ONLINE

ProductionBATCH

Z Enterprise Info Server

• Data Virtualization Manager• CICS/IMS/WAS/DB2 • Cognos BI, QMF, SPSS• AI/ML/Watson/Analytics• Guardium Data Activity Monitor• ADDI

• CICS• IMS/TM• WAS• DB2 • IMS/DM• VSAM• Transactional

• CICS• IMS/TM• WAS• DB2• IMS/DM• VSAM • IDAA Loader• ETL• Batch

I/O Subsystem

z/OS LPAR 1 z/OS LPAR 2 z/OS LPAR 3

CP

CP

zIIP zIIP zIIP zIIPCF

CF

CF

HiperSockets

REStful Services:• DB2 Connect & DDF• z/OS Connect• CICS TX Gateway• IMS Connect• IBM API Connect• IBM Open Analytics• Z Common Data Provider

CPs, zIIPs, Crypto CPs, zIIPs, Crypto

OSA

Other z/OS, z/Linux Systems

DistributedSystemsO

SA

OSA

DB2VSAM IMS SMF

Stream &Unstructured

ContentIDAA

Requesting Applications:• ESB + API• Fraud Detection• Watson, Machine Learning• Data Crawler, Ingestion, • Mobile Apps, Marketing• Analytics, Reporting• Risk & Compliance• KPI Dashboards• Low Code App Development• Micro Services• Hybrid Cloud Apps• Data Usage Monitoring

• …

Data Synchronization

SQLQueries

ZPARMScan direct ‘ALL’ zEDS queries to

IDAA

DaaS

DB2 Views

ETL orReplication

Could be done from here also

Netezza

Don’t ETL, Use data in place until

you know what you need.

OracleMongoPostgresSQL ServerHadoopEtc…

Distributed Databases

Pervasive Encryption, RACF, zSecure, Guardium Data Activity Monitor

z/OS + z/Linux

Consider implementing a separate, mostly read-only, z/OS LPAR as an Enterprise Information Server (zEIS)to provide data-as-a-service for the entire enterprise with Data Virtualization Manager

IDAA

IBM STG

Enterprise Data Architecture

Next Steps

14

• Understand your data access requirements and pain points

• Get an Executive Sponsor (CIO), understand cost justification requirements ( get a free IBM BVA assessment)

• Present Data Virtualization concept, socialize, gain commitment to proceed, (free IBM z Workshop)

• Identify a POC with a LOB, document criteria for success, commitment to move forward

• Present allocate a new z/OS zEIS LPAR: CPs, (2) zIIPs, memory, OSA ports, Crypto, ICF

• Data Virtualization POC’s:• Demonstrate z/OS Connect RESTful assess to DB2 & VSAM data via CICS or DVM• Demonstrate access from z/OS to distributed databases • Convert a small existing ETL workload to direct data access • Provide direct data access to a BU who has requested ETL• Use DVM to access SMF data & develop a utilization report• Use DVM o join DB2 & VSAM & distributed databases for a BU, analyze cost VS ETL• Develop a metadata catalog with DVM• Address new reporting or analytics BU backlog & build it• Demonstrate data access from, z/Linux apps• Build a z/OS CICS micro service with APIs

• Take advantage of New Offering ‘Container’ special pricing for new HW & SW from IBM

Thank You

FASTER

IBM STG

Enterprise Data Architecture

16

Characteristic(1) Operational

Systems(2) Informational

Systems(3) AnalyticSystems

Primary Use Mission Critical

Future

Executive

StrategicTactical

Time OrientationPresent

Data Set

Past

Primary Users Line Management

UpdateRead Only-Update

Access Orientation

Access Intent

Table (s) Record

Access Method

Daily

SessionTransaction, Batch,

Periodic

Query, Batch,

Last Save

Daily, Incremental

Unit of Work Last RefreshRecovery Unit

Backup Cycle

Each major category of data store has its own unique functional usage, backup, recovery, refresh, and tuning

characteristics.