View
29
Download
0
Category
Preview:
Citation preview
Stan Muse, System z Client Architect
IBM Global Markets, Financial Services Sector
smuse@us.ibm.com
770-380-2468
Enterprise Data Architecture and IBM Data Virtualization Manager:Data-as-a-Service for the Enterprise
December 7, 2018
IBM STG
Enterprise Data Architecture
Problem Statement
2
For decades clients have been copying data from systems of recordfor analysis and for providing applications services
• ETL processes are complex and consume vast amounts of resources• And still the business units do not have the data they really need
• Data duplication is very costly: Storage, Staff, Software Licenses• COOs and CFOs can not quantify these costs in terms of TCO
• Costly platforms, like Teradata, Exadata, and others consume IT growth budget• This workload could be as big as production online or batch systems
• Data latency is a huge problem for many applications, like fraud detection & trading• Day or week old data is not useful, losses continue to grow
• Data security is compromised with thousands of data feeds to who knows where• Who needs to hack a mainframe? Lack of data security results in expensive data breaches
• Enterprise data architecture has become very complex or unmanageable• Server sprawl is a common problem with runaway costs
• Distributed-oriented CIOs use this to accomplish their get-off-the-mainframe strategy• First copy the data, then rewrite the applications that use it
• Clients are struggling with how to provide mainframe Data as a Service (DBaaS)• This is key to the success of Watson, Hybrid Cloud, and many other new strategic solutions
• Result: Enterprise Data Management and data access has become IT’s costliest and most visible problem• Business growth is stifled
IBM STG
Enterprise Data Architecture
Client Justification for Data Virtualization
3
Clients need less complexity, lower costs, and better security:
• Lower Costs: Mainframe budgets and staff are cut as data and new development go off platformto thousands of distributed servers
• Reduce Complexity: Simplify data architecture, make the mainframe easier to accessand understand by keeping data in place
• Better Data Security: Clients need better data governance & security, with less cost & complexity
• Ability to Monitor Data Usage: most data breaches from the mainframe are from the inside
• Ability to create a chargeback system for data usage for the digital age
• Better Data Governance: Data is growing exponentially, and most clients are not ready
• A Central Data Server for the digital data distribution age
• Avoid long mainframe production systems change control cycle, be more agile for delivering new strategic solutions like machine learning
IBM STG
Enterprise Data Architecture
4
For architectural orientation, we begin with the familiar three layer client-server model as originally defined by Gartner Group in the 1980’s.
PRESENTATION
APPLICATION
DATA
Agenda: Discussion Orientation
IBM System Technology Group
IBM System Technology Group
Gartner Group showed how these 3 layers could be split between tiers, and defined several client-server distributed computing models.
Client
Presentation
Application
Data
Presentation
Application
Data
Presentation
Application
Presentation
Application
Data
Centralized
Computing Distributed
Presentation
Distributed
Application
Distributed
DatabaseDistributed Computing
Application
Data
Application
Data Data
Application
Data
Server
PathIncreasing complexity, support staff, cost
Presentation
(Green Screen) (Thin Client) (Fat Client)
Presentation
Presentation
IBM System Technology Group
Believing that they can reduce IT costs, some large enterprises moving to a Hyper-distributed modelhave dramatically increased cost and complexity, while reducing reliability and availability
IBM STG
Enterprise Data Architecture
8
The Enterprise Data Architecture conceptual model is made up of 7 classifications of data stores, showing the flow of data between them.
Enterprise Data Architecture - data store classifications:
1. Operational
2. Informational
3. Analytic
4. Shared
5. Systemic
6. External
7. Dark
Each of these data store classifications has its own unique requirements for:
• Usage
• Performance
• Recovery
• Security
• Etc.
We will define and discuss each of these in detail…
IBM STG
Enterprise Data Architecture
9
The high level Enterprise Data Architecture with data circulatory system
IBM STG
Enterprise Data Architecture
10
Data VirtualizationManager
The high level Enterprise Data Architecture with data circulatory system
Systems of Record
IBM STG
Enterprise Data Architecture
11
Enterprise Data Architecture data store classifications review:
1. Operational: On-Line Transaction processing and batch systems
2. Informational: Data Warehouse environment including Content & Records management
3. Analytic: Data Search and Analysis & Numerically intensive planning systems
4. Shared: Results of analytic analysis for operational systems usage
5. Systemic: data environment descriptions, parameters, and security authorizations
6. External: Data purchased and imported form external agencies
7. Dark: Archives, backups, image copies of all other data stores
Note: The term “Data Lake” is often used to include any subset of these
A quick review of the Enterprise Data Architecture 7 classifications of data stores:
IBM STG
Enterprise Data Architecture
Some Data Virtualization Use Cases
12
Data Virtualization can replace and simplify current or enable new data uses:
• Separation of distributed ad-hoc query workload from production for resources, tuning, availability
• Everyone accessing and using the same, current (system of record) data - Single version of the truth
• Easier to grant/provide access than existing production Online & Batch LPARs, easier change control for new solutions
• Metadata & defined interfaces, SQL and APIs to enterprise data and micro-services for easier access
• Ability to monitor usage patterns & stop access for intrusion detection, denial of service from distributed applications, cloud
• Access to ad-hoc query tools, JAVA, that are not available on production Online & Batch LPARs
• Provide SQL access to IMS, VSAM, Flat File, SMF, Unstructured and Streaming data with data federation
• Data Exploration for test data management, and best source for ETL when required, or Agile development tool
• Enable New technologies:• Hybrid Cloud access• Machine Learning, Cognitive Computing, Watson• Data federation from IMS DM, VSAM, Flat File, SMF, unstructured data• ISV Industry Solutions data access
• Better data security, eliminate the need for ETL & sending data everywhere• Keep data warehouse, data lakes on the mainframe• Provide a data obfuscation layer for better security or testing• z/OS DB2 Views with DVM can expose (select & project) only data needed from all data sources
Infrastructure simplificationPromotes overallbetter data usage
IBM STG
Enterprise Data Architecture
zEIS LPAR Conceptual Model: A central access point or data bus for all data flowing from the mainframe
Production ONLINE
ProductionBATCH
Z Enterprise Info Server
• Data Virtualization Manager• CICS/IMS/WAS/DB2 • Cognos BI, QMF, SPSS• AI/ML/Watson/Analytics• Guardium Data Activity Monitor• ADDI
• CICS• IMS/TM• WAS• DB2 • IMS/DM• VSAM• Transactional
• CICS• IMS/TM• WAS• DB2• IMS/DM• VSAM • IDAA Loader• ETL• Batch
I/O Subsystem
z/OS LPAR 1 z/OS LPAR 2 z/OS LPAR 3
CP
CP
zIIP zIIP zIIP zIIPCF
CF
CF
HiperSockets
REStful Services:• DB2 Connect & DDF• z/OS Connect• CICS TX Gateway• IMS Connect• IBM API Connect• IBM Open Analytics• Z Common Data Provider
CPs, zIIPs, Crypto CPs, zIIPs, Crypto
OSA
Other z/OS, z/Linux Systems
DistributedSystemsO
SA
OSA
DB2VSAM IMS SMF
Stream &Unstructured
ContentIDAA
Requesting Applications:• ESB + API• Fraud Detection• Watson, Machine Learning• Data Crawler, Ingestion, • Mobile Apps, Marketing• Analytics, Reporting• Risk & Compliance• KPI Dashboards• Low Code App Development• Micro Services• Hybrid Cloud Apps• Data Usage Monitoring
• …
Data Synchronization
SQLQueries
ZPARMScan direct ‘ALL’ zEDS queries to
IDAA
DaaS
DB2 Views
ETL orReplication
Could be done from here also
Netezza
Don’t ETL, Use data in place until
you know what you need.
OracleMongoPostgresSQL ServerHadoopEtc…
Distributed Databases
Pervasive Encryption, RACF, zSecure, Guardium Data Activity Monitor
z/OS + z/Linux
Consider implementing a separate, mostly read-only, z/OS LPAR as an Enterprise Information Server (zEIS)to provide data-as-a-service for the entire enterprise with Data Virtualization Manager
IDAA
IBM STG
Enterprise Data Architecture
Next Steps
14
• Understand your data access requirements and pain points
• Get an Executive Sponsor (CIO), understand cost justification requirements ( get a free IBM BVA assessment)
• Present Data Virtualization concept, socialize, gain commitment to proceed, (free IBM z Workshop)
• Identify a POC with a LOB, document criteria for success, commitment to move forward
• Present allocate a new z/OS zEIS LPAR: CPs, (2) zIIPs, memory, OSA ports, Crypto, ICF
• Data Virtualization POC’s:• Demonstrate z/OS Connect RESTful assess to DB2 & VSAM data via CICS or DVM• Demonstrate access from z/OS to distributed databases • Convert a small existing ETL workload to direct data access • Provide direct data access to a BU who has requested ETL• Use DVM to access SMF data & develop a utilization report• Use DVM o join DB2 & VSAM & distributed databases for a BU, analyze cost VS ETL• Develop a metadata catalog with DVM• Address new reporting or analytics BU backlog & build it• Demonstrate data access from, z/Linux apps• Build a z/OS CICS micro service with APIs
• Take advantage of New Offering ‘Container’ special pricing for new HW & SW from IBM
Thank You
FASTER
IBM STG
Enterprise Data Architecture
16
Characteristic(1) Operational
Systems(2) Informational
Systems(3) AnalyticSystems
Primary Use Mission Critical
Future
Executive
StrategicTactical
Time OrientationPresent
Data Set
Past
Primary Users Line Management
UpdateRead Only-Update
Access Orientation
Access Intent
Table (s) Record
Access Method
Daily
SessionTransaction, Batch,
Periodic
Query, Batch,
Last Save
Daily, Incremental
Unit of Work Last RefreshRecovery Unit
Backup Cycle
Each major category of data store has its own unique functional usage, backup, recovery, refresh, and tuning
characteristics.
Recommended