Upload
gerard-todd
View
216
Download
1
Embed Size (px)
Citation preview
© 2012 IBM Corporation
1
IBM DB2 Analytics AcceleratorSpeed analytics delivery
...powered by Netezza technology
Osama Alkatib, [email protected] z Information Integration IT Specialist
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
DB2 Analytics Accelerator Accelerating decisions to the speed of business
Blending System z and Netezza
technologies to deliver unparalleled,
mixed workload performance for complex
analytic business needs.
Get more insight from your data
• Fast, predictable response times for “right-time” analysis
• Accelerate analytic query response times
• Improve price/performance for analytic workloads
• Minimize the need to create data marts for performance
• Highly secure environment for sensitive data analysis
• Transparent to the application
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
OLTP vs. Analytics – Examples
OLTP - “Transactional” Transactional analytics: Operational BI
Deep analytics: Warehouse / BI / OLAP
Withdrawal from a bank account using an ATM
Approve request to increase credit line based on credit history and customer profile
Regular reporting to central bank – sum of transactions by account
Buying a book at Amazon.com Propose additional books based on similar purchases by other customers
Which books were best-sellers in Europe over the last 2 months?
Check-In for a flight at the airport Offer an upgrade based on frequent flyer history of all passengers and available seats
Marketing campaign to sell more tickets in off-peak times
Hand-over manufactured printers to an oversea-carrier
Optimize shipping by selecting cheapest and most reliable carrier on demand
Trend of printers sold in emerging countries versus established markets.
3
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
Performance & Savings
Accelerating decisions to the speed of business
Queries run faster
• Save CPU resources
• People time
• Business opportunities
Actual customer results, October 2011
Times Faster
Query
Total Rows
Reviewed
Total Rows
Returned Hours Sec(s) Hours Sec(s)Query 1 2,813,571 853,320 2:39 9,540 0.0 5 1,908Query 2 2,813,571 585,780 2:16 8,220 0.0 5 1,644Query 3 8,260,214 274 1:16 4,560 0.0 6 760Query 4 2,813,571 601,197 1:08 4,080 0.0 5 816Query 5 3,422,765 508 0:57 4,080 0.0 70 58Query 6 4,290,648 165 0:53 3,180 0.0 6 530Query 7 361,521 58,236 0:51 3,120 0.0 4 780Query 8 3,425.29 724 0:44 2,640 0.0 2 1,320Query 9 4,130,107 137 0:42 2,520 0.1 193 13
DB2 Only DB2 with
IDAA
DB2 Analytics Accelerator: “we had this up and running in days with queries that ran over 1000 times faster”
DB2 Analytics Accelerator: “we expect ROI in less than 4 months”
5 © 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
Deep DB2 Integration within zEnterprise
DataManager
BufferManager
IRLMLog
Manager
IBMDB2
AnalyticsAccelerator
Applications DBA Tools, z/OS Console, ...
. .
.
Operational Interfaces(e.g. DB2 Commands)
Application Interfaces(standard SQL dialects)
z/OS on System z
Netezza
DB2 for z/OS
Superior availabilityreliability, security,
Workload management
Superior performance on analytic queries
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
DB2 for z/OS Accelerator
Accelerator Data Load
IDAA Studio
IDA
A A
dm
inis
trativ
e S
tore
d P
roc
ed
ure
s
.
.
.
.
.
.
.
.
.
Table A
Part 1
Part 2
Part m
Table C
Table B
Table D
Part 1
Part 2
Part 3
Unload USS Pipe
Unload
Unload
USS Pipe
USS Pipe
CPU FPGA
Memory
CPU FPGA
Memory
CPU FPGA
Memory
CPU FPGA
Memory
Co
ord
inato
r
• 1 TB / h – can vary, depending on CPU resources, table partitioning, …• Update on table partition level, concurrent queries allowed• Trickle-feed update under discussion
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
Query off-load applicabilityIDAA is based on Netezza which supports rich set of SQL and data types
• BI tools such as Cognos have been running on Netezza for years and will run on IDAA as well
Due to very large number of query types and SQL functions, there are some restrictions. Here are the key restrictions:
• No static SQL• Not all DB2 functions, such as HEX, POSITION, SIN, ...• No user defined functions• No correlated table expressions or recursive correlated table expressions• No correlated subquery in the SELECT list • No UTF-16 and MIXED/DBCS EBCDIC• No multiple encoding schemes in the same statement• Not all DB2 special registers: CURRENT PATH, SERVER, SQLID, SCHEMA,
APPLICATION ENCODING SCHEME• Not all DB2 data types such as LOBs, ROWID, XML.
Note: These restrictions could be lifted in a future release
IBM DB2 Analytics Accelerator
8 © 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
Query Execution Process Flow
Optimizer
IDA
A D
RD
A R
equestor
IDAA
Application
ApplicationInterface
Queries executed with IDAA
Queries executed without IDAA
Heartbeat (IDAA availability and performance indicators)
Query execution run-time for queries that cannot be or should
not be off-loaded to IDAA
SPU
Memory
SPU
Memory
SPU
Memory
SPU
Memory
SM
P H
ost
HeartbeatDB2 for z/OS
CPU FPGA
CPU FPGA
CPU FPGA
CPU FPGA
CPU FPGA
CPU FPGA
CPU FPGA
CPU FPGA
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
Workload-Optimized Query Execution
DB2 for z/OS andDB2 Analytics Accelerator
OLTP-like query
OLTP-like query
Light ODS-query
Light ODS-query
Heavy BI QueryHeavy BI Query
Light BI QueryLight BI Query
DB2 Native ProcessingDB2 Native Processing
User control and D
B2 h
euristic
Single and unique system for mixed query workloads
Dynamic decision for most efficient execution platform
New special register QUERY ACCELERATION
– NONE
– ENABLE
– ENABLE WITH FAILBACK
New heuristic in DB2 optimizer
Combines the strengths of both System z and Netezza
Merging operational and data warehouse into a single optimized environment
Optimized processing for BI Workload
9
10 © 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
IBM DB2 Analytics Accelerator V2 Product Components
10Gb
OSA-Express4
10 GbE
Primary
Backup
CLIENT
Data Studio Foundation
DB2 Analytics Accelerator
Admin Plug-in
zEnterprise
Data Warehouse applicationDB2 for z/OS enabled for IBM
DB2 Analytics Accelerator
IBM DB2 Analytics Acelerator
BladeCenter
NetezzaTechnology
Users/Applications
Network
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
SMP Hosts
Snippet BladesTM
(S-Blades, SPUs)
Disk Enclosures
IDAA ServerSQL Compiler, Query Plan, Optimize,Administration
2 front/end hosts, IBM 3650M3 or 3850X5 clustered active-passive 2 Nehalem-EP Quad-core 2.4GHz per host
Processor &streaming DB logicHigh-performance databaseengine streaming joins,aggregations, sorts, etc.e.g. TF12: 12 back/end SPUs (more details on following charts)
Slice of User DataSwap and Mirror partitionsHigh speed data streamingHigh compression rate
EXP3000 JBOD Enclosures 12 x 3.5” 1TB, 7200RPM, SAS (3Gb/s) max 116MB/s (200-500MB/s compressed data)e.g. TF12: 8 enclosures → 96 HDDs 32TB uncompressed user data (→ 128TB)
IDAA powered by Netezza 1000TM Appliance
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
Netezza 1000 Appliance Scalability
1000-3 1000-6 1000-12 1000-24 1000-36 1000-48 1000-72 1000-96 1000-120
Cabinets 1/4 1/2 1 2 3 4 6 8 10
Processing Units 24 48 96 192 288 384 576 768 960
Capacity (TB) 8 16 32 64 96 128 192 256 320
Effective Capacity
(TB)*32 64 128 256 384 512 768 1024 1280
.......
1 10
Capacity = User Data spaceEffective Capacity = User Data Space with compression *: 4X compression assumed
Current IDAA Platforms Future
Predictable, Linear Scalability throughout entire family
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
13
Connectivity Options
Multiple DB2 systems can connect to a single IDAA
A single DB2 system can connect to multiple IDAAs
• residing in the same LPAR• residing in different LPARs• residing in different CECs• being independent (non-data sharing)• belonging to the same data sharing group• belonging to different data sharing groups
Multiple DB2 systems can connect to multiple IDAAs
Full flexibility for DB2 systems:
Better utilization of IDAA resourcesBetter utilization of IDAA resourcesScalabilityScalabilityHigh availabilityHigh availability
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
14
Creating the Hybrid Data ServerCombining the best transaction system with the best analytics system
Transaction Processing Systems (OLTP)
Deep Analytics
z/OS: Recognized leader in
transactional workloads with security, availability
and recoverability
Netezza: Recognized leader in
cost-effective high speed deep analytics
Data Mart Data Mart Data Mart
Data Mart Consolidation
Best in Deep Analytics
Proven appliance leader in high speed analytic systems
Best in Consolidation
Unprecedented mixed workload flexibility and virtualization providing the most options for cost effective consolidation
Best in OLTP and Transactional Analytics
Industry recognized leader in mission critical transaction systems
Together:Destroying the myth that transactional and decision support workloads have to be on separate platforms
Transactional Analytics
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
Free Workload Evaluation/Sizing
1
CustomerDatabaseCustomerDatabase
Documentationand REXX procedure
Documentationand REXX procedure
Data package(mainly unload
data sets)
Data package(mainly unload
data sets)IBM lab
DatabaseIBM lab
Database
Pre-process andload
Pre-process andload
2 3
Quick Workload Test Tool
Quick Workload Test Tool
Report
Assessment
Customer Collecting information from dynamic
statement cache, supported by step-by-step instruction and REXX script (small effort for customer)
Uploading compressed file (up to some MB) to IBM FTP server
IBM / Center of Excellence Importing data into local database Quick analysis based on known DB2
Analytics Accelerator capabilities
Report for a first assessment:
Acceleration potential for
Queries
Estimated time
CP cost
Report for a first assessment:
Acceleration potential for
Queries
Estimated time
CP cost
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
IDAA - Fast Time to Value
Faster Time To
Value
Productivity Benefits Reduced IT Operational Costs due to reduced
complexity Enable “Train of Thought” business processes Over time, once initial IDAA environment is
setup, DBA time freed up for additional projects
– No indexes = less DBA time
– Less database and storage administration
– Less overall application tuning
Delay Capacity Upgrades Offload eligible DB2 workload Reduce Storage Costs
– 1/3 less required Indexes of target tables– Direct savings of eliminated copies
Avoid/Reduce ISV SW Costs Avoid/Reduce Maintenance Costs
Improved Service Levels on Key
Application and Reporting Environments Fast, predictable response times for “right-time”
analysis Accelerated analytic query response times Improved price/performance for analytic
workloads Highly secure environment for sensitive data
analysis Transparent to the application
++
Faster Time to Deliver to Business Eliminate/Simplify ETL Eliminate Copies of Data Centralize Data Access current data using current structures
– Avoid DataMart/DataWarehouse Design Delays
Dramatically Reduce Development Costs Create New Business Opportunities
==Faster
Time toDeliver
ReduceMIPS & CPU
Consumption
Reduce ISV SW Costs
Productivity Benefits
Delay Capacity Upgrades
++ ++++
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
Fast Time to Value
IBM DB2 Analytics Accelerator (Netezza 1000-12) Production ready - 1 person, 2 days
Table Acceleration Setup … 2 Hours– DB2 “Add Accelerator”– Choose a Table for “Acceleration”– Load the Table (DB2 copy to Netezza)– Knowledge Transfer– Query Comparisons
Initial Load Performance … 400 GB “Loaded” in 29 Min
570 million rows (Loads of 800GB to 1.3TB/Hr)
Actual Query Acceleration … 1908x faster2 Hours 39 Minutes to 5 Seconds
CPU Utilization Reduction35% to ~0%
Actual customer results, October 2011
© 2012 IBM Corporation
Information ManagementSmart Business Analytics on System z
What is the value?
Quickly delivers analytics to operational applications
High speed analytics where the data is generated
Enables train-of-thought analysis with high speed complex queries
Substantially reduces operational costs by removing the need for complex query tuning
Creates a highly secure environment for highly sensitive analysis (EAL5)
Speeds batch reporting cycle to meet stricter SLAs
Enables decision makers to perform business analysis they never dared in the past
Enables query acceleration across multiple applications and systems
Capitalizes on DB2 skills and certification removing the need to learn or convert to another SQL environment