Upload
truonglien
View
240
Download
1
Embed Size (px)
Citation preview
IBM Analytics© 2015 IBM Corporation 1
© 2015 IBM Corporation
DB2 Analytics Accelerator Version 5.1New Functions and Capabilities
Patric Becker, Frank Neumann, Ute Baumbach
Analytics on z Systems CoE, Boeblingen Laboratory
November 2015
© 2015 IBM Corporation2
Legal Disclaimer
• © IBM Corporation 2015. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
© 2015 IBM Corporation3
DB2 Analytics Accelerator V5.1 – Functional Highlights
� Version 5.1 of DB2 Analytics Accelerator opens up a new dimension of analytical processing by introducing accelerator-only tables and in-database analytics .
� Accelerator-only tables can benefit statistics and analytics tools that use temporary data
for reports. The high velocity of execution enables these tools to quickly gather all required data.
� Accelerator-only tables enable acceleration of data transformations implemented via SQL statements. Storing interim results in accelerator-only tables enables subsequent queries or data transformations to process all relevant data on the accelerator with high
speed
� In-database analytics capabilities enable acceleration of predictive analytics
applications. This enables SPSS/Netezza Analytics data mining and in-database modeling to be processed within the Accelerator.
IBM Analytics© 2015 IBM Corporation 2
© 2015 IBM Corporation4
DB2 Analytics Accelerator V5.1 – Functional Highlights (contd.)
� System Temporal and Bitemporal tables support (DB2 11 only)
� Active and history table loaded to Accelerator
� Timestamp(12) data type now supported
� Truncation to timestamp(6), new options for ZPARM QUERY_ACCEL_OPTIONS
� Queries using temporal SQL expressions can be routed to Accelerator
� Encryption of data-in-motion
� Encrypt network traffic between z Systems and the Accelerator using Internet Protocol Security (IPsec).
� Network traffic includes:
� DRDA traffic (e.g. Queries, Table Loads)
� Configuration Console traffic
� Incremental Update traffic
� Requires configuration and enablement on z Systems and on the Accelerator
� Incremental update operation enhancements
� Ability to disable query acceleration automatically for suspended tables
© 2015 IBM Corporation5
DB2 Analytics Accelerator V5.1 – Functional Highlights (contd.)
� Improved installability and maintainability
� Extended compatility between stored procedure version level and accelerator version level
� Enhanced product packaging
� Further supported SQL functions: Datetime arithmetic functions, Inlined scalar UDFs (DB2 11 only)
© 2015 IBM Corporation6
Details of new functions - Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
IBM Analytics© 2015 IBM Corporation 3
© 2015 IBM Corporation7
Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
© 2015 IBM Corporation8
Introducing Accelerator-only table type in DB2 for z/OSCreation (DDL) and access remains through DB2 for z/OS in all cases
Non-accelerator DB2 table
• Data in DB2 only
Accelerator-shadow table
• Data in DB2 and the Accelerator
Accelerator-archived table / partition
• Empty read-only partition in DB2
• Partition data is in Accelerator only
Accelerator-only table (AOT)
• “Proxy table” in DB2
• Data is in Accelerator only
Table 1
Table 4
Table 3
Table 2Table 2
Table 4
Table 3
© 2015 IBM Corporation9
In-database transformation and multi-step processing
� DB2 Analytics Accelerator introduces Accelerator-Only tables to store intermediate or final results of data transformation or multi-step processes
� Accelerator-Only tables enable complex data transformations taking place on DB2 Analytics Accelerator, called “In-database transformation”
� Accelerator-Only tables enable intermediate and final results of reporting or other processes to be generated and stored on DB2 Analytics Accelerator, called “In-database multi-step processing”
� Advantages:
� Accelerate in-database data transformations and data movement processes
� Reduced need of data movement processes to other platforms for data transformation purposes
� Enables multi-step reporting on the Accelerator
� Saves disk space and CPU cost on z Systems currently used for transformations and reporting steps
� Allow data preparation steps for data mining and other advanced analytics to execute on the Accelerator
� Store a set of data in DB2 Analytics Accelerator only, not on DB2 for z/OS, without
using the High Performance Storage Saver functionality
IBM Analytics© 2015 IBM Corporation 4
© 2015 IBM Corporation10
Accelerator-only tables – Technical basics
� AOTs are created and dropped using DB2 DDL statements (CREATE; DROP)
• Accelerator must be started
• QUERY ACCELERATION behavior may have any value during CREATE/DROP
• Syntax:
� CREATE TABLE MYTABLE (...) CCSID ccsid IN ACCELERATOR <ACCEL1>;
� DROP TABLE MYTABLE;
� Recommended to create a database in DB2 to be used for the AOTs
• CREATE TABLE MYTABLE (...) CCSID ccsid IN ACCELERATOR <ACCEL1> IN
DATABASE MYDB;
• Usual authorization necessary to create objects in database
� SELECT and INSERT/UPDATE/DELETE operations using AOTs can only run on the Accelerator
• QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL
• Accelerator-shadow tables, Accelerator-archived tables and other AOTs can be used in the same statement
� Dynamic and static SQL can be used with AOTs
© 2015 IBM Corporation11
Accelerator-only tables – Technical basics
� There are some restrictions for using certain DDL against AOTs
� For example: ALTER TABLE, RENAME TABLE
� Restrictions on clauses/keywords used in DDL statements
� For example: Generated, Partition By, WITH DEFAULT
� Restrictions on data types, column types, usage and code pages
� LOB columns
� LOCK TABLE, reference to MQTs, TRUNCATE
� In general, the same data type restrictions as found in normal accelerated shadow DB2 tables
� AOT counters in IFCIDs 2 (Statistics) and 3 (Accounting)
� See the IBM DB2 Analytics Accelerator for z/OS documentation for a full list
© 2015 IBM Corporation12
Accelerator-only tables – Technical basics
� Queries using AOTs can only run on the Accelerator
� QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL
� INSERTS, UPDATES and DELETES to AOTs can only run in the Accelerator
� QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL
� INSERT FROM SELECT into an AOT
� Can only run on the accelerator
� All tables in the SELECT must reside in the accelerator
� Accelerated shadow DB2 tables
� Archive tables
� AOTs
� QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL
� Cross-Loader to load data from AOTs into DB2 for z/OS
� Different table names� PI49378 may be needed: WHEN USING DB2 CROSSLOADER WITH SET CURRENT QUERY
ACCELERATION, THE SPECIAL REGISTER IS NOT APPLIED TO THE QUERIES THAT ARE RUN
IBM Analytics© 2015 IBM Corporation 5
© 2015 IBM Corporation13
Multi-step reporting applications with DB2 for z/OS BEFORE Accelerator-only tables: Report processing on DB2, source data might reside on the Accelerator already
Temporary results
2
Temporary results
1Temporary results
Reports and Dashboards
n
Multi-Step Report
n
1
2
Reporting Application
Credit Card
Transaction History
Customer
Summary Mart
Credit CardTransaction History
CustomerSummary Mart
© 2015 IBM Corporation14
Multi-step reporting applications with DB2 for z/OSWith Accelerator-only tables: Temporary objects and processing on the Accelerator
Temporary results
2Temporary results
1Temporary results
Reports and Dashboards
n
Multi-Step Report
n
1
2
Reporting Application
Credit CardTransaction History
CustomerSummary Mart
Credit CardTransaction History
CustomerSummary Mart
Data for transactional and analytical processing
© 2015 IBM Corporation15
In-database transformationUsing Accelerator-only tables and ELT logic in the Accelerator
Transaction Processing Systems (OLTP)
Analytics
Advantages: • Simpler to manage
• Better performance and reduced latency Data for transactional and analytical processing
Customer Transactions
CustomerData
Customer Transaction Summary and History AOTs
Customer Summary Mart AOTs
Customer Transactions
CustomerData
ELT logic
IBM Analytics© 2015 IBM Corporation 6
© 2015 IBM Corporation16
Data scientist work areaUsing Accelerator-only tables for ad-hoc analysis
Transaction Processing Systems (OLTP)
Data for transactional and analytical processing
Customer Transactions
CustomerData
Customer Transactions
CustomerData
Work database John
Work Area AOTs
Work database Bob
Work Area AOTs
Data Scientist John
Data Scientist Bob
© 2015 IBM Corporation17
Integrate more data sources for analyticsLoad external data to the accelerator and save combined analytic results in accelerator-only tables
Transaction Processing Systems (OLTP)
AnalyticsData for transactional and analytical processing
Customer Transactions
CustomerData
Combined result in AOTs
Customer Transactions
CustomerData
Related data from
other sourcesExternal
Files
Imagecopy
DB2 Analytics Accelerator Loader
© 2015 IBM Corporation18
• Deeper insight into operational status through faster reportingSupport multistep reporting applications. Third party BI reporting suites, QMF, home grown applications, etc.
• Accelerated campaign tuning for IBM Campaign (Unica)Improved performance for iterative campaign tuning.
• Simplifying data-transformation processesDelivering in-database transformation within DB2 Analytics Accelerator
• Data mart consolidation through flexible data infrastructure Host data marts on z Systems, where the data originates
• Deeper insight into customers and marketsData scientist work area
• Simpler data integration with DB2 Analytics Accelerator Loader for z/OS (load non-DB2 for z/OS data)Assimilate more data sources for analytics to shorten development cycles and speed integration efforts
• In-database analytics to accelerate predictive analyticsImproves the quality of models, speeds calculations and delivers real-time, actionable business processes
What can I do with AOTs?
IBM Analytics© 2015 IBM Corporation 7
© 2015 IBM Corporation19
Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
© 2015 IBM Corporation20
In-database analytics
Enable acceleration of predictive analytics applications
In-database analytics enables SPSS/NetezzaAnalytics (INZA) data mining and in-database
modeling to be processed within the Accelerator
� Accelerates SPSS/Netezza Analytics (INZA) data mining and in-database modeling
� Allows frequent model refreshes to enable adequate scoring
� Reduced need of data movement processes to other platforms for predictive analytics purposes
� Supports the full lifecycle of a real-time analytics solution on a single,
integrated system, combining transactional data, historical data, and predictive analytics
© 2015 IBM Corporation21
In-database analytics – Technical basics
� Set of stored procedures of IBM Netezza In-Database Analytics Package (INZA) are available on the Accelerator for modeling purposes, including:
� Decision Tree
� Regression Tree
� Naive Bayes
� K-means Clustering and TwoStep Clustering
� Stored procedures use accelerator-shadow tables or accelerator-only tables as input and create accelerator-only tables as output
� DB2 for z/OS contains stored procedure wrappers to enable invocation of the
stored procedures from SPSS Modeler 17
� Actual stored procedure execution takes place on Accelerator
� Separate install of the IBM Netezza Analytics stored procedures required on the Accelerator:
� http://www-01.ibm.com/support/knowledgecenter/SS4LQ8_5.1.0/com.ibm.datatools.aqt.doc/installmanual/concept/c_idaa_inst_analytics.html?lang=en
IBM Analytics© 2015 IBM Corporation 8
© 2015 IBM Corporation22
In-database analytics – Technical basics
� SPSS Modeler 17 uses the following INZA stored procedures that now run on the Accelerator:
� dectree - Builds a Decision Tree model by growing and pruning a tree
� grow_dectree - Builds a Decision Tree model
� predict_dectree - Applies a Decision Tree model to generate classification predictions
� prune_dectree - Prunes a previously built Decision Tree model
� regtree - Builds a Regression Tree model by growing and pruning a tree
� grow_regtree - Builds a Regression Tree model
� prune_regtree - Prunes a previously built Regression Tree model
� predict_regtree - Applies a Regression Tree model to generate regression predictions for
a dataset
� naivebayes - Builds a Naive Bayes model
� predict_naivebayes - Applies a Naive Bayes model to generate classification predictions
for a dataset
� kmeans - Builds a Clustering model that clusters the input data into k centers. The
centers are calculated as the mean value of the nearest input data records
� predict_kmeans - Applies a K-means Clustering model to cluster records of a dataset
© 2015 IBM Corporation23
In-database analytics – Technical basics
� two_step - Builds a TwoStep Clustering model that first distributes the input data into a
hierarchical tree structure according to the distance between the data records, then
reduces the tree into k clusters. A second pass over the data associates the input data
records to the next cluster
� predict_twostep - Applies a TwoStep Clustering model to score records of a dataset
� split_data - Randomly splits the input data into two separated subsets
� pmml_model - Stores the given analytics model as PMML document to a table
� export_pmml - Exports the given analytics model as PMML document to a file, or it
exports a model from a PMML table to a file. If no PMML table exists containing the
PMML document for this model, one can be created automatically when requested.
Optionally, instead of writing to a file, the result can be returned by the procedure.
� model_exists - Checks if the given model exists. The model can be searched in the current
or in another given database.
� drop_model - Drops the given model. All managed tables of this model are also dropped.
© 2015 IBM Corporation24
In-database analyticsData preparation (using AOTs) and SPSS modeling in the Accelerator
Transaction Processing Systems (OLTP)
With embedded scoring
Advantages: • Allows fast model refreshes
• Ensures adequate scoring
• Better performance and reduced latency
Data for transactional and analytical processing
Customer Transactions
CustomerData
Customer TxnData Prep AOTs
Customer Transactions
CustomerData
Modeling
ModelModelSPSS Modeler
IBM Analytics© 2015 IBM Corporation 9
© 2015 IBM Corporation25
source(select)
process process typenode
modeling nugget score output
Step 1 DB2 Analytics Accelerator Query Acceleration
Step 2 Step 3 In-database transformationand analytics
Without Accelerator: The SPSS source node selects the input data from DB2 z/OS without any acceleration, then all data preparation is done in SPSS Modeler, followed by the modeling algorithm in
SPSS modeler, then the model is scored and scoring results are inserted into a DB2 z/OS result table
Source tables are accelerator-shadow tables and the accelerator processes the (complex)select statement
Step 1
Data preparation is pushed down into the accelerator producing an accelerator-only table for thetype node
Step 2
INZA modeling algorithm is called and executed in the accelerator on an accelerator-onlytable
Step 3
Data Source Access and Preparation Modelling Scoring
Source datasource process process t
(select) (select) Result data
In-database analyticsData preparation (using AOTs) and SPSS modeling in the Accelerator
With Accelerator:
© 2015 IBM Corporation26
Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
© 2015 IBM Corporation27
Temporal Table support in DB2 Analytics Accelerator
� With V4 PTF 5
� Application(Business)-period temporal tables
� With V5.1, but for DB2 11 only
� System-period temporal tables
• Active table
• History table: contains the historical versions of the rows in the active
table
• Uses TIMESTAMP(12) for Begin and End columns
� Bitemporal tables
• Contains both Business Time temporal and System time temporal
definition
Temporal Table Concept: DB2 supports time-based data management that allows you to insert, update, delete, and query data in the past, the present, and future while keeping a complete history of what you knew and when you knew it.
IBM Analytics© 2015 IBM Corporation 10
© 2015 IBM Corporation28
Timestamp(12) support in DB2 Analytics Accelerator
� With DB2 Analytics Accelerator V5.1 tables with timestamp(12) columns can be added
and loaded to the Accelerator with the truncation of timestamp(12) to timestamp(6).
• Impact: Queries running on the Accelerator return different results than on DB2 for z/OS for these columns
� Introduction of two new options 5 and 6 for ZPARM QUERY_ACCEL_OPTIONS to
control the truncation of timestamp precision > 6.
• Option 5: queries referencing system temporal tables are eligible.
• Option 6: queries referencing an expression or regular (non system temporal) table
columns with a TIMESTAMP(12) data type are eligible.
Netezza NPS doesn't support timestamp(12). Workaround: MAPPING timestamp precision 12 to timestamp precision 6 on the Accelerator.
© 2015 IBM Corporation29
System Time Temporal Query Routing
� Both active and history tables with Timestamp(12) can be loaded to the Accelerator
� Queries on system temporal tables are routed to the Accelerator when ZPARM
QUERY_ACCEL_OPTIONS is set to 5 or 6
• Special query rewrite is applied for the following 3 temporal SQL:
• FOR SYSTEM_TIME AS OF expr
• FOR SYSTEM_TIME FROM expr1 TO expr2
• FOR SYSTEM_TIME BETWEEN expr1 AND expr2
� For Bitemporal tables the support is straightforward since both System Time and Business Time temporal tables anc clauses are supported
© 2015 IBM Corporation30
Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
IBM Analytics© 2015 IBM Corporation 11
© 2015 IBM Corporation31
Encryption of data …
� Encryption of data “at rest”
� Already available by default with N3001 self-encrypting disks (SEDs)
� Addresses risk of stolen disks
� Encryption of “data-in-motion”
� Encrypt network traffic between z Systems and the Accelerator using Internet Protocol Security (IPsec)
� Network traffic includes:
� DRDA traffic (e.g. queries and results, table loads)
� Configuration Console traffic (passwords)
� Incremental Update traffic
� Requires configuration and enablement on z Systems and on the
Accelerator
� Addresses network sniffing (interception of TCP/IP packets)
© 2015 IBM Corporation32
Do you need encryption of data-in-motion ?
� Recommended setup is private data network, also see http://www-01.ibm.com/support/docview.wss?uid=swg27038935
• This is still the best and recommended alternative to ensure security
• Private data network has no performance impact (> 2 TB/hour load thruput)
� Sample domains for encryption of data-in-motion:
• Some customers have multiple data centers and connect the remote IDAA over a non-
private network (routed, accessible from intra-net)
• Some customer have compliance rules (security policy) to encrypt all network traffic
� However Ipsec
• Is complex to administer/setup
• Reduces network bandwidth
• All incoming traffic to one accelerator combined is currently limited to approx. 700-900 GB/hour
� Get help from z/OS Communications Server L2/L3 if you are not a network expert!
© 2015 IBM Corporation33
Encryption of data in DataStudio
Icon Description
Disks are not encrypted.Network traffic is not encrypted
Disks are not encrypted.Network traffic is encrypted for at least one z/OS system.
Disks are encrypted.Network traffic is not encrypted.
Disks are encrypted.Network traffic is encrypted for at least one z/OS system.
One of the certificates used for network encryption is expiring in 17 days.orOne of the certificates used for network encryption has expired.
Click on details link
shows details about certificates and keys
Click on details link
shows details about certificates and keys
IBM Analytics© 2015 IBM Corporation 12
© 2015 IBM Corporation34
Encryption, keys, certificates
It is more complex than this• Security Association (SA) with lifetime
• Authentication phase is implemented by a component called "IPsec Key Exchange" (IKE), needs to be encrypted as well
• Key and certificate management• Configuration restart required for an initially
unencrypted connection
Public/private keys are used to encrypt and decrypt dataCertified by trusted certificate authority (CA)
© 2015 IBM Corporation35
Prereqs for z/OS LPAR and Accelerator
� z/OS
• Communication server V 1.13 or higher or V2.1 or higher supporting IKEV2
• Set up and configure the TCP/IP stack of communication server with IKED (=Internet
Key Exchange daemon), NSSD (=Network security services daemon), policy agent (=manages policy definitions and dependent services), TRMD (=Traffic Regulation Management daemon), SYSLOGD (=system logging daemon)
• Implementing IPsec in the z/OS communications server - depending on the prior experience with that setup - can be a lengthy, complex process that requires planning and preparation. The system programmer performing the configuration steps on the
z/OS side should be familiar with z/OS communications server
� Accelerator (only N200x (Striper) and N3001 (Mako) models supported)
• Host management version 5.4.1.0 or higher incl. libreswan (installed by IBM service for
existing systems)
� Network
• For existing firewalls in the private data network: configure IKED port 500 the protocols
50 and 51.
© 2015 IBM Corporation36
Configuration sequence
1. Generate a key pair and certificate for the z/OS side and activate it in the keyring owned by NSSD
2. Generate a key pair and certificate for the accelerator peer and export it into a PKCS#12 binary file. Store the file in the USS directory used for IDAA software updates
3. Transfer the PKCS#12 certificate file to the accelerator: IDAA Studio Accelerator View -> Transfer Update
4. Import the transferred certificate into the accelerator NSS database: IDAA configuration console -> (8) -(Menu) Manage Encryption of Data in Motion -> (1) - Import PKCS12 file with RSA keys and certificates
5. Restart the IPsec service to activate an imported certificate: IDAA configuration console ->(8) - (Menu) Manage Encryption of Data in Motion -> (7) - Restart the IPsec service
6. z/OS: Prepare the IPsec policy agent configuration for the LPAR
7. Enable an LPAR for IPsec on the accelerator: IDAA configuration console ->(8) - (Menu) Manage
Encryption of Data in Motion -> (3) - Enable encryption of data in motion
8. z/OS: Activate the prepared IPsec policy on the LPAR (Refresh policy agent)
9. Test your connection and diagnose problems
10. Confirm the IPsec configuration: IDAA configuration console -> (8) - (Menu) Manage Encryption of Data in Motion -> (4) - Confirm enablement of encryption
11. Verify that the network traffic is encrypted: IDAA configuration console -> (8) - (Menu) Manage Encryption of Data in Motion -> (6) - Display the status of encrypted connections
Configuration Details: http://www-
01.ibm.com/support/docview.wss?uid=swg27047011&myns=swgimgmt&mynp=OCSS4LQ8&mync=E&cm_sp=swgimgmt-_-
OCSS4LQ8-_-E
Known Issues: http://www-01.ibm.com/support/docview.wss?uid=swg27047150
IBM Analytics© 2015 IBM Corporation 13
© 2015 IBM Corporation37
Console options for encryption of data and performance implications
Because current RedHat 6 Linux kernel allows only serial de/encryption (single threaded), throughput of Accelerator operations is currently limited to ~700 GB/h
for encrypted traffic. (Compared to 2-4 TB/h for unencrypted traffic)
Because current RedHat 6 Linux kernel allows only serial de/encryption (single threaded), throughput of Accelerator operations is currently limited to ~700 GB/h
for encrypted traffic. (Compared to 2-4 TB/h for unencrypted traffic)
© 2015 IBM Corporation38
Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
© 2015 IBM Corporation39
Incremental update improvements in 2015
• V4.1 PTF 5 (May 2015)
• Suspending faulty tables from replication
• Improved incremental update performance
– Parallel Apply
– Automatic definition of distribution and organization keys
• V4.1 PTF 6 (4Q 2015)
• Continuous operation of replication
• V5.1 (4Q 2015)
• Disable query acceleration for suspended faulty tables
IBM Analytics© 2015 IBM Corporation 14
© 2015 IBM Corporation40
Disable query acceleration for suspended faulty tables
� Available since V4.1 PTF5: “Suspend faulty table”
• Replication of tables may be suspended & mirroring continuous w/out table
• By default: “Suspend Faulty Tables” is on
• Issue: queries still accelerated while faulty table is not replicated anymore
� New capability
• Automatically disable query acceleration for faulty tables
• Generate DSNX message: notify query acceleration is disabled for table
• Configuration
• Off per default (to avoid breaking existing automation)
• Turn on in configuration console per subsystem / database system
© 2015 IBM Corporation41
Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
© 2015 IBM Corporation42
Product packaging and installation enhancements and changes
Keep only applied software package files in HFS (more like SMP/E), automatic clean-up of older versions when SMP/E updates are applied.
Option to switch back to the old behavior so that old files are not deleted.
Keep only applied software package files in HFS (more like SMP/E), automatic clean-up of older versions when SMP/E updates are applied.
Option to switch back to the old behavior so that old files are not deleted.
For DB2 object dropping, re-creation and procedure bind:
New installation program AQTTRIN, invoked from job AQTTIJSP.
Performs required steps without step customization.
For DB2 object dropping, re-creation and procedure bind:
New installation program AQTTRIN, invoked from job AQTTIJSP.
Performs required steps without step customization.
V5.1 available as fully working GA version.
Before, we always requested to install PTF1 at GA time.
Release Notes: http://www-1.ibm.com/support/docview.wss?uid=swg27047096&myns=swgimgmt&mynp=OCSS4LQ8
V5.1 available as fully working GA version.
Before, we always requested to install PTF1 at GA time.
Release Notes: http://www-1.ibm.com/support/docview.wss?uid=swg27047096&myns=swgimgmt&mynp=OCSS4LQ8
AQT_INSTALL_PREFIX environment variable has been removed. Specify an absolute path in the environment variable AQT_HOST_PACKAGE_DIRECTORY. AQT_INSTALL_PREFIX environment variable has been removed. Specify an absolute path in the environment variable AQT_HOST_PACKAGE_DIRECTORY.
IBM Analytics© 2015 IBM Corporation 15
© 2015 IBM Corporation43
Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
© 2015 IBM Corporation44
Installation dependencies
� Sequence of applying updates is crucial
� New flexibility
• Software upgrade of server components when stored procedure level > Accelerator server level(starting with 5.1 stored procedures and accelerator server 4.1)
© 2015 IBM Corporation45
Agenda
� Accelerator-only tables
• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only
• Now available for DB2 for z/OS 10 and DB2 for z/OS 11
� Analytics Server, SPSS, INZA
� Temporal Tables
� Encryption of Data-in-motion
� Incremental Update
� Product Packaging Enhancements
� Stored Procedure and Component Dependencies
� Further SQL support
IBM Analytics© 2015 IBM Corporation 16
© 2015 IBM Corporation46
Further SQL support for query acceleration
� For DB2 11 only, enabled with DB2 APAR PI42662 and NPS 7.2.1
• Datetime arithmetic functions support – adding and subtracting date, time, timestamp types
• Motivation: Different date, time representations within DB2 and NPS lead to datetime arithmetic incompatibilities
• DB2 uses yymmdd representation for data, NPS uses interval representation
• E.g. 115 means 1 month and 15 days on DB2, but 115 days in NPS
• Inlined scalar UDF support
• Definition: An inlined SQL scalar returns a single value each time it is invoked. The returned value is specified in the RETURN statement of the
body of the function.
© 2015 IBM Corporation47
Supported Datetime Arithmetic Examples
Arithmetic (addition and subtraction) with date, time, timestamp types
� SELECT DATEC1 - DATEC2 FROM IDAA.TIMETB FOR FETCH ONLY;
Arithmetic (addition and subtraction) with datetime constants:
� SELECT DATEC1 - '2013-02-01' FROM IDAA.TIMETB FOR FETCH ONLY;
Arithmethic with labeled durations (were already supported before) � SELECT DATEC - 1 MONTH FROM IDAA.TIMETB FOR FETCH ONLY;
© 2015 IBM Corporation48
DDL
CREATE FUNCTION
HCM.GMT_DATE (p_date DATE, p_time TIME, p_gmt_off_set BIGINT)
RETURNS DATE LANGUAGE SQL
DETERMINISTIC READS SQL DATA NO EXTERNAL ACTION
RETURN(
DATE (TIMESTAMP(p_date , p_time) + p_gmt_off_set seconds)
);
Query Reference
WHERE HCM.GMT_DATE( VIEW.DATE, VIEW.TIME, 0) BETWEEN DATE('2014-08-02') AND DATE('2014-09-01')
Inlined Scalar UDF Example
IBM Analytics© 2015 IBM Corporation 17
© 2015 IBM Corporation49
Inlined Scalar UDF Restriction for Acceleration
� Inline scalar UDF DDL cannot be nested. Nested inline scalar UDF in a query is allowed though.
Not allowed
CREATE FUNCTION INSCUDF1(arg1 date)…;
CREATE FUNCTION INSCUDF2(arg1 integer)
…
RETURN arg1*DATE(INSCUDF1(CURRENT_DATE));
Allowed
SELECT C1
…
WHERE INSCUDF3(COL1,INSCUDF4(COL2)) > 12
© 2015 IBM Corporation50