Informatica PowerCenter 8.6Pushdown Optimization

April 11, 2023 2

Pushdown Optimization Overview

Running Pushdown Optimization

Running Source-Side Pushdown Optimization Running Target-Side Pushdown Optimization Running Full Pushdown Optimization

Working With Databases

Using ODBC drivers

Working With Expressions

Operators Variables Functions

Agenda

April 11, 2023 3

Working With Transformations

Aggregator transformation Expression Transformation Filter Transformation Joiner Transformation Lookup Transformation Sorter Transformation Union Transformation Source Qualifier transformation Target

Working With Sessions

Working With partition Working With Target Loads Rules Error Handling, Logging, and Recovery

Agenda

April 11, 2023 4

Working With SQL Overrides

Views Troubleshooting Orphaned Views Rules and Guidelines

Using the $$PushdownConfig Mapping Parameter

Viewing Pushdown Groups

Rules and Guidelines

Agenda

April 11, 2023

Pushdown Optimization Overview

April 11, 2023 6

Pushdown Optimization Overview

Use the Pushdown Optimization Viewer to

preview the SQL statements ,pushdown messages and mapping logic that is push to the

source or target database

Pushes the transformation logic to the source or target database

The Integration Service analyzes the mapping and writes one or more SQL statements based on the mapping transformation logic,

mapping, session and pushdown configuration

April 11, 2023

Running Pushdown Optimization

April 11, 2023 8

Running Source-Side Pushdown Optimization

Integration Service generates SELECT

statement based on the transformation logic for each transformation it

can push to the database

All valid transformation logic are pushed to DB

For SQL or Lookup override views are generated. Select statement runs against the view

The Integration Service analyzes the mapping from the source to the target or until it reaches a downstream transformation it cannot push to the database

April 11, 2023 9

Running Target-Side Pushdown Optimization

The Integration Service analyzes the mapping from the target to the source or until it reaches an upstream transformation it cannot push to the database

Integration Service generates INSERT, DELETE, or UPDATE statement based on the transformation logic

Processes transformation logic up to the point that it can push the transformation logic to the target database

April 11, 2023 10

Running Full Pushdown Optimization

Only when Source and Target are on same Database

INSERT SELECT statement that is run on the database – Incorporating the whole of transformation logic

When a session runs for full pushdown optimization, the

database must run a long transaction if the session

contains a large quantity of data

April 11, 2023

Working With Databases

April 11, 2023

Working With Databases

Pushdown optimization can be configured for the following databases :

Oracle 9.x and above IBM DB2 Teradata Microsoft SQL Server Sybase ASE Databases that use ODBC drivers

April 11, 2023

Working With Databases

When native drivers are used, the Integration Service generates SQL statements using native database SQL

When ODBC drivers are used, the Integration Service generates SQL statements using ANSI SQL

The Integration Service can generate a greater variety of functions when it generates SQL statements using native language instead of ANSI SQL

April 11, 2023

Working With Databases

The database may produce different output than the Integration Service when the following settings and conversions are different:

Nulls treated as the highest or lowest value Sort order Case sensitivity Numeric values converted to character values. Date values converted to character values Precision SYSDATE system variable

April 11, 2023

Working With Databases

Nulls treated as the highest or lowest valueNull value is treated as low in the sorter transformation

Oracle treats null values as the highest value in the sort order

Returned results therefore will differ as null values are treated differently

April 11, 2023

Working With Databases Sort Order

The sort order is case sensitive

Sort Order in Microsoft SQL Server database is not case sensitive

Returned results therefore will differ as sort orders are different.

April 11, 2023

Working With Databases

Case Sensitivity

The filter condition will return the values where col_varchar2 = ‘CA’

Database which are not case sensitive will return rows that match the values ‘Ca,’ ‘ca,’ ‘cA,’ and ‘CA’

Therefore the results will differ in case of pushdown optimization

April 11, 2023

Working With Databases

Numeric values converted to character values

The IN_EMPLOYEE_ID = 1234567890 when converted to character will be ‘1234567890’

Database might convert the number to ‘1.2E9’

Returned results may differ because of different conversion format

April 11, 2023

Working With Databases

Date values converted to character values

The ROUND function on a date, stores the date value in a character column, using the format MM/DD/YYYY HH:MI:SS.US

Oracle stores the date value using default format as DD-MON-YY

Disable pushdown optimization if the date format required is MM/DD/YYYY HH:MI:SS.US

April 11, 2023

Working With Databases

Precision

A transformation Decimal datatype has a precision of 1-28

Teradata Decimal datatype has a precision of 1-18

Therefore the results will differ as database uses a different precision than the Integration Service

April 11, 2023

Working With Databases

SYSDATE system variable

The current date and time for the node running the service process is returned

On Pushdown optimization, the SYSDATE variable returns the current date and time for the machine hosting the database HH:MI:SS.US

If the time zone of the machine hosting the database is not same as the time zone of the machine running the Integration Service process, the results can vary

April 11, 2023

Using ODBC Drivers

When ODBC drivers are used, the Integration Service cannot detect the database type and must generates ANSI SQL

In some cases, ANSI SQL is not compatible with the syntax required for a particular database

Different databases encounters different types of problems

April 11, 2023

Using ODBC Drivers

IBM DB2A session containing a Sorter transformation fails if it is configured for both a distinct and case-insensitive sort and the one of the sort keys is a string datatype.

A session containing a Lookup transformation fails for source-side or full pushdown optimization.

A session that requires type casting fails if the casting is from x to date/time or from float/double to string, or if it requires any other type casting that IBM DB2 databases disallow

April 11, 2023

Using ODBC Drivers

Microsoft SQL Server

A session containing a Sorter transformation fails if it is configured for both a distinct and case-insensitive sort

Sybase ASE

When Sybase ASE 12.5 or below is used, sessions that perform datatype conversions fail. The session fails if Joiner transformation used is configured for a full outer join.

April 11, 2023

Using ODBC Drivers

Teradata

Teradata sessions fail if the session requires a conversion to a numeric datatype and the precision is greater than 18. Teradata sessions fail when full pushdown optimization is used for a session containing a Sorter transformation A sort on a distinct key may give inconsistent results if the sort is not case sensitive and one port is a character port A session containing an Aggregator transformation may produce different results from PowerCenter if the group by port is a string datatype and it is not case-sensitive A session containing a Lookup transformation fails if it is configured for target-side pushdown optimizationA session that requires type casting fails if the casting is from x to date/time. A session that contains a date to string conversion fails.

April 11, 2023

Working With Expressions

April 11, 2023

Working With Expressions

When pushdown optimization is used, the Integration Service converts the expression in the transformation or in the workflow link by determining equivalent operators, variables, and functions in the database.

If there is no equivalent operator, variable, or function, the Integration Service processes the transformation logic.

April 11, 2023

Working With Expressions - Operators

Operator Oracle IBM DB2 Teradata Sybase ASE

Microsoft SQL

Server

ODBC

+ - * / X X X X X X

= > < >= <= <>

X X X X X X

!= X X X X X X

^= X X X X X X

not and or X X X X X X

|| X Source Source Source Source -

% X X X X X -

April 11, 2023

Working With Expressions - Variables

Variable Oracle IBM DB2

Teradata SybaseASE Microsoft SQL

Server

ODBC

SESSSTARTTIME X X X X X X

SYSDATE X X X X X -

WORKFLOWSTARTTIME - - - - - -

April 11, 2023

Working With Expressions - Functions

Function IBM DB2

Microsoft SQL

Server

Oracle Sybase ASE

Teradata ODBC Exceptions

ABS() X X X X X XABORT()

ADD_TO_DATE() X S X S X Cannot push transformation logic to a Teradata database for a transformation that uses an expression containing ADD_TO_DATE to change days, hours, minutes, or seconds.

AES_DECRYPT()

AES_ENCRYPT()

ASCII() X X X XAVG() X X X X X XCEIL() X X X X SCHOOSE()

CHR() X X X XCHRCODE()

April 11, 2023

Working With Expressions - Functions

Function IBM DB2 Microsoft SQL Server

Oracle Sybase ASE

Teradata ODBC

COMPRESS() X X X X X X

CONCAT() S S X S S

COS() X X X X X X

COST() X X X X X X

COSH() X S X S X

COUNT() X X X X X X

CRC32() X X X X X X

CUME() X X X X S

DATE_COMPARE() S S S S S S

DATE_DIFF() X X X X

DECODE() X X X X X X

April 11, 2023

Working With Expressions - Functions

Function IBM DB2 Microsoft SQL Server

Oracle Sybase ASE

Teradata ODBC

DECODE_BASE64() X X X X X X

DECOMPRESS() S S X S S

ENCODE_BASE64() X X X X X X

ERROR() X X X X X X

EXP() X X X X X X

FIRST()

FLOOR() X X X X S

FV()

GET_DATE_PART() X X X X X

GREATEST() X

IIF() X X X X X X

IN() X X X X X X

INDEXOF()

INITCAP() X

INSTR() S X X S S

IS_DATE()

April 11, 2023

Working With Expressions - Functions

Function IBM DB2

Microsoft SQL Server

Oracle Sybase ASE

Teradata ODBC Exceptions

IS_NUMBER()

IS_SPACES()

ISNULL() X X X X X X

LAST() Oracle returns the date up to the second. If the input date contains subseconds, Oracle trims the date to the second

LAST_DAY() X

LEAST() X

LENGTH() X X X X X

LOWER() X X X X X X

LPAD() X

LTRIM() X X X X X If LTRIM in transformation logic, is used the database treats the argument (' ') as NULL, but the Integration Service treats the argument (' ') as spaces

LOG() X S X S S

Note: For details on remaining functions refer to the Workflow Administration Guide

April 11, 2023

Working With Transformations

April 11, 2023

Working With Transformations

The Integration Service can push transformation logic to the source, target, or both.

Some transformations cannot be pushed to the database, and additional rules apply to each transformation that can be pushed to the database

The Integration Service processes the transformation logic if any of the following conditions are true:

The transformation logic updates a mapping variable and saves it to the repository database

The transformation contains a variable port

Override default values for input or output ports

April 11, 2023

Working With Transformations

Summary of Mapping Objects Valid for Pushdown Optimization

Transformation Source-Side and Full Target-Side

Aggregator X -

Application Source Qualifier - -

Custom - -

Expression X X

External Procedure - -

Filter X -

HTTP - -

Java - -

Joiner X -

Lookup X X

Normalizer - -

Rank - -

April 11, 2023

Working With Transformations

Summary of Mapping Objects Valid for Pushdown Optimization

Transformation Source-Side and Full Target-Side

Router - -

Sequence Generator - -

Sorter X -

Source Qualifier X -

SQL - -

Stored Procedure - -

Target X -

Transaction Control - -

Union X -

Update Strategy - -

XML Generator - -

XML Parser - -

XML Source Qualifier - -

April 11, 2023 38

AGG_ITEMIDSOR_OrderID TARGET_ITEM S_ORDER2 (Or acle)

SQ_TARGET_I TEMS_ORDER 1

TARGET_ITEM S_ORDER1 (Or acle)

AGG_OrderID

The Integration Service cannot push the aggregator transformation logic to the database if any of the following conditions are true

Working With Transformations – Aggregator Transformation

A session is configured for incremental aggregation

Transformation logic for the first Aggregator is pushed to the database, and the Integration Service processes the second Aggregator transformation

April 11, 2023

The Integration Service cannot push the aggregator transformation logic to the database if any of the following conditions are true

Working With Transformations – Aggregator Transformation

Any one of the aggregate functions FIRST(), LAST(), MEDIAN(), or PERCENTILE() is used

An output port is not an aggregate or a part of the group by port

Nested aggregate function or Conditional aggregate clause is used

April 11, 2023

Working With Transformations – Expression and Filter Transformation

order12 (Flat File )

SQ_ORDERSORDERS (Oracl e)

EXP_Transform Data

TARGET_ITEM S_ORDER (DB2 )

SQ_ORDER_IT EMS

ORDER_ITEMS (Oracle)

SQ_ITEMS FIL_DisContFlag

JOIN_items

ITEMS (Oracle)

All the valid databases for the pushdown optimization can use all Source Side and Full Pushdown Optimization Type when Filter Transformation is used

All the valid databases for the pushdown optimization can use all three types of Pushdown Optimization when Expression Transformation is used

April 11, 2023

The Integration Service cannot push the Joiner transformation logic to the database if any of the following conditions are true:

Working With Transformations – Joiner Transformation

SQL cannot be generated to represent an outer join combined with a multi-table join that joins greater than two tables

The Joiner is configured for a full outer join and attempt to push transformation logic to a Sybase database

If the Aggregator transformation is placed upstream from a Joiner transformation in the pipeline

April 11, 2023

Guidelines for Lookup transformations in Pushdown Optimization

Working With Transformations – Lookup Transformation

Lookup table and source table must be on the same database for source-side or target-side pushdown optimization

Lookup should not be configured to Dynamic Cache or Policy on multiple match as First or Last Value

To use pushdown optimization, you must configure the Lookup transformation to report an error on multiple matches

Mapping should not contain Lookup transformation downstream from an Aggregator transformation

Database does not uses Powercenter cache during Pushdown Optimization

Unconnected Lookup Transformation can be used

April 11, 2023

Working With Transformations – Lookup Transformation (Snapshot)

Only one branch of the pipeline is processed when Lookup transformation exists in two branches of a pipeline that are joined by a Joiner or Union transformation

When session requires data type conversion target-side pushdown optimization fails

Target-side pushdown optimization fails when lookup table connected to ODBC Connection or Sybase or Microsoft SQL Server database

April 11, 2023

Working With Transformations – Lookup Transformation

Use the following rules and guidelines if the lookup query in a Lookup transformation is overridden :

Configure the session for pushdown optimization with a view. The Integration Service creates a view, runs an SQL query against the view, and then drops the view.

Need to manually remove the views that the Integration Service creates.

To use a lookup override, should have necessary database privileges to create a view.

Cannot append a custom ORDER BY clause to the SQL statement in the lookup override.

The order of the columns in the lookup override must match the order of the ports in the Lookup transformation.

April 11, 2023

Working With Transformations – Sorter Transformation

Guidelines for Sorter transformations in Pushdown Optimization

On distinct sort, the Integration Service pushes the Sorter transformation to the database and processes downstream transformations

When a Sorter transformation is downstream from a Union transformation, the sort key on the Union transformation must be a connected port.

April 11, 2023

Working With Transformations – Union Transformation

Guidelines for Union transformations in Pushdown Optimization

A distinct key port should be connected

Input groups must originate from the same source

All the input groups should be pushed to source database

April 11, 2023

Working With Sessions

April 11, 2023

Working With Sessions

The Integration Service can push transformation logic to the source or target database when it does not conflict with the following session configurations:

Partitioning : When you configure pushdown optimization, the Integration Service merges all the rows into the first partition and passes empty data for each subsequent partition. The Integration Service cannot merge partitions for all partition types.

Target load rules: Some target load rules affect pushdown optimization. The full pushdown optimization cannot be used when some target load rules are configured.

Error handling, logging, and recovery : When the Integration Service pushes transformation logic to the database, it handles errors differently from when it runs a full session.

April 11, 2023

Working With Sessions

The Integration Service processes the transformation logic for the session if any of the following conditions are true:

Run a data profiling or debugging session

Use an external loader and configure the session for target-side pushdown optimization. The Integration Service can push transformation logic to the source database if you use an external loader

Enable row error logging

April 11, 2023

Working With Sessions – Partitioning

When a session configured for pushdown optimization contains multiple partitions, the Integration Service can push the session to the database in the following situations:

If a transformation uses pass-through partitioning, the Integration Service can push transformation logic to the source or target database, or both.

If a transformation uses a key range partition in the Source Qualifier transformation and hash auto-keys partitioning in downstream partition points, the Integration Service can push transformation logic to the source database by merging all rows into the first partition. To merge all rows into the first partition, the end key range for each partition must equal the start range for the next partition, and it cannot overlap with the next partition.

April 11, 2023

Working With Sessions – Partitioning

Sorter transformation with hash auto-keys partitioning

The SQL statement selects items 1313 through 9354, which includes all values in the key range and merges the data from both partitions into the first partition

The SQL statement for the second partition passes empty data

The first key range is 1313 - 3340, and the second key range is 3340 - 9354

April 11, 2023

Working With Sessions – Target Load Rules

Pushdown Optimization with Target Load Options

Target Option Source-Side Optimization Target-Side Optimization Full Optimization

Insert X X X*

Delete X X X**

Update as update X X -

Update as insert X X X*

Update else insert X X -

X* = If “source rows as data-driven” are treated, the Integration Service cannot perform full pushdown optimization. It can push transformation logic upstream or downstream from the Update Strategy or Custom transformation

X** =Although full pushdown optimization can be used when “source rows as delete” are treated, performance gains might not be achieved. If not, use partial pushdown optimization. Also, full pushdown optimization cannot be used and treat source rows as delete if the session includes a Union transformation and the Integration Service pushes transformation logic to a Sybase database0

April 11, 2023

Working With Sessions – Error Handling

When the Integration Service pushes transformation logic to the database, it cannot track errors that occur in the database. As a result, it handles errors differently from when it runs the full session.

When the Integration Service runs a session configured for full pushdown optimization and an error occurs, the database handles the errors.

When the database handles errors, the Integration Service does not write reject rows to the reject file, and it treats the error threshold as though it were set to 1.

April 11, 2023

Working With Sessions – Logging

When transformation logic is pushed to the database, the Integration Service performs the following logging functionality:

The session log does not contain details for transformations processed on the database.

The Integration Service does not write the thread busy percentage to the log for a session configured for full pushdown optimization.

The Integration Service writes the number of loaded rows to the log for source-side, target-side, and full pushdown optimization.

When the Integration Service pushes all transformation logic to the database, the Integration Service does not write the number of rows read from the source to the log.

When the Integration Service pushes transformation logic to the source, the Integration Service writes the number of rows read for each source to the log. However, the number may differ from statistics for the same session run by the Integration Service.

April 11, 2023

Working With Sessions – Recovery

When a session is configured for full pushdown optimization, the Integration Service runs the session on the database. As a result, it cannot perform incremental recovery if the session fails. Instead the database rolls back transactions.

If the database server fails, it rolls back transactions when it restarts. If the Integration Service fails, the database server rolls back the transaction.

When recovery for sessions is performed that contain SQL overrides or lookup overrides, the Integration Service must drop and recreate views

April 11, 2023

Working With SQL Overrides

April 11, 2023

Working With SQL Overrides

When an SQL override for a Source Qualifier transformation is used in a session configured for source or full pushdown optimization with a view, the Integration Service creates a view in the source database based on the override

After it creates the view in the database, the Integration Service generates an SQL query that it can push to the database

The Integration Service runs the SQL query against the view to perform pushdown optimization

To use an SQL override with pushdown optimization, the session must be configured for pushdown optimization with a view.

April 11, 2023

Working With SQL Overrides - Views

The session for pushdown optimization when configured with views, the Integration Service completes the following tasks:

Creates a view: The Integration Service generates the view by incorporating the SQL override query in a view definition. The Integration Service does not parse or validate the SQL override, so the SQL override should be tested before running the session.

Runs an SQL query against the view: After the Integration Service generates a view, the Integration Service runs an SQL query to push the transformation logic to the source. It runs this query against the view created in the database.

Drops the view: When the transaction completes, the Integration Service drops the view it created to run the SQL override query.

April 11, 2023

Working With SQL Overrides - Views

A mapping that searches for 94117 zip codes in a customer database

SQL override which return customers information whose names match variations of the name Johnson, including names such as Johnsen, Jonssen and Jonson

The Integration Service pushes transformation logic for this session to the database, it runs the SQL statement to create a view in the source database

After the session completes, the Integration Service drops the view

After the view is created, the Integration Service runs an SQL query to perform the transformation logic in the mapping

To create a unique view name, the Integration Service appends PM_V to a value generated by a hash function

April 11, 2023

Working With SQL Overrides - Troubleshooting Orphaned Views

The Integration Service might be unable to drop a view if the Integration Service, session, or connectivity fails. The orphaned view does not affect performance.

Complete the following steps to remove extra views from the database:

View the session log: While running the session, the Integration Service writes an event log when it creates the view, and it also writes an event log when it drops the view. If an Integration Service, session, or connection fails when running a session, check the session log to see if Integration Service dropped the view.

Run a query: If the Integration Service did not drop views, run a query to locate all views created by the Integration Service. Views created by the Integration Service use the same prefix.

Drop the view: After locating the views created by the Integration Service, run an SQL statement to manually drop the views from the database.

April 11, 2023

Working With SQL Overrides - Rules and Guidelines

Use the following rules and guidelines when pushdown optimization is configured for a session containing an SQL override:

Do not use an order by clause in the SQL override

When a custom SQL query is created, the SELECT statement must list the port names in the order in which they appear in the transformation.

Use ANSI outer join syntax in the SQL override.

Do not use a Sequence Generator transformation.

If a Source Qualifier transformation is configured for a distinct sort and contains an SQL override, the Integration Service ignores the distinct sort configuration.

April 11, 2023

Working With SQL Overrides - Rules and Guidelines

Use the following rules and guidelines when pushdown optimization is configured for a session containing an SQL override:

If the Source Qualifier contains multiple partitions, specify the SQL override for all partitions.

If a Source Qualifier transformation contains Informatica outer join syntax in the SQL override, the Integration Service processes the Source Qualifier transformation logic.

PowerCenter does not validate the override SQL syntax, so test the SQL override query before it is pushed to database.

When SQL override is created, ensure that the SQL syntax is compatible with the source database.

April 11, 2023

Using the $$PushdownConfig Mapping Parameter

April 11, 2023

Using the $$PushdownConfig Mapping Parameter

To use different pushdown optimization configurations at different times, use the $$PushdownConfig mapping parameter.

The parameter runs the same session using the different types of pushdown optimization.

April 11, 2023

Complete the following steps to configure the mapping parameter:

Create $$PushdownConfig in the Mapping Designer.

When the $$PushdownConfig mapping parameter is added in the Mapping Designer, use the following values:

When the session is configured, choose $$PushdownConfig for the Pushdown Optimization attribute`

Using the $$PushdownConfig Mapping Parameter

Field Value

Name $$PushdownConfig

Type Parameter

Datatype String

Precision or Scale 10

Aggregation n/a

Initial Value None

Description Optional

April 11, 2023

Define the parameter in the parameter file

Enter one of the following values for $$PushdownConfig in the parameter file:

None: The Integration Service processes all transformation logic for the session.

Source: The Integration Service pushes part of the transformation logic to the source database.

Source with View: The Integration Service creates a view to represent the SQL override value, and it runs an SQL statement against this view to push

part of the transformation logic to the source database.

Target: The Integration Service pushes part of the transformation logic to the target database.

Full: The Integration Service pushes all transformation logic to the database.

Full with View: The Integration Service creates a view to represent the SQL override value, and it runs an SQL statement against this view to push part of the transformation logic to the source database. The Integration Service pushes any remaining transformation logic to the target database.

Using the $$PushdownConfig Mapping Parameter

April 11, 2023

Viewing Pushdown Groups

April 11, 2023

The group of transformations that can be processed as one SQL statement is called a pushdown group

The Integration Service creates multiple pushdown groups depending on the number of pipelines, sources, targets, and the type of pushdown optimization used.

If the session has multiple partitions, the Integration Service executes an SQL statement for each partition in the group.

Pushdown groups can be viewed using the Pushdown Optimization Viewer

From Pushdown Optimization Viewer , the transformations that can be pushed to the database can be identified and those that the Integration Service processes

The messages in the Viewer can be used to determine how to edit transformations or mappings to push more transformation logic to the database

The Pushdown Optimization Viewer cannot display the SQL that runs in the session if variables is used or if the session is configured to run on a grid.

Viewing Pushdown Groups

April 11, 2023

Viewing Pushdown Groups

Pipeline1

Pipeline2Pushdown Group 1

Pushdown Group 2

* ->Transformation is part of both pushdown groups

April 11, 2023

Viewing Pushdown Groups

Select the type of optimization.

Select pushdown groups to view

Select a value for the connection variable

Number indicates the pushdown group.

SQL statements areGenerated for

the pushdown group.

In the Workflow Manager, open a session configured for pushdown optimization

In the Mapping tab, select View Pushdown Optimization

The pushdown option in the viewer does not affect the optimization that occurs at run time. To change pushdown optimization for a session, edit the session properties

April 11, 2023

Configuring Sessions for Pushdown Optimization

April 11, 2023

Configuring Sessions for Pushdown Optimization

Review the information in the Pushdown Optimizer Viewer to determine if edition in the mapping is needed

• Select to configure pushdown optimization.

• Select one of these None, To Source, To Source with View, To Target, Full, Full with View, $$PushdownConfig

April 11, 2023 73

April 11, 2023 74