Upload
puneet-kala
View
57
Download
3
Tags:
Embed Size (px)
Citation preview
Key Concepts of ODI
Execution Context Knowledge Modules Models Interfaces Packages Scenarios Procedures Variables
Variables
Defined in Designer Tab of ODI Scope: Global, Project Operations
1. Refresh 2. Declare3. Set4. Evaluate
Interfaces
What1. The destination data store for the data (the
target). This will be chosen from an ODI model.2. The data stores that supply the input data (the
sources). These too will be chosen from ODI models.
How1. The transformations that are applied to the data
during the transition from sources to target (the mappings). These are expressed in SQL.
2. The physical transfer mechanisms that are used between sources and target (the flow). This role is performed by the Knowledge Modules
Packages
Packages are the basic element of orchestration in ODI.
Sequence your interfaces, and define what operation to perform when a step (or interface) fails
Sequential Execution
Models
Models in ODI are used to store the metadata imported from the databases.
Created using Designer Tab in ODI
Procedures
A Procedure is a set of commands that can be executed by an agent.
Procedures should be considered only when what you need to do can't be achieved in an interface
Procedures require you to develop all your code manually, as opposed to interfaces
Scenario
Scenario is the "compiled" version of an ODI object.
It is possible to generate scenarios for packages, procedures, interfaces or variables
Load Plans
Execution of scenarios in a hierarchy of sequential and parallel steps
Scenarios, Packages, interfaces, variables and Procedures can be added to load plans.
It is an executable object that contains steps that needs to be executed.
Knowledge Modules
Issue the join-and-filter query to extract the MySQL data into the staging area in Oracle
Extract the data from the flat file and upload that into the staging area
Combine the data in the staging area and upload it from there into the target data schema
SDE SIL PLP
SDERetail Analytics SDE programs are source dependent extraction programs that extract data from source system, transform data, and load data to Retail Analytics staging tables. SDE programs name have "sde" as the suffix.
SILRetail Analytics SIL programs are source independent loading programs that load data from Retail Analytics staging tables to Retail Analytics base level data mart tables. SIL programs name have "sil" as the suffix.
PLPRetail Analytics PLP programs are post loading programs that load data from Retail Analytics base level tables or Retail Analytics temporary tables, created and populated during batch cycle, to Retail Analytics data mart tables. The PLP programs include ETL maintenance and fact aggregations. PLP programs name have "plp" as the suffix.
Batch Process and Running Jobs
SDE, SIL & PLP each have a job associated with them which are shell process
User has two options, either execute from ODI or execute the batch.
When executing the Batch, please follow:1. Verify in C_LOAD_DATES table for any existing
Entry. 2. Verify Parameter values on C_ODI_PARAMS3. Execute the Batch.
In case of Errors look into : $MMHOME/Errors/Log
Example SDE, SIL
SDE: SDE_RetailColorDimension1. Package: SDE_RetailColorDimension2. Source Table: DIFF_IDS3. Target Table: W_RTL_PRODUCT_COLOR_DS4. Batch Name: prdclrsde.ksh
SIL: SIL_RetailProductColorDimension1. Package: SIL_RetailProductColorDimension2. Source Table: W_RTL_PRODUCT_COLOR_DS3. Target Table: W_RTL_PRODUCT_D4. Batch Name: rtlprodcolorsil.ksh
UAF
The goal of the Universal Adapter Framework (UAF) is to simplify the process of moving source dependent extracts into Retail Insights staging tables for customers
In terms of implementation, the UAF first requires pipe (‘|’) separated value (DAT file) text file extracts to be provided as inputs
Once the DAT files are in place, the UAF can be used to move that data into Retail Insights staging tables through the use of sqlldr
The control files required for sqlldr will be created automatically during the process that is written in ODI.
Data Error
These occur when we encounter "bad" data during data integration tasks
Data type issues, ODI Constraints Keys, References, Conditions Detection of Data errors is done by
setting Flow_Control option in the Flow tab of an IKM
Errors detected move to ‘Error Hospital’ Only data check in RA! ‘Dimension
Identifier Missing’
Error Table
All columns, definitions of the corresponding table. For example: Error occurred while Loading Customer data in Customers Table, hence E$_CUSTOMER Table.
Additional column about control, “S” Static or “F” Flow
Error Message Check Date ODI Object Info Constraint Name Constraint Type ex. FK(Reference Failure) ODI Session
Correcting Error
RECYCLE_ERROR Option in IKM to True Correcting Data in Error Tables and
Executing the Packages.
Example of Execution Errors
These occur when one or more steps in an interface, procedure, package, or scenario fail to complete successfully
Database is down, Which can be handled and anticipated error, hence managed through flow control using KO and OK steps in a package
Unexpected Error during the Execution of a package, Log could be found in Operator Navigator in ODI Studio example: TO_DATA function on an XML source Data Server
Operational Errors
These arise if one or more of the underlying platform-level or infrastructure components encounter some kind of exception.
This area forms part of management and monitoring, especially locating and viewing system component (for example, ODI Agent)
Restart / Recovery
What happens when a Job fails?1. All successful interfaces in the package
commit data to the database, we can restart the package from operational navigator tab. Default behavior to restart from point of failure
2. When we restart a package again, KM takes care of changes.
1. Trunk and Load2. Incremental Update
3. KM written to create a File when successful execution and verify the existence of that file before starting again.
Load Plan Restart Options
If Load Plan is created we have three options for each step.1. Restart from new session2. Restart from failed step3. Restart from Failed Task