Oracle Data Integrator (ODI) 11g_ limitations.pdf

AboutWhite PapersContact

Posted by Uli Bethke on May 18, 2011 in Oracle Data Integrator (ODI)

When I first heard about the new load plan feature in ODI 11.1.1.5 I thought it could replace our customdeveloped scheduling mechanism.

While the load plan feature makes it easier to run scenarios in parallel it became quickly clear that there areinefficiencies in defining dependencies and as a result in exploiting parallelism.

In the example below we are loading two fact tables. Fact 1 has dependencies on Dimension 1 andDimension 2, whereas Fact 2 just has a dependency on Dimension 2.

Fact 1 needs to wait for both Dimension 1 and 2 to finish before it can be loaded. Fact 2 just needs to wait forload of Dimension 2 to finish before it can be loaded itself. So the most efficient load plan would be to loadDimension 1 and Dimension 2 in parallel. Once Dimension 2 has finished loading, processing of Fact 2 kicksin. Once load for Dimension 1 has been kicked off, the load for Fact 1 should be kicked off as well.The problem is that a load plan in ODI can’t be set up like this.

Oracle Data Integrator (ODI) 11g: limitation of load plans | BI-Quotient http://www.business-intelligence-quotient.com/?p=1284

1 of 6 7/19/2013 10:26 AM

In the ODI load plan both fact tables need to wait for both parallely loaded dimensions to finish loadingbefore the fact tables can be loaded themselves in parallel.

This inefficiency in parallelism is typically no problem when you are loading in nightly batch windows andwhen dealing with just a hundred or so scenarios. However, the unnecessary waits may add up. Anotherfeature that I am missing is a feature to limit the number of scenarios that can be executed in a parallel step ata time. As an example you may have ten scenarios in a parallel step. However, you just want five to executein parallel, e.g. because of resource constraints on the database server. Once one of the scenarios has finishedone of the remaining five kicks in and so on.

We have a custom developed solution that has all of the above features and a lot more, and seamlesslyintegrates with ODI. Get in touch to find out more.

Most Popular Posts

April 5, 2009 -- Query hints in OBIEEI have recently come across a post on the OTN forums on how to use hints in OBIEE. OBIEE lets you sp...

October 12, 2009 -- ODI: Automating deployment of scenarios to production in Oracle Data IntegratorIn this post I will show you how you can automatically deploy scenarios in ODI. It is rather cumb...

March 12, 2009 -- Consuming a REST web service with OracleSOAP web services dominate in a SOA environment. However, there is a new kid on the block: REST. Thi...

November 19, 2009 -- ODI Snippets: OdiSqlUnload with headerOdiSqlUnload is a handy tool to quickly dump the content of a table into a CSV file. This is a lot q...

November 17, 2009 -- ODI snippets: Purge Log and shrink spaceWe all know that we should purge the ODI log on a regular basis. if we log everything or if we run n...

7 Comments

G1 on Jun 06, 2013

Hi,

Yes, I think we can achieve this by Nesting parallel and serial steps.Lets say, Dimension_1 and Dimension_2 are in one parallel step.


2 of 6 7/19/2013 10:26 AM

In that parallel step add one serial step, and add Dimension_2 and Fact_2 table in that step serially.

sree on May 29, 2013

one more limitation i can see in load plan is when ever we want to run the load plan from commandline it just invokes the load plan.if you want to see the status we need to go back to operator and seewhether load plan is completed or not

Prasadenc on Jul 31, 2012

Dear all,Do we have a solution for the problem stated above in the latest version of ODI?

Is there any suggestible work around at this point of time for such an issue?

Thanks,Prasad

admin on Jun 21, 2011

This does not solve the problem, as you create a different type of inefficiency. In your solutiondimension_1 and dimension_2 are not loaded in parallel even though there is no dependency and assuch should be loaded in parallel.

Sid on Jun 20, 2011

Uli, but you can group DIMENSION_1 and Fact_1 serially in a package called PACKAGE_1 and callthat package inside the load plan (in place of DIMENSION_1). That should remove the wait.About the other inconvenience I have to agree.

admin on May 26, 2011

Hi Victor.

Are you referring to the Number of concurrent sessions an agent can run. The default value is 1000. Ifso, then this is still available in ODI 11g. If not can you elaborate a bit further.


3 of 6 7/19/2013 10:26 AM

Setting this value will certainly work in most situations. However, if you need more control in selectingwhich scenarios can run in parallel then this setting is not granular enough.

In a more granular approach you would score a scenario based on how resource intensive it is, e.g. 1-10and then set a maximum threshold for parallel execution, e.g. 500.

Cheersuli

odi-usage on May 26, 2011

Hi Uli.

About limitation sessions running in parallel. Did Oracle remove possibilities using agents for that inthat version?In our Pkg_Scheduler we just limit agent to 26 so in third or fourth level of scenario’s hierarchy wehave about no more then 15 sessions running at the same time.

Leave Reply

Name: E-Mail: Website: Comment:

Looking for ODI developers in Dublin.

ODI developers required for contracts in Dublin. EU work permit needed. Get in touch if you areinterested!

About Me

Hi! I'm Uli Bethke. Data warehouse super hero and evangelist. I am based in Ireland.Nerdy stuff I am good at includes: anything data warehouse, data integration, dataquality, and ODI. Read more...


4 of 6 7/19/2013 10:26 AM

Recent Posts

ODI 11g Cookbook – The leading ETL tool now also has the best ETL book Jun 15, 2013

Using ODI user functions to dynamically inject SQL into Interfaces May 19, 2013

Endeca text enrichment. Entities extraction, sentiment analysis, and text tagging withLexalytics customer defined lists. I love this one! Apr 18, 2013

Endeca text tagging: Tagging unstructured IT jobs data against a whitelist of LinkedInskills Apr 01, 2013

Oracle User Group Dublin, 12 March 2013. It’s FREE. Feb 14, 2013

Extreme re-usability in ODI 11g (I can’t believe I am giving away this trick). Jan 07, 2013

How you can launch an ODI scenario through a web service call? Dec 12, 2012

Search Blog

Blogroll

Data Warehousing BooksODI Experts

Tags

Amazon EC2 analytic functions Business Intelligence Business Intelligence Books CDC data warehouse books examples of data mining


5 of 6 7/19/2013 10:26 AM

google hsqldb jdbc lexalytics OBIEE odi 11g book odi api odi beginners odi best practice odi best practice

resources odi book odi documentation odi getting started odi jython ODI performance ODI performance tuning odi resource odi tutorials odi web

services ODM 11gR2 ODM 11g R2 Oracle Oracle Asynchronous Change Data Capture CDC Streams

oracle data integrator beginners oracle data integrator documentation oracle data integrator getting started oracle data integrator

resources Oracle Data Mining Book Oracle Data Mining links oracle odi Oracle Warehouse Builder sql Sunopsis Memory Engine text

mining web scraping

© 2012 BI-Quotient - All rights reserved.


6 of 6 7/19/2013 10:26 AM

Documents

Oracle Data Integrator (ODI) 11g_ limitations.pdf