Different Formats of the Flat File for the Same Target

7/28/2019 Different Formats of the Flat File for the Same Target

1/3

2/7/13 different formats of the flat file for the same target

www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files 1/3

Not endorsed by or affiliated withSAP

Register | Login

Delivering Better Business Decisions (Opens a new window)

Index | About Sponsors | Contact Us | Calendar | FAQ | Rules and Guidelines | Privacy Policy | Search

General Notice: We are proud to introduce the new Blogger Link Program for BOB. If you write a blog related to business intelligence andwould like to register for this program, read the details here. Thanks.General Notice: Upcoming Events: ASUG Ontario: Feb 12. There are 5 ASUG sponsored events in the next 45 days, see calendar for details.

different formats of the flat file for the sametarget

Search this topic... | Search DI: Designer and JobDesign... | Search Box

Register or Loginto Post Forum Index -> Data Integrator -> DI: Designer and Job Design

Author Message

bobj1008Forum Member

Joined: 17 Aug 2010

Posts: 1

Posted: Tue Aug 17, 2010 1:24 pm Post subject: different formats of the flat file for the same target

In our deployment, we use plugin code to extract the csvfiles in the required format. The customers are on

same version of datamart, but they are on different versions of source database - from 3.x to 4.5 depending on

which version of app lication they are using. In 4.0, we introduced a new column email in the user table in the

source da tabase. Accordingly, plugin will add the field in the csv file. But not a ll the customers will get the

upgraded version of plugin at the same time. So ETL code needs to decide which data flow to process depending

on the format of the csv file to load data to the same target table. I made the email field in the target table

nullable but it still expects the same format of the csv file with delimiter for null value.

Need help to achieve this. Can I read the structure of the flat file in DS or get the count of delimiters so tha t I can

use a conditional to use different data flow based on the format of the flat files.

Can I make the email column in the flat file optional?

Thanks much in advance.

Back totop

CirnoSenior Member

Joined: 25 Jul 2008

Posts: 53Location: UndefinedFantastic Object

Posted: Wed Aug 25, 2010 9:26 am Post subject: Re: different formats of the flat file for the same target

For delimited file formats you can set the 'Adaptable schema' flag and it won't complain if you don't have a ll the

columns.

It also won't complain if you are missing a big piece of a line so keep that in mind.

Back to top

CLS69Principal Member

Joined: 11 Jun 2009

Posts: 194Location: Italy

Posted: Mon Aug 30, 2010 5:16 pm Post subject: Re: different formats of the flat file for the same target

In order to understand which version of flat file you are reading, you might also do the following:

1. make an extra file format where you read the who le line in one field

2. read only the first line and perform a set of different "match_regex" of the line so to understand which fields it

contains.

Once you got the version, you know how to process it.

CiaoClaudio

Back to top

ShivbabaPrincipal Member

Posted: Fri Sep 10, 2010 1:34 a m Post subject: Re: different formats of the flat file for the same target
https://www.cia.gov/library/publications/the-world-factbook/geos/it.htmlhttp://www.forumtopics.com/busobj/viewtopic.php?p=752373#752373http://www.forumtopics.com/busobj/viewtopic.php?p=751319#751319http://www.forumtopics.com/busobj/index.phphttp://www.forumtopics.com/busobj/viewforum.php?f=83http://www.forumtopics.com/busobj/viewforum.php?f=117http://www.forumtopics.com/busobj/viewtopic.php?t=161304&view=previoushttp://www.forumtopics.com/busobj/printview.php?t=161304&start=0http://www.forumtopics.com/busobj/viewtopic.php?t=161304&view=nexthttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&start=0&postdays=0&postorder=asc&highlight=csv+fileshttp://www.forumtopics.com/busobj/viewtopic.php?p=848237#848237http://www.forumtopics.com/busobj/index.phphttp://www.forumtopics.com/busobj/about.phphttp://www.forumtopics.com/busobj/index.phphttp://www.forumtopics.com/busobj/redirect.php?id=85http://www.forumtopics.com/busobj/redirect.php?id=85http://www.forumtopics.com/busobj/redirect.php?id=85http://www.forumtopics.com/busobj/redirect.php?id=85http://www.forumtopics.com/busobj/viewtopic.php?p=755155#755155http://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttp://www.forumtopics.com/busobj/viewtopic.php?p=752373#752373https://www.cia.gov/library/publications/the-world-factbook/geos/it.htmlhttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttp://www.forumtopics.com/busobj/viewtopic.php?p=751319#751319http://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttp://www.forumtopics.com/busobj/viewtopic.php?p=749149#749149http://www.forumtopics.com/busobj/viewtopic.php?t=161304&view=nexthttp://www.forumtopics.com/busobj/printview.php?t=161304&start=0http://www.forumtopics.com/busobj/viewtopic.php?t=161304&view=previoushttp://www.forumtopics.com/busobj/viewforum.php?f=117http://www.forumtopics.com/busobj/viewforum.php?f=83http://www.forumtopics.com/busobj/index.phphttp://www.forumtopics.com/busobj/login.phphttp://www.forumtopics.com/busobj/profile.php?mode=registerhttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&start=0&postdays=0&postorder=asc&highlight=csv+fileshttp://www.forumtopics.com/busobj/viewtopic.php?t=204268http://www.forumtopics.com/busobj/viewtopic.php?p=848237#848237http://www.forumtopics.com/busobj/search.phphttp://www.forumtopics.com/busobj/faq.php?mode=privacyhttp://www.forumtopics.com/busobj/faq.php?mode=ruleshttp://www.forumtopics.com/busobj/faq.phphttp://www.forumtopics.com/busobj/calendar.phphttp://www.forumtopics.com/busobj/contact.phphttp://www.forumtopics.com/busobj/sponsors.phphttp://www.forumtopics.com/busobj/about.phphttp://www.forumtopics.com/busobj/index.phphttp://www.forumtopics.com/busobj/redirect.php?id=85http://www.forumtopics.com/busobj/login.phphttp://www.forumtopics.com/busobj/profile.php?mode=registerhttp://www.forumtopics.com/busobj/index.php


2/3



Joined: 01 Jun 2010

Posts: 132

,

I am also facing the same situation....

In my case, I need to compare the header (1st row of the file ) with the columns defined in the input format. The

order and name of the columns defined in the file format must match with order and name of the columns in the

input file. If it doesn't match, I should get some error message out and te rminate the job immediately.

All suggestions invited...

Thanks in advance...

Back to top


Joined: 11 Jun 2009



Cons idering that you the file format definition, you might check the first row against a regex like the following

(assuming tab as column separator):

\t\t....\t

and the file will match if and only if the outcome is 1.

Back to top


Joined: 01 Jun 2010

Posts: 132


Hi,

Thanks for your reply...

I am sorry but I don't know the syntax of match_regex function. Can you please elaborate on how to use this

function..Can you please give the exact syntax...

Back to top


Joined: 11 Jun 2009


Posted: Fri Sep 10, 2010 12:36 pm Post subject: Re: different formats of the flat file for the same target

match_regex(string to match,regex, NULL)

the third parameter set to NULL will fit mostly all the times

of course, in order to match the header, you need an additional file format where the whole row is read in one

field.

hth

Claudio

Back to top


Joined: 01 Jun 2010

Posts: 132

Posted: Sun Sep 12, 2010 11:31 pm Post subject: Re: different formats of the flat file for the same target

OK...

Thanks for the reply..

I will try it out and revert back...

Back to top

FreekIMForum Member

Posted: Tue Sep 21, 2010 2:52 am Post subject: Re: different formats of the flat file for the same target

I am new to data integrator and i am facing a similar problem,

I have got 5 different Excel versions that need to be loaded into 1 source. The files havent got unique
http://www.forumtopics.com/busobj/viewtopic.php?p=757739#757739http://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttp://www.forumtopics.com/busobj/viewtopic.php?p=755470#755470https://www.cia.gov/library/publications/the-world-factbook/geos/us.htmlhttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttp://www.forumtopics.com/busobj/viewtopic.php?p=755336#755336https://www.cia.gov/library/publications/the-world-factbook/geos/it.htmlhttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttp://www.forumtopics.com/busobj/viewtopic.php?p=755246#755246https://www.cia.gov/library/publications/the-world-factbook/geos/us.htmlhttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttp://www.forumtopics.com/busobj/viewtopic.php?p=755241#755241https://www.cia.gov/library/publications/the-world-factbook/geos/it.htmlhttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttps://www.cia.gov/library/publications/the-world-factbook/geos/us.html


3/3



Joined: 21 Sep 2010

Posts: 2

characteristics in the filename. So the different versions must be determined based on the content of the files.

I have loaded all files into a flat table, now based on some unique characteristics within the file i must determine

what format to use for further handling.

How can i make this work?

Back to top

Display posts from previous: All Posts Oldest First Go

Register or Login to Post Forum Index -> Data Integrator -> DI: Designer and Job Design

Page 1 of 1 All times are GMT - 5 Hours

Select a search Go Jump to: DI: Designer and Job Design Go

Index | About | FAQ | RAG | Privacy | Search | Register | Login

Get community updates via Twitter: FollowFollow @BOB_Board@BOB_Board 443 follow ers

Not endorsed by or affiliated with SAPPowered by phpBB phpBB GroupGenerated in 0.7976 seconds using 16 queries. (SQL 0.0123 Parse 0.7319 Other 0.0534)Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17

Hosted by ForumTopics.com | Terms of ServicephpBB Customizations by the phpBBDoctor.comShameless plug for MomentsOfLight.com
http://www.momentsoflight.com/http://www.phpbbdoctor.com/http://www.forumtopics.com/busobj/tos_busobj.phphttp://www.forumtopics.com/http://www.phpbb.com/http://www.forumtopics.com/busobj/login.phphttp://www.forumtopics.com/busobj/profile.php?mode=registerhttp://www.forumtopics.com/busobj/search.phphttp://www.forumtopics.com/busobj/faq.php?mode=privacyhttp://www.forumtopics.com/busobj/faq.php?mode=ruleshttp://www.forumtopics.com/busobj/faq.phphttp://www.forumtopics.com/busobj/about.phphttp://www.forumtopics.com/busobj/index.phphttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&view=nexthttp://www.forumtopics.com/busobj/printview.php?t=161304&start=0http://www.forumtopics.com/busobj/viewtopic.php?t=161304&view=previoushttp://www.forumtopics.com/busobj/viewforum.php?f=117http://www.forumtopics.com/busobj/viewforum.php?f=83http://www.forumtopics.com/busobj/index.phphttp://www.forumtopics.com/busobj/login.phphttp://www.forumtopics.com/busobj/profile.php?mode=registerhttp://www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files#tophttps://twitter.com/intent/user?original_referer=http%3A%2F%2Fwww.forumtopics.com%2Fbusobj%2Fviewtopic.php%3Ft%3D161304%26highlight%3Dcsv%2Bfiles&region=count_link&screen_name=BOB_Board&tw_p=followbutton&variant=2.0https://twitter.com/intent/follow?original_referer=http%3A%2F%2Fwww.forumtopics.com%2Fbusobj%2Fviewtopic.php%3Ft%3D161304%26highlight%3Dcsv%2Bfiles&region=follow_link&screen_name=BOB_Board&tw_p=followbutton&variant=2.0

Documents

Different Formats of the Flat File for the Same Target