DS Training

7/30/2019 DS Training

1/14

DATASTAGE TRAINING

Date:-03/09/2012

Objective:-DataStage Backup is weekly activity tasks which will be perform every Monday after

Production Batch run. Two types of Backup we are taking.

. Configuration Backup

. Code Backup1. Configuration Backup: -

Configuration backup needs to be taken from 4 servers. They are:-

1. Production server (10.144.223.210)2. QA server (10.147.109.205)3. AT/TEST server (10.147.109.202)4. DEV server (10.147.109.201)1. Backup from Production server (10.144.223.210)

Step1:- Logon to code backup server (10.144.223.40) as a datastage user.

Then with the help of winscp toolusing dsadm user logon Production server (10.144.223.210)

Step2:- Go to/root/datastage/projects/

These are the four main projects inside the projects folder.

a.

DWH_EU_prdb. DW_INV_PRDc. EDI_prdd. SHAREreporting_prd

From each projects we will copy two files for configuration backup.

1. uvodbc.config2. DSparams

So from here we will take 8 files, 2 files from each project.

Step3:-We will put these files in the code backup server (10.144.223.40).

Create a folder like Backup_YYYYMMDD inside D:\DataStage_configfiles\Prodand put all 8

files inside this Backup_YYYYMMDD with the following nomenclature :-

1. DWH_EU_ prd _ YYYYMMDD_ uvodbc.config2. DWH_EU_ prd _ YYYYMMDD_ DSparams3. DW_INV_PRD_ YYYYMMDD_ uvodbc.config


2/14

4. DW_INV_PRD_ YYYYMMDD_ DSparams5. EDI_prd_ YYYYMMDD_ uvodbc.config6. EDI_prd_ YYYYMMDD_ DSparams7. SHAREreporting_prd_ YYYYMMDD_ uvodbc.config8. SHAREreporting_prd_ YYYYMMDD_ DSparams

Step4:- Go to/root/software/datastage/Ascential/datastage/DSEngine

Copy 3 files for backup

1. .odbc.ini2. dsenv3. uvconfig

Step5:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).

Create Backup_YYYYMMDD folder.

Paste

1. .odbc_ YYYYMMDD.ini2. dsenv_ YYYYMMDD3. uvconfig_ YYYYMMDD

Step6:- Go to/root/software/oracle/product/10.2.0/network/admin

Copy

1. tnsnames.ora



Paste

1. tnsnames_YYYYMMDD.ora

2. Backup from QA server (10.147.109.205)Step1:- Logon to code backup server (10.144.223.40) as a datastage user.


3/14

Logon QA server (10.147.109.205) using winscp tool as dsadm user.

Step2:- Go to/root/datastage/projects/

These are the four main projects inside the projects folder.

a. DWH_EU_prdb. DW_INV_PRDc. EDI_prdd. SHAREreporting_prd

From each projects we will copy two files for configuration backup.

1. uvodbc.config2. DSparams

So from here we will take 8 files, 2 files from each project.

Step3:-We will put these files in the code backup server (10.144.223.40).

Create a folder like Backup_YYYYMMDD inside D:\DataStage_configfiles\Prodand put all 8

files inside this Backup_YYYYMMDD with the following nomenclature :-

1. DWH_EU_ prd _ YYYYMMDD_ uvodbc.config2. DWH_EU_ prd _ YYYYMMDD_ DSparams3. DW_INV_PRD_ YYYYMMDD_ uvodbc.config4. DW_INV_PRD_ YYYYMMDD_ DSparams5. EDI_prd_ YYYYMMDD_ uvodbc.config6. EDI_prd_ YYYYMMDD_ DSparams7. SHAREreporting_prd_ YYYYMMDD_ uvodbc.config8. SHAREreporting_prd_ YYYYMMDD_ DSparams

Step4:- Go to/root/software/datastage/Ascential/datastage/DSEngine

Copy 3 files for backup

1. .odbc.ini2. dsenv3. uvconfig

Step5:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).Create Backup_YYYYMMDD folder.

Paste

1. .odbc_ YYYYMMDD.ini2. dsenv_ YYYYMMDD3. uvconfig_ YYYYMMDD


4/14

Step6:- Go to/root/software/oracle/product/10.2.0/network/admin

Copy

1. tnsnames.ora



Paste

1. tnsnames_YYYYMMDD.ora3. Backup from AT/TEST server (10.147.109.202)


Use winscp tool to log on to AT/TEST server (10.147.109.202) as dsadm user.

Step2:- Go to /root/software/datastage/Ascential/datastage/DSEngine

Copy

1) .odbc.ini2) dsenv3) uvconfig4) uvodbc.config

Step3:-Create a folder named Backup_YYYYMMDD and paste the copied files on server (10.144.223.40)

at D:\DataStage_configfiles\AT/TEST

Put the files inside this folder Backup_YYYYMMDD with the nomenclature as:-.

1) .odbc_ YYYYMMDD.ini2) dsenv_ YYYYMMDD3) uvconfig_ YYYYMMDD4) uvodbc.config _ YYYYMMDD

Step4:- Go to/root/software/oracleadm/product/10.2.0/network/admin

Copy

1. tnsnames.ora


5/14

Step5:- Create a folder named Backup_YYYYMMDD at D:\DataStage_configfiles\AT/TESTon the code

backup server (10.144.223.40).

Paste the file inside this folder Backup_YYYYMMDD with the nomenclature :-

tnsnames_YYYYMMDD.ora

4. Backup from DEV server (10.147.109.201)


Use winscp tool to log on to AT/TEST server (10.147.109.201) as dsadm user.

Step2:- Go to /root/software/datastage/Ascential/datastage/DSEngine

Copy

1) .odbc.ini2) dsenv3) uvconfig4) uvodbc.config

Step3:-Create a folder named Backup_YYYYMMDD and paste the copied files on server (10.144.223.40)

at D:\DataStage_configfiles\AT

Put the files inside this folder Backup_YYYYMMDD with the nomenclature as:-.

1) .odbc_ YYYYMMDD.ini2) dsenv_ YYYYMMDD3) uvconfig_ YYYYMMDD4) uvodbc.config _ YYYYMMDD

Step4:- Go to/root/software/oracleadm/product/10.2.0/network/admin

Copy

1. tnsnames.ora

Step5:- Create a folder named Backup_YYYYMMDD at D:\DataStage_configfiles\AT on the code backup

server (10.144.223.40). Paste the file inside this folder Backup_YYYYMMDD with the nomenclature:-

tnsnames_YYYYMMDD.ora


6/14

Backup from Prod server in UNIX (10.147.109.210)

Logon through PUTTYinto 10.147.109.210using dsadm as a user.

Put all the files inside this location/software/configbkup/

Goto this path/root/software/oracleadm/product/10.2.0/network/admin

Type :- ls a

List of all the files will come.

Use this command to copy the file and put into desire location.

Type:-c p tnsnames.ora /software/configbkup/tnsnames/tnsnames_YYYYMMDD.ora

Goto this path:-/root/software/datastage/Ascential/datastage/DSEngine

Type :-lsa

List of all the files will come.

Type:-c p .odbc.ini /software/configbkup/odbc/.odbcYYYYMMDD.ini

Type:-c pdsenv /software/configbkup/ dsenv/dsenvYYYYMMDD

Type:-c puvodbc.config /software/configbkup/ uvodbc/ uvodbc YYYYMMDD.config

Type:-c puvconfig /software/configbkup/ uvconfig/uvconfigYYYYMMDD

1. Code Backup from Prod server (10.147.109.210)There are two types of code backup.

1. Manual code backup

2. Automated code backup

1. Manual code backup: - Logon to code backup server (10.144.223.40)

Step1:-Open Administrator client then click on Projects tab, select one project then click properties

tab then click on the unprotected tab then ok.


7/14

Step2:-Same way unprotect the all 4 projects one by one before code back.

Step3:-Open Manager Client, click on Exporttab then click on the Datastage componenttab, select

the path where you want put the backup file. Select the file


8/14

Required path is:-D: \Prod_Datastage_Code_Backups\ Backups_YYYYMMDD\

Here we are exporting only .dsx file one from each project.

SHAREreporting_prd_ YYYYMMDD.dsx

EDI_prd_ YYYYMMDD. dsx

DW_INV_PRD_ YYYYMMDD. dsx

DWH_EU_ prd _ YYYYMMDD. Dsx


9/14

Step4:-After code backup put the projects into protected mode.


10/14

1. Automated code backup:-Logon to code backup server (10.144.223.40)

Goto this path:- C:\Documents and setting\datastage\DS_Backup_xxx.bat

Double click on the backup bat file it will run and take the backup.If you want to verify this backup goto

C:\Prod_Datastage_Code_Backups\ExportedBackups\Prod_Backup.

Here we can check the above code backup has been taken correctly or not.


11/14

Date:-04/09/2012

ntroduction and Project Overview and server configuration

There are 2 DB servers.

SHARE(Hosted on Oracle)

SMART(Hosted on SIEBEL)


12/14

Mail Delivery List

Guy Allen- QA deployment manager Sarah Barker Prod deployment Manager Mark Woods & RosarioDatastage SME Hans & Robert Arnold - DBA

Stages in Batch run

Extraction -> 2. Staging -> 3. BIW -> 4. MV refresh -> 5. End ETL

All the sequences are linked with SeqDW_ETL_Main. We have to start this sequence first and it will automatically

rigger other sequence. If there is any abort during the batch run, the status of the job will be abort. The sequence

elated to concerned job will be in Abort/Restartable state (otherwise Running/Restartable). SeqDW_ETL_Main

will also be in Abort/Restartable state if the sequence inside it will be in Abort/Restartable state. Once ETL Main

s in Abort/Restartable state, Seqrecovery.InstanceName, which is DWH_ONE in production

Seqrecovery.DWH_ONE), will start automatically. Once Seqrecovery.DWH_ONE completed successfully, we can

estart the batch run from SeqDW_ETL_Main if it is an approved error. If Seqrecovery.DWH_ONE fails then we

ave to reset the sequence and manually submit the job. Once seq.recovery completed we can submit

eqDW_ETL_Main.

. Extraction-> Name ofthe sequence is SeqExtSrcToStg. It uploads the data from DB(Oracle and Seibel) to the AIX server

10.144.223.210

1st sequence to run. If there is any issue, then it should be looked into by E-Team(Oracle DB issue) and ES DBA team (Siebel

DBA Issue)[email protected] heading [email protected] the DL

Schemas present in this stage is Oracle_DWH and Siebel_DWH. It takes approximately 1 hr. for the extraction to complete.

. Staging -> Name of the sequence is SeqStgToBDW. It uploads data from AIX server (10.144.223.210) to BDW

database.

First the comparison takes place between data already present in CDC tables and the extracted data. CDC tables are in staging_once schema. DCD tables ID are in CDC_Lookup and are also is in staging_one schema. Schema names are same in QA. BDW_ONE(used for DW_EU_PRD project) and BDW_01(used for DW_ENV_PROD) schemas are also

present in BDW database.

It takes approximately 1:30 Hrs. to complete.
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


13/14

. BIW -> Name of the sequence is SeqBDWtoBIW. There are 2 sequences that run inside this sequence.

o Load Dimension (seqLoadDimension). This sequence runs first as it holds the primary key to thefact tables. It takes 10-15 minutes to complete this sequence

o Load Facts (seqLoadFact). This will update the fact tables. It takes 4-5 hours to complete thesequence.

o This database is being looked in to by Oracle DBA.4. Refresh MV ->

Materialized views are being updated in this stage. Materialized views are stored physically on the server. It takes 30 minutes to complete this stage. Cube build starts after this stage.

. End_ETL -> Name of the sequence is SeqEndETL. It takes 5hours to complete this sequence. No SLA for this but we have to monitor this. This is a house-keeping job which clears dataset created in the scratch space of application server

(10.144.223.210).

Datasets contains temporary data created in different stages of batch run. AnalyzeDBSchema is a job which runs in SeqEndETL. It gathers the table data.

Projects in Datastage:-

DW_EU_PROD This is the only project with SLA. EDE_PRD

o 1st week of the month after batch run, mostly after 3:00PM BST.o Ticket needs to be assigned to DWH queue before starting with the project.

DW_INV_PRDo Order and stock reconciliation are part of this project.o Order Starts at around 5:10 AM BST and stock reconciliation starts at 5:15AM BST from Monday to

Friday.

Share_Reportingo RICOH CAPITAL starts at 12AM & SAR starts at 4AM BST.o Runs from Monday to Friday.o Tactical Projects are part of this.o SLAM and SLA_CHM batch runs on Monday.o SLA and SLA_CH batch runs on 2nd of the month. We will run the weekly batch if requested by the

client.


14/14

o Likewise PPGM runs on Monday and PPG runs on 1st day of every month.o If there is any there is clash between the dates of monthly and weekly run, then we have to de-

schedule the weekly run and only run the monthly run.

***Note :- FG500 is a sequence which runs once a month before 5 th of every month. Its a part of

DW_EU_PROD