DS Training

Embed Size (px)

Citation preview

  • 7/30/2019 DS Training

    1/14

    DATASTAGE TRAINING

    Date:-03/09/2012

    Objective:-DataStage Backup is weekly activity tasks which will be perform every Monday after

    Production Batch run. Two types of Backup we are taking.

    . Configuration Backup

    . Code Backup1. Configuration Backup: -

    Configuration backup needs to be taken from 4 servers. They are:-

    1. Production server (10.144.223.210)2. QA server (10.147.109.205)3. AT/TEST server (10.147.109.202)4. DEV server (10.147.109.201)1. Backup from Production server (10.144.223.210)

    Step1:- Logon to code backup server (10.144.223.40) as a datastage user.

    Then with the help of winscp toolusing dsadm user logon Production server (10.144.223.210)

    Step2:- Go to/root/datastage/projects/

    These are the four main projects inside the projects folder.

    a.

    DWH_EU_prdb. DW_INV_PRDc. EDI_prdd. SHAREreporting_prd

    From each projects we will copy two files for configuration backup.

    1. uvodbc.config2. DSparams

    So from here we will take 8 files, 2 files from each project.

    Step3:-We will put these files in the code backup server (10.144.223.40).

    Create a folder like Backup_YYYYMMDD inside D:\DataStage_configfiles\Prodand put all 8

    files inside this Backup_YYYYMMDD with the following nomenclature :-

    1. DWH_EU_ prd _ YYYYMMDD_ uvodbc.config2. DWH_EU_ prd _ YYYYMMDD_ DSparams3. DW_INV_PRD_ YYYYMMDD_ uvodbc.config

  • 7/30/2019 DS Training

    2/14

    4. DW_INV_PRD_ YYYYMMDD_ DSparams5. EDI_prd_ YYYYMMDD_ uvodbc.config6. EDI_prd_ YYYYMMDD_ DSparams7. SHAREreporting_prd_ YYYYMMDD_ uvodbc.config8. SHAREreporting_prd_ YYYYMMDD_ DSparams

    Step4:- Go to/root/software/datastage/Ascential/datastage/DSEngine

    Copy 3 files for backup

    1. .odbc.ini2. dsenv3. uvconfig

    Step5:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).

    Create Backup_YYYYMMDD folder.

    Paste

    1. .odbc_ YYYYMMDD.ini2. dsenv_ YYYYMMDD3. uvconfig_ YYYYMMDD

    Step6:- Go to/root/software/oracle/product/10.2.0/network/admin

    Copy

    1. tnsnames.ora

    Step7:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).

    Create Backup_YYYYMMDD folder.

    Paste

    1. tnsnames_YYYYMMDD.ora

    2. Backup from QA server (10.147.109.205)Step1:- Logon to code backup server (10.144.223.40) as a datastage user.

  • 7/30/2019 DS Training

    3/14

    Logon QA server (10.147.109.205) using winscp tool as dsadm user.

    Step2:- Go to/root/datastage/projects/

    These are the four main projects inside the projects folder.

    a. DWH_EU_prdb. DW_INV_PRDc. EDI_prdd. SHAREreporting_prd

    From each projects we will copy two files for configuration backup.

    1. uvodbc.config2. DSparams

    So from here we will take 8 files, 2 files from each project.

    Step3:-We will put these files in the code backup server (10.144.223.40).

    Create a folder like Backup_YYYYMMDD inside D:\DataStage_configfiles\Prodand put all 8

    files inside this Backup_YYYYMMDD with the following nomenclature :-

    1. DWH_EU_ prd _ YYYYMMDD_ uvodbc.config2. DWH_EU_ prd _ YYYYMMDD_ DSparams3. DW_INV_PRD_ YYYYMMDD_ uvodbc.config4. DW_INV_PRD_ YYYYMMDD_ DSparams5. EDI_prd_ YYYYMMDD_ uvodbc.config6. EDI_prd_ YYYYMMDD_ DSparams7. SHAREreporting_prd_ YYYYMMDD_ uvodbc.config8. SHAREreporting_prd_ YYYYMMDD_ DSparams

    Step4:- Go to/root/software/datastage/Ascential/datastage/DSEngine

    Copy 3 files for backup

    1. .odbc.ini2. dsenv3. uvconfig

    Step5:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).Create Backup_YYYYMMDD folder.

    Paste

    1. .odbc_ YYYYMMDD.ini2. dsenv_ YYYYMMDD3. uvconfig_ YYYYMMDD

  • 7/30/2019 DS Training

    4/14

    Step6:- Go to/root/software/oracle/product/10.2.0/network/admin

    Copy

    1. tnsnames.ora

    Step7:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).

    Create Backup_YYYYMMDD folder.

    Paste

    1. tnsnames_YYYYMMDD.ora3. Backup from AT/TEST server (10.147.109.202)

    Step1:- Logon to code backup server (10.144.223.40) as a datastage user.

    Use winscp tool to log on to AT/TEST server (10.147.109.202) as dsadm user.

    Step2:- Go to /root/software/datastage/Ascential/datastage/DSEngine

    Copy

    1) .odbc.ini2) dsenv3) uvconfig4) uvodbc.config

    Step3:-Create a folder named Backup_YYYYMMDD and paste the copied files on server (10.144.223.40)

    at D:\DataStage_configfiles\AT/TEST

    Put the files inside this folder Backup_YYYYMMDD with the nomenclature as:-.

    1) .odbc_ YYYYMMDD.ini2) dsenv_ YYYYMMDD3) uvconfig_ YYYYMMDD4) uvodbc.config _ YYYYMMDD

    Step4:- Go to/root/software/oracleadm/product/10.2.0/network/admin

    Copy

    1. tnsnames.ora

  • 7/30/2019 DS Training

    5/14

    Step5:- Create a folder named Backup_YYYYMMDD at D:\DataStage_configfiles\AT/TESTon the code

    backup server (10.144.223.40).

    Paste the file inside this folder Backup_YYYYMMDD with the nomenclature :-

    tnsnames_YYYYMMDD.ora

    4. Backup from DEV server (10.147.109.201)

    Step1:- Logon to code backup server (10.144.223.40) as a datastage user.

    Use winscp tool to log on to AT/TEST server (10.147.109.201) as dsadm user.

    Step2:- Go to /root/software/datastage/Ascential/datastage/DSEngine

    Copy

    1) .odbc.ini2) dsenv3) uvconfig4) uvodbc.config

    Step3:-Create a folder named Backup_YYYYMMDD and paste the copied files on server (10.144.223.40)

    at D:\DataStage_configfiles\AT

    Put the files inside this folder Backup_YYYYMMDD with the nomenclature as:-.

    1) .odbc_ YYYYMMDD.ini2) dsenv_ YYYYMMDD3) uvconfig_ YYYYMMDD4) uvodbc.config _ YYYYMMDD

    Step4:- Go to/root/software/oracleadm/product/10.2.0/network/admin

    Copy

    1. tnsnames.ora

    Step5:- Create a folder named Backup_YYYYMMDD at D:\DataStage_configfiles\AT on the code backup

    server (10.144.223.40). Paste the file inside this folder Backup_YYYYMMDD with the nomenclature:-

    tnsnames_YYYYMMDD.ora

  • 7/30/2019 DS Training

    6/14

    Backup from Prod server in UNIX (10.147.109.210)

    Logon through PUTTYinto 10.147.109.210using dsadm as a user.

    Put all the files inside this location/software/configbkup/

    Goto this path/root/software/oracleadm/product/10.2.0/network/admin

    Type :- ls a

    List of all the files will come.

    Use this command to copy the file and put into desire location.

    Type:-c p tnsnames.ora /software/configbkup/tnsnames/tnsnames_YYYYMMDD.ora

    Goto this path:-/root/software/datastage/Ascential/datastage/DSEngine

    Type :-lsa

    List of all the files will come.

    Type:-c p .odbc.ini /software/configbkup/odbc/.odbcYYYYMMDD.ini

    Type:-c pdsenv /software/configbkup/ dsenv/dsenvYYYYMMDD

    Type:-c puvodbc.config /software/configbkup/ uvodbc/ uvodbc YYYYMMDD.config

    Type:-c puvconfig /software/configbkup/ uvconfig/uvconfigYYYYMMDD

    1. Code Backup from Prod server (10.147.109.210)There are two types of code backup.

    1. Manual code backup

    2. Automated code backup

    1. Manual code backup: - Logon to code backup server (10.144.223.40)

    Step1:-Open Administrator client then click on Projects tab, select one project then click properties

    tab then click on the unprotected tab then ok.

  • 7/30/2019 DS Training

    7/14

    Step2:-Same way unprotect the all 4 projects one by one before code back.

    Step3:-Open Manager Client, click on Exporttab then click on the Datastage componenttab, select

    the path where you want put the backup file. Select the file

  • 7/30/2019 DS Training

    8/14

    Required path is:-D: \Prod_Datastage_Code_Backups\ Backups_YYYYMMDD\

    Here we are exporting only .dsx file one from each project.

    SHAREreporting_prd_ YYYYMMDD.dsx

    EDI_prd_ YYYYMMDD. dsx

    DW_INV_PRD_ YYYYMMDD. dsx

    DWH_EU_ prd _ YYYYMMDD. Dsx

  • 7/30/2019 DS Training

    9/14

    Step4:-After code backup put the projects into protected mode.

  • 7/30/2019 DS Training

    10/14

    1. Automated code backup:-Logon to code backup server (10.144.223.40)

    Goto this path:- C:\Documents and setting\datastage\DS_Backup_xxx.bat

    Double click on the backup bat file it will run and take the backup.If you want to verify this backup goto

    C:\Prod_Datastage_Code_Backups\ExportedBackups\Prod_Backup.

    Here we can check the above code backup has been taken correctly or not.

  • 7/30/2019 DS Training

    11/14

    Date:-04/09/2012

    ntroduction and Project Overview and server configuration

    There are 2 DB servers.

    SHARE(Hosted on Oracle)

    SMART(Hosted on SIEBEL)

  • 7/30/2019 DS Training

    12/14

    Mail Delivery List

    Guy Allen- QA deployment manager Sarah Barker Prod deployment Manager Mark Woods & RosarioDatastage SME Hans & Robert Arnold - DBA

    Stages in Batch run

    Extraction -> 2. Staging -> 3. BIW -> 4. MV refresh -> 5. End ETL

    All the sequences are linked with SeqDW_ETL_Main. We have to start this sequence first and it will automatically

    rigger other sequence. If there is any abort during the batch run, the status of the job will be abort. The sequence

    elated to concerned job will be in Abort/Restartable state (otherwise Running/Restartable). SeqDW_ETL_Main

    will also be in Abort/Restartable state if the sequence inside it will be in Abort/Restartable state. Once ETL Main

    s in Abort/Restartable state, Seqrecovery.InstanceName, which is DWH_ONE in production

    Seqrecovery.DWH_ONE), will start automatically. Once Seqrecovery.DWH_ONE completed successfully, we can

    estart the batch run from SeqDW_ETL_Main if it is an approved error. If Seqrecovery.DWH_ONE fails then we

    ave to reset the sequence and manually submit the job. Once seq.recovery completed we can submit

    eqDW_ETL_Main.

    . Extraction-> Name ofthe sequence is SeqExtSrcToStg. It uploads the data from DB(Oracle and Seibel) to the AIX server

    10.144.223.210

    1st sequence to run. If there is any issue, then it should be looked into by E-Team(Oracle DB issue) and ES DBA team (Siebel

    DBA Issue)[email protected] heading [email protected] the DL

    Schemas present in this stage is Oracle_DWH and Siebel_DWH. It takes approximately 1 hr. for the extraction to complete.

    . Staging -> Name of the sequence is SeqStgToBDW. It uploads data from AIX server (10.144.223.210) to BDW

    database.

    First the comparison takes place between data already present in CDC tables and the extracted data. CDC tables are in staging_once schema. DCD tables ID are in CDC_Lookup and are also is in staging_one schema. Schema names are same in QA. BDW_ONE(used for DW_EU_PRD project) and BDW_01(used for DW_ENV_PROD) schemas are also

    present in BDW database.

    It takes approximately 1:30 Hrs. to complete.

    mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]
  • 7/30/2019 DS Training

    13/14

    . BIW -> Name of the sequence is SeqBDWtoBIW. There are 2 sequences that run inside this sequence.

    o Load Dimension (seqLoadDimension). This sequence runs first as it holds the primary key to thefact tables. It takes 10-15 minutes to complete this sequence

    o Load Facts (seqLoadFact). This will update the fact tables. It takes 4-5 hours to complete thesequence.

    o This database is being looked in to by Oracle DBA.4. Refresh MV ->

    Materialized views are being updated in this stage. Materialized views are stored physically on the server. It takes 30 minutes to complete this stage. Cube build starts after this stage.

    . End_ETL -> Name of the sequence is SeqEndETL. It takes 5hours to complete this sequence. No SLA for this but we have to monitor this. This is a house-keeping job which clears dataset created in the scratch space of application server

    (10.144.223.210).

    Datasets contains temporary data created in different stages of batch run. AnalyzeDBSchema is a job which runs in SeqEndETL. It gathers the table data.

    Projects in Datastage:-

    DW_EU_PROD This is the only project with SLA. EDE_PRD

    o 1st week of the month after batch run, mostly after 3:00PM BST.o Ticket needs to be assigned to DWH queue before starting with the project.

    DW_INV_PRDo Order and stock reconciliation are part of this project.o Order Starts at around 5:10 AM BST and stock reconciliation starts at 5:15AM BST from Monday to

    Friday.

    Share_Reportingo RICOH CAPITAL starts at 12AM & SAR starts at 4AM BST.o Runs from Monday to Friday.o Tactical Projects are part of this.o SLAM and SLA_CHM batch runs on Monday.o SLA and SLA_CH batch runs on 2nd of the month. We will run the weekly batch if requested by the

    client.

  • 7/30/2019 DS Training

    14/14

    o Likewise PPGM runs on Monday and PPG runs on 1st day of every month.o If there is any there is clash between the dates of monthly and weekly run, then we have to de-

    schedule the weekly run and only run the monthly run.

    ***Note :- FG500 is a sequence which runs once a month before 5 th of every month. Its a part of

    DW_EU_PROD