DataStage Enterprise EditionParallel Job Developers Guide

Version 7.5 June 2004 Part No. 00D-023DS75

Published by Ascential Software Corporation. 2004 Ascential Software Corporation. All rights reserved. Ascential, DataStage, QualityStage, AuditStage,, ProfileStage, and MetaStage are trademarks of Ascential Software Corporation or its affiliates and may be registered in the United States or other jurisdictions. Windows is a trademark of Microsoft Corporation. Unix is a registered trademark of The Open Group. Adobe and Acrobat are registered trademarks of Adobe Systems Incorporated. Other marks are the property of the owners of those marks. This product may contain or utilize third party components subject to the user documentation previously provided by Ascential Software Corporation or contained herein. Documentation Team: Mandy deBelin

Table of ContentsHow to Use this GuideDocumentation Conventions ................................................................................... xxx User Interface Conventions ............................................................................. xxxii DataStage Documentation ..................................................................................... xxxiii

Chapter 1. IntroductionDataStage Parallel Jobs ............................................................................................... 1-2

Chapter 2. Designing Parallel JobsParallel Processing ...................................................................................................... 2-1 Pipeline Parallelism ............................................................................................. 2-2 Partition Parallelism ............................................................................................ 2-3 Combining Pipeline and Partition Parallelism ................................................ 2-4 Repartitioning Data ............................................................................................. 2-5 Parallel Processing Environments ............................................................................ 2-6 The Configuration File ............................................................................................... 2-7 Partitioning, Repartitioning, and Collecting Data ................................................. 2-8 Partitioning ........................................................................................................... 2-8 Collecting ............................................................................................................ 2-21 Repartitioning ..................................................................................................... 2-25 The Mechanics of Partitioning and Collecting .............................................. 2-27 Sorting Data ............................................................................................................... 2-29 Data Sets ..................................................................................................................... 2-30 Meta Data ................................................................................................................... 2-30 Runtime Column Propagation ......................................................................... 2-31 Table Definitions ................................................................................................ 2-31 Schema Files and Partial Schemas ................................................................... 2-32 Data Types ........................................................................................................... 2-32 Strings and Ustrings .......................................................................................... 2-36 Complex Data Types ......................................................................................... 2-36 Incorporating Server Job Functionality ................................................................. 2-38 Table of Contents iii

Chapter 3. Stage EditorsShowing Stage Validation Errors ..............................................................................3-7 The Stage Page .............................................................................................................3-7 General Tab ...........................................................................................................3-8 Properties Tab .......................................................................................................3-9 Advanced Tab .....................................................................................................3-12 Link Ordering Tab ..............................................................................................3-15 NLS Map Tab ......................................................................................................3-17 NLS Locale Tab ...................................................................................................3-18 Inputs Page .................................................................................................................3-19 General Tab .........................................................................................................3-20 Properties Tab .....................................................................................................3-20 Partitioning Tab ..................................................................................................3-21 Format Tab ...........................................................................................................3-26 Columns Tab .......................................................................................................3-28 Advanced Tab .....................................................................................................3-48 Outputs Page ..............................................................................................................3-51 General Tab .........................................................................................................3-52 Properties Tab .....................................................................................................3-53 Format Tab ...........................................................................................................3-54 Columns Tab .......................................................................................................3-56 Mapping Tab .......................................................................................................3-58 Advanced Tab .....................................................................................................3-61

Chapter 4. Data Set StageMust Dos ......................................................................................................................4-2 Writing to a Data Set ............................................................................................4-2 Reading from a Data Set ......................................................................................4-2 Stage Page .....................................................................................................................4-3 Advanced Tab .......................................................................................................4-3 Inputs Page ...................................................................................................................4-4 Input Link Properties Tab ...................................................................................4-4 Partitioning Tab ....................................................................................................4-5 Outputs Page ................................................................................................................4-8 Output Link Properties Tab ................................................................................4-8

iv

DataStage Enterprise Edition Parallel Job Developers Guide

Chapter 5. Sequential File StageMust Dos ..................................................................................................................... 5-3 Writing to a File .................................................................................................... 5-3 Reading from a File .............................................................................................. 5-3 Stage Page .................................................................................................................... 5-4 Advanced Tab ....................................................................................................... 5-4 NLS Map Tab ........................................................................................................ 5-5 Inputs Page .................................................................................................................. 5-6 Input Link Properties Tab ................................................................................... 5-7 Partitioning Tab .................................................................................................... 5-8 Input Link Format Tab ...................................................................................... 5-11 Example of Writing a Sequential File .............................................................. 5-24 Outputs Page ............................................................................................................. 5-26 Output Link Properties Tab .............................................................................. 5-27 Reject Links ......................................................................................................... 5-31 Output Link Format Tab ................................................................................... 5-31 Example of Reading a Sequential File ............................................................ 5-45 Using RCP With Sequential Stages ......................................................................... 5-46

Chapter 6. File Set StageMust Dos ..................................................................................................................... 6-3 Writing to a File .................................................................................................... 6-3 Reading from a File .............................................................................................. 6-3 Stage Page .................................................................................................................... 6-4 Advanced Tab ....................................................................................................... 6-4 NLS Map Tab ........................................................................................................ 6-4 Inputs Page .................................................................................................................. 6-5 Input Link Properties Tab ................................................................................... 6-6 Partitioning Tab .................................................................................................... 6-8 Input Link Format Tab ...................................................................................... 6-10 Outputs Page ............................................................................................................. 6-24 Output Link Properties Tab .............................................................................. 6-24 Reject Link Properties ........................................................................................ 6-26 Output Link Format Tab ................................................................................... 6-26 Using RCP With File Set Stages .............................................................................. 6-39

Table of Contents

v

Chapter 7. Lookup File Set StageMust Dos ......................................................................................................................7-3 Creating a Lookup File Set: .................................................................................7-3 Looking Up a Lookup File Set: ...........................................................................7-3 Stage Page .....................................................................................................................7-3 Advanced Tab .......................................................................................................7-4 Inputs Page ...................................................................................................................7-5 Input Link Properties Tab ...................................................................................7-6 Partitioning Tab ....................................................................................................7-7 Outputs Page ................................................................................................................7-9 Output Link Properties Tab ..............................................................................7-10

Chapter 8. External Source StageMust Dos ......................................................................................................................8-2 Stage Page .....................................................................................................................8-3 Advanced Tab .......................................................................................................8-3 NLS Map Tab ........................................................................................................8-4 Outputs Page ................................................................................................................8-4 Output Link Properties Tab ................................................................................8-5 Reject Link Properties ..........................................................................................8-7 Format Tab .............................................................................................................8-7 Using RCP With External Source Stages ................................................................8-20

Chapter 9. External Target StageMust Dos ......................................................................................................................9-2 Stage Page .....................................................................................................................9-3 Advanced Tab .......................................................................................................9-3 NLS Map Tab ........................................................................................................9-4 Inputs Page ...................................................................................................................9-4 Input Link Properties Tab ...................................................................................9-5 Partitioning Tab ....................................................................................................9-6 Format Tab .............................................................................................................9-9 Outputs Page ..............................................................................................................9-22 Using RCP With External Target Stages .................................................................9-22

vi

DataStage Enterprise Edition Parallel Job Developers Guide

Chapter 10. Complex Flat File StageMust Dos ................................................................................................................... 10-2 Stage Page .................................................................................................................. 10-3 File Options Tab ................................................................................................. 10-4 Record Options Tab ......................................................................................... 10-10 Columns Tab ..................................................................................................... 10-11 Layout Tab ........................................................................................................ 10-18 Advanced Tab ................................................................................................... 10-19 NLS Map Tab .................................................................................................... 10-19 Input Page ................................................................................................................ 10-20 Input Link Columns Tab ................................................................................. 10-21 Partitioning Tab ................................................................................................ 10-21 Output Page ............................................................................................................. 10-23 Selection Tab ..................................................................................................... 10-24 Output Link Columns Tab .............................................................................. 10-28 Reject Links ....................................................................................................... 10-29 ................................................................................................................................... 10-29

Chapter 11. SAS Parallel Data Set StageMust Dos ................................................................................................................... 11-2 Writing an SAS Data Set ................................................................................... 11-2 Reading an SAS Data Set .................................................................................. 11-2 Stage Page .................................................................................................................. 11-3 Advanced Tab ..................................................................................................... 11-3 Inputs Page ................................................................................................................ 11-4 Input Link Properties Tab ................................................................................. 11-4 Partitioning Tab .................................................................................................. 11-5 Outputs Page ............................................................................................................. 11-7 Output Link Properties Tab .............................................................................. 11-8

Chapter 12. DB2/UDB Enterprise StageAccessing DB2 Databases ........................................................................................ 12-3 Remote Connection ........................................................................................... 12-5 Handling Special Characters (# and $) ........................................................... 12-6 Examples .................................................................................................................... 12-8

Table of Contents

vii

Looking Up a DB2/UDB Table ........................................................................12-8 Updating a DB2/UDB Table ...........................................................................12-10 Must Dos ..................................................................................................................12-12 Writing a DB2 Database ..................................................................................12-12 Updating a DB2 Database ...............................................................................12-12 Deleting Rows from a DB2 Database ............................................................12-13 Loading a DB2 Database .................................................................................12-13 Reading a DB2 Database .................................................................................12-14 Performing a Direct Lookup on a DB2 Database Table ..............................12-14 Performing an In Memory Lookup on a DB2 Database Table ...................12-15 Stage Page .................................................................................................................12-15 Advanced Tab ...................................................................................................12-15 NLS Map Tab ....................................................................................................12-16 Inputs Page ...............................................................................................................12-17 Input Link Properties Tab ...............................................................................12-18 Partitioning Tab ................................................................................................12-27 Outputs Page ............................................................................................................12-29 Output Link Properties Tab ............................................................................12-30

Chapter 13. Oracle Enterprise StageAccessing Oracle Databases .....................................................................................13-3 Handling Special Characters (# and $) ............................................................13-4 Loading Tables ....................................................................................................13-6 Examples .....................................................................................................................13-7 Looking Up an Oracle Table .............................................................................13-7 Updating an Oracle Table .................................................................................13-9 Must Dos .................................................................................................................. 13-11 Updating an Oracle Database ......................................................................... 13-11 Deleting Rows from an Oracle Database ......................................................13-12 Loading an Oracle Database ...........................................................................13-12 Reading an Oracle Database ...........................................................................13-13 Performing a Direct Lookup on an Oracle Database Table ........................13-13 Performing an In Memory Lookup on an Oracle Database Table ............13-14 Stage Page .................................................................................................................13-14 Advanced Tab ...................................................................................................13-14

viii

DataStage Enterprise Edition Parallel Job Developers Guide

NLS Map ........................................................................................................... 13-15 Inputs Page .............................................................................................................. 13-17 Input Link Properties Tab ............................................................................... 13-17 Partitioning Tab ................................................................................................ 13-25 Outputs Page ........................................................................................................... 13-27 Output Link Properties Tab ............................................................................ 13-28

Chapter 14. Teradata Enterprise StageAccessing Teradata Databases ................................................................................. 14-2 Installing the Teradata Utilities Foundation .................................................. 14-2 Creating Teradata User ..................................................................................... 14-2 Creating a Database Server .............................................................................. 14-3 Teradata Database Character Sets ................................................................... 14-3 Must Dos ................................................................................................................... 14-4 Writing a Teradata Database ............................................................................ 14-4 Reading a Teradata Database ........................................................................... 14-4 Stage Page .................................................................................................................. 14-5 Advanced Tab ..................................................................................................... 14-5 NLS Map ............................................................................................................. 14-6 Inputs Page ................................................................................................................ 14-7 Input Link Properties Tab ................................................................................. 14-7 Partitioning Tab ................................................................................................ 14-10 Outputs Page ........................................................................................................... 14-13 Output Link Properties Tab ............................................................................ 14-13

Chapter 15. Informix Enterprise StageAccessing Informix Databases ................................................................................ 15-2 Considerations for Using the High Performance Loader (HPL) ................ 15-2 Using Informix XPS Stages on AIX Systems .................................................. 15-6 Must Dos ................................................................................................................... 15-6 Writing an Informix Database .......................................................................... 15-6 Reading an Informix Database ........................................................................ 15-7 Stage Page .................................................................................................................. 15-8 Advanced Tab ..................................................................................................... 15-8 Inputs Page ................................................................................................................ 15-9

Table of Contents

ix

Input Link Properties Tab .................................................................................15-9 Partitioning Tab ................................................................................................15-12 Outputs Page ............................................................................................................15-14 Output Link Properties Tab ............................................................................15-15

Chapter 16. Transformer StageMust Dos ....................................................................................................................16-3 Transformer Editor Components ............................................................................16-4 Toolbar .................................................................................................................16-4 Link Area .............................................................................................................16-4 Meta Data Area ...................................................................................................16-4 Shortcut Menus ...................................................................................................16-5 Transformer Stage Basic Concepts ..........................................................................16-6 Input Link ............................................................................................................16-6 Output Links .......................................................................................................16-6 Editing Transformer Stages ......................................................................................16-8 Using Drag and Drop ........................................................................................16-8 Find and Replace Facilities ...............................................................................16-9 Select Facilities ..................................................................................................16-10 Creating and Deleting Columns .................................................................... 16-11 Moving Columns Within a Link .................................................................... 16-11 Editing Column Meta Data ............................................................................. 16-11 Defining Output Column Derivations ..........................................................16-12 Editing Multiple Derivations ..........................................................................16-15 Handling Null Values in Input Columns .....................................................16-18 Defining Constraints and Handling Otherwise Links ................................16-18 Specifying Link Order .....................................................................................16-20 Defining Local Stage Variables .......................................................................16-21 The DataStage Expression Editor ..........................................................................16-24 Expression Format ...........................................................................................16-24 Entering Expressions .......................................................................................16-26 Completing Variable Names ...........................................................................16-27 Validating the Expression ...............................................................................16-27 Exiting the Expression Editor .........................................................................16-27 Configuring the Expression Editor ................................................................16-27 System Variables ...............................................................................................16-28 Guide to Using Transformer Expressions and Stage Variables .................16-28 x DataStage Enterprise Edition Parallel Job Developers Guide

Transformer Stage Properties ................................................................................ 16-31 Stage Page ......................................................................................................... 16-31 Inputs Page ....................................................................................................... 16-35 Outputs Page .................................................................................................... 16-38

Chapter 17. BASIC Transformer StagesMust Dos ................................................................................................................... 17-2 BASIC Transformer Editor Components ............................................................... 17-3 Toolbar ................................................................................................................. 17-3 Link Area ............................................................................................................. 17-3 Meta Data Area .................................................................................................. 17-4 Shortcut Menus .................................................................................................. 17-4 BASIC Transformer Stage Basic Concepts ............................................................. 17-5 Input Link ........................................................................................................... 17-5 Output Links ...................................................................................................... 17-6 Before-Stage and After-Stage Routines ........................................................... 17-7 Editing BASIC Transformer Stages ......................................................................... 17-7 Using Drag and Drop ........................................................................................ 17-7 Find and Replace Facilities ............................................................................... 17-8 Select Facilities .................................................................................................... 17-9 Creating and Deleting Columns .................................................................... 17-10 Moving Columns Within a Link .................................................................... 17-10 Editing Column Meta Data ............................................................................ 17-11 Defining Output Column Derivations .......................................................... 17-11 Editing Multiple Derivations ......................................................................... 17-14 Specifying Before-Stage and After-Stage Subroutines ............................... 17-17 Defining Constraints and Handling Reject Links ....................................... 17-19 Specifying Link Order ..................................................................................... 17-21 Defining Local Stage Variables ...................................................................... 17-22 The DataStage Expression Editor ......................................................................... 17-25 Expression Format ........................................................................................... 17-25 Entering Expressions ....................................................................................... 17-27 Completing Variable Names .......................................................................... 17-28 Validating the Expression ............................................................................... 17-28 Exiting the Expression Editor ......................................................................... 17-29 Configuring the Expression Editor ............................................................... 17-29

Table of Contents

xi

BASIC Transformer Stage Properties ....................................................................17-29 Stage Page ..........................................................................................................17-30 Inputs Page ........................................................................................................17-31 Outputs Page ....................................................................................................17-33

Chapter 18. Aggregator StageExample ......................................................................................................................18-3 Must Dos ....................................................................................................................18-5 Stage Page ...................................................................................................................18-6 Properties Tab .....................................................................................................18-7 Advanced Tab ...................................................................................................18-15 NLS Locale Tab .................................................................................................18-16 Inputs Page ...............................................................................................................18-17 Partitioning Tab ................................................................................................18-17 Outputs Page ............................................................................................................18-19 Mapping Tab .....................................................................................................18-20

Chapter 19. Join StageJoin Versus Lookup ............................................................................................19-3 Example Joins .............................................................................................................19-3 Inner Join .............................................................................................................19-4 Left Outer Join ....................................................................................................19-4 Right Outer Join ..................................................................................................19-5 Full Outer Join ....................................................................................................19-5 Must Dos ....................................................................................................................19-6 Stage Page ...................................................................................................................19-6 Properties Tab .....................................................................................................19-7 Advanced Tab .....................................................................................................19-8 Link Ordering Tab ..............................................................................................19-9 NLS Locale Tab ...................................................................................................19-9 Inputs Page ...............................................................................................................19-10 Partitioning on Input Links ............................................................................19-10 Outputs Page ............................................................................................................19-13 Mapping Tab .....................................................................................................19-14

Chapter 20. Merge StageExample Merge ..........................................................................................................20-3 xii DataStage Enterprise Edition Parallel Job Developers Guide

Must Dos ................................................................................................................... 20-4 Stage Page .................................................................................................................. 20-5 Properties Tab ..................................................................................................... 20-5 Advanced Tab ..................................................................................................... 20-7 Link Ordering Tab .............................................................................................. 20-8 NLS Locale Tab ................................................................................................... 20-8 Inputs Page ................................................................................................................ 20-9 Partitioning Tab .................................................................................................. 20-9 Outputs Page ........................................................................................................... 20-12 Reject Links ....................................................................................................... 20-12 Mapping Tab ..................................................................................................... 20-13

Chapter 21. Lookup StageLookup Versus Join ............................................................................................ 21-6 Example Look Up ..................................................................................................... 21-6 Must Dos ................................................................................................................... 21-9 Using In-Memory Lookup tables .................................................................... 21-9 Using Oracle or DB2 Databases Directly ...................................................... 21-10 Using Lookup Fileset ....................................................................................... 21-12 Lookup Editor Components .................................................................................. 21-13 Toolbar ............................................................................................................... 21-13 Link Area ........................................................................................................... 21-13 Meta Data Area ................................................................................................ 21-13 Shortcut Menus ................................................................................................ 21-14 Editing Lookup Stages ........................................................................................... 21-15 Using Drag and Drop ...................................................................................... 21-15 Find and Replace Facilities ............................................................................. 21-16 Select Facilities .................................................................................................. 21-17 Creating and Deleting Columns .................................................................... 21-18 Moving Columns Within a Link .................................................................... 21-18 Editing Column Meta Data ............................................................................ 21-18 Defining Output Column Derivations .......................................................... 21-19 Defining Input Column Key Expressions .................................................... 21-22 Lookup Stage Properties ........................................................................................ 21-23 Stage Page ......................................................................................................... 21-23 Inputs Page ....................................................................................................... 21-25 Outputs Page .................................................................................................... 21-28 Table of Contents xiii

Lookup Stage Conditions .......................................................................................21-29 The DataStage Expression Editor ..........................................................................21-31 Expression Format ...........................................................................................21-32 Entering Expressions .......................................................................................21-33 Completing Variable Names ...........................................................................21-34 Validating the Expression ...............................................................................21-34 Exiting the Expression Editor .........................................................................21-35 Configuring the Expression Editor ................................................................21-35

Chapter 22. Sort StageExamples .....................................................................................................................22-4 Sequential Sort ....................................................................................................22-4 Parallel Sort .........................................................................................................22-7 Total Sort ............................................................................................................22-10 Must Dos ..................................................................................................................22-13 Stage Page .................................................................................................................22-13 Properties Tab ...................................................................................................22-13 Advanced Tab ...................................................................................................22-17 NLS Locale Tab .................................................................................................22-18 Inputs Page ...............................................................................................................22-18 Partitioning Tab ................................................................................................22-19 Outputs Page ............................................................................................................22-21 Mapping Tab .....................................................................................................22-22

Chapter 23. Funnel StageExamples .....................................................................................................................23-3 Continuous Funnel Example ............................................................................23-3 Sort Funnel Example ..........................................................................................23-6 Sequence Funnel Example ................................................................................23-8 Must Dos ..................................................................................................................23-10 Stage Page .................................................................................................................23-10 Properties Tab ................................................................................................... 23-11 Advanced Tab ...................................................................................................23-12 Link Ordering Tab ............................................................................................23-13 NLS Locale Tab .................................................................................................23-14

xiv

DataStage Enterprise Edition Parallel Job Developers Guide

Inputs Page .............................................................................................................. 23-14 Partitioning on Input Links ............................................................................ 23-15 Outputs Page ........................................................................................................... 23-17 Mapping Tab ..................................................................................................... 23-18

Chapter 24. Remove Duplicates StageExample ...................................................................................................................... 24-2 Must Dos ................................................................................................................... 24-5 Stage Page .................................................................................................................. 24-6 Properties Tab ..................................................................................................... 24-6 Advanced Tab ..................................................................................................... 24-7 NLS Locale Tab ................................................................................................... 24-8 Inputs Page ................................................................................................................ 24-8 Partitioning on Input Links .............................................................................. 24-9 Output Page ............................................................................................................. 24-11 Mapping Tab ..................................................................................................... 24-12

Chapter 25. Compress StageMust Dos ................................................................................................................... 25-2 Stage Page .................................................................................................................. 25-2 Properties Tab ..................................................................................................... 25-2 Advanced Tab ..................................................................................................... 25-3 Input Page .................................................................................................................. 25-3 Partitioning on Input Links .............................................................................. 25-4 Output Page ............................................................................................................... 25-6

Chapter 26. Expand StageMust Dos ................................................................................................................... 26-2 Stage Page .................................................................................................................. 26-2 Properties Tab ..................................................................................................... 26-2 Advanced Tab ..................................................................................................... 26-3 Input Page .................................................................................................................. 26-3 Partitioning on Input Links .............................................................................. 26-4 Output Page ............................................................................................................... 26-4

Table of Contents

xv

Chapter 27. Copy StageExample ......................................................................................................................27-2 Must Dos ....................................................................................................................27-6 Stage Page ...................................................................................................................27-7 Properties Tab .....................................................................................................27-7 Advanced Tab .....................................................................................................27-7 Input Page ...................................................................................................................27-8 Partitioning on Input Links ..............................................................................27-8 Outputs Page ............................................................................................................ 27-11 Mapping Tab ..................................................................................................... 27-11

Chapter 28. Modify StageExamples .....................................................................................................................28-2 Dropping and Keeping Columns .....................................................................28-2 Changing Data Type ..........................................................................................28-3 Null Handling .....................................................................................................28-4 Must Dos ....................................................................................................................28-5 Stage Page ...................................................................................................................28-5 Properties Tab .....................................................................................................28-5 Advanced Tab ...................................................................................................28-12 Input Page .................................................................................................................28-13 Partitioning on Input Links ............................................................................28-13 Outputs Page ............................................................................................................28-15 .............................................................................................................................28-16

Chapter 29. Filter StageSpecifying the Filter ..................................................................................................29-2 Input Data Columns ..........................................................................................29-2 Supported Boolean Expressions and Operators ............................................29-3 String Comparison .............................................................................................29-3 Examples .............................................................................................................29-4 Must Dos ....................................................................................................................29-5 Stage Page ...................................................................................................................29-6 Properties Tab .....................................................................................................29-6 Advanced Tab .....................................................................................................29-7

xvi

DataStage Enterprise Edition Parallel Job Developers Guide

Link Ordering Tab .............................................................................................. 29-8 NLS Locale Tab ................................................................................................... 29-8 Input Page .................................................................................................................. 29-9 Partitioning on Input Links .............................................................................. 29-9 Outputs Page ........................................................................................................... 29-12 Mapping Tab ..................................................................................................... 29-13

Chapter 30. External Filter StageMust Dos ................................................................................................................... 30-2 Stage Page .................................................................................................................. 30-2 Properties Tab ..................................................................................................... 30-2 Advanced Tab ..................................................................................................... 30-3 Input Page .................................................................................................................. 30-4 Partitioning on Input Links .............................................................................. 30-4 Outputs Page ............................................................................................................. 30-6

Chapter 31. Change Capture StageExample Data ............................................................................................................. 31-3 Must Dos ................................................................................................................... 31-4 Stage Page .................................................................................................................. 31-5 Properties Tab ..................................................................................................... 31-5 Advanced Tab ..................................................................................................... 31-8 Link Ordering Tab ............................................................................................ 31-10 NLS Locale Tab ................................................................................................. 31-10 Inputs Page .............................................................................................................. 31-11 Partitioning Tab ................................................................................................ 31-12 Outputs Page ........................................................................................................... 31-14 Mapping Tab ..................................................................................................... 31-15

Chapter 32. Change Apply StageExample Data ............................................................................................................. 32-4 Must Dos ................................................................................................................... 32-5 Stage Page .................................................................................................................. 32-6 Properties Tab ..................................................................................................... 32-6 Advanced Tab ..................................................................................................... 32-9

Table of Contents

xvii

Link Ordering Tab ............................................................................................32-10 NLS Locale Tab .................................................................................................32-10 Inputs Page ............................................................................................................... 32-11 Partitioning Tab ................................................................................................ 32-11 Outputs Page ............................................................................................................32-14 Mapping Tab .....................................................................................................32-15

Chapter 33. Difference StageExample Data .............................................................................................................33-3 Must Dos ....................................................................................................................33-4 Stage Page ...................................................................................................................33-5 Properties Tab .....................................................................................................33-5 Advanced Tab .....................................................................................................33-8 Link Ordering Tab ..............................................................................................33-9 NLS Locale Tab ...................................................................................................33-9 Inputs Page ...............................................................................................................33-10 Partitioning Tab ................................................................................................ 33-11 Outputs Page ............................................................................................................33-13 Mapping Tab .....................................................................................................33-14

Chapter 34. Compare StageExample Data .............................................................................................................34-2 Must Dos ....................................................................................................................34-3 Stage Page ...................................................................................................................34-4 Properties Tab .....................................................................................................34-4 Advanced Tab .....................................................................................................34-6 Link Ordering Tab ..............................................................................................34-7 NLS Locale Tab ...................................................................................................34-7 Inputs Page .................................................................................................................34-8 Partitioning Tab ..................................................................................................34-9 Outputs Page ............................................................................................................ 34-11

Chapter 35. Encode StageMust Dos ....................................................................................................................35-2 Stage Page ...................................................................................................................35-2

xviii

DataStage Enterprise Edition Parallel Job Developers Guide

Properties Tab ..................................................................................................... 35-2 Advanced Tab ..................................................................................................... 35-3 Inputs Page ................................................................................................................ 35-3 Partitioning Tab .................................................................................................. 35-4 Outputs Page ............................................................................................................. 35-6

Chapter 36. Decode StageMust Dos ................................................................................................................... 36-2 Stage Page .................................................................................................................. 36-2 Properties Tab ..................................................................................................... 36-2 Advanced Tab ..................................................................................................... 36-3 Inputs Page ................................................................................................................ 36-3 Partitioning Tab .................................................................................................. 36-4 Outputs Page ............................................................................................................. 36-4

Chapter 37. Switch StageExample ...................................................................................................................... 37-2 Must Dos ................................................................................................................... 37-3 Stage Page .................................................................................................................. 37-4 Properties Tab ..................................................................................................... 37-5 Advanced Tab ..................................................................................................... 37-7 Link Ordering Tab .............................................................................................. 37-8 NLS Locale Tab ................................................................................................... 37-8 Inputs Page ................................................................................................................ 37-9 Partitioning Tab .................................................................................................. 37-9 Outputs Page ........................................................................................................... 37-12 Mapping Tab ..................................................................................................... 37-13 Reject Link ......................................................................................................... 37-13

Chapter 38. SAS StageExample Job ............................................................................................................... 38-2 Must Dos ................................................................................................................... 38-5 Using the SAS Stage on NLS Systems ............................................................. 38-6 Stage Page .................................................................................................................. 38-7 Properties Tab ..................................................................................................... 38-7

Table of Contents

xix

Advanced Tab ................................................................................................... 38-11 Link Ordering Tab ............................................................................................38-12 NLS Map ............................................................................................................38-12 Inputs Page ...............................................................................................................38-13 Partitioning Tab ................................................................................................38-14 Outputs Page ............................................................................................................38-16 Mapping Tab .....................................................................................................38-17

Chapter 39. Generic StageMust Dos ....................................................................................................................39-2 Stage Page ...................................................................................................................39-2 Properties Tab .....................................................................................................39-2 Advanced Tab .....................................................................................................39-3 Link Ordering Tab ..............................................................................................39-4 Inputs Page .................................................................................................................39-4 Partitioning Tab ..................................................................................................39-5 Outputs Page ..............................................................................................................39-7

Chapter 40. Surrogate Key StageKey Space ....................................................................................................................40-2 Examples .....................................................................................................................40-5 Must Dos ....................................................................................................................40-7 Stage Page ...................................................................................................................40-7 Properties Tab .....................................................................................................40-7 Advanced Tab .....................................................................................................40-9 Inputs Page .................................................................................................................40-9 Partitioning Tab ................................................................................................40-10 Outputs Page ............................................................................................................40-12 Mapping Tab .....................................................................................................40-13

Chapter 41. Column Import StageExamples .....................................................................................................................41-2 Must Dos ....................................................................................................................41-7 Stage Page ...................................................................................................................41-7 Properties Tab .....................................................................................................41-7

xx

DataStage Enterprise Edition Parallel Job Developers Guide

Advanced Tab ..................................................................................................... 41-9 Inputs Page .............................................................................................................. 41-10 Partitioning Tab ................................................................................................ 41-10 Outputs Page ........................................................................................................... 41-13 Format Tab ........................................................................................................ 41-13 Mapping Tab ..................................................................................................... 41-26 Reject Link ......................................................................................................... 41-26 Using RCP With Column Import Stages ............................................................. 41-27

Chapter 42. Column Export StageExamples .................................................................................................................... 42-2 Must Dos ................................................................................................................... 42-6 Stage Page .................................................................................................................. 42-7 Properties Tab ..................................................................................................... 42-7 Advanced Tab ..................................................................................................... 42-8 Inputs Page ................................................................................................................ 42-9 Partitioning Tab .................................................................................................. 42-9 Format Tab ........................................................................................................ 42-12 Outputs Page ........................................................................................................... 42-24 Mapping Tab ..................................................................................................... 42-25 Reject Link ......................................................................................................... 42-26 Using RCP With Column Export Stages .............................................................. 42-26

Chapter 43. Make Subrecord StageExamples .................................................................................................................... 43-3 Must Dos ................................................................................................................... 43-6 Stage Page .................................................................................................................. 43-6 Properties Tab ..................................................................................................... 43-6 Advanced Tab ..................................................................................................... 43-7 Inputs Page ................................................................................................................ 43-8 Partitioning Tab .................................................................................................. 43-9 Outputs Page ........................................................................................................... 43-11

Chapter 44. Split Subrecord StageExamples .................................................................................................................... 44-2 Must Dos ................................................................................................................... 44-6 Stage Page .................................................................................................................. 44-6 Table of Contents xxi

Properties Tab .....................................................................................................44-6 Advanced Tab .....................................................................................................44-7 Inputs Page .................................................................................................................44-7 Partitioning Tab ..................................................................................................44-8 Outputs Page ............................................................................................................44-10

Chapter 45. Combine Records StageExamples .....................................................................................................................45-2 Example 1 ............................................................................................................45-3 Example 2 ............................................................................................................45-6 Must Dos ....................................................................................................................45-9 Stage Page ...................................................................................................................45-9 Properties Tab .....................................................................................................45-9 Advanced Tab ................................................................................................... 45-11 NLS Locale Tab ................................................................................................. 45-11 Inputs Page ...............................................................................................................45-12 Partitioning Tab ................................................................................................45-13 Outputs Page ............................................................................................................45-15

Chapter 46. Promote Subrecord StageExamples .....................................................................................................................46-3 Example 1 ............................................................................................................46-3 Example 2 ............................................................................................................46-5 Must Dos ....................................................................................................................46-8 Stage Page ...................................................................................................................46-9 Properties Tab .....................................................................................................46-9 Advanced Tab .....................................................................................................46-9 Inputs Page ...............................................................................................................46-10 Partitioning Tab ................................................................................................46-10 Outputs Page ............................................................................................................46-13

Chapter 47. Make Vector StageExamples .....................................................................................................................47-2 Example 1 ............................................................................................................47-2 Example 2 ............................................................................................................47-5

xxii

DataStage Enterprise Edition Parallel Job Developers Guide

Must Dos ................................................................................................................... 47-7 Stage Page .................................................................................................................. 47-8 Properties Tab ..................................................................................................... 47-8 Advanced Tab ..................................................................................................... 47-8 Inputs Page ................................................................................................................ 47-9 Partitioning Tab .................................................................................................. 47-9 Outputs Page ........................................................................................................... 47-12

Chapter 48. Split Vector StageExamples .................................................................................................................... 48-2 Example 1 ............................................................................................................ 48-3 Example 2 ............................................................................................................ 48-5 Must Dos ................................................................................................................... 48-7 Stage Page .................................................................................................................. 48-8 Properties Tab ..................................................................................................... 48-8 Advanced Tab ..................................................................................................... 48-8 Inputs Page ................................................................................................................ 48-9 Partitioning Tab .................................................................................................. 48-9 Outputs Page ........................................................................................................... 48-12

Chapter 49. Head StageExamples .................................................................................................................... 49-2 Head Stage Default Behavior ........................................................................... 49-2 Skipping Data ..................................................................................................... 49-4 Must Dos ................................................................................................................... 49-5 Stage Page .................................................................................................................. 49-5 Properties Tab ..................................................................................................... 49-5 Advanced Tab ..................................................................................................... 49-7 Inputs Page ................................................................................................................ 49-8 Partitioning Tab .................................................................................................. 49-8 Outputs Page ........................................................................................................... 49-10 Mapping Tab ..................................................................................................... 49-11

Chapter 50. Tail StageExamples .................................................................................................................... 50-2

Table of Contents

xxiii

Must Dos ....................................................................................................................50-4 Stage Page ...................................................................................................................50-4 Properties Tab .....................................................................................................50-4 Advanced Tab .....................................................................................................50-6 Inputs Page .................................................................................................................50-6 Partitioning Tab ..................................................................................................50-7 Outputs Page ..............................................................................................................50-9 Mapping Tab .....................................................................................................50-10

Chapter 51. Sample StageExamples .....................................................................................................................51-2 Sampling in Percent Mode ................................................................................51-2 Sampling in Period Mode .................................................................................51-6 Must Dos ....................................................................................................................51-8 Stage Page ...................................................................................................................51-8 Properties Tab .....................................................................................................51-8 Advanced Tab ...................................................................................................51-10 Link Ordering Tab ............................................................................................ 51-11 Input Page ................................................................................................................. 51-11 Partitioning on Input Links ............................................................................51-12 Outputs Page ............................................................................................................51-14 Mapping Tab .....................................................................................................51-15

Chapter 52. Peek StageMust Dos ....................................................................................................................52-2 Stage Page ...................................................................................................................52-2 Properties Tab .....................................................................................................52-2 Advanced Tab .....................................................................................................52-4 Link Ordering Tab ..............................................................................................52-6 Inputs Page .................................................................................................................52-6 Partitioning Tab ..................................................................................................52-7 Outputs Page ..............................................................................................................52-9 Mapping Tab .....................................................................................................52-10

xxiv

DataStage Enterprise Edition Parallel Job Developers Guide

Chapter 53. Row Generator StageExamples .................................................................................................................... 53-2 Using a Row Generator Stage in Default Mode ............................................ 53-2 Example of Specifying Data to be Generated ................................................ 53-4 Example of Generating Data in Parallel ......................................................... 53-7 Must Dos ................................................................................................................... 53-9 Stage Page .................................................................................................................. 53-9 Advanced Tab ..................................................................................................... 53-9 Outputs Page ........................................................................................................... 53-10 Properties Tab ................................................................................................... 53-10

Chapter 54. Column Generator StageExample ...................................................................................................................... 54-2 Must Dos ................................................................................................................... 54-7 Stage Page .................................................................................................................. 54-7 Properties Tab ..................................................................................................... 54-7 Advanced Tab ..................................................................................................... 54-9 Input Page .................................................................................................................. 54-9 Partitioning on Input Links ............................................................................ 54-10 Outputs Page ........................................................................................................... 54-12 Mapping Tab ..................................................................................................... 54-13

Chapter 55. Write Range Map StageExample ...................................................................................................................... 55-2 Must Dos ................................................................................................................... 55-4 Stage Page .................................................................................................................. 55-4 Advanced Tab ..................................................................................................... 55-4 NLS Locale Tab ................................................................................................... 55-5 Inputs Page ................................................................................................................ 55-6 Input Link Properties Tab ................................................................................. 55-7 Partitioning Tab .................................................................................................. 55-7

Chapter 56. Parallel Jobs on USSDeployment Options ................................................................................................ 56-1 Deploy Under Control of DataStage ............................................................... 56-2

Table of Contents

xxv

Deploy Standalone .............................................................................................56-4 Implementation Details ............................................................................................56-5 Directory Structure .............................................................................................56-5 Generated Files ...................................................................................................56-6 Configuration Files ............................................................................................56-7 Running Jobs on the USS Machine .........................................................................56-8 Deploying and Running from DataStage .......................................................56-8 Deploying from DataStage, Running Manually ............................................56-9 Deploying and Running Manually ................................................................56-10

Chapter 57. Managing Data SetsStructure of Data Sets ................................................................................................57-1 Starting the Data Set Manager .................................................................................57-3 Data Set Viewer ..........................................................................................................57-4 Viewing the Schema ...........................................................................................57-5 Viewing the Data ................................................................................................57-6 Copying Data Sets .................................