55
ADABAS Extraction & Change Data Capture NatWorks, Inc. Chris S. Bradley Presented by

ADABAS Extraction & Change Data Capture

  • Upload
    lihua

  • View
    78

  • Download
    3

Embed Size (px)

DESCRIPTION

ADABAS Extraction & Change Data Capture. Presented by. Chris S. Bradley. NatWorks, Inc. The Question…. Where Do You Need YOUR ADABAS Data Today?. DATA. DATA. Adabas. Data Warehouse Extraction. The Problems…. End-User Extraction Data Warehouse Extraction - PowerPoint PPT Presentation

Citation preview

Page 1: ADABAS Extraction & Change Data Capture

ADABASExtraction

&Change Data Capture

NatWorks, Inc.

Chris S. Bradley

Presented by

Page 2: ADABAS Extraction & Change Data Capture

2 TSI 4/05

The Question…

Where Do You Need YOUR

ADABASData Today?

Page 3: ADABAS Extraction & Change Data Capture

3 TSI 4/05

The Problems… End-User Extraction Data Warehouse

Extraction Web Publishing / Data

Exchange

Adabas

DATA

End-User Extraction

DATA

Data Warehouse Extraction

Web Publishing /Data Exchange

Page 4: ADABAS Extraction & Change Data Capture

4 TSI 4/05

The Message…

A Software AG customer who has ADABASADABAS & NATURALNATURAL

already has the best tools to handleallall

requirements forData Extraction & Change Data Capture

Page 5: ADABAS Extraction & Change Data Capture

5 TSI 4/05

ADABAS - Two Major Issues

How to Access ADABAS ADABAS data structures

Page 6: ADABAS Extraction & Change Data Capture

6 TSI 4/05

Accessing ADABAS Data

ADABAS

ADABAS Direct Calls

The Only Way to communicate directly to ADABAS is through Direct CallsDirect Calls

Page 7: ADABAS Extraction & Change Data Capture

7 TSI 4/05

Accessing ADABAS Data continued

ADABAS

ADABAS Direct Calls

Option #1

Imbedded

Direct Calls

Option #2

SQL / ODBC

Option #3

Natural

Option #4

ADABAS

Utilities

Option #5

Vendor

Utilities

Page 8: ADABAS Extraction & Change Data Capture

8 TSI 4/05

ADABAS Data Structures

All standard data formats are supported: alphanumeric, binary, fixed point, floating point,

packed decimal, unpacked decimal, ... Supports two basic field types

Elementary fields “recurring fields” (MUs)

Consecutive fields may be grouped A group may be repeated - Periodic Groups (PEs) Periodic Groups may contain one or more Multiple-

Value Fields

Page 9: ADABAS Extraction & Change Data Capture

9 TSI 4/05

ADABAS Data Structures continued

ADABAS has unique data structures:Multi-Valued Fields - “MUs” (array structure) Periodic-Groups - “PEs” (table structure) MUs in PEs (multi-dimensional structure)

ADABAS has some “difficult” data types IBM STCK-based Date and TimeStamp fields

EBCDIC to ASCIIPacked FieldsSign Byte Handling

Page 10: ADABAS Extraction & Change Data Capture

10 TSI 4/05

ADABAS Extraction Facts FACT #1

NATURAL was developed by Software AG specifically to access ADABAS

FACT #2NATURAL represents the most widely used AND best understood solution for accessing ADABAS

FACT #3What NATURAL should not be used to do, ADABAS Utilities handle (with support from NATURAL)

FACT #4Extraction / CDC should be done in BATCH

Page 11: ADABAS Extraction & Change Data Capture

11 TSI 4/05

ADABAS Extraction - Conclusions Natural allows flexible ADABAS Access Natural easily handles all ADABAS data

structures Natural easily handles all ADABAS data types Natural will always work with ADABAS

NaturalNatural for maximum flexibilityfor maximum flexibility

ADABAS Utilities for maximum performanceADABAS Utilities for maximum performance

Page 12: ADABAS Extraction & Change Data Capture

12 TSI 4/05

Solving ADABAS Data Access

ADABAS

ADABAS Direct Calls

Option #3

Natural

Option #4

ADABAS

Utilities

Embrace What Exists (ROI) Best Understood Solution High Performing Solution

Page 13: ADABAS Extraction & Change Data Capture

13 TSI 4/05

The Real Problem

NO GENERATIONGeneration is needed for: • Required Natural Objects• Required ADABAS Parameters• Required JCL / Script Processes

Page 14: ADABAS Extraction & Change Data Capture

a

Data Extraction Solutionfor

ADABAS

Page 15: ADABAS Extraction & Change Data Capture

15 TSI 4/05

Leveraging Proven Technology

ADABAS™

ADABAS™UtilitiesPredict™ Natural™

EXCEL ACCESS DesktopTools XML/XSL Load Ready

Data

DB2™

Load ReadyData

RDBMSof Choice

XMLTamino™

Targetof Choice

ETL Toolof Choice

ADASAVBackup

ADABAS Extraction

Page 16: ADABAS Extraction & Change Data Capture

16 TSI 4/05

The NatQuery Extraction SolutionNatQuery works by acting as an on-demand Natural Programmer.

From a graphical user interface a User is enabled to:

Create Query Specifications Generate Natural Data Extraction programs from

these Specifications Submit Generated Extract programs for execution Remotely monitor execution status Download Extracted Data Load extracted data into MS Access, MS Excel or

convert data into XML (with optional XSL)

Page 17: ADABAS Extraction & Change Data Capture

17 TSI 4/05

The NatQuery Extraction Solution

Administration Component

NatQuery

End-userComponent

Generation Component

Internally, NatQuery can be thought of as having three components:

Administrative Component End-User Component Generation Component

Page 18: ADABAS Extraction & Change Data Capture

18 TSI 4/05

The NatQuery Extraction Solution

Administration Component

NatQuery

End-userComponent

Generation Component

The Administrative Component is used by an Administrator to capture information that is specific to the platform, environment, and ADABAS data files that NatQuery will be used against.

The Administration Component provides NatQuery with the ability to capture application-specific intelligence.

Page 19: ADABAS Extraction & Change Data Capture

19 TSI 4/05

The NatQuery Extraction Solution

Administration Component

End-userComponent

Generation Component

End-user

RequiredFiles / Fields

DesiredTarget

User-specifiedSelect Logic

OptionalVariables

Natural Program

The End-User Component allows for the easy entry of extract specifications.

The Generation component generates a Natural program from a

specification.

Page 20: ADABAS Extraction & Change Data Capture

20 TSI 4/05

The NatQuery Extraction SolutionServer Environment

Workstation Environment

DATA

Natural

Natural Program

Natural Program

DATA

UserExtract

Specification

Access Excel XML

NatQueryNatQueryDATA

Other Environment

Adabas

ExtractionWithNatural

...

Page 21: ADABAS Extraction & Change Data Capture

21 TSI 4/05

Adabas

The NatQuery Extraction SolutionServer Environment

Workstation Environment

DATA

Natural

Natural Program

Natural Program

DATA

UserExtract

Specification NatQueryNatQueryDATA

Other Environment

ADACMPParameters

ADACMPUtility

ExtractionWithADABAS UtilitiesAndNatural

Page 22: ADABAS Extraction & Change Data Capture

22 TSI 4/05

The NatQuery Extraction Solution

Better control of requests Request execution can be easily scheduled Impact on online production applications are controlled

More efficient execution environment over “online” Significantly Less overhead

NatQuery handles Job Control Language (JCL) Template approach provides easy set-up /

maintenance Dynamic substitution makes templates executable

Requests Execute in “batch”

Page 23: ADABAS Extraction & Change Data Capture

23 TSI 4/05

NatQuery Administration Overview

Administration Component

NatQuery

End-userComponent

Generation Component

JCL

1) Provide Site-Specific Job Control Language (JCL)

JCL Templates Provide:• Integration to Natural / ADABAS• Dynamic Process Customization

Page 24: ADABAS Extraction & Change Data Capture

24 TSI 4/05

NatQuery Administration Overview

2) Capture Natural Data Definition Modules (DDMs)

JCL

DDMsDDMs are obtained:• Automatically via a User Request• Manually via an Import function

Administration Component

NatQuery

End-userComponent

Generation Component

Page 25: ADABAS Extraction & Change Data Capture

25 TSI 4/05

NatQuery Administration Overview

3) Capture Expanded DDM Info

JCL

DescriptorStatistics

FileRelationships

OccurrenceInformation

File I/OParameters

PredictMetadata

DDMs

Administration Component

NatQuery

End-userComponent

Generation Component

Administration Information provides NatQuery with application-specific generation intelligence

Page 26: ADABAS Extraction & Change Data Capture

26 TSI 4/05

NatQuery Generation Overview

Administration Component

End-userComponent

Generation Component

End-user

RequiredFiles / Fields

DesiredTarget

User-specifiedSelect Logic

OptionalVariables

Natural Program

The End-User Component allows for the easy entry of extract specifications.

The Generation Component converts an extract specification to an optimized Natural program.

Workstation Environment

Page 27: ADABAS Extraction & Change Data Capture

27 TSI 4/05

Server Integration Overview File Transfer Protocol (FTP) Integration

Direct FTP into batch Indirect FTP into batch Just FTP (Manual Execution)

Manual Integration IND$FILE (IBM) Manual FTP Other Methods...

Workstation Environment

NatQuery

Mainframe Environment

AutomatedCommunication

Is Achieved UsingStandard FTP

Page 28: ADABAS Extraction & Change Data Capture

28 TSI 4/05

FTP

Direct FTP Integration Overview

Server EnvironmentServer Environment User Submits Request Program is generated and imbedded into JCL /

Script Program and JCL / Script is FTP’ed to the Server Local Log File is written

Workstation Environment

NatQuery

JES (MVS), POWER (VSE)

Batch

RequestOutput

RemoteLog

Request Executes Execution updates Remote Log, creates

Output User Retrieves Output

Output automatically FTP’ed to workstation

UserRequest

UserRequestNatural Program

RequestOutput

LocalLog

User Builds QueryUser Builds Query Specification

Page 29: ADABAS Extraction & Change Data Capture

29 TSI 4/05

FTP

In-Direct FTP Integration Overview

Server EnvironmentServer Environment

Workstation EnvironmentWorkstation Environment

Batch Natural

NatQuery FTP Server

UserRequest

RequestOutput

UserLog

Server Submits Request Execution updates remote log, creates output

User Retrieves Output Output automatically FTP’ed to workstation User

RequestUser

RequestNatural Program

RequestOutput

NatQuery

LocalLog

User Builds Query Specification User Submits Request

Program is generated and imbedded into JCL / Script

Program and JCL / Script is FTP’ed to the Server Local Log File is written

Page 30: ADABAS Extraction & Change Data Capture

30 TSI 4/05

FTP

Just FTP Integration Overview

Server Environment

Workstation Environment

Batch Natural

UserRequest

RequestOutput

UserLog

User Manually Submits Request Execution updates remote log, creates output

User Retrieves Output Output automatically FTP’ed to workstation User

RequestUser

RequestNatural Program

RequestOutput

NatQuery

LocalLog

User Builds Query Specification User Submits Request

Program is generated and imbedded into JCL / Script

Program and JCL / Script is FTP’ed to the Server Local Log File is written

Page 31: ADABAS Extraction & Change Data Capture

31 TSI 4/05

NatQuery Integration to ETL Tools

Workstation Environment

DSX Generation DataStage Exchange file (DataStage proprietary format) Allows for Full Integration of Predict Metadata

CFD Generation COBOL File Definition (in copybook format)

Generation Component

NatQuery ETL tool

NatQuery Generates Descriptions of Extract Layout

ImportImport

DATA

DSXFilesCFDFilesSGTFiles

Page 32: ADABAS Extraction & Change Data Capture

32 TSI 4/05

NatQuery Features Optimized Access to Source File(s), Based on User-

Entered Selection Criteria Automatic determination and generation of best access method

– Descriptors, Super-Descriptors, Sub-Descriptors…– Read Logical, Read Physical, Find, Get– Zero coding effort required– Full Sensitivity of Suppression

Autmomatic Support for Multi-Fetch (Pre-Fetch) Automated Integration to Server (FTP)

Download DDMs (direct support for SYSTRANS utility) Automatic Generation of required Descriptor Statistics Automated Upload, Execute and Download of Results Automated Extraction of PREDICT Meta Data

Page 33: ADABAS Extraction & Change Data Capture

33 TSI 4/05

NatQuery Features - continued...

Full handling of All ADABAS Field Types Date, TimeStamp, Packed, Integer, Binary, ...

Data Conversion at Extract LevelData Conversion at Extract Level Conversion of ADABAS formats to ASCII equivalents Full Ability to handle Sign Byte for numeric fields

Full handling of All ADABAS “recurring” dataFull handling of All ADABAS “recurring” data MUs, PEs, MUs in PEs Administratively defined defaults and maximums User over-ride of defaults (within allowed maximums) Ability to “Flatten” or “Concatenate” at field level

Page 34: ADABAS Extraction & Change Data Capture

34 TSI 4/05

NatQuery Features - continued...

First-Name Last-Name Address-Line City ...

Suite 100454 South Main Street

“Flattening” a recurring data structure

OneSourceRecord

TwoExtractRecords Chris Bradley 2 Suite 100 Northfield ...

Chris Bradley 1 454 South Main Street Northfield ...

NatQuery built Index

Two occurrences

Page 35: ADABAS Extraction & Change Data Capture

35 TSI 4/05

NatQuery Features - continued...

First-Name Last-Name Address-Line City ...

“Concatenating” a recurring data structure

OneSourceRecord

OneExtractRecord

Chris Bradley 454 South Main Street Suite 100 Northfield ...

Two occurrencesSuite 100454 South Main Street

Page 36: ADABAS Extraction & Change Data Capture

36 TSI 4/05

NatQuery Features - continued...

Direct Integration to ETL Tools Automatic generation of interface files

– DSX and / or CFD files

Automatic Linking of up to 5 “Primary” files No User Knowledge Required

Support for Automatic “Look-up” files Code-to-Text conversions, Administratively defined Look-ups are “transparent” to the user

Page 37: ADABAS Extraction & Change Data Capture

37 TSI 4/05

NatQuery Features - continued...

Support “Full Extract” or “Intersection Set” Multi-File Flexible Extraction

Abilitity to Define Variables Redefines, Constants, Expressions, Compress and

Dynamic (date-based and user input)

Direct Support for ADABAS utilities ADACMP, ADASEL, ADACDC

Use of “Batch” Provides Controllable Extraction

Page 38: ADABAS Extraction & Change Data Capture

38 TSI 4/05

NatQuery Features - continued...

Full Manipulation of Query Specifications Save, Save As, Delete Query specifications stored with a long and short query

description

User Specifiable Data Extract Limits “Test” Extracts

Administratively Contolled User Data Limits Ability to disallow Read Physical Ability to set Record Limits

One Tool - Dual Use End-User Extraction Data Warehouse Extraction

Page 39: ADABAS Extraction & Change Data Capture

39 TSI 4/05

NatQuery Features - continued...

Minimal Mainframe Footprint Just Natural, JCL and mainframe disk space Existing ADABAS utilities (optional)

Extraction Capability to Any Data Source Natural Can Talk to ADABAS, VSAM, DB2...

Integration to Desktop Tools MS Access MS Excel XML (with optional XSL) Tab Delimited or User-Specified Delimiters

Page 40: ADABAS Extraction & Change Data Capture

40 TSI 4/05

NatQuery Features - continued...

Integration to PREDICT All Field-Level PREDICT Metadata is made available within

NatQuery Administrator can create their own

Ability to Trace I/O generation review I/O generation process

Full Support of Native Security Natural Security ADABAS Security

Automatic Update Ability Allows for centralized roll-out of new versions

...

Page 41: ADABAS Extraction & Change Data Capture

theChange Data Capture Solution

for

ADABASADABASNatWorks, Inc.

Page 42: ADABAS Extraction & Change Data Capture

42 TSI 4/05

Leveraging Proven Technology

ADABAS™

ADABAS™UtilitiesNatural™

EXCEL ACCESS DesktopTools XML/XSL Load Ready

Data

Load ReadyData

RDBMSof Choice

ETL Toolof Choice

ADABASPLOG

ADABAS Change Data Capture & Transaction Auditing

Page 43: ADABAS Extraction & Change Data Capture

43 TSI 4/05

The Source of ADABAS Changes

ADABAS’ transaction recovery mechanism 100% data integrity, all transactions recorded

True “Point-in-Time” snapshot of ADABAS Changed Data Available w/o ADABAS Access

PLOG 2

ADABAS Protection Log (PLOG)

Adabas

Mainframe Environment

PLOG 1

Page 44: ADABAS Extraction & Change Data Capture

44 TSI 4/05

ADABAS CDC (Change Data Capture)

PLOG contains all transactions against all Files PLOG is in compressed format

The same compression used by ADABAS PLOG data is stored in Variable-Length records

Different from file to file and within same file PLOG data requires “conversion”

EBCDIC to ASCII, date / time formats

Issues in accessing PLOG:

Page 45: ADABAS Extraction & Change Data Capture

45 TSI 4/05

ADASEL utility (ADABAS 6): “Splits” PLOG transactions into separate files

– One File for each requested ADABAS File– Handles Expanded Files

Decompresses PLOG records ADACDC utility (ADABAS 7):

everything ADASEL does direct delivery of Delta changes

ADABAS CDC

ADABAS utilities solve most PLOG issues:

Page 46: ADABAS Extraction & Change Data Capture

46 TSI 4/05

NatCDC converts variable-length to fixed-length User-Specified number of MU and PE occurrences PLOG Header converted

– IBM STCK time, Expanded File ISNs are normalized NatQuery generates all required objects

All Parameters and Programs

ADABAS CDC

NatCDC / NatQuery solves remaining issues:

Page 47: ADABAS Extraction & Change Data Capture

47 TSI 4/05

NatCDC Base Components

Workstation Environment Server Environment

RawPLOGData

Disk

Tape

ADASEL

NatQuery

NatCDC SORT

ADASEL / ADACDC utility supplied with ADABAS

NatCDC utility (Single Optimized Natural Program)

System Sort Program

Page 48: ADABAS Extraction & Change Data Capture

48 TSI 4/05

NatCDC Processing Overview

Workstation Environment Server Environment

RawPLOGData

Disk

Tape

RawFILEData

Disk

Tape

FixedLength

Data

Disk

Tape

DWHCDCData

Disk

TapeGeneratedParameters

ADASEL

NatQuery

NatCDC

GeneratedParameters

GeneratedProcessingProgram

SORT

One JCL Stream for each file(or expanded file chain)

One JCL Streamfor each 20 files

DDM

GeneratedParameters

Page 49: ADABAS Extraction & Change Data Capture

49 TSI 4/05

NatCDC Features 100% Data Integrity

All transactions handled, even Backouts Simple Mainframe Installation

One Single Natural object program (NatCDC) One Natural Program for each file One JCL Stream per file

The Fastest and Most Trusted Decompression SAG knows their own compression the best Performance is Critical

– CDC is a frequently occurring activity

Page 50: ADABAS Extraction & Change Data Capture

50 TSI 4/05

NatCDC Features

Variable-Length to Fixed Length conversion Final Layout is User-Determined Recurring Fields Padded or Truncated

– Exception Reports Produced Automatically Full Handling of all ADABAS data structures

MUs, PEs, and MUs in PEs Automatic format translations:

– EBCDIC to ASCII– Date and Timestamp– Sign handling of all numeric-based fields

Page 51: ADABAS Extraction & Change Data Capture

51 TSI 4/05

NatCDC Features

Full Support for Expanded FilesPhysical to Logical ISN conversion

Full Generation of all Required ObjectsAll ParametersAll ProgramsAll JCL

Data is supplied with Standard HeaderTransaction Date, Time, ISN, Seq#, ...

Page 52: ADABAS Extraction & Change Data Capture

52 TSI 4/05

NatCDC Features Field Selection Options

C* values available as dataFields may be selectively omitted

Integration to ETL ToolsCFD generation “DSX” generation (Ascential DataStage)

Time Differential Handling OptionsAutomaticManual

Page 53: ADABAS Extraction & Change Data Capture

53 TSI 4/05

NatCDC Features Data Output Options

Logical Last – Single Record flagged as Store, Update or Delete

Logical First and Last– One or two records flagged as Before or After

All Extensive Reporting Options

Occurrence Exception Processing Store, Update, Delete Counts Total Before and After images ...

Page 54: ADABAS Extraction & Change Data Capture

54 TSI 4/05

NatCDC Benefits

Cost EffectiveUses vendor supplied utilityUses Natural

One Tool - Dual UseData Warehouse Change Data Capture

(CDC)End-User Extraction

...

Page 55: ADABAS Extraction & Change Data Capture

Simple ideas with enormous potential.

www.treehouse.com | [email protected]

and