IMS Database Primer

IMS Database Primer

Original Material Created by:

Dennis EichelbergerIT SpecialistIMS Support – Washington System Center

Modified for the Canadian Eastern Region IMS User Groups by:

Gary Wicks, IT Specialist - IMS Lab Services

Contents

• Short history and business positioning for IMS• Architectural structural fundamentals• Value descriptions for IMS• DB hierarchical structures• Defining databases• Defining programs• ODBM• SQL access and the Catalog• IMS DB education available2

3

Information Management System (IMS)

Originally created for the NASA Apollo Projectto put a man on the moon by IBM and Client corroborations

Success!

4

Where the World Depends on IMS• Manufacturing

– Aerospace, equipment, automobile, …

• Banking• Telecommunications• Bulk Transportation: Air and Rail• Travel: Air, Car and Rail• Health Providers• Insurance• Finance/Credit Card Processing• Package delivery• Government• Retail• …

IMS Architecture Fundamental Concepts• Database management (IMS DB component)

– Separated from application programming• IMS realized that moving the definition of the data out of the program would have substantial benefits when many programs could access the same database

• Transaction management (IMS TM component)

– Separated from application programming• The sending and receiving of input and output messages was removed from programs

– Allows the same program to access the message regardless of what type of device was used to send in the message

– IMS's message queues were critical to its initial design

• Multiple region online system

– Applications run in their own regions• Also called partitions, address spaces, …

5

Possible Xmemory access from and between IMS address spacesDB2 APPC IRLM Others

DL/ISAS

CTRL

...

DRA

CCTL MPP BMP IFP

Multiple application threads per address space for each CCTL and ODBA region

Your application programs run in the green address spaces in this bottom row

JMP JBP

MQSeries RRS

Appl Region

ODBA

WLM

SPOC

TCPIP

VTAM Console

WEBSPHERE

IXGLOGR

IMS Connect(BPE)

...

Tools

...

DBRC CQS(BPE)

OMCSL(BPE)

RMCSL(BPE)

SCICSL(BPE)

ODBMCSL(BPE)

IMS Address Space Structures

6

Compatibility: The Foundation of IMS Success• Application compatibility

– Programs written in 1969 still work– IMS does not require recompiles of applications for new IMS releases

• Database compatibility– Databases do not require upgrades for new IMS releases

• Application programs are independent of target database type and access method utilized

• IMS has allowed users to grow their applications in lieu of forcing major conversions

7

Adaptability • And yet IMS also keeps up with technological trends

– IMS provides a TCP/IP socket server – IMS Connect• For access to IMS transactions and databases

– Allows IMS applications to be either clients or servers – Delivers adapters that run in JEE, SOAP, etc. environments

• Pluggable components that provide easy interfaces to IMS resources – Supports type-2 and type-4 universal drivers for access to IMS DB

• IMS provides application programming interfaces– Unique to IMS

• DL/I calls to access IMS resources: message queues and data• IMS applications can be written in: COBOL, PL/I, Assembler, C/C++, Pascal, ADA, REXX, Java

– Portable – Java classes and JDBC 8

Programming Interface• Database definitions reside as IMS meta data outside of programs

– DBDs • Defines the database structure

– PSBs• Views of application sensitivity to segments within databases• Contain access definitions (get, insert, delete, replace)

• Programs– DL/I calls used for database access

• Program not aware of data sets and physical characteristics of data

• DL/I calls used for input/output messages– Program independent of communication protocols– IMS applications get messages from a queue, process them, and

insert them back to the queue

9

What New Users of IMS Need to Understand• IMS Databases (this is todays introduction)

• IMS Online Systems and Transaction Management

• IMS Connect

• Application Programming Interfaces to IMS

• IMS in a Parallel Sysplex®

10

JJ

II

H 2

GGF

E

DDDCC

IMS Database Structure• IMS databases are hierarchical

A

B G

C

F

IE

J

H 1D

Root segment – Segments have parents and children• Segments are accessed through their parents

• Parents and Children are relatedD is accessed through A and B

– Segments have twins• Multiple B segments under one A segment• Twins are related • H2 is accessed via H1

- Segment pointers are updated by IMS at the same time the segment is updated

A segment type (e.g. A, B, or C) is similar to a relational table.A segment occurrence, one of the segments, is similar to a row in a relational table.An IMS database is similar to several tables which are joined.

Tree structure

11

IMS Database Structure …• Full Function database types

– Hierarchical indexed direct (HIDAM)– Partitioned HIDAM (PHIDAM)– Generalized Sequential (GSAM)– Hierarchical indexed sequential (HISAM)*– Hierarchical Sequential (HSAM) *– Hierarchical direct (HDAM)– Partitioned HDAM (PHDAM)

* Simple version available

• Fast Path database types– Data Entry Database (DEDB)– Main Storage Database (MSDB)

• MSDBs are root-only and stored & accessed only in memory

§ Indexed databases– HIDAM, PHIDAM, HISAM– Roots are ordered in key sequence

§ Non-indexed databases– HDAM, PHDAM, DEDB– Randomization routines are used to

order roots

Structure is chosen based on what makes most sensefor access patterns, e.g., sequential versus random

12

Value and use of Hierarchical Structures• GSAM

Usage Characteristics

Insert sequentially – time order Delete unavailable

Used for IMS BMP and Batch jobs requiring restartability

13

Value and use of Hierarchical Structures• HSAM

Usage Characteristics Not available

Insert in key sequence Secondary Indexing

Logical relationships

Variable length segments

Update not available DLET or REPL call

Used for low volume processing e.g. data archival

14

Value and use of Hierarchical Structures• HISAM

Usage CharacteristicsSame size records

Complete record fits into logical defined recordFew Deletes or Inserts Delete / Insert Available

Root Delete Space is reclaimed in primary area, not reclaimed in overflow

Dependent Segment Marked deleted, Space not available

Database reorganization required to reclaim physical space

15

Value and use of Hierarchical Structures• HDAM

Usage CharacteristicsSegments are unordered Position based on key randomization

Segment relationships via pointers Dependents may be in separate blocks

Delete removes segment Space is reclaimed

High activity Speedy segment retrieval and update

16

HDAM Database Randomization

17

RAP1 RAP2 RAP3

SEG12 SEG36 SEG82

SEG04 SEG01 SEG95

Block 3RAP1 RAP2 RAP3

SEG15 SEG06 SEG58

SEG65 SEG47 SEG13


SEG02 SEG68 SEG77

SEG47 SEG02 SEG81

Block 1

RAP1 RAP2 RAP3

SEG01 SEG14 SEG55

SEG01 SEG01 SEG73

Block 4

RAP1 RAP2 RAP3

SEG71 SEG53 SEG01

SEG85 SEG84 SEG29


SEG05 SEG04 SEG90

SEG48 SEG19 SEG76

Block 5

DATABAS1

RandomizerDFSHDC40

RMNAME=(DFSHDC40,3,3,25)

Value and use of Hierarchical Structures• HIDAM

Usage CharacteristicsTwo DBDs required 1 = Index 2 = Data area

Root Segments are sequentially ordered Index may be processed separately

Segment relationships via pointers Dependents may be in separate blocks

Delete removes segment Space is reclaimed

High activity Speedy segment retrieval and update

18

Large DB Size Support Through HALDB and Fast Path DEDBs• HALDB (PHDAM and PHIDAM) and DEDB databases

– Database is composed of multiple database data sets• HALDB (High Availability Large Database) calls these datasets partitions• DEDB calls these datasets areas

– HALDB may have up to 1001 partitions• Each partition may have up to 10 4GB data sets

– Over 40 terabytes!• In IMS 14 the OSAM data sets may be up to 8GB

– Over 80 terabytes

– DEDBs may have up to 2048 areas• Each area has one data set up to 4GB

– Over 8 terabytes

Application programs are unaware of the partitions or areas

The end user accesses their data regardless of position or residence

19

Data Base Secondary Index use

• IMS databases may also have Secondary indexes

– Indexes are based on different fields in any segment

– Example:• Secondary index into course database based on student ID– One could find all the classes in which a student is enrolled

Course

Instructor Class

Student

Sec. IndexStudent ID

ID

20

Data Base Logical Relationships• Full Function database may have logical relationships

– Logical databases are formed from multiple physical DBs– Example:

• Logical relationship between employee and department DBs

• Examples of logical databases

Employee

Depart. No.Salary History

Department

Member

Employee

Depart. No. | Depart.Salary History

Member

Member | Employee

Depart. No.Salary History

Department

21

Defining Databases• Database Definition - DBD

– Database definition is used to define:• Database type • Dataset type (primary / overflow)• Number of Segments• Hierarchical relationship of the segments• Fields within a Segment• Key fields• Data types of the fields• Logical relationships within or outside of the database• Indexing

22

Defining DatabasesDBD NAME=DI21PART, X

ACCESS=(HISAM,VSAM), XPASSWD=NO, XDATXEXIT=NO, X

DATASET DD1=DI21PART, XDEVICE=3380, XOVFLW=DI21PARO, XBLOCK=(100,10), XSIZE=(4096), XRECORD=(678)

SEGM NAME=PARTROOT, XEXTERNALNAME=PARTROOT, XPARENT=0, XBYTES=50, XRULES=(LLL,LAST), X

FIELD NAME=(PARTKEY,SEQ,U), XEXTERNALNAME=PARTKEY, XBYTES=17, XSTART=1, XTYPE=C, XDATATYPE=CHAR

SEGM NAME=STANINFO, XEXTERNALNAME=STANINFO, XPARENT=PARTROOT, XBYTES=85, XRULES=(LLL,LAST),… X

23

Defining ProgramsProgram Specification Block - PSB

– Program Specification is used to define Program Control Blocks - PCB:– Program Control Blocks define

• Database(s) to be accessed• Processing options to access it with

– Get, Insert, Delete, …• Segments the program is aware of• Fields within the segment the program is aware of• In general, the program view of the accessed database(s)

24

Defining DatabasesDBD NAME=DI21PART, X

ACCESS=(HISAM,VSAM), XPASSWD=NO, XDATXEXIT=NO,


SEGM NAME=PARTROOT, XEXTERNALNAME=PARTROOT, XPARENT=0, XBYTES=50, XRULES=(LLL,LAST),


SEGM NAME=STANINFO, XEXTERNALNAME=STANINFO, XPARENT=PARTROOT, XBYTES=85, XRULES=(LLL,LAST), …

DBPCB00 PCB TYPE=DB,DBDNAME=DI21PART,PROCOPT=A,KEYLEN=44 SENSEG NAME=PARTROOT,PARENT=0 SENSEG NAME=STANINFO,PARENT=PARTROOT SENSEG NAME=STOKSTAT,PARENT=PARTROOT SENSEG NAME=CYCCOUNT,PARENT=STOKSTAT SENSEG NAME=BACKORDR,PARENT=STOKSTAT PSBGEN LANG=ASSEM,PSBNAME=DFHSAM04,MAXQ=1 END

DBPCB01 PCB TYPE=DB,DBDNAME=DI21PART,PROCOPT=G,KEYLEN=43 SENSEG NAME=PARTROOT,PARENT=0 SENSEG NAME=STANINFO,PARENT=PARTROOT SENSEG NAME=STOKSTAT,PARENT=PARTROOT PSBGEN LANG=COBOL,PSBNAME=DFSSAM01,MAXQ=1 END

25

Identifying Databases to Program Relationships• DataBase generation - DBD• Program Specification Block generation - PSB• Application Control Block - ACB

– Connects DBDs and PSBs– A PSB may access multiple databases– Multiple PSBS may access the same database– Application changes or additions may be implemented without an IMS restart.

• Affected resources may need to be quiesced for control block updates• The Online Change process is normally utilized for this

Program 1 DBD

Program 1 PSB

Program 2 ACBProgram 1 ACB

26

Database Access• All IMS Database access is though a standard programming

– DL/1 using DLI

• Data Language 1 through the Data Language Interface

– AIBTDLI Application Interface – REXX, Java using PCB labels– ASMTDLI Assembler programs– CTDLI C programs– CEETDLI C programs using PCB labels– CBLTDLI COBOL programs– PASTDLI Pascal programs– PLITDLI PL/1 programs

27

Database Access

• DL/1 call– CALL CBLTDLI USING GU dbpcb i/oarea argument

– Translates to:• LLZZtrancode ‘search argument’ landing on the Message Queue.• IMS sees ‘trancode’ and finds the (Scheduler Message Block) SMB related to that transaction

• Loads the program to start in a dependent region• Passes the input message to the program

28

Open Database Manager Access - ODBM • Implemented with Common Service Layer• Allows database access outside of dependent region• Some components are:

– IMS Universal Database resource adapter (DRDA)– IMS Universal JDBC driver– IMS Universal DL/I driver– Open Database Access interface

(ODBA)– ODBM CSL interface

• Security & Access throughIMS Connect

ClientApplication

z/OS

IMSCONNECT

Exit

IMS

Database

manager

SCI OperationsManager

IMSCONNECTExtensions

ODBM DLI

Universal DLI

WebSphereApplicationServer WAS

CSLDMI

JDBC Driver

ODBM

Database Database

RECON

Database

29

IMS Catalog• Database Considerations

– Traditional database definitions can ‘miss’ the entirety of the data • Define a segments total length

– Also can define a Key / Sequence field• The remainder is defined as ‘space’• Relies on the application program to map the ‘space’ using copy books

– Relational views of traditional IMS databases using SQL or JDBC• Require search fields to be defined in the meta data

– IMS meta data is stored in the IMS Catalog

30

IMS Catalog• IMS Version 12 added an IMS Catalog

– IMS meta data • Single source of trusted repository of meta data• Loaded and updated from IMS ACBLIB• Use IMS 14 with the IMS Explorer for Development to create DDL

– Send DDL directly to the IMS Catalog– IMPORT DEFN SOURCE(CATALOG) to activate (TYPE 2 Command)

– Uses HALDB for its layout • Registered to DBRC• Uses four dataset groups • Database maintenance procedures apply

– Image Copy– Reorganization– Recovery

– Loaded and updated from IMS ACBLIB31

Defining Databases for Eventual Use in the CatalogDBD NAME=DI21PART, X

ACCESS=(HISAM,VSAM), XPASSWD=NO, XDATXEXIT=NO,


SEGM NAME=PARTROOT, XEXTERNALNAME=PARTROOT, XPARENT=0, XBYTES=50, XRULES=(LLL,LAST),


FIELD NAME=PARTNAME, XEXTERNALNAME=PARTNAME, XBYTES=24, XSTART=18, XTYPE=C, XDATATYPE=CHAR

SEGM NAME=STANINFO, X

Meta data update

1. Add fields needed to the DBD source to regen

2. FTP Source to DBD3. Generate the DBD4. Generate the ACB5. Perform an Online

Change

32

Expansion into SQL access• Understanding that the world has moved towards the relational view of databases and SQL to access them

– IMS also provides the ability to view the hierarchical database structure in a relational way

• The IMS Enterprise Suite Explorer– Creates an SQL View of the Hierarchical database

» Metadata mapping from the IMS Catalog

• And a set of java universal drivers that interpret the calls made by java programs– Local to z/OS– From remote environments

33

SQL calls to IMS databases• IMS Version 13 added direct SQL access using COBOL Version 5.1

– Builds upon the catalog trusted meta data– Allows a COBOL application program to make SQL calls to access IMS hierarchical databases• Most SQL calls are supported• IMS 14 adds more SQL support

PartitionA

PartitionB

PartitionC

PartitionA

PartitionB

Catalog Database

Catalog Secondary Index

Catalog

IMS

IMS ExplorerFor Development

ACBs

34

IMS Data Base Courses AvailableIn order to begin to master IMS DB concepts definitions and operational control and recovery, the following courses are available:

• CM220 IMS Physical Organization of Databases. 4 Day Lecture / Lab• CM301 IMS DB Performance and Tuning. 5 Day Lecture / Lab• CM20 DBRC. 5 Day Lecture / Lab• CM46 HALDB. 1 Day Lecture • CM50 IMS Data Sharing. 3 Day Lecture / Lab

– We will now introduce each course and give a sample of information contained within

Contact your IBM representative to learn more35

IMS Physical Organization Course Objectives• Code the Database Directories (DBD) and Program Specification Blocks (PSB) for physical databases• Code the DBDs and PSBs to implement secondary indexing• Use the appropriate IMS utilities to reorganize and recover physical databases, including those with secondary indexes• Use the DL/I test program• Prepare a job stream to load a database using any of the IMS access methods

36

Start with the Most Desirable Block (MDB): CI/BLOCK requiring least amount of I/O to perform pointer maintenance during insert activity

1. Check MDB for free space2. Check 2nd MDB for free space3. Block in buffer pool from same track4. Block in buffer pool from same cylinder5. Block on same track (bit map)6. Block on same cylinder (bit map)7. Block in buffer pool within +/- scan cylinder8. Block within +/- scan cylinder (bit map)9. Block in buffer pool at end of data set10. Block at end of data set (bit map) – then new dataset extent11. Any block in data set (bit map)

E S D S / O S A M

BIT MAP

MDB

DATABASE DATA SET

BIT MAP

MDB

BUFFER POOL

HDAM HIDAM

Sample from DB Physical Organization Course

HD SpaceSearch Algorithm

37

IMS DB Performance and Tuning Course Objectives• Analyze performance data about the IMS database environment• Choose IMS access methods and options that provide the best database performance• Improve performance by selecting database buffer pools and buffer pool options and with appropriate data set access method and storage attributes• Implement the optimum performance options for VSAM data sets at define and execution time• Evaluate the need for secondary indexes and select options to improve their performance

38

PROGRAM ELAPSED TIME

- Operating System (MVS-z/OS) PAGING delays- Operating System (MVS-z/OS) DISPATCHING delay

IWAIT TIME

DATABASE I/OSYNC PT PROC

LOCKING ACTIVITY WITHIN PROGRAM

ISOLATION OR IRLM

NOT-IWAIT TIME

DLI CALL PROC DATABASE OPENLOGGING I/OLWAWAIT: LOG BUFFERSLATCH REQUESTSBUFFER MGMTUSER EXITSOTHER ...

DLI ELAPSED TIMENOT-DLI TIME

APPL PGM PROC

DB2 CALLS

Sample from DB Performance and Tuning Course

39

DBRC Course Objectives• Install, implement, and manage the DBRC (Database Recovery Control) component of IMS (Information Management System)

• Identify and list the basic elements of DBRC • Describe the processing of DBRC records • Describe the data sharing environment• Describe the subsystem interaction and database authorization process • Register full-function, fast path, HALDB, non-recoverable databases, and their data sets

• Define and register database data set groups, data groups, and recovery groups

• Define and register change accumulation groups • Describe installation tasks for DBRC and RECON data sets

40

CHAN2

BCS

VOL456

P.RECON2

CACHECU2

CHAN3

BCS

VOL789

P.RECON3

CACHECU3

BCS

VOL123

P.RECON1

CACHECU1

CHAN1

Specify DASD fast write (DFW) for the RECON data sets

RECON Placement

Sample from DBRC Course

41

HALDB Course Objectives• Define HALDB databases and their partitions

• Understand database administrator responsibilities in managing HALDB

• Develop procedures for the support and recovery of HALDB

• Migrate databases to the use of HALDB

• Understand application design issues for HALDB

42

• If partition PB with high key 4000000 needs to be split:– Unload partition PB• HD Unload or HPUnload

– Define new partition PE • Sets PINIT flag forPB and PE

– Initialize partitions PB and PE– Reload partitions PE and PB• HD Reload or HP Load

– Partitions PA, PC, and PDare not affected

PA2000000

PB4000000

PC6000000

PD8000000

PA2000000

PE3000000

PC6000000

PD8000000

PB4000000

HD Unload

HD Reload

Define new partition

Splitting a HALDB Partition

Sample from HALDB Course

43

Data Sharing Course Objectives• Define IMS subsystems, IRLMs, and Coupling Facilities to use block level data sharing• Design recovery procedures for failures in the block level data sharing environment• Monitor the performance of a block level data sharing environment• Explain IRLM and Coupling Facility roles in supporting block level data sharing

44

1Subsystem A: GHU ACME (Holds DB record lock on 100 (RBA of Root or RAP)

2Subsystem A: REPL ACME (Holds block lock on Block2)

3Subsystem B: GU ZENITH (Holds DB record lock on 121)

4 Subsystem B: REPL ZENITH (Must wait till block lock on Block2 is released)

Database Records Database on DASD

100

ACME NAIL

121

ZENITH SCREW

Block1

Block2

Block3

100 121

ACME ZENITH

NAIL SCREW

Sample from Data Sharing Course• Full Function Locking Example

45

Summary• IMS is a database manager

– Supports hierarchical databases– Manages highly available and large database– May be accessed from IMS TM, CICS, WAS, DB2, distributed, any platform

• IMS is a transaction manager– Multi-address space architecture– Processes messages– Schedules and manages programs– May be accessed via VTAM, TCP/IP, MQ, …– May be accessed from any platform– Accesses IMS and DB2 databases– This is another ‘world’ to learn about

46

Reference Information

• More information– IMS home page

• www.ibm.com/ims– IMS Primer SG24-5352

• Available at www.ibm.com/redbooks– An Introduction to IMS

• Available on Amazon.com, textbookx.com, ibmpressbooks.com, etc.

47

Technology

IMS Database Primer