Informix Data Storage andInformix Data Storage andData Data InIn’’ss and Outand Out’’ss

Informix User Forum 2006

Washington, DC (VA) December 8-9, 2006

James EdmistonJames EdmistonQuest Information Systems, Inc.Quest Information Systems, Inc.

james@questinfosys.comjames@questinfosys.com

Commercial Break…

• Join the IIUG– It’s free– www.iiug.org

• IDUG (IDIG) 2007– May 6-10, 2007, San Jose, CA

• Forum 2008– Stay tuned…

We’ve come a long way…

OutlinePart 1• Background and Inspiration• Something is just not right…• A tale of 2 SANsPart 2• Informix Data Storage• Creating, Managing and Monitoring• Observing GuidelinesPart 3• Informix Data Movement tools and utilities• Getting Data “In” and “Out”

Purpose

• What I found at a client site– Lack of Critical and temp dbspace separation– Little or no use of fragmentation (big tables)– Lots of chunks in 1 data dbspace– Inability to control specific disk placements– Use of RAID 5 (dev and test)

• RAID 5– www.baarf.com– BAARF - Battle Against Any RAID Five, Four, Free (3)

• Offer practical advice and recommendations

> onstat -d

Dbspacesaddress number flags fchunk nchunks flags owner namec0000000a774fe60 1 0x1 1 1 N informix rootdbsc0000000a9c55e98 2 0x1 2 1 N informix physdbsc0000000a9c6f9b8 3 0x1 3 3 N informix logdbsc0000000a9c6fb38 4 0x2001 4 2 N T informix tmpdbs1c0000000a9c6fcb8 5 0x2001 5 2 N T informix tmpdbs2c0000000a9c6fe38 6 0x2001 6 2 N T informix tmpdbs3c0000000a9c70028 7 0x2001 7 2 N T informix tmpdbs4...

Chunksaddress chunk/dbs offset size free bpages flags pathnamec0000000a7751028 1 1 0 50000 43876 PO-- /dev/infx/ck_01_1c0000000a9c56028 2 2 50000 125000 447 PO-- /dev/infx/ck_01_1c0000000a9c561c0 3 3 175000 849000 447 PO-- /dev/infx/ck_01_1c0000000a9c56358 4 4 0 256000 255381 PO-- /dev/infx/ck_01_2c0000000a9c564f0 5 5 256000 256000 255373 PO-- /dev/infx/ck_01_2c0000000a9c56688 6 6 512000 256000 255373 PO-- /dev/infx/ck_01_2c0000000a9c56820 7 7 768000 256000 255381 PO-- /dev/infx/ck_01_2

Example

Example 2 (same installation…)Dbspacesaddress number flags fchunk nchunks flags owner name...c0000000a9c701a8 8 0x1 8 107 N informix pstable1c0000000a9c70328 9 0x1 13 45 N informix psindex1c0000000a9c704a8 10 0x1 157 12 N informix pstable2

Chunksaddress chunk/dbs offset size free bpages flags pathnamec0000000a9c569b8 8 8 0 1024000 17898 PO-- /dev/infx/ck1c0000000a9c56b50 9 8 0 1024000 6482 PO-- /dev/infx/ck2c0000000a9c56ce8 10 8 0 1024000 16490 PO-- /dev/infx/ck3c0000000a9c56e80 11 8 0 1024000 28314 PO-- /dev/infx/ck4c0000000a9c57028 12 8 0 1024000 372 PO-- /dev/infx/ck5c0000000a9c57b50 19 8 0 1024000 372 PO-- /dev/infx/ck6c0000000a9c57ce8 20 8 0 1024000 67036 PO-- /dev/infx/ck7c0000000a9c57e80 21 8 0 1024000 29389 PO-- /dev/infx/ck8c0000000a9c58028 22 8 0 1024000 19971 PO-- /dev/infx/ck9c0000000a9c581c0 23 8 0 1024000 70158 PO-- /dev/infx/ck10...

SAN Example 2…

Part 2Part 2Informix Data StorageInformix Data Storage

Informix User Forum 2006

Washington, DC (VA) December 8-9, 2006

Physical • Chunk

– Largest unit of physical disk dedicated to database server data storage

– Raw disk devices or regular file system files (cooked)

– Raw disk chunks perform better due to less OS overhead

– Maximum size prior to IDS 9.4 was 2GB– Maximum size in IDS 9.4 and higher is 4TB with

use of large chunks (onmode –BC) – v10 eliminates 2GB max default and need to use

onmode -BC

Physical (cont)

• Pages– Unit of disk storage read from and written to– 2k bytes most Unix, 4k on AIX & Windows– Configurable Page size introduced in IDS 10

• Multiple of platform page size up to 16k

Physical (cont)

• Extents– Physical unit of storage used to allocate

storage space for a database table– Consists of a collection of contiguous pages – Default initial Extent size and

Next Extent size = 16k– Important to manage extent sizes and

number of extents– Manage with EXTENT SIZE, NEXT SIZE in

create and alter table

Logical• Dbspaces

– Logical unit of storage– Can have between 1 and 32,767 chunks– Contains databases, tables, indexes, logical

logs and physical log, temporary space– Important to Separate:

• Root• Log spaces• Temp • Data• Indexes

Dbspaces• Root Dbspace

– Initial dbspace containing reserved pages and internal tables

– Default dbspace location for database creation

– Configured in ONCONFIG:# Root Dbspace Configuration

ROOTNAME rootdbs # Root dbspace nameROOTPATH /dev//informix/chunk1 # Path for device containing root dbspaceROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)ROOTSIZE 2048000 # Size of root dbspace (Kbytes)

# Disk Mirroring Configuration Parameters

MIRROR 1 # Mirroring flag (Yes = 1, No = 0)MIRRORPATH /dev/informix2/chunk1 # Path for device containing mirrored rootMIRROROFFSET 0 # Offset into mirrored device (Kbytes)

Dbspaces• Logical Logs

– Move out of root dbspace to a dbspace on disk not shared by active tables or the physical log

• Physical Log– Move out of root dbspace…– onparams – move physical log, add logical logs

• Dbspaces for Data, Indexes• Temp Dbspaces

– If more than 1, place on different disks– List in DBSPACETEMP ONCONFIG parameter

DBSPACETEMP tmp1dbs,tmp2dbs,tmp3dbs,tmp4dbs,tmp5dbs,tmp6dbs,tmp7dbs,tmp8dbs

Temporary dbspaces (cont)

Experience…• Unexpected behavior

– 9.4 on Linux– Complex join in view– Used rootdbs for sort

• -565 Cannot read sorted rows.• -131 ISAM error: no free disk space.

– Mistakenly thought it was temp dbspaces, but it was rootdbs

Creating, Managing and Monitoring

• Creating and Managing– onspaces - command line utility– On-Monitor– GUI utilities

• Server Studio• ISA• IDSAdmin (Coming soon…)

• Monitoring– onstat– SMI queries– GUI utilities

Creating and Managing• onspacesonspaces -c -d dbspce1 -p /dev/raw_dev1 -o 0 -s 2097150 -m /dev/raw_dev2 0

• onmonitorDBSPACES: Create BLOBSpace Mirror Drop Info Add_chunk datasKip Status ExitCreate a new dbspace.

-----------------------------On-Line------- Press CTRL-W for Help. --------

• ISA, Server Studio

– Create, add, drop

Press ESC to build a new DBspace.Press Interrupt to cancel the option and return to the Dbspaces menu.Press F2 or CTRL-F for field level help.

CREATE DBSPACE

Dbspace Name [ ] Mirror [ ] Temp [ ]

Dbspace Page Size [ 2] Kbytes

PRIMARY CHUNK INFORMATION:

Full Pathname [ ]

Offset [ 0] Kbytes Size [ 0] Kbytes

MIRROR CHUNK INFORMATION:

Full Pathname [ ]

Offset [ 0] Kbytes

On-Monitor

Dbspacesaddress number flags fchunk nchunks flags owner name0EACB7D8 1 0x20001 1 1 N informix rootdbs0EBDBE90 2 0x20001 2 1 N informix ol_james0EACBAE0 3 0x28001 3 1 N S informix sbspace0EACBC30 4 0x20001 4 1 N informix dbs_data14 active, 2047 maximum

Chunksaddress chunk/dbs offset size free bpages flags pathname0EACB928 1 1 0 12800 5309 PO-- \IFMX\rootdbs0EBDBA28 2 2 0 12800 12697 PO-- \IFMX\ol_james0EBDBBA0 3 3 0 12800 11879 11879 POS- \IFMX\sbspace

Metadata 868 554 8680EBDBD18 4 4 0 500000 431989 PO-- \IFMX\dbs_data14 active, 2047 maximum

Expanded chunk capacity mode: disabled

onstat –d (spaces and chunks showing size and free space)

Also:onstat –D (spaces and chunks, shows reads and writes)onstat -g iof (shows chunk reads and writes)

Monitoring

oncheck –pe = Database and table storage in a chunk> oncheck –pe ol_james

DBspace Usage Report: ol_james Owner: informix Created: 01/26/2006

Chunk Pathname Size Used Free2 \IFMX\ol_james 12800 792 12008

Description Offset Size------------------------------------------------------------- -------- --------RESERVED PAGES 0 2CHUNK FREELIST PAGE 2 1ol_james:'informix'.TBLSpace 3 50stores_demo:'informix'.systables 53 8stores_demo:'informix'.syscolumns 61 16stores_demo:'informix'.sysindices 77 24stores_demo:'informix'.systabauth 101 8stores_demo:'informix'.syscolauth 109 8

.

.

.stores_demo:'jgedmiston'.zip_ix 704 4stores_demo:'jgedmiston'.catalog 708 8stores_demo:'jgedmiston'.msgs 724 8stores_demo:'jgedmiston'.idxmsgs_enus 732 5stores_demo:'jgedmiston'.idxmsgs_frfr 737 5FREE 742 103ol_james:'informix'.TBLSpace 845 50FREE 895 11905

Total Used: 792Total Free: 12008

ExtentsC:\demo>oncheck -pt stores_demo:customer

TBLspace Report for stores_demo:jgedmiston.customer

Physical Address 2:52Creation date 08/20/2006 15:18:58TBLspace Flags 801 Page Locking

TBLspace use 4 bit bit-mapsMaximum row size 134Number of special columns 0Number of keys 0Number of extents 1Current serial value 129First extent size 8Next extent size 8Number of pages allocated 8Number of pages used 2Number of data pages 1Number of rows 28Partition partnum 2097201Partition lockid 2097201

dbspace pages_size pages_used pages_free percent_free

dbs_data 500000 68011 431989 86.40rootdbs 12800 7499 5301 41.41sbspace 12800 12246 554 4.33ol_james 12800 103 12697 99.20

------------------------------------------------------------------------------- Module: @(#)dbsfree.sql 1.4 Date: 97/07/18-- Author: Lester B. Knutsen Email: lester@advancedatatools.com-- Advanced DataTools Corporation-- Discription: Displays free space in all dbspaces like Unix "df -k " command------------------------------------------------------------------------------- Note:On some versions (e.g 7.3) need to use the 7.2 version of-- this script to correctly display the truncated dbspace name.-- If the dbspace name is blank - use the 72 version.

database sysmaster;

select name[1,8] dbspace, -- name truncated to fit on one linesum(chksize) Pages_size, -- sum of all chuncks size pagessum(chksize) - sum(nfree) Pages_used,sum(nfree) Pages_free, -- sum of all chunks free pagesround ((sum(nfree)) / (sum(chksize)) * 100, 2) percent_free

from sysdbspaces d, syschunks cwhere d.dbsnum = c.dbsnumgroup by 1order by name[6,8], name[1,5];--order by 1;

SMI (sysmaster) queries

Server Studio

ISA

ISA

Guidelines• rootdbs - Keep it clean

– Don’t allow any user databases to be created in root– Allow extra space for system table growth

• Critical dbspaces– Move physical log and logical logs to their own dbspace

(preferably on different disks)• Temp dbspaces

– Must specify in ONCONFIG DBSPACETEMP or rootdbs will be used…???

– Used for creation of temporary tables and sorting• ORDER BY, GROUP BY, UNIQUE, etc.

– Use multiple temp dbspaces and place on separate disks

Guidelines (cont)• Minimize number of chunks per dbspace• Use table fragmentation for large, high use

tables• Reduce number of table extents• *Take advantage of large chunks to minimize the

number of chunks/dbspaces on larger disks• *Utilize the configurable pages size for tables

with large row size• Test

Part3: Getting Data In and OutPart3: Getting Data In and Out

Informix User Forum 2006

Washington, DC (VA) December 8-9, 2006

Comparison of data migration tools…

YesNoYs, or from a non Informix database

YesYesMove Data Between OS?

EasiestMore difficultMost difficultModerateModerateEase of Use

Logging optional

Logging must be turned off

Logging optional

Logging optional

Logging optional

Logging Status

TextBinaryTextTextTextType of File

Disk onlyDisk or tapeDisk, tape, or pipe

Disk onlyDisk or tapeLocation of Data

Can modifyCannot modifyCan modifyCan modifyCan modifyDatabase Schema

Any data in the specified format usually produced by UNLOAD

Must be produced by onunload

Any ASCII or COBOL data, User can create custom capabilities

Any data in the format specified by the input file

Usually produced by dbexport

Source of Data

ModerateFastFastSlowModeratePerformance

Partial or complete table

Table or database

Partial or complete table

Partial or complete table

Database onlyGranularity of data

UNLOAD/ LOAD

onunload/onload

HPLdbloaddbexport/dbimport

unload/load• Use of SQL• Limit/filter with “where” clause• Extraction of whatever data that can be queried

in an SQL statement• Delimited data

– Default is a “|” (pipe, vertical bar) if DBDELIMITER has not been set to something different

– Can use other delimiters if specified, except “/”, “0-9”, “a-z” and “A-Z”

• ASCII as output (or input)

dbschemaUSAGE:

dbschema [-q] [-t tabname] [-s user] [-p user] [-r rolename] [-f procname][-hd tabname] -d dbname [-w passwd] [-seq sequence] [-l [num]][-u [ia] udtname [all]] [-it [Type]] [-ss [-si]] [filename]

-q Suppress the db version from header

-t table name or "all" for all tables

-s synonyms created by user nameor "all" for all users

-p permissions granted to user nameor "all" for all users

-r create and grant of the roleor "all" for all roles :Not a valid option for SE

-f SPL routine nameor "all" for all SPL routines

-hd Histograms of the distribution for columns ofof a specified table, a specific table column,or "all" for all tables.

-d database name

-w database password

(cont)-seq generate sequence specific syntax

-u Prints the definitions of user-defined data types

-ui Prints the definitions of user-defined data types,including type inheritance

-ua Prints the definitions of user-defined data types,including all functions and casts defined over a type

-u all Directs dbschema to include all the tablesin the display of distributions

-it Type of isolation can be DR, CR, CS or RR

-l set lock mode to wait [number] optional

-ss generate server specific syntax

-si excludes the generation of index storage clauses fornon-fragmented tables

filename is the nameof file that the SQLscript goes in.

dbexport/dbimport• dbexport unloads a database into ASCII files for

later import into another database• Creates a schema file of the database –

database.sql• Creates unload of each table to single file - table.unl• dbimport creates and populates a database from

dbexport• The schema file can be edited if necessary• Use same environment variables • If the database is a logging database, import with

no logging, then use ondblog or ontape to change logging status

dbexport/dbimport (cont)

• Use –X option to export binary data in char fields• Can export to tape• On disk, creates a directory – database.exp

– Default location is from where the command is issued– Able to specify different location with -o parameter

• Quiz…

Usage:dbexport <database> [-X] [-c] [-q] [-d] [-ss][{ -o <dir> | -t <tapedev> -b <blksz> -s <tapesz> [-f <sql-command-file>] }]NOTE: arguments to dbexport are order independent.

onpladm (onpload)usage : onpladm <command>

create job <jobname> {options}create map <mapname> {options}create project <projectname> {options}create object {options}delete job <jobname> {options}delete map <mapname> {options}delete format <formatname> {options}delete query <queryname> {options}delete filter <filtername> {options}delete device <devicename> {options}delete machine <machinename> {options}delete project <projectname> {options}modify object {options}describe job <jobname> {options}describe map <mapname> {options} describe format <formatname> {options}describe query <queryname> {options} describe filter <filtername> {options}describe device <devicename> {options}describe machine <machinename> {options}describe project <projectname> {options}

(cont)

list job {options}list map {options}list format {options}list query {options}list filter {options}list device {options}list machine {options}list project {options}list defaults {options}run job {options}run project {options}configure defaults {options}

Summary

• Informix Data Storage– Follow recommended guidelines– Separate and spread I/O– Monitor and manage

• Moving Data In and Out– Choose utility to meet your need– Flexible and easy to use

References

• Informix Administrator’s Guide– Chapter on Data Storage– Chapter on Managing Disk Space

• Informix Migration Guide– Data Migration Tools– Data Movement Utilities

Informix Data Storage andInformix Data Storage andData Data InIn’’ss and Outand Out’’ss

Informix User Forum 2006

Washington, DC (VA) December 8-9, 2006

James EdmistonJames EdmistonQuest Information Systems, Inc.Quest Information Systems, Inc.

james@questinfosys.comjames@questinfosys.com