DB-16: In Any Case, the Devil’s in the DataServer Details

DB-16: In Any Case, the Devil’s in the DataServer Details

David MoloneyPrincipal,

Progress OpenEdge

That which we persist in doing becomes easier, not that the task itself has become easier, but that our ability to perform it has improved.

Ralph Waldo Emerson (1803 - 1882)

http://www.quotationspage.com/quote/2095.html




http://www.quotationspage.com/myquotations.php?add=2095

http://www.quotationspage.com/quote/2095.html#email

© 2007 Progress Software Corporation2 DB-16: In Any Case, the Devil’s in the DataServer Details

Agenda

A Story Terminology & Technology Sync-up The Devils in the Details

• Case Studies–ABL & DataServer–ABL & the RDBMS–DataServer & RDBMS–DataServer & API Access Drivers

DataServer Performance Summary & Questions


A Story

Mary and the three DataServers

Once Upon A Time …


Mary meets Compiler

For Each

Display Cust

ABL


Client meets Server

OpenEdge™

DatabaseOpenEdge™

Database


Where’s the love ? Where’s my pie ?

OracleOracleMicrosoft

SQL Server

Microsoft

SQL Server

IBM

DB2

IBM

DB2


For Each What ?

For Each

Display Cust

ABL

Habla“For Each” ?


Hmmm….

For Each

Display Cust

ABL


I’ve got Bingo !

For Each

Display Cust

ABL


OE DB OE DB

Trusted friends

For Each

Display Cust

ABL

ODBCDriver

Manager

Driver

Data

Source

Informix SybaseDB2

DriverDriver

Data

Source

Data

Source

OCIAPI

SQL*Net

Oracle

Data

Source


Mary makes a house call

For Each

Display Cust

ABL

Hey Man …It’s all about the

“For Each”

OE DB OE DB


Bravo !


The End


That was then …

For Each

Display Cust

ABL

OE DB OE DB


Terminology & Technology Sync-up

OpenEdge

DatabaseOpenEdge

Database

Oracle

SQL ServerOracle

SQL Server

MS SQL

ServerMS SQL

Server



OpenEdge

DatabaseOpenEdge

Database

Oracle

SQL ServerOracle

SQL Server

DataServer Broker

SQL

--

--

MS SQL

ServerMS SQL

Server



OpenEdge

DatabaseOpenEdge

Database

INT CHAR

INT CHAR

1 J

2 Alsop

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

DataServer Broker

SQL

--

--

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates

Item

-no

Item

_no



OpenEdge

DatabaseOpenEdge

Database

INT CHAR

INT CHAR

1 J

2 Alsop

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

DataServer Broker

SQL

--

--

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates

Item

-no

Item

_no



OpenEdge

DatabaseOpenEdge

Database

INT CHAR

INT CHAR

1 J

2 Alsop

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

DataServer Broker

SQL

--

--

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates

Item

-no

Item

_no


Sync-up: 4-Tier Perspective

OpenEdge

DatabaseOpenEdge

Database

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

DataServer Broker

SQL

--

--

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates INT CHAR

INT CHAR

1 J

2 Alsop


Sync-up: Self-Service Client 3-Tier Perspective

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates

OpenEdge

DatabaseOpenEdge

Database

INT CHAR

INT CHAR

1 J

2 Alsop


OpenEdge Client

DataServer

Sync-up: Application-centric

SELECT UPPERCASE (name) FROM Customer

FOR EACH Customer

Oracle

CBA

SELECT TO UPPER(name) FROM Customer

OpenEdge

DB2ZYX

321

SELECT TO UPPER(name) FROM Customer SELECT

UPPER(name) FROM Customer

---

---

---

Application Centric - Just another Oracle OCI or ODBC Client

SQL

Server

SQL

Server

OpenEdgeOpenEdgeSchema

PULL

Schema

PUSH


Sync-up: Client-Server Perspective

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates

OpenEdge

DatabaseOpenEdge

Database

INT CHAR

INT CHAR

1 J

2 Alsop

DataServer Broker

SQL

--

--


Sync-up: Component Perspective

OpenEdge

DatabaseOpenEdge

Database

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates

INT CHAR

INT CHAR

1 J

2 Alsop


Sync-up: DataServer Component Layers

Progress ABL

Dataserver LayerDataserver Layer

Dataserver Client

Dataserver Server

MS SQL Srvr

ODBC

Oracle

OCI


OpenEdge Reference ArchitectureFor Service Oriented Business Applications

Design for longevity, flexibility, and competitive advantage

PresentationPresentation

Business ComponentsBusiness Components

Data AccessData Access

Data SourcesData Sources

Co

mm

on

Infrastru

cture

Co

mm

on

Infrastru

cture

Enterprise ServicesEnterprise Services


Case Study: Components


Appreciate intricacies amongst components Demonstrate the somewhat “arbitrary”

diversity DataServer issues Create opportunities to discuss important

aspects of the DataServer architecture. Promote the interest of a problem-solver

audience

Case Study: Goals


DataServer deference given to the database• On Security

• On Transaction Control

• On Lock Management

• On Cursor Consistency

Case Study: Prerequisite & Disclaimer


I ran a pro-to-<dataserver> migration and compiled some code against the schema holder it built. Everything compiled just fine.

Then I built a separate schema holder, pulled definitions from the very same SQL database, and connected to it. But, against this schema holder, the compiled code fails.

Why would it matter how we create the schema holder as long as the structure of the SQL database remains exactly the same !

The Devils in the Details: Case Study #1

+ +


DataServer Migration

OpenEdge

DatabaseOpenEdge

Database

INT CHAR

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

DataServer Broker

SQL

--

--

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates Item-no

Item_no

OpenEdge

DatabaseOpenEdge

Database

INT CHAR

1 J

2 Alsop

Item_no

Item-no


Why don’t my schema holder “pull” definitions match the OpenEdge Database they were derived from ?

The script that built the SQL Database came directly from OpenEdge Database !

The Devils in the Details: Case Study #1 (cont.)


DataServer Pushed & Pulled Independently

OpenEdge

DatabaseOpenEdge

Database

INT CHAR

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

DataServer Broker

SQL

--

--

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates Item_no

Item_no

OpenEdge

DatabaseOpenEdge

Database

INT CHAR

1 J

2 Alsop

Item_no

Item-no


Pull & Adjust Schema

OpenEdge

DatabaseOpenEdge

Database

INT CHAR

Oracle

SQL ServerOracle

SQL Server

INT VARCHAR2

5 L

6 Ellison

DataServer Broker

SQL

--

--

MS SQL

ServerMS SQL

Server

INT VARCHAR

3 B

4 Gates Item-no

Item_no

OpenEdge

DatabaseOpenEdge

Database

INT CHAR

1 J

2 Alsop

Item_no

Item-no



We deployed a new schema holder to our customers and their production applications started crashing with no warning. We didn’t make any code changes but if we do, the application still crashes, only with a different error.

What would cause the DataServer’s inconsistent behavior and how can we get an indication of the problem ?

+ ++



I compiled my application against an OpenEdge Database. Then, I did the same against my Schema Holder. Why did the DataServer r-code grow larger than my OpenEdge database r-code ?

++



During a migration, I’m getting truncation errors loading data into my SQL Server database . I elected the “load data” option. The schema copied fine but migrating data from my OpenEdge database to a DataServer schema holder fails. Why would the migration create definitions too small for my data ?

+ +



I’m migrating an OpenEdge database to a DataServer schema. When I elect to load data, I get NULL-constraint and unique-constraint violations.

+ +


Record Write & Availability

DEFINE BUFFER xcust FOR cust.CREATE cust.cust-num = 111.FIND xcust

WHERE xcust.cust-num = 111.

The Devils in the Details: Case Study #6 (Intro.)

+ +


Record Write & Availability

DEFINE BUFFER xcust FOR cust.CREATE cust.cust-num = 111.VALIDATE cust. /* or RELEASE cust. */

FIND xcust WHERE xcust.cust-num = 111.

The Devils in the Details: Case Study #6 (Intro.)

+ +



I’m getting a STOP condition trying to assign a column value after a FIND statement. The application exits after the ASSIGN. How can I be sure this isn’t corrupting my database ?

DO TRANSACTION:FIND FIRST cust.ASSIGN name = FILL(“a”,35) NO-ERROR.

END.

+ +



+ +

But, I’m still getting the error condition

DO TRANSACTION:FIND FIRST cust.ASSIGN name = FILL(“a”,35)

NO-ERROR.VALIDATE cust.

END.



+ +

DO TRANSACTION:FIND FIRST cust.ASSIGN name = FILL(“a”,35) /* NO-ERROR */.VALIDATE cust NO-ERROR.IF error-status:error THEN DO: <some error processing> UNDO, LEAVE.END.

END.



+ +

But, I’m still getting the error condition

DO TRANSACTION ON ERROR UNDO,LEAVE:FIND FIRST cust.ASSIGN name = FILL(“a”,35) NO-ERROR.VALIDATE cust.

CATCH Progress.Lang.AppError ae: MESSAGE "Inside AppError Catch".

IF ae:GetMessage(1) <> ? THEN MESSAGE ae:GetMessage(1) ae:GetMessageNum(1).

ELSE MESSAGE "ReturnError" ae:returnvalue view-as alert-box.

DELETE OBJECT ae. END CATCH.

CATCH Progress.Lang.ProError pe: MESSAGE "Inside ProError Catch". REPEAT i = 1 TO pe:NumMessages: PUT UNFORMATTED " Error Number: " pe:GetMessageNum(i)

FORMAT ">>>>>9" SKIP "Message: " pe:GetMessage(i) FORMAT "x(73)" SKIP.

END. DELETE OBJECT pe. END CATCH.

END.



+ +

Warning:

DO TRANSACTION:FIND FIRST cust EXCLUSIVE-LOCK.ASSIGN name = FILL(“a”,35) NO-ERROR.VALIDATE cust.

CONTENTION EXPOSURE !

END.



I have two clients running simultaneously.Both create records and lock records exclusively on the same table. Why are they getting “table is use by another user” errors ?

DO TRANSACTION:FIND FIRST cust EXCLUSIVE-LOCK.IF AVAILABLE cust THEN ASSIGN name =

“Bob”. REPEAT:FIND NEXT cust EXCLUSIVE-LOCK.IF NOT AVAILABLE cust THEN LEAVE.Cnt = Cnt + 1.ASSIGN name = “Bob” + STRING(cnt).

END.END.

+ +



DO TRANSACTION: FIND FIRST cust EXCLUSIVE-LOCK. IF AVAILABLE cust THEN ASSIGN name = “Bob”. REPEAT: DO TRANSACTION: FIND NEXT cust EXCLUSIVE-LOCK. IF NOT AVAILABLE cust THEN LEAVE. Cnt = Cnt + 1. ASSIGN name = “Bob” +

STRING(cnt). END. END. END.END.

+ +



DO TRANSACTION:FIND FIRST cust EXCLUSIVE-LOCK.IF AVAILABLE cust THEN ASSIGN name = “Bob”.

END.REPEAT:

DO TRANSACTION: FIND NEXT cust EXCLUSIVE-LOCK. IF NOT AVAILABLE cust THEN LEAVE. Cnt = Cnt + 1. ASSIGN name = “Bob” +

STRING(cnt). END.

END.END.

+ +



Another example regarding cursor consistency

DEFINE VARIABLE num AS INT INITIAL 103.DO TRANSACTION: FIND cust WHERE cust = num

EXCLUSIVE-LOCK. ASSIGN name = “Bob”.END.

FIND cust WHERE cust-num = num.DISPLAY name.

+ +



Parenthesis around names in the WHERE clause of our join is [slow/fast], generates multiple queries and sends index hints with the queries.

• For each customer, each order WHERE (order.custnum) = (cust.custnum):

Oracle documentation says joins won’t pass index hints on a server join. If we just remove the parenthesis:

• For each customer, each order WHERE order.custnum = cust.custnum:

We only get one query, [better/worse] performance and no hints.

+


The Devils in the Details: Case Study #9 - Part I

In our Oracle DataServer application, one of our users got a locked record error:

“<table> in use by <user> on <tty>. Wait or press CTRL-C to exit. (121)“

Pressing CTRL-C does not actually do anything. Our client script isn’t trapping CTRL-C, so what is ?

NOTE: stty on Unix shows Ctrl-C is mapped

+ +


The Devils in the Details: Case Study #9 - Part II

Why did I get a “System Error 126” on Windows connecting through the ODBC DataServer ?

"Specified driver could not be loaded due to system error 126"

+ +


Our Oracle DataServer application is running fine. Why does our application all of a sudden have a problem with an invalid cursor and then exit ?


++


We migrated our legacy Oracle DataServer application from an earlier Progress implementation, that used “shadow columns”, to OpenEdge where it uses Function-based indexes. Why are we now seeing performance problems that appear to be related to Function Based Indexes ?


++


DataServer Rule:

USE-INDEX and BY clauses affect the SQL ORDER BY clause. This guarantees the order of the results but does NOT guarantee the index selections of an execution plan.


++


We’ve converted our legacy DB2/400 database to run against the ODBC DataServer. Some of our existing tables don’t have indexes so we describe indexes in the schema holder that satisfy the DataServer’s ROWID requirements. But when we display results for these tables, from a query like the following, why do some records show up twice in our result set ?

FOR EACH <table>:

DISPLAY <table>.

END.


++ +


FOR EACH <table> SHARE-LOCK:

DISPLAY <table>.

END.


++ +


FOR EACH <table> EXCLUSIVE-LOCK:

DISPLAY <table>.

END.

___________________________________

SELECT <columns> FROM <table> WHERE

<key-components> = <key-value>

-- or --

<RECID> = <recid-value>


++ +


DataServer Rule:

Uniqueness is critical to proper functioning of DataServer cursors. Random access reads and all transactional activity is dependent on being able to locate an individual row.


++ +


Why can’t data access through a DataServer be a fast as OpenEdge native access ?

The Devils in the Details: Performance Study


FIND FIRST Customer NO-LOCK NO-ERROR.IF AVAILABLE Customer THEN Cnt = 1.REPEAT: FIND NEXT Customer NO-ERROR. IF NOT AVAILABLE (Customer) THEN LEAVE. Cnt = Cnt + 1.END.

OPEN QUERY q FOR EACH Customer NO-LOCK.REPEAT: GET NEXT q. IF NOT AVAILABLE Customer THEN LEAVE. Cnt = Cnt + 1.END.CLOSE QUERY q.



Replace FIND statements with FORs & QUERYs• FIND FIRST customer -> FOR FIRST customer: END. • FIND LAST order -> bBuffer:FIND-LAST().

Be explicit about lock type

Field Lists• FOR EACH customer FIELDS(cust-num name) NO-LOCK:• Make sure to include all fields you reference. This compiles:

FIND FIRST customer FIELDS(cust-num) WHERE CAN-FIND(FIRST order WHERE order.st = cust.st)

Write JOIN-BY-SQLDB queries• FOR EACH customer, EACH order:

Index Reposition• OPEN QUERY q1 FOR EACH order INDEXED-REPOSITION• REPOSITION q1 to recid myid



Query structure• Don’t get fancy

– WHERE (city + STRING(“,”) + state) = …– Complex joins may require a client join or client

selection

• WHERE col1 = INTEGER(‘123’) is better than WHERE STRING(col1) = “123”

• Try to make your BY clause and the expected INDEX selection compatible to avoid reordering: WHERE col1 = <> AND col2 = <> By col1, BY col2



Query Tuning• CACHE-SIZE(<size>)• REVERSE-FROM• (no-index-hint hint “run fast”)• NO-BIND-WHERE• NO-UNIQUE-ORDER-ADDED

Database Optimizations• Indexes, covering idx’s (clustered idx (MSS) &

“included” columns– MSS2005)• Updated Statistics• Index rebuild/reorg & Fill Factors, etc.



Remember you can always conditionalize:• Compile Time

&GLOBAL-DEFINE DB-TYPE ORACLE&IF DEFINED ({&ORACLE}) &THEN …

• Run TimeIF DBTYPE(dbname) = “PROGRESS” THEN RUN OpenEdge-optimized-code

• ELSE RUN DataServer-optimized-code



Use stored procedures:

• RUN STORED-PROC send-sql-statement hdl1 = PROC-HANDLE (“select name, cust_num FROM customer”).FOR EACH proc-text-buffer WHERE PROC-HANDLE = hdl1: DISPLAY proc-text.END.CLOSE STORED-PROC send-sql-statement rtn-stat = PROC-STATUS WHERE PROC-HANDLE = hdl1.

• DEF VAR ttHndl AS HANDLEDEF TEMP-TABLE tt1 ….ttHndl = TEMP-TABLE tt1:HANDLE.RUN STORED-PROC send-sql-statement (“select name, cust_num FROM customer”) LOAD-RESULT-INTO ttHndl.

• DEF VAR ttHndl AS HANDLE EXTENT 2.DEF TEMP-TABLE tt1 … ttHndl[1] = TEMP-TABLE tt1:HANDLE.DEF TEMP-TABLE tt2 … ttHndl[2] = TEMP-TABLE tt2:HANDLE.RUN STORED-PROC send-sql-statement (“select name FROM customer; select order-num FROM order”) LOAD-RESULT-INTO ttHndl.



In Summary

Know thy Client Component (ABL & DataServer)

Know thy access component (drivers & configuration)

Know thy database component (data manager & engine)


For More Information, go to…

PSDN

• “DataServer Best Practices”• http://www.psdn.com/library/servlet/KbServlet/

download/1320-102-620/ds_best_practices.pdf

Exchange 07:• DB-21 Data Management and Platforms

Roadmap and Info Exchange

(Wednesday 1:30-3:00 Room #200)


Questions?


Thank you for your time!


Documents

DB-16: In Any Case, the Devil’s in the DataServer Details