Increase the Flexibility of Increase the Flexibility of MySQL-based SOA Frameworks MySQL-based SOA Frameworks with a Data Access Layerwith a Data Access Layer

Mike FrostProduct ManagerDataDirect Technologies

DataDirect Technologies OverviewDataDirect Technologies Overview

• #1 independent supplier of data access technologies

• Any application, any platform, any data source

• Standards leader

• Leading the market in addressing changing data access requirements

• A rich history, strong backing:

• 20 years in business• Formerly Intersolv, Merant, NEON• Independent operating company of Progress

Software (Nasdaq: PRGS)

• MySQL partner since 2007

MySQLOracleDB2SQL ServerSybaseInformixTeradataXMLAdabasIMSVSAMCICSIAMIDMSEDI

ODBCJDBCADO.NETXQueryWeb Services

AgendaAgenda

• Where are we today?• SOA• Data access

• Data access problems in SOA environments

• Solving the problems

• What does the future hold?

AgendaAgenda

• Where are we today?• SOA• Data access

• Data access problems in SOA environments

• Solving the problems

• What does the future hold?

The current SOA landscapeThe current SOA landscape

• Many corporate IT organizations are evaluating, planning, or implementing SOA environments• Common framework for development of new applications• Leverages standards such as XML, SOAP, etc.

• A collection of business goals drives this decision• Control IT costs• Increase overall business nimbleness and flexibility• Develop and deploy new applications faster• Reuse business logic common to multiple applications• Optimize use of hardware / software investments• Build bridges between heterogeneous environments

The SOA visionThe SOA vision

1101010

0101010

1010100

0101001

0101010

1000100

1111000

0001111

0010101

01

1101010

0101010

1010100

0101001

0101010

1000100

1111000

0001111

0010101

01

1101010010101

0101010001010

0101010101000

1001111000000

1110100011101

0101101010011

1010101000101

0101000110101

0011010101110

1001101010010

1011101001001

11001001101

1101010

0101010

1010100

0101001

0101010

1000100

1111000

0001111

0010101

01

11010100101

01010101000

10100101010

10100010011

11000000111

1001010101

11010100

10101010

10100010

10010101

01010001

00111100

00001111

00101010

1

1101010010

1010101010

0010100101

0101010001

0011110000

0011110010

1010

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

The current data access landscapeThe current data access landscape

• Defined interfaces• OCI, ct-lib, etc.• ODBC• JDBC• ADO.NET• OLE DB

• Interfaces have evolved – ground breaking changes are not functional capabilities

• All of these interfaces do roughly do the same thing• Provide APIs to connect to data sources, issue queries, return

data, etc.• Reuse legacy business knowledge

Data access in traditional application architecturesData access in traditional application architectures

MySQL

Data access in the real worldData access in the real world

OracleMySQL MS SQLServer

DB2 /z/OS

business logic

AgendaAgenda

• Where are we today?• SOA• Data access

• Data access problems in SOA environments

• Best practices for data access in SOA environments

• What does the future hold?

Data access in a typical SOA architectureData access in a typical SOA architecture

MySQL

What problems does this causeWhat problems does this cause for SOA environments? for SOA environments?

• Performance / scalability requirements underachieved• “Best in class”/reusable services initially designed for 50 users

might be deployed in a SOA environment with 500 users• Lack of understanding of the mechanics of data access

• Deployment slowdowns• A small deployment headache quickly becomes a huge snafu as

the number of services increases• Services typically built in a DBMS-centric fashion, limiting

reusability

• Higher system TCO• Reducing heterogeneity is time-consuming and difficult• Consolidation works only if identical or better performance can

be achieved

Why does this happen?Why does this happen?

• Starting with what is easiest / most comfortable• What works for MySQL may not be right for other data sources (and

vice-versa)

• Data access is usually an afterthought or checklist item• “SOA experts” provide SOA guidelines – not data architects

• “SOA-enablement” = expose applications as web services

…over the traditional data access model• Tightly coupled• Complex State Machine• Mostly synchronous• Connection based• Well defined API• Relational model driven

• SELECT then Fetch model

• SOA presents a very different approach….• Loosely coupled• Stateless• Synchronous /

Asynchronous• Message based• XML data interchange• “RPC” model

AgendaAgenda

• Where are we today?• SOA• Data access

• Data access problems in SOA environments

• Solving the problems

• What does the future hold?

How can these problems be solved?How can these problems be solved?

Best Practices for data access in a SOA environment:

1. Decouple business logic and data access logic

2. Optimize the use of heterogeneous environments

3. Addressing persistent data access performance problems

4. Streamline data access with data integration techniques

1. Decouple data access logic from business logic1. Decouple data access logic from business logic

• Can be done today with any standards-based data access• Use ODBC, JDBC, ADO.NET, or any other API• Multiple technologies exist to facilitate this architecture today

• Leverage and focus skills of existing developers• Allows application developers to focus on application

development• Leaves data access in the hands of the data access experts• Applications built faster and better

• Selection of the right database middleware is essential for success• Build all applications with a strong foundation in data access• “A chain is only as strong as its weakest link”

Consider changing this approach…Consider changing this approach…

Data Access layer

ShippingApplication

InvoicingApplication

ContractsApplication

MySQL

……to this approach: a Data Access Layerto this approach: a Data Access Layer

MySQL

Data Access Layer

ODBC, JDBC, ADO.NET, etc.

ShippingApplication

InvoicingApplication

ContractsApplication

Key to building a successful Data Access LayerKey to building a successful Data Access Layer

High quality database middleware

• Reliability

• Consistency across data sources

• High performance & scalability

• True conformance to standards

• Robust feature and functionality support

What about SOA environments?What about SOA environments?

MySQL

From Data Access Layer to Data Access ServiceFrom Data Access Layer to Data Access Service

• It is possible to “SOA-enable” the data access layer• In a SOA environment, the data access layer becomes a data

access service• Services can now access data presented as a web service

• Avoid accessing data directly inside every service• Forces business logic experts to become data access experts• Results in extremely complicated deployment scenarios• Reduces scalability and performance

• One of the reasons to adopt SOA is to share “best in class” services across a set of applications• Accessing data in an optimized manner IS a “best in class”

service

• A data access service is “best in class”, scalable, easy to deploy, and reusable

2. Optimize the use of 2. Optimize the use of heterogeneous environmentsheterogeneous environments

• Tendency to reduce complexity by standardizing on as few environments/data sources as we can imagine• Typically on those environments/data sources we know best or

offer the lowest TCO

• Consolidation is often seen as a way to cut IT costs• But many IT flexibility problems start because of unforeseen

narrowness in the consolidated architecture• These problems waste many of the cost savings of consolidation

• Optimizing can help minimize consolidation pain• Reliability of new, consolidated architecture• Difficulty achieving identical or better performance• Inconsistency of implementation due to slight differences in data

source capabilities• Higher, rather than lower, system TCO

Don’t reduce complexity Don’t reduce complexity just for the sake of doing sojust for the sake of doing so

Oracle

Data Access Layer

OCI

ShippingApplication

InvoicingApplication

ContractsApplication

SOA environments can handle SOA environments can handle heterogeneity if data access is decoupledheterogeneity if data access is decoupled

OracleMySQL MS SQL Server

DB2 /z/OS

business logic

Optimization vs. Consolidation?Optimization vs. Consolidation?

• First question – does the existing system work?

• If so, first strategy is to assume the SOA principle of ‘reuse’ should apply – in particular if the system is complex• Migrating and consolidating a complex working system onto an

alternative system will cost orders of magnitude more than you think it will

• Most “unsolvable” problems will be encountered only after you are in the details or are at deployment time

• If consolidation seems to be the answer, assume that achieving identical performance is the first goal and benchmark* extensively before agreeing to consolidate

3. Addressing persistent data access 3. Addressing persistent data access performance problemsperformance problems

The SOA Data Access Service Lifecycle:

• Design Database Schema (tables, partitions, indexes, etc)

• Design architecture of the application framework

• Design individual applications/services

• Build/Test/Deploy individual applications/services

• Load Test

• Panic: performance is not acceptable – how do I fix it?

Data access code must be written well Data access code must be written well to avoid performance bottlenecks to avoid performance bottlenecks

• Standards-based APIs provide numerous ways to do the exact same thing

• Documentation and coding guidelines that address performance are difficult to find• Even if found, they may be too general to be helpful

• Bad coding techniques account for many of the data access performance problems people encounter

• Education is the key to avoiding some of the most common, yet avoidable blunders• Simple techniques can improve performance by > 100%

Example: Calling a Stored ProcedureExample: Calling a Stored Procedure

• Examine every API call … example below performs well over 100% faster

“{call getMikesBeerInventory (‘Hellbender Barleywine’,‘Pisgah Brewing’,2008)}”

or

“{call getMikesBeerInventory (?, ?, ?)}”

Understand the lifecycle Understand the lifecycle of solving data access performance problemsof solving data access performance problems

• The “bottleneck” will start in the database• Design an efficient schema and tune

the database

• The “bottleneck” will move to the application or service• Understand the performance issues

surrounding database middleware and then design/tune performance-aware data access code

• The “bottleneck” will shift to your data access middleware• Choose database middleware that

can be tuned with runtime tuning options

• Adjust those options to match your application and environment

How database middleware works conceptuallyHow database middleware works conceptually

What’s really going on?What’s really going on?

Database middleware can make a differenceDatabase middleware can make a differencein terms of throughputin terms of throughput

Database middleware can determine how Database middleware can determine how well an application or service will scalewell an application or service will scale

4. Streamline data access with data 4. Streamline data access with data integration techniquesintegration techniques

• Most SOA environments begin with simple services doing simple things• Looking up an order using an order id

• As the environment expands and the comfort level increases, the complexity of services increases• Retrieve an incoming EDI order, validate client information in

internal system and retrieve customer number, place order in another internal system using customer number, generate and send receipt back to client via EDI

• Now “accessing” several data sources in a single service, an EDI feed, a data source that stores the client information, a data source where the order is placed, and then another EDI request to send a response

A typical SOA-based data integration problemA typical SOA-based data integration problem

The user submits a request for a report

about their stock holdings

The web server needs to fetch user’s personal data, stock holdings and live stock data to compile a report to send

back to the user A public service offers live (delayed) stock

prices

Some obvious pieces …Some obvious pieces …

MySQL

Solve complex data integration Solve complex data integration problems with XQueryproblems with XQuery

MySQL

XQJ + XQueryImplementation

Advanced services require multiple Advanced services require multiple operations to access dataoperations to access data

• The overhead involved to sequentially process data inside services is extensive• At some point, data is usually marshaled to/from an XML

format• Data must be transformed from one data model (XML for

example) to another (relational for example)• Code must be written to do these transformations of data• Multiple invocations of data access services are needed to

fulfill the request

• Thinking differently, it is possible to design to integrate the data using XQuery instead of sequentially access the different data stores

Why integrate data with XQuery?Why integrate data with XQuery?

• Simplifies coding

• Encapsulates business logic and data integration logic into a single step

• Improves performance• No marshalling/unmarshalling of data• Optimized access to data• Optimized joining/integration of data

AgendaAgenda

• Where are we today?• SOA• Data access

• Data access problems in SOA environments

• Solving the problems

• What does the future hold?

What does the future hold for data What does the future hold for data access in SOA environments?access in SOA environments?

• Emerging standards will transform data access services into robust data application services (DAS)

• Organizations will begin to regard implementation of a DAS as key to their IT infrastructure

• Database vendors and third party ISVs will begin to compete at offering a complete DAS solution• Some are already offer solutions that are just a layer on top of

existing database middleware components (some low-quality)• Since DAS represents a new abstraction layer, it makes more

sense to optimize the connectivity between the DAS and back-end data sources

• High quality DAS implementations will allow organizations to maximize their SOA investment

What sorts of standards?What sorts of standards?

• SDO (Service Data Objects) • Unified and consistent data access to heterogeneous data

sources• Simplified programming model for application programmers• Enables SOA-enabled applications and services to work

consistently across heterogeneous data sources

• SDO is an emerging standard that can form the foundation for a DAS solution• Supported by IBM, BEA, Oracle, SAP, Sybase and...• Version 2 of the specification has been released with

implementations in C++, Java, Perl, .NET, and…• Large standards group run through OASIS

• Other standards and specifications

The SOA vision…The SOA vision…

1101010

0101010

1010100

0101001

0101010

1000100

1111000

0001111

0010101

01

1101010

0101010

1010100

0101001

0101010

1000100

1111000

0001111

0010101

01

1101010010101

0101010001010

0101010101000

1001111000000

1110100011101

0101101010011

1010101000101

0101000110101

0011010101110

1001101010010

1011101001001

11001001101

1101010

0101010

1010100

0101001

0101010

1000100

1111000

0001111

0010101

01

11010100101

01010101000

10100101010

10100010011

11000000111

1001010101

11010100

10101010

10100010

10010101

01010001

00111100

00001111

00101010

1

1101010010

1010101010

0010100101

0101010001

0011110000

0011110010

1010

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

101010001010110111010101000001010110101111011000101000010011001010111010100101001010111100100101010110010100011010101000101011011101010100000101011010111101100010100001001100101011

combined with DAScombined with DAS

What should you do?What should you do?

• Make data access a first class citizen of any SOA environment

• Don’t wait for “magic beans” to build your data access environment the right way - apply the Best Practices explained here today1. Decouple business logic and data access logic

2. Optimize the use of heterogeneous environments

3. Addressing persistent data access performance problems

4. Streamline data access with data integration techniques

• Keep an eye on emerging standards that can form the foundation of a complete data services solution

Questions?Questions?