Migration From Oracle to PostgreSQL

Copyright©2016 NTT corp. All Rights Reserved.

Migration from Oracle to PostgreSQL - The problems and the solutions -

Kazuki Uehara NTT OSS Center March 19, 2016

Copyright(c)2016 NTT Corp. All Rights Reserved.

2 Copyright©2016 NTT corp. All Rights Reserved.

• Self Introduction

• Introduction of Today's topics

• Problems about Database migration.

• OSS Products that support the Database migration

• Conclusion

Agenda


• Kazuki Uehara

• From Japan

• Hobby

• Travelling by bicycle

• Taking pictures

• Work

• technical support

• technical consulting

• functional verification and performance evaluation

• Products

• PostgreSQL

• pgpool-II、Slony-I

Who am I?


• Who we are? • NTT（Nippon Telegraph and Telephone Corporation）

• National flagship carrier in Japan

• What NTT OSS Center is doing？ • Promotes the adoption of OSS by the group companies

• Total support

• support desk, Introduction support, Product maintenance

• R&D

• developing OSS and related tools with the communities

• Deals with about 60 OSS products.

About us

NTT group

subsidiary

about 900 companies

NTT NTT OSS Center


• The world's most advanced OSS DBMS.

• Continued featuere/performance improvement by the community.

We involve in PostgreSQL

600 systems

for the last seven years.

Number of adoptions of PostgreSQL

in the group companies

year


PostgreSQL vs Oracle

PostgreSQL 9.5 Oracle 12c

SQL ○ (ISO SQL 2011) ○ (ISO SQL 2011)

stored procedure ○ (PL/pgsql,Java,perl,...) ○ (PL/sql,Java,...)

trigger ○ ○

Online Backup ○ ○

partitionig ○ ○

Replication ○

(Synchronous/Asynchronous) ○

(Synchronous/Asynchronous)

HA Cluster ○ (pacemaker,pgpool-II,...) ○ (VCS, MSCS,...)

clustered systems with shared disk storage

× ○ (RAC)

License BSD Named User Plus License

/ Processor Lincense

License Fee ○ free of charge × Compensation

• No big difference.

• You can use PostgreSQL at many systems.


• Focus on Database migration technique from Oracle to PostgreSQL.

• What is system migration? • Migration is the process of transferring the system and

data to another environment.

• Three categories of system migration: • rehost ・・・To replace only platform hardware

• rewrite ・・・To replace OS and programing languages

• rebuild ・・・Remake the entire system

• Database migration is needed in 'rewrite' or 'rebuild'.

Today's topics

1. Problems in the Database migration

2. How you can solve them


• Introduction

• Today's topic

• Problem about Database migration

• Why you need Database migration?

• The items to be considered for the Database migration

• Issues on Database migration

• OSS Products that support the DB migration

• Conclusion


• Your system is obsolete

• Support of HW expired.

• Maintenance costs rise due to aging of equipment.

• Your Database is expensive or poorly operating

• You want to cut the license cost

• You want to improve performance of middleware.

• You can downsize HW.

• You want to use new features.

• You found your commercial Database was over spec

• didn't use the proper functionalities of the commercial product.

Why you need Database migration?


The items to be considered for the Database migration

AP Server

1. Logical design of DB

• ER Diagram

2. Physical design of DB

• arrangement of the data

3. Data

• dump, restore

4. Operation procedures

• maintenance tool (backup, batch)

5. SQL used in application

• the dedicated SQL of commercial product.

DB Server

DB

（Oracle → PostgreSQL）

Logical

design

Physical

design

application

SQL a

SQL b

…

data

Client /

Web

DataBase

Administrator

Operationnal

procedures

entity

attribute

table

index

column relationship

…

…

1 2

5 3

4


The items to be considered for the Database migration

AP Server

1. Logical design of DB

• ER Diagram

2. Physical design of DB

• arrangement of the data

3. Data

• dump, restore

4. Operation procedures

• maintenance tool (backup, batch)

5. SQL used in application

• the dedicated SQL of commercial product.

DB Server

DB

（Oracle → PostgreSQL）

Logical

design

Physical

design

application

SQL a

SQL b

…

data

Client /

Web

DataBase

Administrator

Operationnal

procedures

entity

attribute

table

index

column relationship

…

…

1 2

5 3

4


1. It will require significant cost to estimate for the migration of SQL.

• The migration cost go up when we spend too much time on estimation.

• You can provide the rough estimate in order to reduce cost.

2. It will require significant cost to modify incompatible SQL.

• You have to find incompatible SQL from a lot of sources. In addition, you have to consider for each how should you modify.

What is the problems?

The risk of losing profits on migration exists.


• This is sample source for Oracle.

• If you use it on PostgreSQL, what should you modify?

• There are three places in this source that require to modify.

What kind of SQL should you modify?

import java.sql.*;

import java.util.*;

public class test02 {

String sqlString = "DELETE mytbl";

public ResultSet testMethod() throws SQLException {

ResultSet rs = stmt.execute(sqlString);

ResultSet rs2= stmt.executeQuery("SELECT sysdate FROM dual");

return rs;

}

}






import java.sql.*;

import java.util.*;






return rs;

}

}

DELETE statement requires FROM clause in PostgreSQL.






import java.sql.*;

import java.util.*;






return rs;

}

}


PostgreSQL doesn't have sysdate.






import java.sql.*;

import java.util.*;






return rs;

}

}


PostgreSQL doesn't have sysdate.

It doesn't exist DUAL table in PostgreSQL


• Introduction

• Today's topic

• Problem about Database migration

• OSS Products that support the Database migration

• db_syntax_diff

• orafce

• Case study

• Conclusion


• Two OSS products as solutions.

• db_syntax_diff • This tool was made by us as migration supporting tool.

• Extracts incompatible SQL from application's source of Oracle.

• Using this tool, anyone can be easily review source of application.

• We can review in a relatively short time even for large scale systems.

• orafce • This is contrib module for PostgreSQL.

• It is an emulation tool for PostgreSQL to use compatibility functions and operators with Oracle RDBMS.

To solve the problem …


An overall outline of db_syntax_diff

ｓｒｃ

db_syntax_diff

XML Output file

• This file is XML format.

Outline of processing

• to do parsing using original parser.

• draw a comparison between the results of

parsing and the contents of dictionary file.

dictionary

file What is dictionary file? • The list of incompatible 'SQL'. • You can modify this file.

ｓｒｃｓｒｃ

Input files • You can input single file or directory. • C source（ProC）, Java source, JSP source, SQL file


• Install the required package using the yum.

• expand the files got from GitHub.

• set the environment variable.

How to use db_syntax_diff 1/4

# yum install perl perl-XML-SAX.noarch xalan-j2 perl-Parse-Yapp

perl-XML-NamespaceSupport.noarch perl-XML-LibXML.x86_64

$ tar -xvf db_syntax_diff.tar.gz

$ vi ~/.bash_profile

export CLASSPATH=/usr/share/java/xalan-j2.jar:$CLASSPATH

export CLASSPATH=/usr/share/java/xalan-j2-serializer.jar:$CLASSPATH

export PATH=$HOME/db_syntax_diff/src:$PATH

export PERL5LIB=$HOME/db_syntax_diff/src/lib

$ source ~/.bash_profile

※If you use xalan-java 2.7.1 later, you have to set CLASSPATH for serializer.jar.



$ db_syntax_diff.pl --help

db_syntax_diff version 2.0

The SQL analyzer for converting to PostgreSQL.

Usage:db_syntax_diff.pl [-e encodingname][-d definition-file]

[-i inputsourcedir[,suffix1[,suffix2]...] ]

[-o outfile][-f filterword][-m modename]

[-I includedir[,includedir1[,includedir2]...]][-h][-v [loglevel]]

[inputfilename]...

-e encodingname, --encoding=encodingname File encoding. The value which can be

specified is "utf8" and "shiftjis" and "eucjp". [default: eucjp]

-d definition-file, --define=definition-file Definition-file file name.

-i inputsourcedir, --input=inputsourcedir Input-file directry.

-o outfile, --output=outfile Output-file file name. [default: STDOUT]

-f filterword, --filter=filterword Pattern filterword. The value which can be specified is

"oracle8" and "oracle8i". [default: ALL]

-m modename, --mode=modename File type of source file. The value which can be

specified is "c" and "sql" and "cpp" and "java". [default: java]

-I includedir, --Include=includedir Add the directory includedir to the list of directories

to be searched for header files. [default: ./]

-h, --help Print usage and exit.

-v, --verbose Print progress of the practice to STDERR. The value which can be

specified is "1" and "3" and "5" and "7". [default: none]

inputfilename Input-file file name.


• I introduce the result as a simple example.

• target file : sample.java

• Run this command.


$ db_syntax_diff.pl -m java -e utf8 ~/tmp/sample.java -o ~/result.xml

import java.sql.*;

import java.util.*;






return rs;

}

}


• This is a part of result.


<?xml version="1.0" encoding="UTF-8"?>

<REPORT file_number="1" start_time="2016/2/25 19:12:06" finish_time="2016/2/25 19:12:07">

<METADATA>

<PARAMETER>-m java -e utf8 /home/uehara/tmp/sample.java -o /home/uehara/result.xml</PARAMETER>

</METADATA>

<FILE name="/home/uehara/tmp/sample.java" string_item_number="3" report_item_number="4" item_number="7">

<STRING_ITEM line="testMethod:sqlString:7">

<TARGET>DELETE mytbl</TARGET>

</STRING_ITEM> ・・・

<REPORT_ITEM id="SQL-107-008" type="SQL" level="LOW2">

<SOURCE>

<CLASS>test02</CLASS>

<METHOD>testMethod</METHOD>

<LINE>5</LINE>

<COLUMN>7</COLUMN>

<VARIABLE>sqlString</VARIABLE>

</SOURCE>

<STRUCT>!(?:[^¥w¥d_]|¥A)FROM(?:[^¥w¥d_]|¥z)</STRUCT>

<TARGET>DELETE mytbl</TARGET> <MESSAGE>FROMの省略は未サポートです。</MESSAGE>

</REPORT_ITEM>

<REPORT_ITEM id="SQL-119-706" type="SQL" level="LOW1"> ・・・

Which file?

What kind of incompatible SQL?

Simple report is output.

What type? How difficult?

What line/column?

What SQL statement?


dictionary

file

Specifications of the wrapper tool

ｓｒｃ

db_syntax_diff

Output files

• This file is XML format.

• It is aggregated in 4 patterns.

Outline of processing

• This tool operates db_syntax_diff.

• In addition, This tool makes CSV files

from XML file.

ｓｒｃｓｒｃ Input files

• You can input single file or directory.

csv

XML

db_syntax_diff_wrapper

configuration

file


• result.csv

• file path of target file

• Number of line

• Number of columns

• ID (db_syntax_diff)

The results of wrapper tool

• The classification

• Difficulty of migration

• simple report

• target query

• Middle • You can't simple migrate.

• High • migration is very difficult.

• Low1 • It's only necessary to delete or replace.

• Low2 • You can't replace, but migration is easy.

Difficulty


• You can reduce the modification cost of SQL.

• The number of incompatible SQL decrease by using orafce.

• Installation Instructions (Only 3 steps)

1. You can get a RPM file from PostgreSQL.org.

• http://yum.postgresql.org/9.5/redhat/rhel-7-x86_64/repoview/orafce95.html

• Latest version(3.2.1) has been published.

2. Install orafce RPM.

3. After, you just run a query "CREATE EXTENSION orafce".

How to use orafce

# rpm -ivh orafce95-3.2.1-1.rhel7.x86_64.rpm

Updating / installing...

1:orafce95-3.2.1-1.rhel7 ################################# [100%]


A simple example

-bash-4.2$ psql postgres psql (9.5.1) Type "help" for help. postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));

ERROR: type "varchar2" does not exist LINE 1: CREATE TABLE bar(i int,v VARCHAR2(20)); ^ postgres=# postgres=# CREATE EXTENSION orafce; CREATE EXTENSION postgres=# postgres=# CREATE TABLE bar(i int,v VARCHAR2(20)); CREATE TABLE postgres=# postgres=# ¥d bar Table "public.bar" Column | Type | Modifiers --------+--------------+----------- i | integer | v | varchar2(20) | postgres=#


A simple example



PostgreSQL doesn't have

data type 'VARCHAR2' .


A simple example



PostgreSQL doesn't have

data type 'VARCHAR2' .

You can use

data type 'VARCHAR2'

in PostgreSQL.


IT infrastructure Client

SW

Web/AP/DB

Job Management/Backup

Other system

Server Enclosure

Case study 1/3

• Billing system

• A managing system of business contract information

• Migration from commercial product to RHEL / Apache / mod_jk / JBoss EAP / PostgreSQL

intracompany

network

intracompany

network

target number of files Number of lines

SQL/DDL

about 740 files

about 250KL

Java

Pro*C

Scale of Database migration


• We judged that there are not big problem in database migration by using db_syntax_diff.

• We extracted 10000 pieces of incompatible SQL.

• But difficulty of alomost the whole incompatible SQL was Low1 or Low2.

Case study 2/3

# difficulty SQL/DDL Java Pro*C

1 Low1 4,186 1 11

2 Low2 5,361 798 94

3 Middle 20 0 0

4 High 0 0 0

total 9,567 799 105

tool's output • Low1

• It's only necessary to delete or replace.

• Low2 • You can't replace, but

migration is easy.

• Middle • You can't simple migrate.

• High • migration is very difficult.


• We can reduce the number of incompatible SQL by orafce.

• In this case, it can reduce to 7000.

• Based on our own experience, PostgreSQL can use 73% of Oracle's SQL by orafce

Case study 3/3


• The migration of SQL is the most difficult process in Database migration.

1. It will require significant cost to estimate for the migration of SQL.

2. It will require significant cost to modify incompatible SQL.

• These issues can be solved by db_syntax_diff and orafce.

• Try Database migration, don’t hesitate.

Conclusion

• If you have interest, please help the development of

db_syntax_diff.

• To start with translate manual into English.


Thank you!

Technology

Migration From Oracle to PostgreSQL