View
231
Download
0
Category
Preview:
Citation preview
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
1/24
npr.org
Migration From Oracle to MySQL
An NPR Case Study
By Joanne Garlow
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
2/24
Overview Background
Database Architecture
SQL Differences
Concurrency Issues
Useful MySQL Tools
Encoding Gotchas
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
3/24
Background NPR (National Public Radio)
Leading producer and distributor of radio programming
All Things Considered, Morning Edition, Fresh Air, Wait, Wait,
Dont Tell Me, etc.
Broadcasted on over 800 local radio stations nationwide
NPR Digital Media
Website (NPR.org) with audio content from radio programs
Web-Only content including blogs, slideshows, editorial columns About 250 produced podcasts, with over 600 in directory
Mobile apps and sites
Syndication
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
4/24
High-Level System Architecture
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
5/24
Limitations of the Oracle Architecture Reached capacity of single system to support our
load
Replication outside our budget
Databases crashes were becoming frequent
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
6/24
Database Architecture Goals Redundancy
Scalability
Load balancing
Separation of concerns
Better security
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
7/24
High-Level System Architecture
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
8/24
Updated by our Content Management System
Transaction Oriented
Resource Contention
Highly Normalized
Isolation from main website
Read-only by our webservers
Horizontally scalable
Database Architecture
Main
InnoDB
AMG
MyISAMPUBLIC
InnoDB
STATIONS
InnoDB
Main
RO slave
Main
RO slave
ContentMgmt System
Web Servers
Scripts Backup
RO slave
Read and updated only by our website
Low resource contention
Small tables or log tables Short Transactions
Updated by a nightly script
Read-only by our ContentManagement System
Need fast full text queries
(replacing Oracle Text)
Large tables
Updated by a quarterly script
Read-only from our website Some log type information written
Low resource contention
No transactions
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
9/24
Issues When Converting SQL MySQL is case sensitive
Oracle outer join syntax (+) -> OUTER JOIN clause
Oracle returns a zero to indicate zero rows updated
MySQL returns TRUE (1) to indicate it successfully
updated 0 rows
MySQL sorts null to the top, Oracle sorts null to the
bottom
Use order by colName desc for sorting asc with nulls at
bottom
M SQL has Limit clause YAY!
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
10/24
Replacing Oracle Sequences Initialize a table with a single row:
CREATE TABLE our_seq (
id INT NOT NULL
);
INSERT INTO our_seq (id) VALUES (120000000);
Do the following to get the next number in the sequence:
UPDATE our_seq SET id=LAST_INSERT_ID(id+1);
SELECT LAST_INSERT_ID();
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
11/24
Replacing Oracle Sequences For updating many rows at once, get the total number of unique IDs you need first:
SELECT @totalRows := COUNT(*) FROM...
Then update npr_seq by that many rows:UPDATE npr_seq SET id=LAST_INSERT_ID(id+@totalRows);
and store that ID into another variable:SELECT @lastSeqId := LAST_INSERT_ID();
Then use the whole rownum workaround described above to get a unique value foreach row:INSERT INTO my_table (my_primary_id . . . ) SELECT@lastSeqId - (@rownum:=@rownum+1), . . . FROM (SELECT@rownum:=-1) r, . . .
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
12/24
Converting Functions NVL() -> IFNULL() or COALESCE() DECODE() -> CASE() or IF() Concatenating strings || -> CONCAT()
test || null returns test in Oracle
CONCAT(test,null) returns null in MySQL LTRIM and RTRIM -> TRIM() INSTR() works differently.
Use LOCATE() for Oracles INSTR() with occurrences = 1. SUBSTRING_INDEX() and REVERSE() might also work.
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
13/24
Converting Dates sysdate -> now()
Adding or subtracting
In Oracle 1 subtracts a day
In MySQL - 1 subtracts a milisecond must use
interval
TRUNC() -> DATE() TO_DATE and TO_CHAR -> STR_TO_DATE and
DATE_FORMAT
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
14/24
Update Differences You can't update a table that is used in the WHERE
clause for the update (usually in an "EXISTS" or asubselect) in mysql.UPDATE tableA SET tableA.col1 = NULLWHERE tableA.col2 IN
(SELECT tableA.col2FROM tableA A2, tableBWHERE tableB.col3 = A2.col3 AND
tableB.col4 = 123456);
You can join tables in an update like this (Mucheasier!):UPDATE tableAINNER JOIN tableB ON tableB.col3 = tableA.col3SET tableA.col1 = NULLWHERE tableB.col4 = 123456;
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
15/24
RANK() and DENSE_RANK() We really found no good MySQL equivalent for these
functions
We used GROUP_CONCAT() with an ORDER BY
and GROUP BY to get a list in a single column over a
window of data
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
16/24
Collation You can set collation at the server, database, table or
column level.
Changing the collation at a higher level (say on the
database) wont change the collation for preexisting
tables or column.
Backups will use the original collation unless you
specify all the way down to column level.
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
17/24
Concurrency Issues In our first round of concurrency testing, our system
ground to a halt!
Deadlocks
Slow Queries
MySQL configuration
sync_binlog = 1 // sync to disk, slow but safe
innodb_flush_log_at_trx_commit = 1 // write each
commit
transaction_isolation = READ-COMMITTED
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
18/24
Useful MySQL Tools MySQL Enterprise Monitor
http://www.mysql.com/products/enterprise/
MySQL GUI Tools Bundle:
http://dev.mysql.com/downloads/gui-tools/5.0.html
MySQL Query Browser similar to Oracles SQL
Developer
MySQL Administrator
http://www.mysql.com/products/enterprise/http://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
19/24
Innotop and innoDB Status innotop
http://code.google.com/p/innotop
Helped us identify deadlocks and slow queries (dont
forget the slow query log!)
In mysql, use
show engine innodb status\G;
Useful for contention and locking issues
http://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotop7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
20/24
Query Profiling Try the Query Profiler with Explain Plan when
debugging slow queries
http://dev.mysql.com/tech-resources/articles/using-new-
query-profiler.html
http://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.html7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
21/24
Concurrency Solution Tuning our SQL and our server configuration helped
Turns out that the RAID card we were using had no
write cache at all. Fixing that allowed us to go live.
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
22/24
Encoding Gotchas Switched from ISO-8859-1 to UTF-8
Migration Tool
Issues with characters that actually were not ISO-8859-
1 in our Oracle database
Lack of documentation for the LUA script produced by
the migration GUI
Update encoding end to end
JSPs, scripts (Perl), PHP, tomcat (Java)
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
23/24
Continuing Issues Bugs with innodb locking specific records (as
opposed to gaps before records)
Uncommitted but timed out transactions
Use innotop or show engine innodb status\G; and
look for threads waiting for a lock but no locks blocking
them
Requires MySQL reboot
7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation
24/24
Questions?
Joanne Garlow
jgarlow@npr.org
http://www.npr.org/blogs/inside
mailto:jgarlow@npr.orghttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidemailto:jgarlow@npr.orgmailto:jgarlow@npr.orgmailto:jgarlow@npr.orgRecommended