Upload
command-prompt-inc
View
634
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Bruce MomjianPg_Migrator allows data to be transfered between major Postgres versions without a dump/restore. This talk explains the internal workings of pg_migrator and includes a pg_migrator demonstration
Citation preview
Rapid Upgrades With Pg_Migrator
BRUCE MOMJIAN,ENTERPRISEDB
March, 2010
AbstractPg_Migrator allows migration between major releases of Postgreswithout a data dump/reload. This presentation explains howpg_migrator works.
Creative Commons Attribution License http://momjian.us/presentations
Traditional Postgres Upgrade Options� pg_dump (logical dump)/restore
� Slony
Rapid Upgrades With Pg-Migrator 1
Why Upgrading Postgres Is Complex� New features often require system table changes
� However, the internal data format rarely changes
Rapid Upgrades With Pg-Migrator 2
Why Pg_Migrator� Very fast upgrades
� Optionally no additional disk space
pg_migrator installs new system tables while using data files from theprevious Postgres version.
Rapid Upgrades With Pg-Migrator 3
How It Works: Initial Setup
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
System Tables
clog
New Cluster
Rapid Upgrades With Pg-Migrator 4
Decouple New Clog Via Freezing
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
4 7
8
9
pg_class
User Tables
System Tables
clog
New Cluster
5Freeze
X X
Rapid Upgrades With Pg-Migrator 5
Transfer Clog and XID
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
System Tables
clog
New Cluster
controldata controldataxid
Rapid Upgrades With Pg-Migrator 6
Get Schema Dump
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
System Tables
clog
New Cluster
pg_dumpall - -schema
Rapid Upgrades With Pg-Migrator 7
Reserve TOAST OIDs Using Relfilenodes (pre-9.0)
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
26
18
System Tables
clog
New Cluster
This is necessary because heap references to TOAST tables contain theTOAST oids for easy lookup.Rapid Upgrades With Pg-Migrator 8
Restore Schema In New Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
2640
39
31
System Tables
clog
New Cluster
pg_dumpall - -schema
30
32
33
34
35
36
37
41
42
43
44
45
47
18
Rapid Upgrades With Pg-Migrator 9
Connect TOAST Placeholders To the Proper Relations(pre-9.0)
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
26
39
31
System Tables
clog
New Cluster
30
32
33
34
35
36
37
42
43
44
45
47
18
41
40
X
Rapid Upgrades With Pg-Migrator 10
Copy User Heap/Index Files
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
2640
39
31
System Tables
clog
New Cluster
30
32
33
34
35
36
37
42
43
44
45
47
18
41
Rapid Upgrades With Pg-Migrator 11
Complete
1
2
3 6
5
4 7
8
9
pg_class
User Tables
15 21 27
262014
13 19 25
241812
17 2311
10 16 22
System Tables
clog
Old Cluster
1
2
3 6
5
4 7
8
9
pg_class
User Tables
2640
39
31
System Tables
clog
New Cluster
30
32
33
34
35
36
37
42
43
44
45
47
18
41
Rapid Upgrades With Pg-Migrator 12
How It Works: In Detail� Check for cluster compatability
– locale– encoding– integer datetimes (default changed from 8.3 -> 8.4)
� Use pg_dumpall to dump old cluster schema (no data)
� Freeze all new cluster rows (remove reference to clog entries)
� Rename tablespaces to *_old (pre-9.0)
� New cluster uses old xid counter value (see freeze above)
– Set system table frozen xids to match the current xid
� Create new users/databasesRapid Upgrades With Pg-Migrator 13
� Collect cluster information
� Install support functions that call internal backend functions
� Create placeholder files to reserve relfilenode file names (pre-9.0)
� Create schema in new cluster
� Adjust new cluster to use reserved relfilenode names (pre-9.0)
– Delete placeholder toast relfilenode files– Remove new cluster toast tables– Create new cluster toast table using reserved relfilenode– Assign new toast tables with proper relfilenodes to relations
� Copy or link files from old cluster to new cluster
– Toast tables have the same relfilenodes as in the old cluster
� Warn about any remaining issues, like REINDEX requirements
Rapid Upgrades With Pg-Migrator 14
Sample Run: Performing Consistency Checks
Performing Consistency Checks
-----------------------------
Checking old data directory (/u/pgsql.old/data) ok
Checking new data directory (/u/pgsql/data) ok
Checking for columns with user-defined composite types ok
Checking for columns with user-defined array types ok
Checking for columns with user-defined enum types ok
Checking for /contrib/isn with bigint-passing mismatch ok
Checking for invalid ’name’ user columns ok
Checking for tsquery user columns ok
Creating script to adjust sequences ok
Creating catalog dump ok
Checking for presence of required libraries ok
| If pg_migrator fails after this point, you must
| re-initdb the new cluster before continuing.
| You will also need to remove the ".old" suffix
| from /u/pgsql.old/data/global/pg_control.old and old
| tablespace directory names.
Many of these checks are unnecessary when migrating to Postgres 9.0.Rapid Upgrades With Pg-Migrator 15
Sample Run: Performing Migration
Performing Migration
--------------------
Adding ".old" suffix to tablespace directory names ok
Adding ".old" suffix to old global/pg_control ok
Analyzing all rows in the new cluster ok
Freezing all rows on the new cluster ok
Deleting new commit clogs ok
Copying old commit clogs to new server ok
Setting next transaction id for new cluster ok
Resetting WAL archives ok
Setting frozenxid counters in new cluster ok
Creating databases in the new cluster ok
Creating placeholder relfiles for toast relations
/u/pgsql/data/base/16397/16586
Adding support functions to new cluster ok
Restoring database schema to new cluster ok
Restoring relations to use fixed toast file names
/u/pgsql/data/base/16397/16586
Removing support functions from new cluster ok
Restoring user relation files
/u/pgsql.old/data/base/11511/2613
Setting next oid for new cluster ok
Creating script to delete old cluster ok
Adjusting sequences okRapid Upgrades With Pg-Migrator 16
Sample Run: Completion
Upgrade complete
----------------
| Optimizer statistics and free space information
| are not transferred by pg_migrator so consider
| running:
| vacuumdb --all --analyze
| on the newly-upgraded cluster.
| Running this script will delete the old cluster’s data files:
| /u/postgres/pg_migrator_output/delete_old_cluster.sh
Rapid Upgrades With Pg-Migrator 17
Possible DataFormat Changes
Change Conversion Method
clog none
heap page header, including bitmask convert to new page format on read
tuple header, including bitmask convert to new page format on read
data value format create old data type in new cluster
index page format reindex, or recreate index methods
TOAST page format convert to new page format on read
Rapid Upgrades With Pg-Migrator 18
Migration Timings
Migration Method Minutesdump/restore 300.0dump with parallel restore 180.0pg_migrator in copy mode 44.0pg_migrator in link mode 0.7
Database size: 150GB, 850 tables
The last duration is 44 seconds.
Timings courtesy of
Stefan Kaltenbrunner
(mastermind on IRC)
Rapid Upgrades With Pg-Migrator 19
Conclusion
Rapid Upgrades With Pg-Migrator 20