Rapid Upgrades with Pg_Upgrade

Rapid Upgrades With Pg_Upgrade

BRUCE MOMJIAN

January, 2015

Pg_Upgrade allows migration between major releases of Postgreswithout a data dump/reload. This presentation explains howpg_upgrade works.Creative Commons Attribution License http://momjian.us/presentations

1 / 24

PostgreSQL the database…

◮ Open Source Object Relational DBMS since 1996

◮ Distributed under the PostgreSQL License

◮ Similar technical heritage as Oracle, SQL Server & DB2

◮ However, a strong adherence to standards (ANSI-SQL 2008)

◮ Highly extensible and adaptable design

◮ Languages, indexing, data types, etc.◮ E.g. PostGIS, JSONB, SQL/MED

◮ Extensive use throughout the world for applications andorganizations of all types

◮ Bundled into Red Hat Enterprise Linux, Ubuntu, CentOSand Amazon Linux

Rapid Upgrades With Pg_Upgrade 2 / 24

PostgreSQL the community…

◮ Independent community led by a Core Team of six

◮ Large, active and vibrant community

◮ www.postgresql.org

◮ Downloads, Mailing lists, Documentation

◮ Sponsors sampler:

◮ Google, Red Hat, VMWare, Skype, Salesforce, HP andEnterpriseDB

◮ http://www.postgresql.org/community/


www.postgresql.org

http://www.postgresql.org/community/

EnterpriseDB the company…

◮ The worldwide leader of Postgres based products andservices founded in 2004

◮ Customers include 50 of the Fortune 500 and 98 of theForbes Global 2000

◮ Enterprise offerings:

◮ PostgreSQL Support, Services and Training◮ Postgres Plus Advanced Server with Oracle Compatibility◮ Tools for Monitoring, Replication, HA, Backup & Recovery

Community

◮ Citizenship

◮ Contributor of key features: Materialized Views, JSON, &more

◮ Nine community members on staff


EnterpriseDB the company…


Traditional Postgres Upgrade Options

◮ pg_dump (logical dump)/restore

◮ Slony


Why Upgrading Postgres Is Complex

◮ New features often require system table changes

◮ However, the internal data format rarely changes


Why Pg_Upgrade

◮ Very fast upgrades

◮ Optionally no additional disk space

pg_upgrade installs new system tables while using data files fromthe previous Postgres version.


How It Works: Initial Setup

1

2

3 6

5

4 7

8

9

pg_class

User Tables and Indexes

15 21 27

262014

13 19 25

241812

17 2311

10 16 22

System Tables and Indexes

clog

Old Cluster

1

2

3 6

5

4 7

8

9

pg_class



clog

New Cluster


Decouple New Clog Via Freezing

1

2

3 6

5

4 7

8

9

pg_class


15 21 27

262014

13 19 25

241812

17 2311

10 16 22


clog

Old Cluster

1

2

3 6

4 7

8

9

pg_class



clog

New Cluster

5Freeze

X X


Transfer Clog and XID

1

2

3 6

5

4 7

8

9

pg_class


15 21 27

262014

13 19 25

241812

17 2311

10 16 22


clog

Old Cluster

1

2

3 6

5

4 7

8

9

pg_class



clog

New Cluster

controldata controldataxid


Get Schema Dump

1

2

3 6

5

4 7

8

9

pg_class


15 21 27

262014

13 19 25

241812

17 2311

10 16 22


clog

Old Cluster

1

2

3 6

5

4 7

8

9

pg_class



clog

New Cluster

pg_dumpall − −schema


Restore Schema In New Cluster

1

2

3 6

5

4 7

8

9

pg_class


15 21 27

262014

13 19 25

241812

17 2311

10 16 22


clog

Old Cluster

1

2

3 6

5

4 7

8

9

pg_class


26


clog

New Cluster

pg_dumpall − −schema

17 23

24

25

27

18

10

11

12

13

14

15

16

19

20

21

22


Copy User Heap/Index Files

1

2

3 6

5

4 7

8

9

pg_class


15 21 27

262014

13 19 25

241812

17 2311

10 16 22


clog

Old Cluster

1

2

3 6

5

4 7

8

9

pg_class


26


clog

New Cluster

1610

11

13

12

14

15

19

20

21

22

23

24

25

27

18

17


Complete

1

2

3 6

5

4 7

8

9

pg_class


15 21 27

262014

13 19 25

241812

17 2311

10 16 22


clog

Old Cluster

1

2

3 6

5

4 7

8

9

pg_class


26


clog

New Cluster

24

25

27

10

11

12

13

14

15

16

17

18

19

20

21

22

23


How It Works: In Detail

◮ Check for cluster compatability

◮ locale◮ encoding

◮ Use pg_dumpall to dump old cluster schema (no data)

◮ Freeze all new cluster rows (remove reference to clog entries)

◮ New cluster uses old xid counter value (see freeze above)

◮ Set system table frozen xids to match the current xid

◮ Create new users/databases

◮ Collect cluster information

◮ Install support functions that call internal backend functions

◮ Create schema in new cluster

◮ Copy or link files from old cluster to new cluster

◮ Warn about any remaining issues, like REINDEXrequirements


Sample Run: Performing Consistency Checks

Performing Consistency Checks-----------------------------Checking current, bin, and data directories okChecking cluster versions okChecking database user is a superuser okChecking for prepared transactions okChecking for reg* system OID user data types okChecking for invalid indexes from concurrent index builds okChecking for contrib/isn with bigint-passing mismatch okCreating catalog dump okChecking for presence of required libraries okChecking database user is a superuser ok

Checking for prepared transactions ok

If pg_upgrade fails after this point, you must re-initdb thenew cluster before continuing.


Sample Run: Performing MigrationPerforming Upgrade------------------Analyzing all rows in the new cluster okFreezing all rows on the new cluster okDeleting files from new pg_clog okCopying old pg_clog to new server okSetting next transaction ID for new cluster okResetting WAL archives okSetting frozenxid counters in new cluster okCreating databases in the new cluster okAdding support functions to new cluster okRestoring database schema to new cluster okRemoving support functions from new cluster ok

Adding ".old" suffix to old global/pg_control ok

If you want to start the old cluster, you will need to removethe ".old" suffix from /u/pgsql.old/data/global/pg_control.old.Because "link" mode was used, the old cluster cannot be safely

started once the new cluster has been started.

Linking user relation filesok

Setting next OID for new cluster okCreating script to analyze new cluster okCreating script to delete old cluster ok


Sample Run: Completion

Upgrade Complete----------------Optimizer statistics are not transferred by pg_upgrade so,once you start the new server, consider running:

analyze_new_cluster.sh

Running this script will delete the old cluster’s data files:delete_old_cluster.sh


Possible DataFormat Changes

Change Conversion Method

clog none

heap page header, including bitmask convert to new page format on read

tuple header, including bitmask convert to new page format on read

data value format create old data type in new cluster

index page format reindex, or recreate index methods

TOAST page format convert to new page format on read


Speed Comparison

Migration Method Minutesdump/restore 300.0dump with parallel restore 180.0pg_upgrade in copy mode 44.0pg_upgrade in link mode 0.7

Database size: 150GB, 850 tables

The last duration is 44 seconds.

Timings courtesy of

Stefan Kaltenbrunner

(mastermind on IRC)


Release History

◮ 9.0 focused on stability

◮ 9.1 focused on performance for databases with manyrelations

◮ 9.2 focused on improved debugging and reliability fornon-standard configurations

◮ 9.3 focused on performance via parallelism and reducedfsync activity

◮ 9.4 dramatically reduced memory usage


Additional Resources…

◮ Postgres Downloads:

◮ www.enterprisedb.com/downloads

◮ Product and Services information:

◮ [email protected]


www.enterprisedb.com/downloads

[email protected]

Conclusion

http://momjian.us/presentationsRapid Upgrades With Pg_Upgrade 24 / 24

Technology

Rapid Upgrades with Pg_Upgrade