Upload
postgresopen
View
1.191
Download
9
Embed Size (px)
Citation preview
Consistent State www.consistentstate.com [email protected]
PostgreSQL Backup and Recovery Methods
Kevin Kempter Chief Data Architect
Wednesday, September 18, 13
-or-How NOT to be this guy
2
Wednesday, September 18, 13
Session Topics✓Overview✓Backup Options✓Restore Options✓Point in Time Recovery
3
Wednesday, September 18, 13
Overview• Multiple backup methods• Multiple backup file formats• Many recovery choices / options if a
pg_restore compatible format is used• PITR• PITR is the base construct for WAL
shipping (Warm Standby)4
Wednesday, September 18, 13
PostgreSQL Backup Options✓pg_dump✓pg_dumpall
5
Wednesday, September 18, 13
pg_dump✓ Utility to dump a snapshot of a single
database
✓ Multiple output options
✓ Non-blocking
✓ Creates a “Consistent” backup - even if the database is in use
6
Wednesday, September 18, 13
pg_dump• Syntax
• pg_dump [connection-options] [dump-options] [dbname]
• Connection Options• -h, --host=HOSTNAME • -p, --port=PORT • -U, --username=NAME• -w, --no-password• -W, --password (should happen automatically)• --role=ROLENAME (do SET ROLE before dump)
• Environment Variables• PGDATABASE• PGHOST• PGOPTIONS• PGPORT• PGUSER
7
Wednesday, September 18, 13
pg_dump - Common Options• -s [--schema-only]
• Dump schema (DDL) only, no data
• -a [--data-only]• Dump data only - no DDL
• -c [--clean]• Generate drop statements for all created objects
• -C [--create]• Generate a “CREATE DATABASE” statement
• -n schema [--schema=schema]• Only dump the specified schema, wildcard characters are allowed, also multiple -n’s are
allowed
8
Wednesday, September 18, 13
pg_dump - Common Options (continued)• -N schema [--exclude-schema=schema]
• Exclude specified schema
• -F format [--format=format]• Output format• p (Plain) plain sql file #default• c (Custom) custom binary format• t (tar) tar format• d (directory) Creates a directory with one file per table/blob, plus a TOC file in a binary
format that pg_restore can read
• -o [--oids]• Dump table OID’s
• -O [--no-owner]• Do not generate ownership commands
9
Wednesday, September 18, 13
pg_dump - Common Options (continued)
• -t table [--table=table]• Only dump the specified table, wildcard characters are allowed, also multiple -t’s are
allowed, over-rides -n and -N options
• -x [--no-privileges] [--no-acl]• Do not dump access privileges
• --inserts • Generate INSERT statements
• --disable-triggers• disable triggers during restore (for a data only restore) when doing a data-only dump
• --lock-wait-timeout=timeout• fail if shared lock on an object cannot be acquired within timeout time
10
Wednesday, September 18, 13
pg_dump - Common Options (continued)
• -Z 0..9 [--compress=0..9]• Specify compression level for custom format or plain format (not supported for tar
format)
• -v [--verbose]
• -V [--version]
11
Wednesday, September 18, 13
pg_dump - Examples
$ pg_dump -C --inserts prod1_db > prod1_db.sqlCreates a dump of insert statements including a create database statement $ pg_dump --data-only --table=customer -Fc prod1_db > prod1_db.cust.fc.dmpDump the customer table data (data only) in a custom format from the prod1_db database
$ pg_dump -s prod1_db > prod1_db.ddl_only.sqlCreates a DDL only dump of the prod1_db database
$ pg_dump --schema=gold -Ft prod1_db > prod1_db.gold_schema.dmpCreates a dump of the gold schema in the prod1_db database in a tar format
12
Wednesday, September 18, 13
pg_dump Summary
13
Wednesday, September 18, 13
pg_dumpall✓ Utility to dump a snapshot of a full
database cluster (or cluster-wide constructs)
✓ Dumps only to plain sql format
✓ Non-blocking
✓ Creates a “Consistent” backup - even if the database is in use
14
Wednesday, September 18, 13
pg_dumpall• Syntax
• pg_dump [connection-options] [dump-options]
• Connection Options• -h, --host=HOSTNAME • -p, --port=PORT • -U, --username=NAME• -w, --no-password• -W, --password (should happen automatically)• --role=ROLENAME (do SET ROLE before dump)
• Environment Variables• PGDATABASE• PGHOST• PGOPTIONS• PGPORT• PGUSER
15
Wednesday, September 18, 13
pg_dumpall - Common Options• -s [--schema-only]
• Dump schema (DDL) only, no data
• -a [--data-only]• Dump data only - no DDL
• -c [--clean]• Generate drop statements for all created objects
• -o [--oids]• Dump table OID’s
• -O [--no-owner]• Do not generate ownership commands
16
Wednesday, September 18, 13
pg_dumpall - Common Options (continued)• -r [--roles-only]
• Dump only CREATE ROLE data
• -t [--tablespaces-only]• Dump only CREATE TABLESPACE data
• -g [--globals-only]• Dump Global Structures (Roles and Tablespaces)
• --no-tablespaces• Do NOT dump CREATE TABLESPACE Data
• --inserts • Generate INSERT statements
17
Wednesday, September 18, 13
pg_dumpall - Common Options (continued)
• --disable-triggers• disable triggers during restore (for a data only restore) when doing a data-only dump
• --lock-wait-timeout=timeout• fail if shared lock on an object cannot be acquired within timeout time
• -v [--verbose]
• -V [--version]
18
Wednesday, September 18, 13
pg_dumpall - Examples
$ pg_dumpall -g > prod1_db_cluster.global_structures.sqlCreates a cluster dump containing only the cluster global structures $ pg_dumpall --tablespaces-only > prod1_db_cluster.tablespaces.sqlDump the cluster tablespaces
$ pg_dumpall --no-tablespaces > prod1_db_cluster.no_tablespaces.sqlCreates a dump of the cluster without any tablespace references
$ pg_dumpall -a > prod1_db_cluster.data_only.sqlCreates a dump of the cluster - data only
19
Wednesday, September 18, 13
pg_dumpall summary
20
Wednesday, September 18, 13
PostgreSQL Restore Options✓psql✓pg_restore
21
Wednesday, September 18, 13
Restoring with psql
$ psql -ef prod1_db.sql > load_db.log 2>&1
$ pg_dump prod1_db | psql -h qa_server
$ pg_dumpall -g | psql -h dev_server -p 5433 > load_dev.log 2>&1
$ pg_dump prod1_db | psql -e test_db > load_test_db.log 2>&1
22
Wednesday, September 18, 13
pg_restore✓ Utility to restore a data file created by
pg_dump
• Works only with non plain text file formats
23
Wednesday, September 18, 13
pg_restore• Syntax
• pg_restore [connection-option...] [option...] [filename]
• Connection Options• -h, --host=HOSTNAME • -p, --port=PORT • -U, --username=NAME• -w, --no-password• -W, --password (should happen automatically)• --role=ROLENAME (do SET ROLE before dump)
• Environment Variables• PGDATABASE• PGHOST• PGOPTIONS• PGPORT• PGUSER
24
Wednesday, September 18, 13
pg_restore - Common Options• -d dbname [--dbname=dbname]
• -C [--create]• Create the specified database before restore
• -c [--clean]• Generate drop statements for all created objects
• -s [--schema-only]• Dump schema (DDL) only, no data
• -a [--data-only]• Dump data only - no DDL
25
Wednesday, September 18, 13
pg_restore - Common Options (continued)• -n namespace [--schema=schema]
• Restore only objects in the specified schema
• -O [--no-owner]• Do not restore ownership of objects
• -I index [--index=index]• Restore specified index only
• -P function-name(argtype [, ...]) [ --function=function-name(argtype [, ...]) ]• Restore specified function only
• -T trigger [--trigger=trigger]• Restore specified trigger only
26
Wednesday, September 18, 13
pg_restore - Common Options (continued)
• -t table [--table=table]• Restore specified table only
• --no-tablespaces• Do not restore any TABLESPACES
• -F format [--format=format]• Output format• c (Custom) custom binary format• t (tar) tar format• d (directory)
27
Wednesday, September 18, 13
pg_restore - Common Options (continued)
• -t table [--table=table]• Restore specified table only
• --no-tablespaces• Do not restore any TABLESPACES
• -j number-of-jobs [--jobs=number-of-jobs]• Use parallel jobs to perform the restore
• --disable-triggers• Disable triggers during the restore (for a data only restore)
• -e [--exit-on-error]• Exits upon any error
28
Wednesday, September 18, 13
pg_restore - Common Options (continued)
• -l [--list]• Create a list (TOC) file
• -L list-file [--use-list=list-file]• Restore based on the specified list file
• -V [--version]
• -v [--verbose]
29
Wednesday, September 18, 13
pg_restore - Examples
$ pg_restore -a -Fc -d prod2_db prod1_db.fc.dmp
Restores data only from a custom formatted file into database prod2_db $ pg_restore -c --schema=gold_partners -v -Ft -d prod2_db prod.tar.dmp
Cleans (removes data & structures first) then restores the gold_partners schema from a tar formatted file into the prod2_db database (with verbose output)
$ pg_restore --schema-only -d qa1_db -Fc -j 10 prod1_db.fc.dmp
Restores the schema only (DDL) from a custom formatted file into the qa1_db database using 10 parallel streams to do the restore
30
Wednesday, September 18, 13
Restoring via a list file• pg_restore can create a list file
from a pg_dump file
• List file will contain one line per needed operation such as:
• CREATE TABLE
• COPY
• CREATE INDEX
• List file can be modified as desired to create a custom restore
31
Wednesday, September 18, 13
Create a list file from the pg_dump file
$ pg_dump -Ft db1 > db1.fc.dmp
$ pg_restore -Ft -l db1.dmp > db1.lst
32
Wednesday, September 18, 13
Sample list file header
;; Archive created at Tue Sep 10 09:42:24 2013; dbname: testdb; TOC Entries: 34; Compression: -1; Dump Version: 1.12-0; Format: CUSTOM; Integer: 4 bytes; Offset: 8 bytes; Dumped from database version: 9.2.4; Dumped by pg_dump version: 9.2.4;;
33
Wednesday, September 18, 13
Sample list file contents
; Selected TOC Entries:;1981; 1262 16386 DATABASE - testdb_old postgres6; 2615 2200 SCHEMA - public postgres1982; 0 0 COMMENT - SCHEMA public postgres1983; 0 0 ACL - public postgres181; 3079 11730 EXTENSION - plpgsql1984; 0 0 COMMENT - EXTENSION plpgsql168; 1259 16411 TABLE public testdb_jasper_metrics_tables postgres169; 1259 16414 TABLE public testdb_jasper_metrics_tables_tmp1 postgres170; 1259 16417 TABLE public testdb_metrics_activity postgres171; 1259 16423 TABLE public testdb_metrics_database postgres172; 1259 16426 TABLE public testdb_postgres_metrics_bgwriter postgres173; 1259 16429 TABLE public testdb_postgres_metricsio_user_tables postgres174; 1259 16432 TABLE public testdb_testdb_gf_metrics_tables postgres175; 1259 16435 TABLE public testdb_testdb_transition_metrics_tables postgres176; 1259 16438 TABLE public testdb_testdb_transition_metrics_tables_tmp1 postgres177; 1259 16441 TABLE public testdb_testdb_transition_metrics_tables_tmp2 postgres 34
Wednesday, September 18, 13
Sample list file contents (cont)
178; 1259 16444 TABLE public idle_conn_metrics postgres179; 1259 16447 TABLE public total_conn_metrics postgres180; 1259 16450 TABLE public waiting_conn_metrics postgres1964; 0 16411 TABLE DATA public testdb_jasper_metrics_tables postgres1965; 0 16414 TABLE DATA public testdb_jasper_metrics_tables_tmp1 postgres1966; 0 16417 TABLE DATA public testdb_metrics_activity postgres1967; 0 16423 TABLE DATA public testdb_metrics_database postgres1968; 0 16426 TABLE DATA public testdb_postgres_metrics_bgwriter postgres1969; 0 16429 TABLE DATA public testdb_postgres_metricsio_user_tables postgres1970; 0 16432 TABLE DATA public testdb_testdb_gf_metrics_tables postgres1971; 0 16435 TABLE DATA public testdb_testdb_transition_metrics_tables postgres1972; 0 16438 TABLE DATA public testdb_testdb_transition_metrics_tables_tmp1 postgres1973; 0 16441 TABLE DATA public testdb_testdb_transition_metrics_tables_tmp2 postgres1974; 0 16444 TABLE DATA public idle_conn_metrics postgres1975; 0 16447 TABLE DATA public total_conn_metrics postgres1976; 0 16450 TABLE DATA public waiting_conn_metrics postgres
35
Wednesday, September 18, 13
Restore via list file - example
$ pg_dump -Ft prod_db > prod_db.fc.dmp
$ pg_restore -Ft -l prod_db.dmp > prod_db.lst
$ createdb qadb3
Edit prod_db.lst as needed / desired
$ pg_restore -L prod_db.lst -Ft -d qadb3 prod_db.dmp
36
Wednesday, September 18, 13
Restore Options
Summary
37
Wednesday, September 18, 13
Point In Time Recovery(PITR)
38
Wednesday, September 18, 13
PITR Overview
•PITR Backups • Archiving the WAL segments• Making Base Backups
•PITR Recovery• Restore the last Base Backup• Prepare the recovered system data directory• Create a recovery.conf file• Start the postmaster
39
Wednesday, September 18, 13
PITR Setup• Enable / set the following parameters in the
postgresql.conf file:• wal_level = archive (or hot_standby)• archive_mode = on• archive_command = 'valid archive command'
Can be any valid shell command (including scripts)• archive_timeout = [timeout]
• Special archive_command (and recovery.conf file) tags• %p = full path (absolute path) and the file name of the WAL
segment to be archived• %f = only the file name of the WAL segment• %% = insert a % character in the command string.
40
Wednesday, September 18, 13
PITR Example• Enable / set the following parameters in the postgresql.conf file:
• wal_level = archive • archive_mode = on• archive_command = 'cp %p /stage/wal/%f'
Can be any valid shell command (including scripts)• archive_timeout = 0
• mkdir /stage/wal
• chown postgres:postgres /stage/wal
• Re-start the Server
41
Wednesday, September 18, 13
PITR Example - create transactions• Execute SQL commands / transactions • Enable access, turn on applications, etc
• This should force the creation of multiple archived WAL files in the /stage/wal directory
• WAL segments are copied when:• The WAL segment is full (see checkpoint_segments)• Number of seconds specified in archive_timeout has passed
42
Wednesday, September 18, 13
PITR Example - create base backup
• Execute pg_start_backup$ psql pitr_test # select pg_start_backup ('tag') ;
• Archive the cluster data directory (and any related tablespaces)
$ tar -czvf /backups/pitr/<date>.data.tar.gz ./datarsyncother copy methods
• Execute pg_stop_backup$ psql pitr_test # select pg_stop_backup () ;
43
Wednesday, September 18, 13
PITR Example - create more transactions
• Execute SQL commands / transactions
• The application, user connections, etc will continue to generate transactions (and archived WAL segments)
• Verify the creation of additional archived WAL files in the /stage/wal directory
44
Wednesday, September 18, 13
PITR - recovery.conf file (common options)
Recovery settings are placed in the file 'recovery.conf'
• restore_command (string) must return nonzero
• restore_command = 'cp /stage/wal/%f %p'
• restore_command = '/usr/local/bin/restore_shell.sh %p %f'
45
Wednesday, September 18, 13
PITR - recovery.conf file (common options)
recovery_target_time (timestamp)
• specifies the time stamp up to which recovery will proceed.
• recovery_target_time and recovery_target_xid are mutually exclusive
• The default is to recover to the end of the WAL log.
46
Wednesday, September 18, 13
PITR Recovery
(1) If available copy the original cluster data directory to an alternate location
if space is an issue at least copy the old pg_xlog dir it may contain additional unarchived WAL segments
(2) Ensure the postmaster is not running
47
Wednesday, September 18, 13
PITR Recovery
If your backup was an rsync to a second server then skip steps 3 & 4
(3) Remove the cluster data directory and any tablespace directories
(4) Restore your last system backup • make sure permissions are retained
• If you're using tablespaces then verify that the symbolic links in pg_tblspc/ were restored
48
Wednesday, September 18, 13
PITR Recovery
(5) Remove any wal segments from the pg_xlog dir that were restored from the backup
If you didn't backup pg_xlog then create it, make sure you re-establish it as a symbolic link if needed If needed also re-create the pg_xlog/archive_status directory
(6) Copy the files from the original pg_xlog dir (if available) into the new pg_xlog dir
do a copy as opposed to a move in case you need to start over
49
Wednesday, September 18, 13
PITR Recovery
(7) Create a recovery command (recovery.conf) in the cluster data directory.
(8) [Optional] Temporarily modify pg_hba.conf to prevent ordinary users from connecting until the recovery is complete
(9) Start the server. The server will go into recovery mode via the recovery.conf file.
Once the recovery is complete then the server will become available and rename the recovery.conf file to recovery.done
If an error interrupts the recovery (or stops the server) then simply re-starting the server will restart the recovery
50
Wednesday, September 18, 13
PITR Recovery
(10) Verify the recovery.If the database was not recovered properly (or to a state that you desire) then go back to step 1
(11) restore the pg_hba.conf to its original state and run a pg_ctl reload (if it was modified for the recovery)
51
Wednesday, September 18, 13
Questions?
Thank You!
52
Wednesday, September 18, 13