Embed Size (px)
DESCRIPTION
Hot Stand By - Setup, Failover and Rebuilding the Master Step by Step Process explained!
Citation preview
Hot Stand BySetup, Failover and Rebuilding the
Master
3/6/2014
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
2
Prepare the Instance Install Postgres on Servers which are going to hold Primary and
Secondary database
Setup and configure the database cluster on Primary servers
In this example: Primary DB Server:
dbserver1 192.168.160.147 Data directory: /opt/PostgresPlus/9.2AS/data Port: 5444
Stand by DB Server dbserver2 192.168.160.150 Data Directory: /opt/PostgresPlus/9.2AS/data2 Port: 5222
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
3
Edit postgresql.conf and pg_hba.conf on master wal_level = hot_standby (mandatory) max_wal_senders = 3 (mandatory to be set to a positive
integer)
wal_keep_segments = 128 (optional/depending on load) replication_timeout = 5 sec (optional) hot_standby = on (effective only for hot stand by
server)
Add entry in pg_hba.conf• host replication all 192.168.0.0/16 trust
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
4
Take a Backup and restore on Secondary Take a cold backup of Primary DB
Instance/Cluster Restore the same for creating the DB
cluster/instance on Secondary server
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
5
Create recovery.conf in Secondary Server standby_mode = 'on'
primary_conninfo = 'host=192.168.160.147 port=5444 user=enterprisedb password=ashnik'
restore_command = 'scp [email protected]:/mnt/arch/%f %p'
# optional# needs archiving command to be enabled on primary
recovery_target_timeline = 'latest' #optional
trigger_file = '/opt/PostgresPlus/9.2AS/data2/recover.trigger'
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
6
Start the servers
Start the secondary server There will be a warning in log files that primary server is not
available, ignore that Start the primary server
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
7
Test Replication
On Primary:edb=# insert into replication_test values (2);INSERT 0 1
On Secondary:edb=# select * from replication_test; test_column ------------- 1 2(2 rows)
Secondary server is read-only:edb=# insert into replication_test values (3);ERROR: cannot execute INSERT in a read-only transaction
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
8
Triggering the Failover To, Trigger a failure on Primary and create the recovery trigger file (manually, but
can be scripted too)
touch opt/PostgresPlus/9.2AS/data2/recover.trigger
Logic to script the above step:
while( pg_ctl -h 192.168.160.147 –p 5444 -c "select 1 “)
{
; #do nothing
}
touch opt/PostgresPlus/9.2AS/data2/recover.trigger
Once completed, the recovery.conf will change to recover.done
Connect to secondary db and execute insert to confirm the failover edb=# insert into replication_test values (4);INSERT 0 1
Or execute select pg_is_in_recovery(); (output must be “f”) to confirm recovery is completed
Point the database/Virtual IP to new database server
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
9
Triggering the Switchover Disconnect all the application from Primary Node
Shutdown the primary database
To, Trigger a failure on Primary and create the recovery trigger file
touch opt/PostgresPlus/9.2AS/data2/recover.trigger
Once completed, the recovery.conf will change to recover.done
Connect to secondary db and execute insertedb=# insert into replication_test values (4);INSERT 0 1
Or execute select pg_is_in_recovery(); (output must be “f”) to confirm recovery is completed
Point the database/Virtual IP to new database server
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
10
Rebuilding the Master Let’s remove the data directory on Old Primary (dbserver1)
rm -Rf /opt/PostgresPlus/9.2AS/data
Restore the primary server from Secondary server (Switched to Master now: dbserver2):
pg_basebackup -D /opt/PostgresPlus/9.2AS/data -h 192.168.160.150 -W -p 5222 -Fp --xlog-method=stream#This step does not need down time
Change the port number in postgresql.conf on restored data directory
Change the recovery.done to recovery.conf in data directory copied just now and make changes in configuration to point to new master
Start the old primary instance as new hot standby
Confi
denti
al in
form
ation
, for
inte
rnal
use
on
ly
11
Monitoring the Replication
Check if the current node is master or slave:• SELECT pg_is_in_recovery();
See the current snapshot on master and slave:• SELECT txid_current_snapshot();
