Upload
command-prompt-inc
View
1.103
Download
5
Tags:
Embed Size (px)
DESCRIPTION
Selena DeckelmannBucardo is a mature replication system for PostgreSQL that supports asynchronous replication for both master-slave and multi-master systems. Originally designed for slow and unreliable networks, it has remarkable recovery ability, an easy to use command-line interface and development is active! Uses for Bucardo include: a slave read-only database, multi-master replication, data warehousing and just having fun moving your data around! Will include overview replication for PostgreSQL in general, a tour of features, a basic configuration walk through and the much feared live demo!
Citation preview
endpoint.com
Bucardo:Replication with tiny little goats
http://www.flickr.com/photos/kevincollins/
Selena [email protected]
endpoint.com
★ Software Engineerhttp://endpoint.com
★ User Group Organizer http://pugs.postgresql.org/pdxpug
★ Conference Organizerhttp://opensourcebridge.org
Hi!
endpoint.com
SEAPUG?
• subscribe: email to: [email protected] body: subscribe seapug
• Talk to me after!
endpoint.com
http://www.telegraph.co.uk/science/science-news/4409958/Extinct-ibex-is-resurrected-by-cloning.html
endpoint.com
What can it do?
http://www.flickr.com/photos/8602783@N06/2294028540
endpoint.com
Master-slave
• Scaling/Redundancy
• UPGRADES (8.x -> 8.4)
• Reporting databases
• Data warehousing
• Now handles SEQUENCES!
endpoint.com
Master-master
• Write two two databases!
• Sync type: SWAP
• Conflict resolution through hooks!
endpoint.com
Custom code
• Respond to DML changes
• Filter
• Run arbitrary code!
endpoint.com
STORY TIME!
item onhandskis 10
itemskiitemskiitem
skiitemskiitem
skiitemskiitem
skiitemskiitem
skiitemski
on hand = count(item)
endpoint.com
itemski
new!
bucardo
count(items)and other stuff
prod dbreporting db
itemski
on hand+1
new!
custom code
NOTIFY
NO
TIFY
sync
A
sync Bsync C
goat courtesy of: http://www.flickr.com/photos/nuritwilde/394603312
endpoint.com
Strengths
• Drop-in, no changes to Postgres
• Control daemon/DB can run anywhere
• EXTREME async
• No locking of slaves
• Command-line controls rock!
endpoint.com
The Fine Print
• No smooth DDL handling
• Not for failover (aka high availability)
• bloat: pg_listener, q, bucardo_delta, bucardo_track
• No locking of slaves
• No Windows support
• One developer
http://www.flickr.com/photos/ncarey/135901120/
endpoint.com
http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling
endpoint.com
Installing
http://www.flickr.com/photos/kendrick/361173694
endpoint.com
Bucardo Requirements
• Postgres (8.3 or higher recommended)
• Perl & some packages:
• DBD::Pg 2.0
• DBI 1.51
• DBIx::Safe 1.2.4
• ExtUtils::MakeMaker 6.32
• pl/perlu and pl/pgsql
endpoint.com
Simple set upMASTER setup:bucardo_ctl add db MyDB name=masterbucardo_ctl add all tablesbucardo_ctl add all sequences
SLAVE setup:bucardo_ctl add db MyDB name=slave port=6543bucardo_ctl add herd all_tablespsql# INSERT INTO herdmap (herd, goat) SELECT ‘all_tables’, id FROM goat WHERE (reltype = ‘table’ OR reltype = ‘sequence’)
AND db = ‘master’;
endpoint.com
Setting up, cont.
TEST:
bucardo_ctl validate all
SYNC:
bucardo_ctl add sync delta type=pushdelta source=all_tables targetdb=slave
(installs triggers!)
endpoint.com
Setting up
• Initiate a one-time-copy: onetimecopy=[1|2]
• Set up ongoing syncs: DEFAULT (can set timeouts)
• MANY configuration options
endpoint.com
Admin stuff
• .bucardorc (set things like: dbport)
• Only need one ‘bucardo’ database
• Doesn’t need to be your prod databases
• System uses the ‘bucardo’ schema for ease of admin, removal
endpoint.com
Smooth upgrades!
bucardo_ctl upgrade
• Stop bucardo (bucardo_ctl stop)
• applies DDL changes, functions
• see also UPGRADE file
endpoint.com
Stuff to watch out for
• search_path for ‘bucardo’ user
• PL/Perlu & pl/pgsql must be installed
• Uses prepared statements, turn off with:
bucardo_ctl update db MyDB server_side_prepares=0
endpoint.com
Internals
http://www.flickr.com/photos/chelseagirlphotos
endpoint.com
Internalsbucardo=# \d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- bucardo | audit_pid | table | bucardo bucardo | audit_pid_id_seq | sequence | bucardo bucardo | bucardo_config | table | bucardo bucardo | bucardo_custom_trigger | table | bucardo bucardo | bucardo_custom_trigger_id_seq | sequence | bucardo bucardo | bucardo_log_message | table | bucardo bucardo | bucardo_rate | table | bucardo bucardo | customcode | table | bucardo bucardo | customcode_id_seq | sequence | bucardo bucardo | customcode_map | table | bucardo bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo(21 rows)
endpoint.com
goatsTable "bucardo.goat" Column | Type |----------------------+--------------------------+ id | integer | db | text | schemaname | text | tablename | text | reltype | text | pkey | text | qpkey | text | pkeytype | text | has_delta | boolean | ping | boolean | ...
endpoint.com
goats
# bucardo_ctl list tablesTable: archive.course_logging DB: odw_master PK: noneTable: archive.messages DB: odw_master PK: noneTable: archive.student_assignments DB: odw_master PK: noneTable: archive.student_courses DB: odw_master PK: noneTable: logging.activity DB: odw_master PK: activity_name (varchar)Table: logging.course_logging DB: odw_master PK: course_logging_id (int4)Table: public.ecash_transaction DB: odw_master PK: trans_id (int4)Table: public.feedback DB: odw_master PK: date (timestamp)Table: public.holidays DB: odw_master PK: customernum|month|day|year (int4|int4|int4|int4)...
endpoint.com
bucardo_ctl list sequencesSequence 63: logging.logging_seqSequence 64: public.trans_id_seqSequence 65: public.num_seqSequence 66: public.upload_seq
endpoint.com
bucardo databasebucardo=# \d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+---------... bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo(21 rows)
endpoint.com
Internals
# bucardo_ctl statusDays back: 3 User: bucardo Database: bucardo Port: 8000 PID of Bucardo MCP: 6876Name Type State PID Last_good Time I/U/D Last_bad Time========+=====+=====+====+=========+=====+=====+========+====a_delta | P |idle |6886|1m4s |0s |0/0/0|unknown | b_delta | P |idle |6885|1m4s |0s |0/0/0|9h32m47s|0s c_delta | P |idle |6887|1m4s |0s |0/0/0|unknown | d_delta | P |idle |6888|54s |0s |0/0/0|2m6s |3s e_delta | P |idle |6890|1m4s |0s |0/0/0|9h32m47s|0s f_delta | P |idle |6889|1m4s |0s |0/0/0|9h32m47s|0s
endpoint.com
# bucardo_ctl status a_deltaDays back: 3 User: bucardo Database: bucardo Port: 8000======================================================================Sync name: a_deltaCurrent state: idle (PID = 6888)Type: pushdeltaSource herd/database: all_a / a_masterTarget database: a_slaveTables in sync: 26Last good: 3m 34s (time to run: 0s)Last good time: Oct 16, 2009 17:52:20 Target: a_slaveIns/Upd/Del: 20 / 0 / 50Last bad: 4m 47s (time to run: 3s)Last bad time: Oct 16, 2009 17:51:08 Target: a_slaveLatest bad reason: ?PID file: /var/run/bucardo/bucardo.ctl.sync.a_delta.pidPID file created: Fri Oct 16 17:52:09 2009Status: activeLimitdbs: 0Priority: 0Checktime: noneOverdue time: 00:00:00Expired time: 00:00:00Stayalive: yes Kidsalive: yesRebuild index: 0 Do_listen: yesPing: yes Makedelta: no Onetimecopy: 0
endpoint.com
How Bucardo replicates
endpoint.com
bucardo_add_delta
bucardo_triggerkick_[sync]
bucardo_delta
INSERT
NOTIFY!
Triggers IN
SERT
, UPD
ATE,
DEL
ETE
endpoint.com
NOTIFY / LISTEN
• Simple interprocess communication
psql# LISTEN cupcakes;psql# NOTIFY cupcakes;Asynchronous notification "cupcakes" received from server process with PID 1337.
• Any Postgres process can listen in
endpoint.com
Actual Triggers! Table "public.test" Column | Type | Modifiers --------+-----------------------+----------- test | character varying(15) | not nullIndexes: "test_pkey" PRIMARY KEY, btree ("test")Triggers: bucardo_add_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH ROW EXECUTE PROCEDURE bucardo_add_delta_role() bucardo_triggerkick_a_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH STATEMENT EXECUTE PROCEDURE bucardo_triggerkick_a_delta()
endpoint.com
On your master DB
Tables:
• bucardo_delta table has: ID, txn_time
• bucardo_track table: who’s done what?
endpoint.com
Processes!• MCP, CTL, KID
• MCP: master control process
• CTL: controller (track, kick off/kill KIDs)
• KID: sync processes
endpoint.com
syncs
• A set of tables (herd) that are replicated together
endpoint.com
Initiating syncs
• NOTIFY vs. timeout vs. kick
• NOTIFY - from master db to MCP
• timeout - controller detects
• kick - manual/from MCP to controller
endpoint.com
TRIGGER
Flow of control
NOTIFY MCP CTLNOTIFY
INSERT INTO “q”
KID
SELECT from “q”UPDATE “q”
REPLICATIONEVENT!
NOTIFY
Bucardo
endpoint.com
Administrativa
• Totally ok to kill KIDs
• CTL cleans up after you
• May need to kill open database handles (for unfinished COPYs, etc)
endpoint.com
Sync types
• Fullcopy. Timeout/kick. TRUNCATE or DELETE, then COPY
• Pushdelta. Trigger. One way master-slave.
• Swap. Trigger. Two-way. Conflict resolution.
endpoint.com
Bucardo anti-trigger
• canonical way to disable triggers: ALTER TABLE DISABLE TRIGGER ALL;:(
• pre 8.3 way: UPDATE pg_class..
• SET session_replication_role = ‘replica’;
• Applies to just the current session
• Thanks, Jan!
endpoint.com
Bucardo hooks
• LIVE DATA CHANGES!
• pass in a hashref (ROW or ROWINFO)
• Returned value - bitmapped (a/b or both!)
• Affect the replication event, side effects
• Example: cache invalidation
endpoint.com
Bucardo Hooks
• customcode & customcode_map
• Conflict handlers (SWAP syncs)
• Exception - fix and try again
• Pre and post trigger
endpoint.com
Development
• bucardo.org
• mailing lists: http://bucardo.org/wiki/Bucardo_mailing_list
• Release early, often.
• Please submit patches! bug reports! tests!
endpoint.com
Questions?
endpoint.com
Thanks!
http://www.slideshare.net/selenamarie/bucardo
@selenamarie on Twitter
rss: http://www.chesnok.com/daily