Embed Size (px)
DESCRIPTION
Golconde is a queue based replication solution for PostgreSQL written in Python. It is designed to be loosely coupled and rely upon existing enterprise messaging systems that have STOMP protocol support. Designed to scale easily and with multi-data center implementations in mind, the application and message queues for distribution live outside of the database. By decoupling Golconde from PostgreSQL it is differentiated from existing replication solutions, moving the workload from the database tier, where CPU, RAM and IO overhead can be very expensive, to a commodity layer where the operational cost for performing the data distribution work is much less expensive.
Citation preview
Friday, May 22, 2009
GOLCONDERené Magritte, 1953
Friday, May 22, 2009
GOLCONDE
* UNLESS YOU START FROM A CLEAN SLATE
Friday, May 22, 2009
GOLCONDE
NOT REPLICATION*
* UNLESS YOU START FROM A CLEAN SLATE
Friday, May 22, 2009
GOLCONDE
NOT REPLICATION*
DATA DISTRIBUTION SYSTEM
* UNLESS YOU START FROM A CLEAN SLATE
Friday, May 22, 2009
GOLCONDE
NOT REPLICATION*
DATA DISTRIBUTION SYSTEM
ONE CANONICAL SOURCE
* UNLESS YOU START FROM A CLEAN SLATE
Friday, May 22, 2009
GOLCONDE
NOT REPLICATION*
DATA DISTRIBUTION SYSTEM
ONE CANONICAL SOURCE
ANY NUMBER OF TARGET RELATIONS
* UNLESS YOU START FROM A CLEAN SLATE
Friday, May 22, 2009
GOLCONDE
NOT REPLICATION*
DATA DISTRIBUTION SYSTEM
ONE CANONICAL SOURCE
ANY NUMBER OF TARGET RELATIONS
QUEUE BASED
* UNLESS YOU START FROM A CLEAN SLATE
Friday, May 22, 2009
FEATURES
Friday, May 22, 2009
FEATURES
LIGHT-WEIGHT DAEMON
Friday, May 22, 2009
FEATURES
LIGHT-WEIGHT DAEMON
“AUTOSQL” AND CUSTOM HANDLERS
Friday, May 22, 2009
FEATURES
LIGHT-WEIGHT DAEMON
“AUTOSQL” AND CUSTOM HANDLERS
TWO METHODS OF OPERATION
Friday, May 22, 2009
FEATURES
LIGHT-WEIGHT DAEMON
“AUTOSQL” AND CUSTOM HANDLERS
TWO METHODS OF OPERATION
REAL-TIME VISUALIZATION
Friday, May 22, 2009
WHY GOLCONDE?
Friday, May 22, 2009
WHY GOLCONDE?
GOOD REPLICATION TOOLS EXIST
SLONY, LONGDISTE, BUCARDO, ETC
Friday, May 22, 2009
WHY GOLCONDE?
GOOD REPLICATION TOOLS EXIST
SLONY, LONGDISTE, BUCARDO, ETC
NO “GOLDEN HAMMER”
Friday, May 22, 2009
WHY GOLCONDE?
GOOD REPLICATION TOOLS EXIST
SLONY, LONGDISTE, BUCARDO, ETC
NO “GOLDEN HAMMER”
CURRENT TOOLS ARE RESOURCE INTENSIVE
LIVE ON THE DATABASE TIER
LIVE IN THE DATABASE
Friday, May 22, 2009
WHY GOLCONDE?
Friday, May 22, 2009
WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION
Friday, May 22, 2009
WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION
CROSS DATA-CENTER IMPLEMENTATIONS
Friday, May 22, 2009
WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION
CROSS DATA-CENTER IMPLEMENTATIONS
MESSAGE BROKERS ARE MORE EFFICIENT FOR QUEUES
Friday, May 22, 2009
WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION
CROSS DATA-CENTER IMPLEMENTATIONS
MESSAGE BROKERS ARE MORE EFFICIENT FOR QUEUES
DATABASE HARDWARE IS EXPENSIVE ($$$)
Friday, May 22, 2009
WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION
CROSS DATA-CENTER IMPLEMENTATIONS
MESSAGE BROKERS ARE MORE EFFICIENT FOR QUEUES
DATABASE HARDWARE IS EXPENSIVE ($$$)
QUEUE AND CONSUMER GRADE HARDWARE ARE NOT
Friday, May 22, 2009
WHY GOLCONDE?
NEEDED A SCALE OUT SOLUTION
CROSS DATA-CENTER IMPLEMENTATIONS
MESSAGE BROKERS ARE MORE EFFICIENT FOR QUEUES
DATABASE HARDWARE IS EXPENSIVE ($$$)
QUEUE AND CONSUMER GRADE HARDWARE ARE NOT
NEED CUSTOMIZABLE WORKFLOW FOR DISPARATE RELATIONS
Friday, May 22, 2009
REQUIREMENTS
Friday, May 22, 2009
REQUIREMENTS
STOMP SUPPORTING MESSAGE BROKER
Friday, May 22, 2009
REQUIREMENTS
STOMP SUPPORTING MESSAGE BROKER
POSTGRESQL 8.3* OR HIGHER
Friday, May 22, 2009
REQUIREMENTS
STOMP SUPPORTING MESSAGE BROKER
POSTGRESQL 8.3* OR HIGHER
PYTHON 2.6
PYYAML
PSYCOPG2
STOMP.PY
Friday, May 22, 2009
STOMP
STREAMING TEXT ORIENTED MESSAGE PROTOCOL
INTEROPERABLE WIRE FORMAT
SUPPORTED BY MULTIPLE BROKERS
HTTP://STOMP.CODEHAUS.ORG/
Friday, May 22, 2009
AMQP
ADVANCED MESSAGE QUEUING PROTOCOL
OPEN STANDARD WIRE PROTOCOL
EMERGING STANDARD SUPPORTED BY MULTIPLE BROKERS
HTTP://WWW.AMQP.ORG/
Friday, May 22, 2009
HTTP://WWW.RABBITMQ.COM/
OpenAMQP
QPID
RedHat Enterprise MRG
Friday, May 22, 2009
ARCHITECTURE
Friday, May 22, 2009
ARCHITECTURE
START OF ACTION LIFECYCLE
CLIENT-BASED: ENQUEUE FROM APPLICATION
TRIGGER-BASED: ENQUEUE FROM POSTGRESQL
Friday, May 22, 2009
ARCHITECTURE
START OF ACTION LIFECYCLE
CLIENT-BASED: ENQUEUE FROM APPLICATION
TRIGGER-BASED: ENQUEUE FROM POSTGRESQL
MULTI-THREADED PYTHON DAEMON
Friday, May 22, 2009
LEXICON
Friday, May 22, 2009
LEXICON
DESTINATIONS ARE THE CANONICAL RELATIONS
Friday, May 22, 2009
LEXICON
DESTINATIONS ARE THE CANONICAL RELATIONS
ARE ACTED UPON IN CLIENT-BASED WORKFLOWS
Friday, May 22, 2009
LEXICON
DESTINATIONS ARE THE CANONICAL RELATIONS
ARE ACTED UPON IN CLIENT-BASED WORKFLOWS
ENQUEUES DATA IN TRIGGER-BASED WORKFLOWS
Friday, May 22, 2009
LEXICON
DESTINATIONS ARE THE CANONICAL RELATIONS
ARE ACTED UPON IN CLIENT-BASED WORKFLOWS
ENQUEUES DATA IN TRIGGER-BASED WORKFLOWS
TARGETS ARE THE DISTRIBUTED RELATIONS
Friday, May 22, 2009
ARCHITECTURE
Friday, May 22, 2009
ARCHITECTURE
DATA IS PASSED IN JSON ENCODED PACKETS
Friday, May 22, 2009
ARCHITECTURE
ADD (INSERT)
DATA IS PASSED IN JSON ENCODED PACKETS
COMMANDS:
Friday, May 22, 2009
ARCHITECTURE
ADD (INSERT)
DELETE
DATA IS PASSED IN JSON ENCODED PACKETS
COMMANDS:
Friday, May 22, 2009
ARCHITECTURE
ADD (INSERT)
DELETE
SET (UPSERT)
DATA IS PASSED IN JSON ENCODED PACKETS
COMMANDS:
Friday, May 22, 2009
ARCHITECTURE
ADD (INSERT)
DELETE
SET (UPSERT)
UPDATE
DATA IS PASSED IN JSON ENCODED PACKETS
COMMANDS:
Friday, May 22, 2009
MESSAGE EXAMPLES
Friday, May 22, 2009
MESSAGE EXAMPLES
{"ACTION": "ADD", "DATA": {"FRIEND_ID": 47, "USER_ID": 16, "STATUS_ID": 1}}
Friday, May 22, 2009
MESSAGE EXAMPLES
{"ACTION": "ADD", "DATA": {"FRIEND_ID": 47, "USER_ID": 16, "STATUS_ID": 1}}
{"ACTION": "DELETE", "RESTRICTION": {"FRIEND_ID": 11, "USER_ID": 126}}
Friday, May 22, 2009
MESSAGE EXAMPLES
{"ACTION": "ADD", "DATA": {"FRIEND_ID": 47, "USER_ID": 16, "STATUS_ID": 1}}
{"ACTION": "DELETE", "RESTRICTION": {"FRIEND_ID": 11, "USER_ID": 126}}
{"ACTION": "SET", "DATA": {"FRIEND_ID": 112, "USER_ID": 111, "TIMESTAMP": "WED APR 1 13:56:54 2009", "STATUS_ID": 1}}
Friday, May 22, 2009
MESSAGE EXAMPLES
{"ACTION": "ADD", "DATA": {"FRIEND_ID": 47, "USER_ID": 16, "STATUS_ID": 1}}
{"ACTION": "DELETE", "RESTRICTION": {"FRIEND_ID": 11, "USER_ID": 126}}
{"ACTION": "SET", "DATA": {"FRIEND_ID": 112, "USER_ID": 111, "TIMESTAMP": "WED APR 1 13:56:54 2009", "STATUS_ID": 1}}
{"ACTION": "UPDATE", "RESTRICTION": {"FRIEND_ID": 5, "USER_ID": 41}, "DATA": {"TIMESTAMP": "WED APR 1 13:56:38 2009", "STATUS_ID": 3}}
Friday, May 22, 2009
AUTOSQL
Friday, May 22, 2009
AUTOSQL
DEFAULT HANDLER TYPE FOR GOLCONDE
Friday, May 22, 2009
AUTOSQL
DEFAULT HANDLER TYPE FOR GOLCONDE
EXAMINES PG_CATALOG DATA FOR SCHEMA
Friday, May 22, 2009
AUTOSQL
DEFAULT HANDLER TYPE FOR GOLCONDE
EXAMINES PG_CATALOG DATA FOR SCHEMA
CACHES SCHEMA
Friday, May 22, 2009
AUTOSQL
DEFAULT HANDLER TYPE FOR GOLCONDE
EXAMINES PG_CATALOG DATA FOR SCHEMA
CACHES SCHEMA
GENERATES SQL
Friday, May 22, 2009
AUTOSQL
DEFAULT HANDLER TYPE FOR GOLCONDE
EXAMINES PG_CATALOG DATA FOR SCHEMA
CACHES SCHEMA
GENERATES SQL
REQUIRES SAME SCHEMA
Friday, May 22, 2009
CUSTOM HANDLERS
THAR BE DRAGONS
Friday, May 22, 2009
CUSTOM HANDLERS
ALLOWS FOR DIFFERENT SCHEMAS FOR THE SAME DATA
EXAMPLE:
LEGACY SCHEMA VS NEW SCHEMA
DIFFERENT FOCUSED TARGET RELATIONS
WARNING: NEED TO UNDERSTAND INTERNAL GOLCONDE FLOW
THAR BE DRAGONS
Friday, May 22, 2009
CLIENT-BASED FLOW
Friday, May 22, 2009
CLIENT-BASED FLOW
CLIENT APPLICATION ENQUEUES
Friday, May 22, 2009
CLIENT-BASED FLOW
CLIENT APPLICATION ENQUEUES
GOLCONDE PERFORMS TRANSACTION ON CANONICAL DESTINATION RELATION
Friday, May 22, 2009
CLIENT-BASED FLOW
CLIENT APPLICATION ENQUEUES
GOLCONDE PERFORMS TRANSACTION ON CANONICAL DESTINATION RELATION
SERIALLY DISTRIBUTES TO TARGET QUEUES
Friday, May 22, 2009
CLIENT-BASED FLOW
Friday, May 22, 2009
CLIENT-BASED FLOW
TARGET HANDLER DEQUEUES
Friday, May 22, 2009
CLIENT-BASED FLOW
TARGET HANDLER DEQUEUES
PERFORMS TRANSACTION ON TARGET RELATIONS
Friday, May 22, 2009
TRIGGER-BASED USAGE
Friday, May 22, 2009
TRIGGER-BASED USAGE
REQUIRES PL/PYTHON
Friday, May 22, 2009
TRIGGER-BASED USAGE
REQUIRES PL/PYTHON
TRIGGER-UTIL.PY BUILDS AND INSTALLS TRIGGERS
Friday, May 22, 2009
TRIGGER-BASED USAGE
REQUIRES PL/PYTHON
TRIGGER-UTIL.PY BUILDS AND INSTALLS TRIGGERS
TRIGGER IS INSTALLED ON THE CANONICAL TABLE
Friday, May 22, 2009
TRIGGER-BASED USAGE
REQUIRES PL/PYTHON
TRIGGER-UTIL.PY BUILDS AND INSTALLS TRIGGERS
TRIGGER IS INSTALLED ON THE CANONICAL TABLE
NO MODIFICATION OF CLIENT APPLICATION REQUIRED
Friday, May 22, 2009
TRIGGER-BASED FLOW
Friday, May 22, 2009
TRIGGER-BASED FLOW
FIRES AFTER INSERT/UPDATE/DELETE
Friday, May 22, 2009
TRIGGER-BASED FLOW
FIRES AFTER INSERT/UPDATE/DELETE
ENQUEUES INTO TARGET QUEUES
Friday, May 22, 2009
TRIGGER-BASED FLOW
FIRES AFTER INSERT/UPDATE/DELETE
ENQUEUES INTO TARGET QUEUES
GOLCONDE PERFORMS TRANSACTIONS ON THE TARGET RELATIONS
Friday, May 22, 2009
CONFIGURATION
YAML BASED
HTTP://WWW.YAML.ORG/
Friday, May 22, 2009
CONFIGURATION
YAML BASED
PARAMETERS FOR:
HTTP://WWW.YAML.ORG/
Friday, May 22, 2009
CONFIGURATION
YAML BASED
PARAMETERS FOR:
LOGGING
HTTP://WWW.YAML.ORG/
Friday, May 22, 2009
CONFIGURATION
YAML BASED
PARAMETERS FOR:
LOGGING
HTTP DAEMON FOR REALTIME MONITORING & VISUALIZATION
HTTP://WWW.YAML.ORG/
Friday, May 22, 2009
CONFIGURATION
YAML BASED
PARAMETERS FOR:
LOGGING
HTTP DAEMON FOR REALTIME MONITORING & VISUALIZATION
DESTINATION / TARGET GROUPS
HTTP://WWW.YAML.ORG/
Friday, May 22, 2009
STATISTICS & VISUALIZATION
Friday, May 22, 2009
STATISTICS & VISUALIZATION
BUILT-IN HTTP SERVER
Friday, May 22, 2009
STATISTICS & VISUALIZATION
BUILT-IN HTTP SERVER
STATS REQUESTS RETURNS JSON
USE WITH STAPLR, NAGIOS, ETC
Friday, May 22, 2009
STATISTICS & VISUALIZATION
BUILT-IN HTTP SERVER
STATS REQUESTS RETURNS JSON
USE WITH STAPLR, NAGIOS, ETC
INTERNAL REAL-TIME VISUALIZATION
Friday, May 22, 2009
CURRENT STATUS
Friday, May 22, 2009
CURRENT STATUS
0.5 BETA RELEASE - 03/02
Friday, May 22, 2009
CURRENT STATUS
0.5 BETA RELEASE - 03/02
0.6 BETA COMING SOON
ADDS REAL-TIME STATS SERVER
BUG FIXES
Friday, May 22, 2009
ROADMAP
Friday, May 22, 2009
ROADMAP
TWO-PHASE COMMIT LIKE BEHAVIOR
ALL TRANSACTIONS MUST COMMIT OR ROLLBACK
ROLLBACK IS DEFINED BY ROLLBACK DATA PACKETS IN AUTOSQL OR HANDLERS
Friday, May 22, 2009
ROADMAP
TWO-PHASE COMMIT LIKE BEHAVIOR
ALL TRANSACTIONS MUST COMMIT OR ROLLBACK
ROLLBACK IS DEFINED BY ROLLBACK DATA PACKETS IN AUTOSQL OR HANDLERS
ADDITIONAL QUEUE SERVERS AND PROTOCOLS
Friday, May 22, 2009
ROADMAP
TWO-PHASE COMMIT LIKE BEHAVIOR
ALL TRANSACTIONS MUST COMMIT OR ROLLBACK
ROLLBACK IS DEFINED BY ROLLBACK DATA PACKETS IN AUTOSQL OR HANDLERS
ADDITIONAL QUEUE SERVERS AND PROTOCOLS
“NATIVE” CLIENT CLASSES IN PHP, PYTHON
Friday, May 22, 2009
QUESTIONS?
CONTACT INFO:
TWITTER: @CRAD
HTTP://GAVINROY.COM
Friday, May 22, 2009
