Galera Replication Demystified: How Does It Work?

Galera Replication Demystified

how does it work ?

about.me/lefred

Who am I ?

Frédéric Descamps@lefred

Frédéric Descamps@lefredWorking for Percona since 2011

Frédéric Descamps@lefredWorking for Percona since 2011Senior Architect

Frédéric Descamps@lefredWorking for Percona since 2011Senior ArchitectManaging MySQL since 3.23

Frédéric Descamps@lefredWorking for Percona since 2011Senior ArchitectManaging MySQL since 3.23devops believer

Frédéric Descamps@lefredWorking for Percona since 2011Senior ArchitectManaging MySQL since 3.23devops believerand I installed my first Galera Cluster in February 2010 ;-)

Galera Replication

Cluster

Galera Replication - Cluster

What is it ?

What does it handle ?

10/262

Standard asyncrhonous replication is server-centric, one serverstreams data to another one. All the nodes have a specific role.

11/262

In Galera, the dataset is synchronized between one or more servers:data-centric

12/262

You can write to any node in your cluster No need to worry abouteventual out-of-sync

13/262

Write events/transactions are sent in parallel

14/262

Cluster Membershipdetermined by the clusterwsrep_cluster_address is just a pointerany node is permitted to join that

knows the cluster namecan find a single active cluster node

15/262

Cluster Membershipdetermined by the clusterwsrep_cluster_address is just a pointerany node is permitted to join that

knows the cluster namecan find a single active cluster node

㫙�㫙�㫙�㫘�㫙�㫘�㫘�㫙�㫙�㫙�㫙�㫘�㫙�㫘�㫘�㫘�㫘�㫙�㫘�㫙�㫙�㫔�㫖�㫔�㫘�㫘�㫙�㫙�㫙�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�

16/262

The cluster manages quorum

and has split-brain protection.

17/262

18/262

19/262

20/262

21/262

22/262

23/262

Replication

24/262

ReplicationDelivers the writeset to all nodes in the cluster

25/262

and all nodes acknowledge the writeset

26/262

and all nodes acknowledge the writeset

27/262

and all nodes acknowledge the writesetCost is ~roundtrip latency to furthest node

28/262

and all nodes acknowledge the writesetCost is ~roundtrip latency to furthest nodeSerialized by Group Communication

29/262

30/262

GTIDNot the same as 5.6 Aynchronous GTID's

31/262

though they appear the same

32/262

though they appear the same939aac77-f7d1-11e3-bd5e-b211d6ab1ec6:1534285

33/262

GTIDs ensure cluster members are consistent with each other

34/262

GTIDs ensure cluster members are consistent with each othernodes joining a cluster have their GTIDs checked

35/262

GTIDs ensure cluster members are consistent with each othernodes joining a cluster have their GTIDs checked

GTIDs can be used to compare downed nodes to each other

36/262

GTIDThe highest GTID is the most recently written

Generally the best practice it to bootstrap the node with the most recentdata

37/262

38/262

39/262

40/262

Global Transaction IDsinitial dataset

bfb912e5-f560-11e2-0800-1eefab05e57d:0

41/262

bfb912e5-f560-11e2-0800-1eefab05e57d:0

first change/transaction/writeset

bfb912e5-f560-11e2-0800-1eefab05e57d:1

42/262

bfb912e5-f560-11e2-0800-1eefab05e57d:0

first change/transaction/writeset

bfb912e5-f560-11e2-0800-1eefab05e57d:1

undefined GTID

00000000-0000-0000-0000-000000000000:-1

43/262

Global Transaction IDs : Galera vsMySQL 5.6

44/262

45/262

46/262

In MySQL 5.6㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫘�㫘�㫕�㫕�㫕�㫘�㫔�㫙�㫘�㫙�㫔�㫙�㫘�㫙�㫙�㫘�㫙�㫔�㫙�㫙�㫙�㫙�㫙�㫙�㫘�㫘�㫔�㫘�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�

47/262

In Galera㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�

48/262

In Galera㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫔�㫙�㫘�㫙�㫔�㫙�㫘�㫙�㫙�㫘�㫙�㫔�㫙�㫙�㫙�㫙�㫙�㫙�㫘�㫘�㫔�㫘�

49/262

In Galera㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫔�㫙�㫘�㫙�㫔�㫙�㫘�㫙�㫙�㫘�㫙�㫔�㫙�㫙�㫙�㫙�㫙�㫙�㫘�㫘�㫔�㫘�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫘�㫘�㫕�㫕�㫘�㫕�㫕�㫘�㫘�㫕�㫕�㫕�㫕�㫕�

50/262

GTID Assignment

The UUID section, 128-bit, is generated during bootstrapping to identifythe cluster.

Generated by mixing the timer value and pseudo-random numbers(depending primarily from the timer and PID), but, currently, there is nouse of NIC's MAC-address although in theory it may be done that way.

More info: https://gist.github.com/lefred/88a2cec88d03854d9934

51/262

GTID Assignment (3)

The seqno, 64-bit, is incremented only when the transaction passescertification and is ready for commit.

For the curious, the algorithm used to derive sequence number is TotemSingle-ring Ordering protocol.

Before that, there is already communication between the nodes, groupcommunication is used to define a group-channel id which is a locallymaintained counter by each node in sync with the group.

52/262

Serialization of writesets

53/262

54/262

55/262

56/262

57/262

58/262

59/262

60/262

61/262

62/262

63/262

64/262

65/262

66/262

RolesWe have 4 distinct roles in Galera:

2 for replication2 for state transfer

67/262

Replication RolesWithin the cluster, all nodes are equal

68/262

Replication RolesWithin the cluster, all nodes are equal'master/donor node'

69/262

The node a given transaction was written and committed on.

70/262

The node a given transaction was written and committed on.'slave/joiner node'

71/262

The node that received the given transaction via Galerareplication.

72/262

The terms master and slave in this context are only relevant for agiven transaction

73/262

The terms master and slave in this context are only relevant for agiven transactionWriteset: Galera’s term for a transaction. One or more RBR rowchanges

74/262

State Transfer RolesNew nodes joining an existing cluster get provisioned automatically

75/262

Joiner = Mew node

76/262

Joiner = Mew nodeDonor = Node giving a copy of the datadir

77/262

State Snapshot transfer

78/262

State Snapshot transferFull backup of Donor to Joiner

79/262

Incremental Snapshot transfer

80/262

Incremental Snapshot transferOnly changes since node left cluster

81/262

WritesetRBR payload (black box to Galera)

82/262

WritesetRBR payload (black box to Galera)Replication keys (generated by master/donor node)

83/262

Primary keys

84/262

Primary keysUnique keys

85/262

Primary keysUnique keysForeign Keys

86/262

Primary keysUnique keysForeign KeysTable names

87/262

Primary keysUnique keysForeign KeysTable namesSchema names

88/262

Primary keysUnique keysForeign KeysTable namesSchema names

Keys are what make certification possible

89/262

Replication ?It consists in 4 operations:

ApplyReplicationCertificationCommit

The order differs with the node's role

90/262

Replication Order on Master/Donor1. Apply

2. Replication3. Certification4. Commit

91/262

Replication Order on Slave/Joiner1. Replication (from master/donor)

2. Certification3. Apply4. Commit

92/262

CertificationCan this writeset be applied?

93/262

Based on unapplied earlier transactions on master/donor

94/262

Based on unapplied earlier transactions on master/donorSuch conflicts must come from other nodes

95/262

Happens on every node

96/262

Happens on every nodeShould be deterministic on every node

97/262

Happens on every nodeShould be deterministic on every nodeResults are not reported to the cluster

98/262

Pass: enter apply queue (commit success on master/donor)

99/262

Pass: enter apply queue (commit success on master/donor)Fail: drop transaction (or return deadlock on master/donor)

100/262

Serialized by group communication sequence (and GTID will besynchronized following the same sequence)

101/262

Serialized by group communication sequence (and GTID will besynchronized following the same sequence)Cost based on # of keys or # of rows

102/262

ApplyApply is done on slave nodes after certification

103/262

ApplyApply is done on slave nodes after certificationCan be parallelized

104/262

if wsrep_slave_threads > 1

105/262

if wsrep_slave_threads > 1if there are no other writesets with conflicting keys also beingapplied

106/262

Cost: size of transaction

107/262

Cost: size of transactionGenerates brute force aborts on local node for conflicts

108/262

CommitFinal local InnoDB commit

109/262

i.e., innodb_flush_log_at_trx_commit

110/262

i.e., innodb_flush_log_at_trx_commitGTID gets generated

111/262

i.e., innodb_flush_log_at_trx_commitGTID gets generatedDone by applier threads on slaves/joiners

112/262

i.e., innodb_flush_log_at_trx_commitGTID gets generatedDone by applier threads on slaves/joinersDone by client thread on master/donor

113/262

i.e., innodb_flush_log_at_trx_commitGTID gets generatedDone by applier threads on slaves/joinersDone by client thread on master/donorinnodb_flush_log_at_trx_commit=1 not required generally for PXC!

114/262

Galera Replication (autocommit)

115/262

116/262

117/262

118/262

119/262

120/262

121/262

Galera Replication (full transaction)

122/262

123/262

124/262

125/262

126/262

127/262

128/262

129/262

130/262

Optimistic LockingTraditional locking

131/262

132/262

133/262

134/262

135/262

136/262

Optimistic LockingOptimistic Locking

137/262

138/262

139/262

140/262

141/262

142/262

143/262

Certification Failure

144/262

Trx1 is open on 㫗�㫙�㫘�㫘�㫕�Trx2 is open on 㫗�㫙�㫘�㫘�㫕�

145/262

㫗�㫙�㫘�㫘�㫕� gets 㫖�㫗�㫗�㫗�㫗�㫗�

146/262

Synchronous replication

147/262

Certification testsrun in isolation on each node

148/262

Certification tests:asynchronous

149/262

Synchronous replicationdeterministic

150/262

Certification succeeds

151/262

Certified transaction goes to the apply queue

152/262

153/262

On 㫗�㫙�㫘�㫘�㫕�, a successful cert test, means an actual 㫘�㫙�㫙�㫙�㫘�㫙�

154/262

and transactions in the apply queue (㫗�㫙�㫘�㫘�㫕� & 㫗�㫙�㫘�㫘�㫕�) are executedasynchronously

155/262

On 㫙�㫙�㫘�㫘�㫕� we commit the transaction

156/262

Synchronous replication

157/262

158/262

159/262

160/262

161/262

162/262

163/262

164/262

Brute Force Abort (bfa)

165/262

166/262

167/262

168/262

169/262

170/262

171/262

Local Certification Failure (lcf)

172/262

173/262

174/262

175/262

176/262

177/262

178/262

179/262

180/262

181/262

182/262

183/262

Certification Errors: summary

184/262

185/262

186/262

Flow ControlAbility of any node in the cluster to ask the rest of the nodes topause writes while it catches up

187/262

Flow ControlAbility of any node in the cluster to ask the rest of the nodes topause writes while it catches upFeedback mechanism for replication process

188/262

Flow ControlAbility of any node in the cluster to ask the rest of the nodes topause writes while it catches upFeedback mechanism for replication processONLY caused by 㫙�㫙�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫙�㫘�㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫘� exceeding a node's㫘�㫘�㫘�㫙�㫘�㫙�㫘�㫙�

189/262

Flow ControlAbility of any node in the cluster to ask the rest of the nodes topause writes while it catches upFeedback mechanism for replication processONLY caused by 㫙�㫙�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫙�㫘�㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫘� exceeding a node's㫘�㫘�㫘�㫙�㫘�㫙�㫘�㫙�CAN pause the entire cluster and look like a cluster stall !

190/262

Tuning Flow ControlToo low:

191/262

frequent FC from any and all nodes in the cluster

192/262

frequent FC from any and all nodes in the clusterToo high:

193/262

increase in replication conflicts in multi-node writings

194/262

increase in replication conflicts in multi-node writingsincrease in apply lag on nodes

195/262

increase in replication conflicts in multi-node writingsincrease in apply lag on nodesincrease in commit lag on nodes

196/262

One node with FC issues

197/262

One node with FC issuesdeal with that node -bad hardware? too slow ?

198/262

Flow Control

199/262

Flow Control

200/262

Flow Control

201/262

Flow Control

202/262

Flow Control

203/262

Flow Control

204/262

Flow Control

205/262

Flow Control

206/262

Flow Control

207/262

Flow Control

208/262

Flow Control

209/262

Flow Control

210/262

Flow Control

211/262

Flow Control

212/262

Flow Control

213/262

Flow Control

214/262

Flow Control

215/262

Flow Control

216/262

Flow Control

217/262

Flow Control

218/262

Flow Control

219/262

State TransferThere are two types of State Transfer in Galera:

220/262

1. SST (Snapshot State Transfer): full data copy

221/262

1. SST (Snapshot State Transfer): full data copyrsyncmysqldumpxtrabackup

222/262

2. IST (Incremental State Transfer): only copy the missing events

223/262

It's always better to try to avoid SST!

224/262

It's always better to try to avoid SST!

㫙�㫙�㫙�㫘�㫙�㫘�㫙�㫙�㫙�㫘�㫘�㫙�㫙�㫙�㫙� can be used to specify the donor

225/262

State Transfer: grastate.datWhen MySQL starts it checks 㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫘�㫕�㫘�㫘�㫙� file (in datadir)

226/262

State Transfer: grastate.datWhen MySQL starts it checks 㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫘�㫕�㫘�㫘�㫙� file (in datadir)This file placeholds GTID between MySQL restarts

227/262

State Transfer: grastate.datWhen MySQL starts it checks 㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫘�㫕�㫘�㫘�㫙� file (in datadir)This file placeholds GTID between MySQL restartsContains UUID and Seqno

228/262

㫔�㫔�㫗�㫖�㫗�㫖�㫗�㫖�㫔�㫙�㫘�㫙�㫘�㫘�㫔�㫙�㫙�㫘�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫙�㫕�㫔�㫕�㫕�㫕�㫙�㫙�㫘�㫘�㫕�㫔�㫔�㫔�㫔�㫘�㫘�㫘�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫙�㫘�㫙�㫙�㫙�㫕�㫔�㫔�㫔�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫘�㫙�㫕�

229/262

㫔�㫔�㫗�㫖�㫗�㫖�㫗�㫖�㫔�㫙�㫘�㫙�㫘�㫘�㫔�㫙�㫙�㫘�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫙�㫕�㫔�㫕�㫕�㫕�㫙�㫙�㫘�㫘�㫕�㫔�㫔�㫔�㫔�㫘�㫘�㫘�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫙�㫘�㫙�㫙�㫙�㫕�㫔�㫔�㫔�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫘�㫙�㫕�

node shutdown cleanly

230/262

grastate.dat - example㫔�㫔�㫗�㫖�㫗�㫖�㫗�㫖�㫔�㫙�㫘�㫙�㫘�㫘�㫔�㫙�㫙�㫘�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫙�㫕�㫔�㫕�㫕�㫕�㫙�㫙�㫘�㫘�㫕�㫔�㫔�㫔�㫔�㫘�㫘�㫘�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫙�㫘�㫙�㫙�㫙�㫕�㫔�㫔�㫔�㫕�㫕�㫘�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫘�㫙�㫕�

231/262

grastate.dat - example㫔�㫔�㫗�㫖�㫗�㫖�㫗�㫖�㫔�㫙�㫘�㫙�㫘�㫘�㫔�㫙�㫙�㫘�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫙�㫕�㫔�㫕�㫕�㫕�㫙�㫙�㫘�㫘�㫕�㫔�㫔�㫔�㫔�㫘�㫘�㫘�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫘�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫘�㫕�㫕�㫕�㫘�㫕�㫕�㫕�㫕�㫕�㫘�㫕�㫙�㫘�㫙�㫙�㫙�㫕�㫔�㫔�㫔�㫕�㫕�㫘�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫘�㫙�㫕�

node is running or did not shutdown cleanly

232/262

grastate.dat - example㫔�㫔�㫗�㫖�㫗�㫖�㫗�㫖�㫔�㫙�㫘�㫙�㫘�㫘�㫔�㫙�㫙�㫘�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫙�㫕�㫔�㫕�㫕�㫕�㫙�㫙�㫘�㫘�㫕�㫔�㫔�㫔�㫔�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫔�㫔�㫔�㫔�㫔�㫔�㫔�㫔�㫔�㫙�㫘�㫙�㫙�㫙�㫕�㫔�㫔�㫔�㫕�㫕�㫘�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫘�㫙�㫕�

233/262

grastate.dat - example㫔�㫔�㫗�㫖�㫗�㫖�㫗�㫖�㫔�㫙�㫘�㫙�㫘�㫘�㫔�㫙�㫙�㫘�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫙�㫙�㫕�㫔�㫕�㫕�㫕�㫙�㫙�㫘�㫘�㫕�㫔�㫔�㫔�㫔�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫕�㫔�㫔�㫔�㫔�㫔�㫔�㫔�㫔�㫔�㫙�㫘�㫙�㫙�㫙�㫕�㫔�㫔�㫔�㫕�㫕�㫘�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫘�㫙�㫕�

node aborted, SST on next restart

234/262

State Transfer: ISTWhen a node starts, it knowns the UUID of the cluster it belonged andthe last sequence number it applied.

So, it sends that position to the other members of the cluster and if anode can send the next events (ws/trx), IST will be performed, if none,then SST will be triggered.

235/262

Galera CacheThose events are stored on the 㫘�㫘�㫙�㫘�㫙�㫘�㫕�㫘�㫘�㫘�㫘�㫘� file.

236/262

preallocated file with a specific size

237/262

preallocated file with a specific sizeused to store the writesets in circular buffer style

238/262

preallocated file with a specific sizeused to store the writesets in circular buffer styledefault size is 128M

239/262

preallocated file with a specific sizeused to store the writesets in circular buffer styledefault size is 128Mcan be increase via provider option 㫘�㫘�㫘�㫘�㫘�㫘�㫕�㫙�㫘�㫚�㫘�

240/262

㫙�㫙�㫙�㫘�㫙�㫘�㫙�㫙�㫙�㫙�㫘�㫘�㫘�㫙�㫘�㫙�㫙�㫙�㫘�㫙�㫙�㫙�㫔�㫖�㫔�㫔�㫘�㫘�㫘�㫘�㫘�㫘�㫕�㫙�㫘�㫚�㫘�㫖�㫕�㫗�㫔�

241/262

㫙�㫙�㫙�㫘�㫙�㫘�㫙�㫙�㫙�㫙�㫘�㫘�㫘�㫙�㫘�㫙�㫙�㫙�㫘�㫙�㫙�㫙�㫔�㫖�㫔�㫔�㫘�㫘�㫘�㫘�㫘�㫘�㫕�㫙�㫘�㫚�㫘�㫖�㫕�㫗�㫔�Galare Cache is mmaped (I/O buffered to memory)

242/262

㫙�㫙�㫙�㫘�㫙�㫘�㫙�㫙�㫙�㫙�㫘�㫘�㫘�㫙�㫘�㫙�㫙�㫙�㫘�㫙�㫙�㫙�㫔�㫖�㫔�㫔�㫘�㫘�㫘�㫘�㫘�㫘�㫕�㫙�㫘�㫚�㫘�㫖�㫕�㫗�㫔�Galare Cache is mmaped (I/O buffered to memory)㫙�㫙�㫙�㫘�㫙�㫘�㫙�㫙�㫘�㫘�㫙�㫘�㫘�㫘�㫘�㫘�㫘�㫘�㫘�㫘�㫙�㫙�㫙�㫙�㫙� provide the first seqno present inthe cache for that node

243/262

Galera Cache & IST

244/262

Galera Cache & IST

245/262

Galera Cache & IST

246/262

Galera Cache & IST

247/262

Galera Cache & IST

248/262

Galera Cache & IST

249/262

Galera Cache & IST

250/262

Galera Cache & IST

251/262

Galera Cache & IST

252/262

Galera Cache & IST

253/262

Galera Cache & IST

254/262

Galera Cache & IST

255/262

Galera Cache & IST

256/262

Galera Cache & IST

257/262

Galera Cache & IST

258/262

Galera Cache & IST

259/262

Galera Cache & IST

260/262

Galera Cache & IST

261/262

Thank you !

262/262

Galera Replication Demystified: How Does It Work?

Software

Thermodynamics - Free160592857366.free.fr/joe/ebooks/Mechanical Engineering Books... · Corporate Finance Demystified Databases Demystified Diabetes Demystified ... Thermodynamics

Tema4la galera

ɷIv therapy demystified (demystified nursing)

Galera Cluster: Synchronous Multi-Master Replication for MySQL HA

Fala galera

Demystified Series - stringworld.rustringworld.ru/files/McMahon_D._String_theory_demystified.pdf · Demystified Series Accounting Demystified Advanced Calculus Demystified Advanced

FOSDEM 2012: Synchronoues multi-master replication with Galera

SignServer Enterprise Cloud Edition Cluster Configuration Guide · 2018-11-01 · Galera replication uses the following ports for communication: 3306: For MySQL client connections

Installing Galera Cluster with MariaDB...Installing Galera Cluster with MariaDB Introduction Galera Cluster Overview Installing Software Conﬁguring Nodes Opening Ports Starting Galera

Early Scheduling in Parallel State Machine Replication · MySQL Group Replication Galera Cluster Blockchain, ... Parallel State Machine Replication 5 R1 R4 R2 R5 R3 R0 scheduler worker

Percona XtraDB Cluster€¦ · WSREP and Galera • WSREP API is a project to develop generic replication plugin interface for databases (WriteSet Replication) • Galera is a wsrep

Galera with Docker: How Synchronous Replication and Linux Containers mesh together

AdvancedCalculus - cloudflare-ipfs.com...AS P.NET 2.0 Demystified Aslronomy Demystified Audio Demystified Biology Demystified Biotechnology Demystified Business Calculus Demystified

MySQL Servers Working as a Team - Replication or Galera ... · MySQL Servers Working as a Team - Replication or Galera Cluster Open Source Data Center Conference, April 26th - 28th,

METEOROLOGY DEMYSTIFIEDs2.bitdl.ir/Ebook/Geology/Meteorology Demystified - S... · 2019. 5. 26. · Quantum Mechanics Demystified Relativity Demystified Robotics Demystified Six Sigma

Standard Replication & Galera Cluster...Standard Replication & Galera Cluster Introduction Purpose & Advantages Standard Replication Layout Conﬁguring Replication Galera Basics Conﬁguring

OpenStack High Availability Technologies€¦ · MySQL Galera cluster • Synchronous multi-master cluster for MySQL/InnoDB database • Database replication is not simply replication

MySQL Servers Working as a Team - Replication or Galera ......MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA 1 / 42 MySQL Servers Working

Physics Demystified : A Self-Teaching Guide (Demystified)

Galera Cluster Documentation€¦ · Galera Cluster Documentation, Releases 3.x and 4.x • Flow Control (page 22) Galera Cluster manages the replication process using a feedback