Upload
aaronmorton
View
478
Download
0
Embed Size (px)
Citation preview
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
CASSANDRA LONDON FEBRUARY 2016
LIGHTENING TALK: INTRODUCTION TO
INCREMENTAL REPAIR
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
Aaron Morton@aaronmorton
Co-Founder & Team Member
About The Last Pickle.
Work with clients to deliver and improve Apache Cassandra based
solutions.Based in New Zealand, USA,
Australia, France.
Repair Incremental Repair
Repair
:(
Why Repair?
It is the way to ensure consistent data on disk.
Why Repair?
Hinted Handoff and (non-blocking) Read Repair are
optimisations.
Why Repair?
Consistency ensures Tombstones are distributed,
preventing deleted data returning.
A Repair Of Two Halves
Step 1:Calculate a hash tree to
describe the data.(aka Merkle Tree)
A Repair Of Two Halves
Step 2:Stream differences between
nodes.
gc_grace_seconds
Repair will ignore Tombstones older than gc_grace_seconds.
So run repair every gc_grace_seconds.*
RepairIncremental Repair
Why Incremental Repairs?
SSTables are immutable, so repair was processing data
that had not changed.
Why Incremental Repairs?
Repairing less data, more frequently, reduces the impact
on the cluster.
repairedAt
Track repaired timestamp per SSTable.
Ignore SSTables that have been repaired.
repairedAt
Set after when all data in the SSTable was repaired.
repairedAt$ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 0
$ bin/nodetool repair --incremental --parallel
$ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 1454500502306
Anti Compaction
Splits partially repaired SSTables into Repaired and
Un-repaired SSTables.
Anti Compaction$ ls -lah keyspace1/standard1 keyspace1-standard1-ka-9-Data.db keyspace1-standard1-ka-10-Data.db
$ bin/nodetool repair --incremental --parallel -st 3074457345618258602 -et -9223372036854775808
$ ls -lah keyspace1/standard1 keyspace1-standard1-ka-11-Data.db keyspace1-standard1-ka-12-Data.db keyspace1-standard1-ka-13-Data.db keyspace1-standard1-ka-14-Data.db
Anti Compaction$ tools/bin/sstablemetadata keyspace1-standard1-ka-11-Data.db SSTable: ... Repaired at: 1454504607623
$ tools/bin/sstablemetadata keyspace1-standard1-ka-12-Data.db SSTable: ... Repaired at: 0
$ tools/bin/sstablemetadata keyspace1-standard1-ka-13-Data.db SSTable: ... Repaired at: 1454504607623
$ tools/bin/sstablemetadata keyspace1-standard1-ka-14-Data.db SSTable: ... Repaired at: 0
Compaction
Two sets of SSTables are maintained: Repaired and Un-
repaired.
Compaction
Compaction Strategy runs in each SSTable set.
Migration Option 1
1. Disable compaction.2. Run a classic full repair3. Stop the node4. stablerepairedset to mark all the SSTables from before step 1
Migration Option 2
bin/nodetool repair --incremental --parallel
More Information
http://www.datastax.com/dev/blog/more-efficient-repairs
Thanks.
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
Aaron Morton@aaronmorton
Co-Founder & Team Memberwww.thelastpickle.com
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License