26
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License CASSANDRA LONDON FEBRUARY 2016 LIGHTENING TALK: INTRODUCTION TO INCREMENTAL REPAIR Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton Co-Founder & Team Member

Cassandra London March 2016 - Lightening talk - introduction to incremental repair

Embed Size (px)

Citation preview

Page 1: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

CASSANDRA LONDON FEBRUARY 2016

LIGHTENING TALK: INTRODUCTION TO

INCREMENTAL REPAIR

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

Aaron Morton@aaronmorton

Co-Founder & Team Member

Page 2: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

About The Last Pickle.

Work with clients to deliver and improve Apache Cassandra based

solutions.Based in New Zealand, USA,

Australia, France.

Page 3: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Repair Incremental Repair

Page 4: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Repair

:(

Page 5: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Why Repair?

It is the way to ensure consistent data on disk.

Page 6: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Why Repair?

Hinted Handoff and (non-blocking) Read Repair are

optimisations.

Page 7: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Why Repair?

Consistency ensures Tombstones are distributed,

preventing deleted data returning.

Page 8: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

A Repair Of Two Halves

Step 1:Calculate a hash tree to

describe the data.(aka Merkle Tree)

Page 9: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

A Repair Of Two Halves

Step 2:Stream differences between

nodes.

Page 10: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

gc_grace_seconds

Repair will ignore Tombstones older than gc_grace_seconds.

So run repair every gc_grace_seconds.*

Page 11: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

RepairIncremental Repair

Page 12: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Why Incremental Repairs?

SSTables are immutable, so repair was processing data

that had not changed.

Page 13: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Why Incremental Repairs?

Repairing less data, more frequently, reduces the impact

on the cluster.

Page 14: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

repairedAt

Track repaired timestamp per SSTable.

Ignore SSTables that have been repaired.

Page 15: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

repairedAt

Set after when all data in the SSTable was repaired.

Page 16: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

repairedAt$ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 0

$ bin/nodetool repair --incremental --parallel

$ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 1454500502306

Page 17: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Anti Compaction

Splits partially repaired SSTables into Repaired and

Un-repaired SSTables.

Page 18: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Anti Compaction$ ls -lah keyspace1/standard1 keyspace1-standard1-ka-9-Data.db keyspace1-standard1-ka-10-Data.db

$ bin/nodetool repair --incremental --parallel -st 3074457345618258602 -et -9223372036854775808

$ ls -lah keyspace1/standard1 keyspace1-standard1-ka-11-Data.db keyspace1-standard1-ka-12-Data.db keyspace1-standard1-ka-13-Data.db keyspace1-standard1-ka-14-Data.db

Page 19: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Anti Compaction$ tools/bin/sstablemetadata keyspace1-standard1-ka-11-Data.db SSTable: ... Repaired at: 1454504607623

$ tools/bin/sstablemetadata keyspace1-standard1-ka-12-Data.db SSTable: ... Repaired at: 0

$ tools/bin/sstablemetadata keyspace1-standard1-ka-13-Data.db SSTable: ... Repaired at: 1454504607623

$ tools/bin/sstablemetadata keyspace1-standard1-ka-14-Data.db SSTable: ... Repaired at: 0

Page 20: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Compaction

Two sets of SSTables are maintained: Repaired and Un-

repaired.

Page 21: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Compaction

Compaction Strategy runs in each SSTable set.

Page 22: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Migration Option 1

1. Disable compaction.2. Run a classic full repair3. Stop the node4. stablerepairedset to mark all the SSTables from before step 1

Page 23: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Migration Option 2

bin/nodetool repair --incremental --parallel

Page 24: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

More Information

http://www.datastax.com/dev/blog/more-efficient-repairs

Page 25: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Thanks.

Page 26: Cassandra London March 2016  - Lightening talk - introduction to incremental repair

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

Aaron Morton@aaronmorton

Co-Founder & Team Memberwww.thelastpickle.com

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License