Manage your compactions before they manage you!

Manage your compactions before they manage you!

About Pythian

18 Years of Data infrastructure• management consulting• 200+ Top brands• 6000+ databases under• management• Over 300 DBA’s, in 29

countries• Top 5% of DBA work force,

9 • Oracle ACE’s, 2 Microsoft

MVP’• s, 1 Cassandra MVP• Oracle, Microsoft, MySQL,• Datastax partners,

Netezza,• Hadoop and MongoDB plus• UNIX Sysadmin and Oracle • apps

2© 2015. All Rights Reserved.

About Me

• Cassandra Consultant– First contact was 0.8

• Cassandra MVP & Datastax Certified Architect

• Lisbon Cassandra Meetup• Passion for distributed

systems• Loves a good challenge• Waterpolo is my sport

• @cjrolo3© 2015. All Rights Reserved.

1 Why Compact

2 Compaction strategies and tuning

3 General tuning

4 Take aways, compacted

5 Q&A


Compaction Impact

• High I/O usage• Temporary increase on disk space• High CPU usage• Increase latency on operations!


Commonly heard

• "My system is compacting 100% of the time"• "All disk I/O is used by compaction"• "Compaction is far behind"• "Hundreds or thousands of SSTables!"


• Quick recap on the write path– Commitlog -> Memtable -> SSTable– SSTables are immutable

Why do we compact?


• Tombstones• Row duplicates• Rows spread across multiple SSTables• Consolidation of data is imperative

Why do we compact? (2)


Size-tiered compaction

• The original compaction type!• "This strategy triggers a minor compaction when there are a

number of similar sized SSTables on disk as configured by the table subproperty, min_threshold. A minor compaction does not involve all the tables in a keyspace"

• When to use: write once data, write heavy scenarios, limited I/O available


Tuning size-tiered

• CQL Properties:– bucket_high– bucket_low– cold_reads_to_omit– max_threshold– min_threshold– min_sstable_size


Levelled compaction

• Strategy appeared in Cassandra 1.0. • "The leveled compaction strategy creates SSTables of a fixed,

relatively small size (160 MB by default) that are grouped into levels. Within each level, SSTables are guaranteed to be non-overlapping. Each level (L0, L1, L2 and so on) is 10 times as large as the previous."

• When to use: Random reads, Reads that are latency sensitive, highly updated rows


Tuning levelled compaction


• CQL Properties:– sstable_size_in_mb

Time-series compaction


• Available since Cassandra 2.0.11, 2.1.1• "DateTieredCompactionStrategy stores data written within a

certain period of time in the same SSTable"• When to use? Time series data!

• Time-Series Tuning


• CQL Properties:– base_time_seconds– max_sstable_age_days– max_threshold– min_threshold– timestamp_resolution

DTCS Quirks...

• Out-of-order data– Hints– Clients not in sync– Repairs– Someone inserted out-of-order data...


Monitoring Compaction

• CFStats• Nodetool Compactionstats• CompactionManagerMBean:

– CompletedTasks: Number of completed compactions since the last start of this Cassandra instance

– PendingTasks: Number of estimated tasks remaining to perform– ColumnFamilyInProgress: The table currently being compacted.– BytesTotalInProgress: Total number of data bytes (index and filter are not included) being

compacted.– BytesCompacted: The progress of the current compaction.

• Strace• iotop


Disk Tuning

• Compaction means large I/O• Big RAID stripes• SSDs!!• Dedicated non-striped disks• No SAN/NAS• I/O scheduler can have some impact• Some linux settings can be used for emergencies.


Take Aways, compacted

• What did we learn here...– Selecting the proper compaction strategy can improve your cluster

performance• Doing the opposite can create serious issues...

– Monitor your compactions!– You can try compactions strategies out without changing your tables!


Q&A

• Thanks for listening!• Questions?


Thank you

Software

Manage your compactions before they manage you!