Manage your compactions before they manage you!

  • Published on
    08-Feb-2017

  • View
    314

  • Download
    1

Embed Size (px)

Transcript

Manage your compactions before they manage you!

About Pythian18 Years of Data infrastructuremanagement consulting200+ Top brands6000+ databases undermanagementOver 300 DBAs, in 29 countriesTop 5% of DBA work force, 9 Oracle ACEs, 2 Microsoft MVPs, 1 Cassandra MVPOracle, Microsoft, MySQL,Datastax partners, Netezza,Hadoop and MongoDB plusUNIX Sysadmin and Oracle apps

2015. All Rights Reserved.

About MeCassandra ConsultantFirst contact was 0.8Cassandra MVP & Datastax Certified ArchitectLisbon Cassandra MeetupPassion for distributed systemsLoves a good challengeWaterpolo is my sport

@cjrolo

2015. All Rights Reserved.

1Why Compact2Compaction strategies and tuning3General tuning4Take aways, compacted5Q&A

2015. All Rights Reserved.

Compaction ImpactHigh I/O usageTemporary increase on disk spaceHigh CPU usageIncrease latency on operations!

2015. All Rights Reserved.

Commonly heard"My system is compacting 100% of the time""All disk I/O is used by compaction""Compaction is far behind""Hundreds or thousands of SSTables!"

2015. All Rights Reserved.

Quick recap on the write pathCommitlog -> Memtable -> SSTableSSTables are immutable

Why do we compact?

2015. All Rights Reserved.

TombstonesRow duplicatesRows spread across multiple SSTablesConsolidation of data is imperativeWhy do we compact? (2)

2015. All Rights Reserved.

Size-tiered compactionThe original compaction type!"This strategy triggers a minor compaction when there are a number of similar sized SSTables on disk as configured by the table subproperty, min_threshold. A minor compaction does not involve all the tables in a keyspace"When to use: write once data, write heavy scenarios, limited I/O available

2015. All Rights Reserved.

Tuning size-tieredCQL Properties:bucket_highbucket_lowcold_reads_to_omitmax_thresholdmin_thresholdmin_sstable_size

2015. All Rights Reserved.

Levelled compactionStrategy appeared in Cassandra 1.0. "The leveled compaction strategy creates SSTables of a fixed, relatively small size (160 MB by default) that are grouped into levels. Within each level, SSTables are guaranteed to be non-overlapping. Each level (L0, L1, L2 and so on) is 10 times as large as the previous."When to use: Random reads, Reads that are latency sensitive, highly updated rows

2015. All Rights Reserved.

Tuning levelled compaction

2015. All Rights Reserved. CQL Properties:sstable_size_in_mb

Time-series compaction

2015. All Rights Reserved. Available since Cassandra 2.0.11, 2.1.1"DateTieredCompactionStrategy stores data written within a certain period of time in the same SSTable"When to use? Time series data!

Time-Series Tuning

2015. All Rights Reserved. CQL Properties:base_time_secondsmax_sstable_age_daysmax_thresholdmin_thresholdtimestamp_resolution

DTCS Quirks...Out-of-order dataHintsClients not in syncRepairsSomeone inserted out-of-order data...

2015. All Rights Reserved.

Monitoring CompactionCFStatsNodetool CompactionstatsCompactionManagerMBean:CompletedTasks: Number of completed compactions since the last start of this Cassandra instancePendingTasks: Number of estimated tasks remaining to performColumnFamilyInProgress: The table currently being compacted.BytesTotalInProgress: Total number of data bytes (index and filter are not included) being compacted.BytesCompacted: The progress of the current compaction. Straceiotop

2015. All Rights Reserved.

Disk TuningCompaction means large I/OBig RAID stripesSSDs!!Dedicated non-striped disksNo SAN/NASI/O scheduler can have some impactSome linux settings can be used for emergencies.

2015. All Rights Reserved.

Take Aways, compactedWhat did we learn here...Selecting the proper compaction strategy can improve your cluster performanceDoing the opposite can create serious issues...Monitor your compactions!You can try compactions strategies out without changing your tables!

2015. All Rights Reserved.

Q&AThanks for listening!Questions?

2015. All Rights Reserved.

Thank you