Upload
sematext-group-inc
View
4.102
Download
0
Embed Size (px)
Citation preview
OCTOBER 11-14, 2016 • BOSTON, MA
Tuning Solr and its Pipeline for LogsRafał Kuć and Radu Gheorghe
Software Engineers, Sematext Group, Inc.
3
01Agenda
Designing a Solr(Cloud) cluster for time-series data
Solr and operating system knobs to tune
Pipeline patterns and shipping options
4
01Time-based collections, the single best improvement
14.10
indexing
5
01Time-based collections, the single best improvement
14.10
indexing
15.10
6
01Time-based collections, the single best improvement
14.10
indexing
15.10
7
01Time-based collections, the single best improvement
14.10 15.10 ... 21.10
indexing
8
01Time-based collections, the single best improvement
14.10 15.10 ... 21.10
indexing
Less merging ⇒ faster indexing
Quick deletes (of whole collections)
Search only some collections
Better use of caches
9
01Load is uneven
BlackFriday
SaturdaySunday
10
01Load is uneven
Need to “rotate” collections fast enough to work with this (otherwise indexing and
queries will be slow)
These will be tiny
BlackFriday
SaturdaySunday
11
01
If load is uneven, daily/monthly/etc indices are suboptimal:
you either have poor performance or too many collections
Octi* is worried:
* this is Octi →
12
01Solution: rotate by size
indexing
logs01
Size limit
13
01Solution: rotate by size
indexing
logs01
Size limit
14
01Solution: rotate by size
indexing
logs01logs02
Size limit
15
01Solution: rotate by size
indexing
logs01logs02
Size limit
16
01Solution: rotate by size
indexing
logs01
logs08...logs02
17
01Solution: rotate by size
indexingPredictable indexing and search performance
Fewer shards
logs01
logs08...logs02
18
01Dealing with size-based collections
logs01
logs08...logs02
app (caches results)
stats
2016-10-11 2016-10-13 2016-10-12 2016-10-14
2016-10-18doesn’t matter,
it’s the latest collection
19
01
Size-based collections handle spiky load better
Octi concludes:
20
01Tiered cluster (a.k.a. hot-cold)
14 Oct
11 Oct
10 Oct
12 Oct
13 Oct
hot01 cold01 cold02
indexing, most searches longer-running (+cached) searches
21
01Tiered cluster (a.k.a. hot-cold)
14 Oct
11 Oct
10 Oct
12 Oct
13 Oct
hot01 cold01 cold02
indexing, most searches longer-running (+cached) searches
⇒ Good CPU and IO* ⇒ Heap. Decent IO for replication&backup
* Ideally local SSD; avoid network storage unless it’s really good
22
01Octi likes tiered clusters
Costs: you can use different hardware for different workloadsPerformance (see costs): fewer shards, less overheadIsolation: long-running searches don’t slow down indexing
23
01AWS specificsHot tier:
c3 (compute optimized) + EBS and use local SSD as cache* c4 (EBS only)
Cold tier:d2 (big local HDDs + lots of RAM)m4 (general purpose) + EBSi2 (big local SSDs + lots of RAM)
General stuff:EBS optimizedEnhanced NetworkingVPC (to get access to c4&m4 instances)
* Use --cachemode writeback for async writing: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/lvm_cache_volume_creation.html
PIOPS is best but expensive
HDD - too slow (unless cold=icy)
⇒ General purpose SSDs
24
01EBS
Stay under 3TB. More smaller (<1TB) drives in RAID0 give better, but shorter IOPS bursts
Performance isn’t guaranteed ⇒ RAID0 will wait for the slowest disk
Check limits (e.g. 160MB/s per drive, instance-dependent IOPS and network)
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.htmlhttp://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSPerformance.html
3 IOPS/GB up to 10K (~3TB), up to 256kb/IO, merges up to 4 consecutive IOs
25
01Octi’s AWS top picks
c4s for the hot tier: cheaper than c3s with similar performance
m4s for the cold tier: well balanced, scale up to 10xl, flexible storage via EBS
EBS drives < 3TB, otherwise avoids RAID0: higher chances of performance drop
26
01Scratching the surface of OS options
Say no to swap
Disk scheduler: CFQ for HDD, deadline for SSD
Mount options: noatime, nodiratime, data=writeback, nobarrier
For bare metal: check CPU governor and THP*
because strict ordering is for the weak
* often it’s enabled, but /proc/sys/vm/nr_hugepages is 0
27
01Schema and solrconfig
Auto soft commit (5s?)
Auto commit (few minutes?)
RAM buffer size + Max buffered docs
Doc Values for faceting+ retrieving those fields (stored=false)
Omit norms, frequencies and positions
Don’t store catch-all field(s)
28
01Relaxing the merge policy*Merges are the heaviest part of indexing
Facets are the heaviest part of searching
Facets (except method=enum) depend on data size more than # of segments
Knobs:
segmentsPerTier: more segments ⇒ less merging
maxMergeAtOnce < segmentsPerTier, to smooth out those IO spikes
maxMergedSegmentMB: lower to merge more small segments, less bigger ones
* unless you only do “grep”. YMMV, of course. Keep an eye on open files, though
⇒ fewer open files
29
01Some numbers: more segments, more throughput (10-15% here)
10 segmentsPerTier10 maxMergeAtOnce
50 segmentsPerTier50 maxMergeAtOnce
need to rotate before this drop
30
01Lower max segment (500MB from 5GB default)
less CPU fewer segments
31
01There’s more...
SPM screenshots from all tests + JMeter test plan here:
https://github.com/sematext/lucene-revolution-samples/tree/master/2016/solr_logging
We’d love to hear about your own results!
correct spelling: sematext.com/spm
32
01
Increasing segments per tier while decreasing max merged segment (by an order of magnitude) makes indexing better and
search latency less spiky
Octi’s conclusions so far
33
01Optimize I/O and CPU by not optimizing
Unless you have spare CPU & IO (why would you?)
And unless you run out of open files
Only do this on “old” indices!
34
01Optimizing the pipeline*
logslog shipper(s)
Ship using which protocol(s)?
Buffer
Route to other destinations?
Parse
* for availability and performance/costs
Or log to Solr directly from app(i.e. implement a new, embedded log shipper)
35
01A case for buffers
performance & availability
allows batches and threads when Solr is down or can’t keep up
36
01Types of buffers
Disk*, memory or a combination
On the logging host or centralized
* doesn’t mean it fsync()s for every message
file or local log shipperEasy scaling; fewer moving partsOften requires a lightweight shipper
Kafka/Redis/etc or central log shipperExtra features (e.g. TTL)One place for changes
bufferapp
bufferapp
37
01Multiple destinations
* or flat files, or S3 or...
input buffer
processing
Outputs need to be in sync
input Processing may cause backpressure
processing
*
38
01Multiple destinations
input
Solroffset
HDFSoffset
input
processing
processing
39
01
Just Solr and maybe flat files? Go simple with a local shipper
Custom, fast-changing processing & multiple destinations? Kafka as a central buffer
Octi’s pipeline preferences
40
01Parsing unstructured data
Ideally, log in JSON*
Otherwise, parse
* or another serialization format
For performance and maintenance(i.e. no need to update parsing rules)
Regex-based (e.g. grok)Easy to build rulesRules are flexibleTypically slow & O(n) on # of rules, but:
Move matching patterns to the top of the listMove broad patterns to the bottomSkip patterns including others that didn’t match
Grammar-based(e.g. liblognorm, PatternDB)
Faster. O(1) on # of rulesNumbers in our 2015 session: sematext.com/blog/2015/10/16/large-scale-log-analytics-with-solr/
41
01Decide what gives when buffers fill up
input shipper
Can drop data here, but better buffer
when shipper buffer is full, app can block or drop data
Check:Local files: what happens when files are rotated/archived/deleted?UDP: network buffers should handle spiky load*TCP: what happens when connection breaks/times out?UNIX sockets: what happens when socket blocks writes**?
* you’d normally increase net.core.rmem_max and rmem_default
** both DGRAM and STREAM local sockets are reliable (vs Internet ones, UDP and TCP)
42
01
Octi’s flow chart of where to log
critical?
UDP. Increase network buffers on destination, so it can handle spiky
traffic
Paying with RAM or IO?
UNIX socket. Local shipper with memory
buffers, that can drop data if needed
Local files. Make sure rotation is in place or you’ll run out of disk!
no
IO RAM
yes
43
01Protocols
UDP: cool for the app, but not reliable
TCP: more reliable, but not completely
Application-level ACKs may be needed:
No failure/backpressure handling needed
App gets ACK when OS buffer gets it ⇒ no retransmit if buffer is lost*
* more at blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html
sender receiverACKs
Protocol Example shippers
HTTP Logstash, rsyslog, Fluentd
RELP rsyslog, Logstash
Beats Filebeat, Logstash
Kafka Fluentd, Filebeat, rsyslog, Logstash
44
01
Octi’s top pipeline+shipper combos
rsyslog
app UNIX socket
HTTP
memory+disk buffercan drop
app
fileKafka
consumer
Filebeat
HTTP
simple & reliable
45
01Conclusions, questions, we’re hiring, thank you
The whole talk was pretty much only conclusions :)
Feels like there’s much more to discover. Please test & share your own nuggets
http://www.relatably.com/m/img/oh-my-god-memes/oh-my-god.jpg
Scary word, ha? Poke us:@kucrafal @radu0gheorghe @sematext...or @ our booth here