Scaling SolrCloud to a large number of Collections
Shalin Shekhar Mangar, Lucidworks Inc. [email protected]
twitter.com/shalinmangar
Apache Solr has a huge install base and tremendous momentum.
SOLR most widely used search solution on the planet. 8M+
total downloads
Solr is both established & growing
250,000+monthly downloads
Solr has tens of thousands of applications in production.
You use Solr everyday.
Largest community of developers.
2500+open Solr jobs.
Solr scalability is unmatched.
• box.com (Dropbox for business) • 10TB+ Index Size • 10 Billion+ Documents • 100 Million+ Daily Requests
Solr scalability is unmatched.
The traditional search use-case
• One large index distributed across multiple nodes
• A large number of users sharing the data
• Searches across the entire cluster
Example: Product CatalogMust search across all products
Subset of optional features in Solr to enable and simplify horizontal scaling a search index
using sharding and replication. !
Goals scalability, performance, high-availability,
simplicity, and elasticity
What is SolrCloud?
Terminology• ZooKeeper: Distributed coordination service that provides centralised configuration,
cluster state management, and leader election
• Node: JVM process bound to a specific port on a machine
• Collection: Search index distributed across multiple nodes with same configuration
• Shard: Logical slice of a collection; each shard has a name, hash range, leader and replication factor. Documents are assigned to one and only one shard per collection using a hash-based document routing strategy
• Replica: A copy of a shard in a collection
• Overseer: A special node that executes cluster administration commands and writes updated state to ZooKeeper. Automatic failover and leader election.
Collection with 2 shards across 4 nodes with replication factor 2
iv
Jetty (node 2, port 8984)
Solr webapp
logstash4solr shard1 Replica
Java VM
iv
Jetty (node 1, port 8983)
Solr webapp
logstash4solr shard1 Leader
Java VM
iv
Jetty (node 4, port 8986)
Solr webapp
logstash4solr shard2 Replica
Java VM
iv
Jetty (node 3, port 8985)
Solr webapp
logstash4solr shard2 Leader
Java VM
Sharding
Replication
Replication
Zookeeper 1
Zookeeper 2
Zookeeper 3
Leader ElectionCentralized
configuration management
ZooKeeper Ensemble
HTTP APIs XML/JSON/CSV/PDF Java/Ruby/Python/PHP
Millions of documents, millions of users
“The limits of the possible can only be defined by going beyond them into the impossible” — Arthur C. Clarke
The curious case of multi-tenant platforms
• Multi-tenant platform for storage and search
• Thousands of tenant applications
• Each tenant application has millions of users
One SolrCloud collection per tenant• Searches are specialised to a user’s data or the
tenant application’s dataset
• Some tenants create a lot of data, others very little
• Some use CPU intensive geo-spatial queries, some just perform simple full text searches and sorting
• Some are write-heavy, others read-heavy
• Some have text in a different natural language
Measure and optimise
• Analyze and find missing features
• Setup a performance testing environment on AWS
• Devise tests for stability and performance
• Find bugs and bottlenecks and fix ’em
Problem #1: Cluster state and updates• The SolrCloud cluster state has information about the
collections, their shards and replicas
• All nodes and (Java) clients watch the cluster state
• Every state change is notified to all nodes
• Limited to (slightly less than) 1MB by default
• 1 node bounce triggers a few 100 watcher fires and pulls from ZK for a 100 node cluster (three states: down, recovering, active)
Solution - Split cluster state and scale
• Each collection gets it’s own state node in ZK
• Nodes selectively watch only those states which they are a member of
• Clients cache state and use smart cache updates instead of watching nodes
• http://issues.apache.org/jira/browse/SOLR-5473
Problem #2: Overseer performance
• Thousands of collections create a lot of state updates
• Overseer falls behind and replicas can’t recover or can’t elect a leader
• Under high indexing/search load, GC pauses can cause overseer queue to back up
Solution - Improve the overseer• Harden the overseer code against ZooKeeper
connection loss (SOLR-5325)
• Optimise polling for new items in overseer queue (SOLR-5436)
• Dedicated overseers nodes (SOLR-5476)
• New Overseer Status API (SOLR-5749)
• Asynchronous execution of collection commands (SOLR-5477, SOLR-5681)
Problem #3: Moving data around• Not all users are born equal - A tenant may have a
few very large users
• We wanted to be able to scale an individual user’s data — maybe even as it’s own collection
• SolrCloud can split shards with no downtime but it only splits in half
• No way to ‘extract’ user’s data to another collection or shard
Solution: Improved data management
• Shard can be split on arbitrary hash ranges (SOLR-5300)
• Shard can be split by a given key (SOLR-5338, SOLR-5353)
• A new ‘migrate’ API to move a user’s data to another (new) collection without downtime (SOLR-5308)
Problem #4: Exporting data
• Lucene/Solr are designed for finding top-N search results
• Trying to export full result set brings down the system due to high memory requirements as you go deeper
Solution - Distributed deep paging
New ‘cursorMark’ feature for deep paging (SOLR-5463)
–twitter.com/UweSays
“The JVM is completely irresponsible and can only be killed with ‘kill -9’”
JVM Bugs!
“Testing scale” at scale
• Performance goals: 6 billion documents, 4000 queries/sec, 400 updates/sec, 2 seconds NRT sustained performance
• 5% large collections (50 shards), 15% medium (10 shards), 85% small (1 shard) with replication factor of 3
• Target hardware: 24 CPUs, 126G RAM, 7 SSDs (460G) + 1 HDD (200G)
• 80% traffic served by 20% of the tenants
How to manage large SolrCloud clusters
• Developed Solr Scale Toolkit
• Fabric based tool to setup and manage SolrCloud clusters in AWS complete with collectd and SiLK
• Backup/Restore from S3. Parallel clone commands.
• Open source!
• https://github.com/LucidWorks/solr-scale-tk
Gathering metrics and analysing logs• LucidWorks SiLK (Solr + Logstash + Kibana)
• collectd daemons on each host
• rabbitmq to queue messages before delivering to log stash
• Initially started with Kafka but discarded thinking it is overkill
• Not happy with rabbitmq — crashes/unstable
• Might try Kafka again soon
• http://www.lucidworks.com/lucidworks-silk
Generating data and load• Custom randomized data generator (re-producible
using a seed)
• JMeter for generating load
• Embedded CloudSolrServer (Solr Java client) using JMeter Java Action Sampler
• JMeter distributed mode was itself a bottleneck!
• Not open source (yet) but we’re working on it!
Numb3rs• 30 hosts, 120 nodes, 1000 collections, 8B+ docs,
15000 queries/second, 2000 writes/second, 2 second NRT sustained over 24-hours
• More than 3x the numbers our client needed
• Unfortunately, we had to stop testing at that point :(
• Turned out they had a 95-5 traffic ratio than a 80-20 ratio so actual performance is even better :)
• Our biggest cluster cost us just $120/hour :)
Not over yet• We continue to test performance at scale
• Published indexing performance benchmark, working on others
• 15 nodes, 30 shards, 1 replica, 157195 docs/sec
• 15 nodes, 30 shards, 2 replicas, 61062 docs/sec
• http://searchhub.org/introducing-the-solr-scale-toolkit/
Our users are also pushing the limits
https://twitter.com/bretthoerner/status/476830302430437376
Up, up and away!
https://twitter.com/bretthoerner/status/476838275106091008
Not over yet• SolrCloud continues to be improved
• SOLR-6220 - Replica placement strategy
• SOLR-6273 - Cross data center replication
• SOLR-5656 - Auto-add replicas
• SOLR-5986 - Don’t allow runaway queries to harm the cluster
• Many, many more
Questions?• Shalin Shekhar Mangar
• twitter.com/shalinmangar
• meetup.com/Bangalore-Apache-Solr-Lucene-Group/
• www.meetup.com/Bangalore-Baby-Apache-Solr-Group/