The Internet in Database: A Cassandra Use Case

The Internet in a DatabaseA Cassandra Use Case

Data on the Web

DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET

● 48 billion pages on the Internet

● 56 million GB of data

● Incredibly powerful connections

● 70% of useful data is unstructured

● User generated data + facts


Too Much Data…


● Modern search engines

○ Unstructured data

○ Unconnected data

○ Unnormalized data

Search


● Goals

○ Collect vast amounts of data through web crawling

○ Normalize and deduplicate data

○ Make it searchable and meaningful


● Speed

● Scale

● Adaptable

Needs

● Very fast

○ Log-structured storage

● Easily scalable

○ Decentralized rings

● Completely adaptable

○ Schema-less key/value store


The Solution


…Almost

● Useful searching was missing

○ Secondary indexes not flexible

○ No free text searches

○ No (reasonable) range queries


● Pros: Full control over indexing

● Cons: Not scalable

What We Needed


● Reasons to go with DSE

○ Combines Cassandra and Solr

○ Constant refinements and integrations

○ Support

Putting It All Together


Normalization

Cassandra

Solr

Cassandra

Solr

Cassandra

Solr

Load Balancing

Our Stack

Web Crawling


Cassandra / Solr Setup

● 3 column families / 3 cores

○ Locations○ Products○ People

● 73,114,909 records


● 29,818,644 records

● Interesting data

○ Reviews○ Revenue○ Contact information

● Businesses vs. Locations

○ Unique key○ Location specific user data

Data: Locations


Data: Products

● 18,470,005 records


○ Categories○ Price○ Reviews

● Challenges

○ Too many unique keys


Data: People

● 24,826,260 records


○ Work History○ Education History○ Location

● Challenges

○ Normalization○ Identification


Challenges

● Memory

● Speed

● Space

● Representation


Challenges: Memory

● Multi-minute garbage collection

● Exponential increase in frequency

● Virtual memory confusion

● Solr + Cassandra

● Heap Size vs Buffer Cache

● Bash Scripts


Challenges: Speed

● Upgrade

○ Better memory management○ Smaller index size

● Reduce index size

● Future: Solaris


Challenges: Speed

● Providing a real-time service

● Issues

○ Solr not inherently real time○ Search speeds○ I/O


Challenges: Speed

● Solr Solution: DSE integration leverages

○ Cassandra's speed○ Cassandra's caches○ Cassandra's distribution○ Solr caches less useful


Challenges: Speed

● Search complexity solution

○ Text vs String indexing○ Uniqueness vs Flexibility○ Leveraging Cassandra


Challenges: Speed

● I/O Solution

○ Cassandra's built in mapping○ Increase disk access speeds (SSDs)

■ Not cost effective○ Future: Solaris


Challenges: Space

● Field corruption

○ Caused by improper encoding○ Exponential growth○ Fills up Solr index

● Locate, inspect & remove corrupt records


Challenges: Space

● Solr index issue

○ No compression (vs Cassandra)○ Must adjust indexing

● Key things to keep in mind

○ Size of fields○ Scale vs Flexibility○ Index as little as possible


Challenges: Representation

● Cassandra is flat

● Actual data is not flat

○ Reviews○ Price information

● Many different output formats

○ CSV, JSON, XML, etc.


● Solution: Flatten when possible

○ E.g. Address object -> Separate fields● Internal subgroup representation

○ Composite keys (Occasionally)■ Known subgroups■ Non multiple subgroups

○ Dynamic fields■ Composite field + Dynamic tag■ E.g. review.text_<tag>




● Robust and adaptable conversion package

● JSON -> Internal

○ Solr returns JSON● Internal -> CSV, JSON, XML

○ User defined views○ Specify field groupings○ Specify partitioning



● Memory Usage

● Speed

● Space

● Containers

Future Work


Future Work: Memory

● Java 7 G1 (Garbage First) Collector

○ Ideal for large heaps○ Big Data Sets○ Bursty Workloads


Future Work: Speed

● Solaris Kernel Scheduler > Linux Kernel Scheduler

○ (At large number of cores)● Drastically increase iops

○ Cache reads (L2ARC) on PCIe SSD (~800 MB/s)○ Cache writes (ZIL) on PCIe SSD (~800 MB/s)○ Reduce needed size of SSD

■ More smaller SSDs in ZFS pool○ Fewer moving parts


Future Work: Space

● Caching at PCIe, Storing on SATA III

○ Cheaper larger storage via ZFS pools○ Easier to grow

● ZFS Compression (LZ4)

○ Replaces Cassandra's Snappy compression○ Very fast lossless compression (400 Mb/s per core)○ Scales to multiple CPUs○ Hits the ram speed limit


Future Work: Containers

● OS Level virtualization

○ Resource control○ Boundary separation

● More control over cassandra resources

● Better snapshots (whole machine)

● Hardware abstracted out

○ Many disks represented as single space○ Easily add or remove hardware

Questions?https://www.datafiniti.net

http://blog.datafiniti.net@datafiniti


Addendum 1

ZFS Comparison

Name Ratio (MB/s) Compression (MB/s)

Decompression (MB/s)

LZ4 (r97) 2.084 410 1810

LZO 2.06 2.106 409 600

QuickLZ 1.5.1b6 2.237 373 420

Snappy 1.1.0 2.091 323 1070

LZF 2.077 270 570

zlib 1.2.8 -1 2.730 65 280

LZ4 HC (r97) 2.720 25 2040

zlib 1.2.8 -6 3.099 21 300

Technology

The Internet in Database: A Cassandra Use Case