Download pdf - Direct Scale-Out Flash for Splunk - unthinkablestorage.com · Apeiron Architecture •The ADS1000 was designed ground-up to leverage applications such as Splunk, which are fully storage

Direct Scale-Out Flash for Splunk

7/25/2017 Apeiron Proprietary and Confidential 1

2

Apeiron Addresses

Apeiron was founded to address three inflection points in the storage market:

• Move to application managed scale out storage - Applications understand internal storage protocols

• Demand for real-time queries on massive data sets - Captive storage cannot manage the required capacity, storage networking is required

• The rapid adoption of NVMe as the standard SSD interface – NVMeperformance creates two new bottlenecks; the controller and the fabric protocol

05/03/2017

Apeiron’s NVMe over Ethernet technology is the ONLY Generally Available product capable of networking not only NVMe, but Intel’s next generation storage: Optane SSD

3

All mentioned brand names are registered trademarks and property of their respective owners. “3D XPoint is a trademark of Intel Corporation in the U.S. and/or other countries”

The World’s Only Universal NVMe Platform

The density and performance roadmap of NVMe SSD’s is accelerating

Only the NVMe roadmap provides unmatched performance and scale; SAS based drives cannot compete

• Unlike captive storage, Apeiron enables independent scaling of servers and storage

• Compatible with ANY commercial NVMedrive

• Data resides on the appropriate NVMeSSD type for its type and value

• ADS economies make it feasible to have all data reside on NVM

• Only Apeiron enables the application to leverage the most appropriate SSD

Apeiron Architecture

• The ADS1000 was designed ground-up to leverage applications such as Splunk, which are fully storage aware

• The system is “controller-less”, there are no devices between the server and the NVMe SSDs - 100% of switching is integrated

• By allowing the application to manage storage as designed, Apeiron has removed a massive scaling, performance and management bottleneck

Apeiron’s performance enables real-time queries on years of data with an 80% hardware consolidation

Each ADS enclosure includes 32-40GbE ports, and 24 NVMe SSDs

Enclosures inter-connect to enable multiple PB’s of capacity

Each ADS server provides 4-40GbE ports in to the ADS environment

5

ADS1000 Performance (2U)

Capacity 38/76/154/184/360TB

Latency (NAND LIMITATION) 100µs

Protocol Overhead <3µs (roundtrip)

Bandwidth sustained 72 GB/s (drive limited)

Random 4K reads 18.4 M IOPS

ADS1000 Scale-out NVMe Solution Unmatched Performance, Scalability and Efficiency

6Apeiron Proprietary and Confidential

Unmatched Performance & Consolidation

• Apeiron provides a fully integrated storage fabric with 24 NVMe drives in a 2U storage enclosure - No need to procure, deploy and manage external switching

• Each 2U building block provides 18.4M IOPS and up to 96GB/s of throughput, up to 184TB (going to 360TB in June 2017)

• The Apeiron architecture provides a massive performance advantage over controller based arrays – ADS1000 enables real time queries on years of data, while ingesting 10’s of TB’s per day

• Footprint reduction of 80% over legacy storage, with a 150% improvement in server performance – Lowest Splunk TCO in the industry

Fully Integrated switching eliminates external fabrics

Scales to 1000’s of NVMe drives and hundreds of application servers


How Does it Scale? 100% linear scalability…

562,592.77 436,948.40326,784.43217,671.94

108,956.58

- 100,000.00 200,000.00 300,000.00 400,000.00 500,000.00 600,000.00

IOPS

IOPS NVMe-NAND

1 Server 2 Servers 3 Servers 4 Servers 5 Servers

8,790.516,827.325,106.013,401.121,702.45

0.00 1,000.00 2,000.00 3,000.00 4,000.00 5,000.00 6,000.00 7,000.00 8,000.00 9,000.0010,000.00

THROUGHPUT

Throughput NVMe-NAND


0.570.580.590.590.59

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

LATENCY

Latency NVMe-NAND



443,453.90290,801.5268,096.5054,875.04

0.00 100,000.00 200,000.00 300,000.00 400,000.00

IOPS

IOPS Optane vs. NAND

NAND With 256K NAND Without 256K Optane With 256K Optane Without 256K

36,533.40

36,200.245,915.44

5,609.72

0.00 10,000.00 20,000.00 30,000.00 40,000.00

THROUGHPUT

Throughput [MB/s] Optane vs. NAND


0.1370.181

1.761.44

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

LATENCY

Latency [ms] Optane vs. NAND


Any NVMe? Even Optane?

ADS1000 Deployment Testing of PetaByte Scale Splunk Clusters

January, 2017



WWT ATC Validation-VM, Bare Metal and ASA

• 75 Physical Index Servers, three switching protocols

• 7 Days of Real-Time queries• 3TB/Day of Ingestion (customer license is for 10)• 12 unnecessary storage controllers• 4, 40U Racks of physical space

• 5x reduction in Index Servers, 4x footprint reduction

• 4-9x query performance means 4-9x the number of ES queries per day

• 8-10TB/Day Ingest (3x legacy architecture)

1.8

PB

R

AW

15

Ind

ex

Serv

ers

ES, ad-hoc and Search Heads


ESG Dashboard recording 10TB/Day with 100% healthy indexing queues

ESG Dashboard showing 100% healthy CPU and Memory on all indexers

ESG Dashboard proving 40-50 concurrent queries running at all times during data ingestion


ESG Dense Query Example: Bare Metal, Virtualized and ADS1000

Type Records / Second Total Time (S) / Records

Bare Metal 12,388 4,834 / 58,919,340

Virtualized Environment 10,528 5,596 / 58,556,553

Apeiron ADS1000 86,341 620 / 53,531,420

Dense Search: index = * | top user, 2 days

The Apeiron Advantage: 7x dense query performance over even a bare metal NAS deployment

All tests are based upon Flash storage

ADS1000 Query Performance Vs. Splunk Reference

Search Events FoundSplunk Reference1

Search Time [s]Apeiron ADS1000

Search Time [s]ADS1000

Advantage

Rare23 11.2 2.1 5x Faster

115 11.2 6.1 1.8x Faster

Super Sparse

26,802 1,112.2 12.6 88x Faster

180,850 1,112.2 19.2 58x Faster

Sparse155,459,317 31,091.9 2,842.2 11x Faster

1,126,745,647 225,349.1 14,985.3 15x Faster


1 Ref: http://docs.splunk.com/Documentation/Splunk/6.4.3/Capacity/Referencehardware

The purpose-built ADS1000 enables massive hardware and software consolidation while providing unmatched performance

http://docs.splunk.com/Documentation/Splunk/6.4.2/Capacity/Referencehardware