Direct Scale-Out Flash for Splunk - ?· Apeiron Architecture •The ADS1000 was designed ground-up to…

  • View
    220

  • Download
    2

Embed Size (px)

Transcript

  • Direct Scale-Out Flash for Splunk

    7/25/2017 Apeiron Proprietary and Confidential 1

  • 2

    Apeiron Addresses

    Apeiron was founded to address three inflection points in the storage market:

    Move to application managed scale out storage - Applications understand internal storage protocols

    Demand for real-time queries on massive data sets - Captive storage cannot manage the required capacity, storage networking is required

    The rapid adoption of NVMe as the standard SSD interface NVMeperformance creates two new bottlenecks; the controller and the fabric protocol

    05/03/2017

    Apeirons NVMe over Ethernet technology is the ONLY Generally Available product capable of networking not only NVMe, but Intels next generation storage: Optane SSD

  • 3

    All mentioned brand names are registered trademarks and property of their respective owners. 3D XPoint is a trademark of Intel Corporation in the U.S. and/or other countries

    The Worlds Only Universal NVMe Platform

    The density and performance roadmap of NVMe SSDs is accelerating

    Only the NVMe roadmap provides unmatched performance and scale; SAS based drives cannot compete

    Unlike captive storage, Apeiron enables independent scaling of servers and storage

    Compatible with ANY commercial NVMedrive

    Data resides on the appropriate NVMeSSD type for its type and value

    ADS economies make it feasible to have all data reside on NVM

    Only Apeiron enables the application to leverage the most appropriate SSD

  • Apeiron Architecture

    The ADS1000 was designed ground-up to leverage applications such as Splunk, which are fully storage aware

    The system is controller-less, there are no devices between the server and the NVMe SSDs - 100% of switching is integrated

    By allowing the application to manage storage as designed, Apeiron has removed a massive scaling, performance and management bottleneck

    Apeirons performance enables real-time queries on years of data with an 80% hardware consolidation

    Each ADS enclosure includes 32-40GbE ports, and 24 NVMe SSDs

    Enclosures inter-connect to enable multiple PBs of capacity

    Each ADS server provides 4-40GbE ports in to the ADS environment

  • 5

    ADS1000 Performance (2U)

    Capacity 38/76/154/184/360TB

    Latency (NAND LIMITATION) 100s

    Protocol Overhead

  • 6Apeiron Proprietary and Confidential

    Unmatched Performance & Consolidation

    Apeiron provides a fully integrated storage fabric with 24 NVMe drives in a 2U storage enclosure - No need to procure, deploy and manage external switching

    Each 2U building block provides 18.4M IOPS and up to 96GB/s of throughput, up to 184TB (going to 360TB in June 2017)

    The Apeiron architecture provides a massive performance advantage over controller based arrays ADS1000 enables real time queries on years of data, while ingesting 10s of TBs per day

    Footprint reduction of 80% over legacy storage, with a 150% improvement in server performance Lowest Splunk TCO in the industry

    Fully Integrated switching eliminates external fabrics

    Scales to 1000s of NVMe drives and hundreds of application servers

  • 7/25/2017 Apeiron Proprietary and Confidential 7

    How Does it Scale? 100% linear scalability

    562,592.77 436,948.40326,784.43217,671.94

    108,956.58

    - 100,000.00 200,000.00 300,000.00 400,000.00 500,000.00 600,000.00

    IOPS

    IOPS NVMe-NAND

    1 Server 2 Servers 3 Servers 4 Servers 5 Servers

    8,790.516,827.325,106.013,401.121,702.45

    0.00 1,000.00 2,000.00 3,000.00 4,000.00 5,000.00 6,000.00 7,000.00 8,000.00 9,000.0010,000.00

    THROUGHPUT

    Throughput NVMe-NAND

    1 Server 2 Servers 3 Servers 4 Servers 5 Servers

    0.570.580.590.590.59

    0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

    LATENCY

    Latency NVMe-NAND

    1 Server 2 Servers 3 Servers 4 Servers 5 Servers

  • 7/25/2017 Apeiron Proprietary and Confidential 8

    443,453.90290,801.5268,096.5054,875.04

    0.00 100,000.00 200,000.00 300,000.00 400,000.00

    IOPS

    IOPS Optane vs. NAND

    NAND With 256K NAND Without 256K Optane With 256K Optane Without 256K

    36,533.40

    36,200.245,915.44

    5,609.72

    0.00 10,000.00 20,000.00 30,000.00 40,000.00

    THROUGHPUT

    Throughput [MB/s] Optane vs. NAND

    NAND With 256K NAND Without 256K Optane With 256K Optane Without 256K

    0.1370.181

    1.761.44

    0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

    LATENCY

    Latency [ms] Optane vs. NAND

    NAND With 256K NAND Without 256K Optane With 256K Optane Without 256K

    Any NVMe? Even Optane?

  • ADS1000 Deployment Testing of PetaByte Scale Splunk Clusters

    January, 2017

    7/25/2017 Apeiron Proprietary and Confidential 9

  • 7/25/2017 Apeiron Proprietary and Confidential 10

    WWT ATC Validation-VM, Bare Metal and ASA

    75 Physical Index Servers, three switching protocols

    7 Days of Real-Time queries 3TB/Day of Ingestion (customer license is for 10) 12 unnecessary storage controllers 4, 40U Racks of physical space

    5x reduction in Index Servers, 4x footprint reduction

    4-9x query performance means 4-9x the number of ES queries per day

    8-10TB/Day Ingest (3x legacy architecture)

    1.8

    PB

    R

    AW

    15

    Ind

    ex

    Serv

    ers

    ES, ad-hoc and Search Heads

  • 7/25/2017 Apeiron Proprietary and Confidential 11

    ESG Dashboard recording 10TB/Day with 100% healthy indexing queues

    ESG Dashboard showing 100% healthy CPU and Memory on all indexers

    ESG Dashboard proving 40-50 concurrent queries running at all times during data ingestion

  • 7/25/2017 Apeiron Proprietary and Confidential 12

    ESG Dense Query Example: Bare Metal, Virtualized and ADS1000

    Type Records / Second Total Time (S) / Records

    Bare Metal 12,388 4,834 / 58,919,340

    Virtualized Environment 10,528 5,596 / 58,556,553

    Apeiron ADS1000 86,341 620 / 53,531,420

    Dense Search: index = * | top user, 2 days

    The Apeiron Advantage: 7x dense query performance over even a bare metal NAS deployment

    All tests are based upon Flash storage

  • ADS1000 Query Performance Vs. Splunk Reference

    Search Events FoundSplunk Reference1

    Search Time [s]Apeiron ADS1000

    Search Time [s]ADS1000

    Advantage

    Rare23 11.2 2.1 5x Faster

    115 11.2 6.1 1.8x Faster

    Super Sparse

    26,802 1,112.2 12.6 88x Faster

    180,850 1,112.2 19.2 58x Faster

    Sparse155,459,317 31,091.9 2,842.2 11x Faster

    1,126,745,647 225,349.1 14,985.3 15x Faster

    7/25/2017 Apeiron Proprietary and Confidential 13

    1 Ref: http://docs.splunk.com/Documentation/Splunk/6.4.3/Capacity/Referencehardware

    The purpose-built ADS1000 enables massive hardware and software consolidation while providing unmatched performance

    http://docs.splunk.com/Documentation/Splunk/6.4.2/Capacity/Referencehardware