RESEARCH
G2M Research Multi-Vendor Webinar NVMe/TCP™- The Eventual NVMe over Fabric Protocol Winner? February 26, 2019
2 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Webinar Agenda
9:00-9:05 Ground Rules and Webinar Topic Introduction(G2M Research)
9:06-9:25 Sponsoring Vendor presentations on topic (5 minute each) 9:26-9:39 Key Question 1 (2-minute question; 3 minutes response per
vendor) 9:40-9:41 Audience Survey 1 (2 minutes) 9:42-9:55 Key Question 2 (2-minute question; 3 minutes response per
vendor) 9:56-9:57 Audience Survey 2 (2 minutes) 9:58-10:11 Key Question 3 (2-minute question; 3 minutes response per
vendor)10:12-10:13 Audience Survey 3 (2 minutes) 10:14-10:18 Audience Q&A (8 minutes) 10:19-10:20 Wrap-Up
RESEARCH
G2M Research Introductionand Ground RulesMike HeumannManaging Partner, G2M Research
4 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Tom SpencerSr. Director, ProductMarketingSolarflarewww.solarflare.com
Panelists
Host/Emcee: Mike HeumannManaging PartnerG2M Researchwww.g2minc.com
Rob DavisVP, StorageTechnologiesMellanoxwww.mellanox.com
Eliot RosenProduct MarketingDirectorBroadcomwww.broadcom.com
Nishant LodhaSr. ProductMarketing ManagerMarvellwww.marvell.com
5 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
What is NVMe® over TCP (NVMe/TCP™)?
NVMe/TCP is a new binding for NVMe over Fabrics (NVMe-oF™)that was recently approved by NVMe.org
NVMe-oF commands are sent over standard TCP/IP sockets Each NVMe queue pair is mapped to a TCP connection TCP provides a reliable transport layer for the
NVMe queueing model NVMe/TCP provides an alternative transport to
NVMe-oRoCE and NVMe-oiWARP for Ethernet,and to NVMe-oFC (Fibre Channel) andNVMe-oIB (InfiniBand)
6 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
NVMe/TCP Characteristics
Advantages NVMe/TCP can run on existing TCP/IP Networks NVMe/TCP is compatible with existing Ethernet
management apps
Disadvantages
NVMe/TCP has higher latency than NVMe-oRoCE
Like NVMe-oRoCE and NVMe-IB transports, NVMe/TCP will (typically) run on converged SAN/ LAN networks (not a dedicated SAN)
7 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Questions We Will Explore In This Webinar
What are the use cases for NVMe/TCP, and what strengths does NVMe/TCP bring to these use cases vis-à-vis other NVMe-oF protocols?
How important are NVMe/TCP’s advantages and disadvantages for its use cases in real-world deployment scenarios?
To what extent will NVMe/TCP replace other Ethernet-based NVMe-oF protocols, and how long will it take to do so?
RESEARCH
MellanoxRob DavisVice President,Storage Technologieswww.mellanox.com
9© 2018 Mellanox Technologies | Confidential
Mellanox NVMe over TCP Support
Mellanox supports NVMe over TCP across our Ethernet products including ConnectX NICs and BlueField SOCs
NVMe/TCP and NVMe/RoCE are complimentary solutions, our products can support both simultaneously
NVMe/RoCE and NVMe/IB have lower and more consistent latencyNVMe/RoCE and NVMe/IB use less CPU cycles
TCP is less optimized for performance and efficiency, but we fully support it and see important use cases for it
10© 2018 Mellanox Technologies | Confidential
NVMe/TCP for Stranded Server Connectivity
Enables NVMe-oF connectivity for legacy systems Example given in article titled “Pavilion compares RoCE and TCP
NVMe over Fabrics performance” https://www.theregister.co.uk/2018/08/16/pavilion_fabrics_performance/ Pavilion provided their customer a mixed solution of servers with
non-RDMA NICs using NVMe/TCP and RDMA NICs using NVMe/RoCE The Pavilion AFA supports both RoCE and TCP NVMe-oF on Mellanox
ConnectX NICs Latency test results showed NVMe/TCP (~200µs) at twice that of
RoCE (~100µs) but half that of NVMe-backed iSCSI (~400µs) This includes AFA controller and NVMe SSD latency
11© 2018 Mellanox Technologies | Confidential
NVMe/TCP for Distance Connectivity
Often networked storage applications need to work over long distancesTCP’s well proven distance
capabilities make NVMe/TCP perfect for these applications We successfully tested this
between the Mellanox Israel and Sunnyvale offices
12© 2018 Mellanox Technologies | Confidential
NVMe/TCP for Control Path
Many data centers have an independent device management network (control path)Usually much lower speedNVMe/TCP enables support for
BMCs with simple IP stacksNVMe-MIAllows for discover controllers on
the control path independent of the data
1Gb100Gb
13© 2018 Mellanox Technologies | Confidential
Performance: NVMe/RoCE vs. NVMe/TCP
Loca
l SSD
Writ
e
RD
MA
Writ
e
TCP
Writ
e
Frac
tion
of IO
s w
ith th
is o
r les
s la
tenc
y
14© 2018 Mellanox Technologies
Importance of Tail Latency in Today’s Datacenters
Most datacenters today need to support interactive real-time requests Online searches generate a small amount of network traffic between the requestor and datacenter, but
the response generates a massive amount of traffic within the datacenter Much of this traffic is related to the core advertising business model of the datacenter’s owner The distributive software architecture that drives these businesses is very susceptible to tail latency
https://www.nextplatform.com/2018/03/27/in-modern-datacenters-the-latency-tail-wags-the-network-dog/
RESEARCH
SolarflareTom SpencerSr. Director, Product Marketingwww.solarflare.com
© 2019 Solarflare Communications, Inc. Confidential & Proprietary 16
OptaneLatency~10us
0
5
10
15
20
25
30
35
40
45
512 4096
Late
ncy
us
Block Size
LinuxKernelNVMe
SPDK
SPDK+
Onload
LinuxKernel
SPDK
SPDK+
Onload
44%Lower
35%Lower
54%Lower
28%Lower
1. Performance
© 2019 Solarflare Communications, Inc. Confidential & Proprietary 17
98% of installed servers connected with TCP
2% of installed servers connected with RDMA
2. Ease of Deployment
© 2019 Solarflare Communications, Inc. Confidential & Proprietary
3. Runs on Legacy Networks
© 2019 Solarflare Communications, Inc. Confidential & Proprietary
4. Ubiquity
© 2019 Solarflare Communications, Inc. Confidential & Proprietary
5. TCP Has Addressed Congestion
BeforeAfter
© 2019 Solarflare Communications, Inc. Confidential & Proprietary
5 Reasons Why to use NVMe-oF TCP
1. Performance
2. Ease of deployment
3. Runs on legacy networks
4. Ubiquity
5. TCP has addressed congestion
23 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
NVMe/TCP™
Ubiquitous in every network
Use existing TCP SW OS infrastructure
Easy to implement– Runs over existing lossy networks– Scalability without concern of congestion management
Can use existing networking configurations
Benefits vs RoCE Disadvantages vs RoCE
Longer latencies – Can be mitigated by limiting tail latency added by network
• Keeping low and consistent with SW algorithms
Performance is CPU bound in kernel mode– Need more CPU cores to achieve similar performance – User space performance is much better
24 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Broadcom solutions
- Common architecture- SmartNIC and Target- NVMe/TCP™ and RoCE
- Offload CPU intensive tasks- RAID/EC, Dedup, Security
- Low power HHHL adapters - Run TCP and RoCE
simultaneously
https://scalegrid.io/blog/comparing-in-memory-databases-redis-vs-mongodb-percona-memory-engine/
Stingray Benefits
25 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Stingray™ - Driving Next Generation NVMeoF Deployments
Applications
HW Accel
100G PCIe BW
De-Dup + RAID
100G RDMA NIC
8x A72 @ 3GHz
Stingray™
Cloud StorageSmartNIC NVMeJBOFs
• 100G NVMeoF target performance
• SDS enablement - Storage services on SmartNIC
• Open programming model on A72’s
• HW acceleration offload for RAID and Security
• Supports both RoCEv2 and TCP
• 16nm for low power (<35W)
• Production adapters available (100G and 50G)
26 © 2019 G2M Communications, Inc.. All rights reserved.
NetXtreme® -E and -S Series NVMe-oF Solutions
Page 26
Solves storage scaling problems with end-to-end solutions Largest OEMs and cloud customers standardize with Broadcom
DeployableTODAY
Thor™ Stingray™Stratus
Whitney+
50G100G
200G SmartNIC
M a r v e l l C o n f i d e n t i a l28
1 2
3
4
5
Universal RDMA + ToE
Ultimate in choice and investment protection with concurrent support for RoCE, RoCEv2 and iWARPand full offload for NVMe/TCP
FC-NVMe
Industry’s most accepted storage transport, now enhanced for NVMe.
Broad Spectrum of Ethernet and Fibre Channel
Connectivity16G and 32GFC , 10/25/50/100GbE
to host the most demanding Storage workloads and deliver
scalability
Seamless Storage Migration
Concurrent offload for NVMe-oF, iSER, iSCSI and FCoE for
Ethernet and FCP and FC-NVMe for Fibre Channel
enables seamless upgrade paths to next gen storage
End to End SolutionsInitiator and Target mode solutions, including SPDK that leverage years of storage and storage networking experience
NVMe-oF
Marvell NVMe-oF Solution
M a r v e l l C o n f i d e n t i a l29
Challenges remain with NVMe-oF™ RDMA!
RNIC Upgrade Required
RDMA Camps
Creates IslandsBackward Compatibility
Forklift Infrastructure and Skillset change required!
Not Automatic
Not Precise
Not for everyone
Congestion
Keeping the network ‘lossless’
RDMA/OFEDexpertise
Skillset Requirements
M a r v e l l C o n f i d e n t i a l30
Marvell NVMe/TCP at Flash Memory Summit 2018
M a r v e l l C o n f i d e n t i a l31
NVMe/TCP - FastPath
App/FS/MQStart of I/O
NVMe-oF
NVMe Core
NVMe/TCP
TCP/IP Stack
NIC
SQ CQNVMe Layer
In-Capsule DataNVMe-oFCapsule
PDU PayloadPDU Header
TCP PayloadTCP/IP Header TCP PayloadTCP/IP
Header
Eth PayloadETH Header Eth PayloadETH
Header Eth PayloadETH Header
Map each QP to TCP connectionConstruct PDU Header, Digest
Segmentation, Header Processing
Protocol and Header Processing
FastLinQ®NVMe/TCP Offload
Engine
Marvell Offloads and Accelerates NVMe/TCP
M a r v e l l C o n f i d e n t i a l
NVMe-oF Performance Comparisons
32
Read Avg Read Tail Write Avg Write Tail
Latency Performance1 Thread :: Queued :: 8KB
NVMe-oF RoCE NVMe-oF SW TCP NVMe-oF HW TCP RoCEv2 SW TCP HW TCP - Current
Million IOPS PerformanceMulti Thread:: 8KB READ
* Initial metrics based on Marvell internal measurements; performance is system dependent. For HW TCP, only target was “offloaded”, initiator was SW driven.
M a r v e l l C o n f i d e n t i a l33
Making the right “fabric” choice!
Not “just” about “fabrics” performance Use CasesCulture and Install Base
RESEARCH
Panel Discussion
35 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Panel Question #1
What are the use cases for NVMe/TCP, and what strengths does NVMe/TCP bring to these use cases vis-à-vis other NVMe-oF protocols?– Mellanox– Solarflare– Broadcom– Marvell
36 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Audience Survey Question #1
What use cases do you believe that NVMe/TCP is best suited for (versus other NVMe-oF protocols)? (check all that apply; 54 responses):
• Databases/database applications: 44%
• Unstructured data/object storage: 52%
• Real-time analytics: 22%
• High-performance computing and modeling: 20%
• DevOps: 24%
• Others: 20%
• I don’t believe that NVMe/TCP is more suited for these use casesthan other NVMe-oF protocols: 11%
37 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Panel Question #2
How important are NVMe/TCP’s advantages and disadvantages for its use cases in real-world deployment scenarios?– Solarflare– Broadcom– Marvell– Mellanox
38 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Audience Survey Question #2
Which of NVMe/TCP’s advantages/disadvantages are important for your use cases (check one answer for each question; 48 responses):
Question Very Important
Somewhat Important
Not Important
Not Enough
Information to Answer
No Opinion
Advantage: Ability to run on existing Ethernet networks 79% 15% 2% 4% 0%
Advantage: Ability to support multicast 23% 40% 15% 13% 10%
Disadvantage: Higher latency than other NVMe-oF protocols 31% 67% 2% 0% 0%
Disadvantage: Not a dedicated SAN 17% 31% 44% 4% 4%
39 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Panel Question #3
To what extent will NVMe/TCP replace other Ethernet-based NVMe-oF protocols, and how long will it take to do so?– Broadcom– Marvell– Mellanox– Solarflare
40 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Audience Survey Question #3
To what extent would your organization consider NVMe/TCP as an alternative to other Ethernet NVMe-oF protocols? (select one; 32 responses):
• We will continue to utilize non-NVMe-oF Ethernet as ourprimary SAN for the next 36 months or greater: 22%
• We are already deploying/planning to deploy NVMe-oFCas our primary SAN: 9%
• We are already deploying/planning to deploy NVMe-oRoCEas our primary SAN: 13%
• We will are considering all NVMe-oF options for our storage deployments in the next 36 months: 47%
• NVMe/TCP is the primary protocol we are considering for ourNVMe-oF deployments during the next 36 months: 9%
41 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH
Audience Q&A
RESEARCH
Thank You For Attending
RESEARCH