The Convergence of Storage and Server Virtualization

© 2007 Solarflare Communications, Inc.

The Convergence of Storage andServer Virtualization

Slide 2 | © 2007 Solarflare Communications, Inc.

About Solarflare Communications

Privately-held, fabless semiconductor company. Founded 2001

Top tier investors: Accel, Foundation, Intel Capital, Oak

Headquartered in Irvine, California, R&D center in Cambridge, UK

Expertise and leadership in Ethernet silicon and software– Highest number of contributions to 10GBASE-T Std IEEE 802.3an

– First production ready, highest-performing 10GBASE-T PHY

– Lowest power, highest performance 10G controller

Single unified 10G Ethernet Silicon and Software platform


Perspective on storage and server virtualization

Manageability, Cost, Scalability

Virtualization takes hold– FibreChannel enabled storage virtualization (1990’s)– VMware, XEN, Microsoft enable server virtualization (2000’s)– CPU chipsets enable more efficient virtualization

Virtualization challenges– Application I/O demands increasing dramatically– Virtual OS introduced new I/O bottlenecks– Cost of supporting multiple networks


Industry solutions

10GBASE-T Enables high performance virtualization at low cost:

– Single, low cost, easy to use, network– High-performance, converged I/O architecture– Expect rapid reduction in price and power (enabling mass

adoption)

iSCSI enables scalable storage on commodity hardware/software– SCSI protocol over TCP– iSCSI products are today mature and stable– 10GBASE-T provides great iSCSI performance (> 1000MB/s)– Storage virtualization and server virtualization are converging


Storage and Virtualization Architecture (SVA)

API which supports acceleration of guest networking and advanced Virtual OSfeatures– migration– hot-plug– stateless offloads

Also supports diverse optional hardware and accelerations– Safe, direct hardware access from guest OS– broadcast and multicast delivery acceleration– local loopback– PCI-IOV and ATS– perfect or imperfect filtering

While retaining the successful OEM/ISV/OS dynamic for the volume market– Open and simple APIs permit innovation– OS and Virtual OS competition and co-existence


I/O in Xen: the front-end and back-end

All 'real' drivers live in Dom0

DomU kernels have pseudo drivers thatcommunicate with Dom0 via the hypervisor

Necessary because only Dom0 is 'trusted'

Dom0 DomU DomU

HardwareHypervisor


I/O with SVA

DomU can access hardware directly + safely– Unprivileged guests can transmit and receive packets directly– Bypassing Dom0 and hypervisor– At least most of the time; still slow path via Dom0

Accelerated paths set up in Dom0– In order safely to restrict DomU’s access

Dom0 DomU DomU

HardwareHypervisor


SVA Network Performance

Back to back SFE4003 A1 Reference design(CX4) (v2.1.122) Intel 5000X chipset,quad core 2.6 GHz Xeon 5355 4GB RAM,1500 byte MTU

Aggregate throughput of 1-4 TCP streams(NetPerf), each in a separate Xenvirtualised Linux 2.6.18 guest.

Direct hardware access increasesbandwidth, reduces CPU overhead andeliminates the I/O bottleneck in the parentdomain.

Responsiveness of machine alsosignificantly improved in the SolarflareAccelerated case.


Bidirectional performance

0

2

4

6

8

1 0

1 2

1 4

1 6

1 guest 2 guests 3 guests 4 guests

Unacce le rated

Acce lerated


Plug-in architecture

Hardware dependent accelerated

plug-in

Front-end driver

FIFOs

Sharedpages

VI

RxQ TxQ EvQ

VI

RxQ TxQ EvQ

VI

RxQ TxQ EvQVI

RxQ TxQ EvQ

Dom0 DomU

In shared pages:•Virtual Interfaces•FIFOs•Tx/Rx buffs•Local MAC tbls

Plug-in

Back-enddriver


Solarflare’s SFC4000

Front-end driver in DomU– Where possible, interacts with h/w directly

• This is known as “fast path”

• Otherwise packets routed via Dom0: “slow path”

• Need to be careful about Tx traffic for local machine

Back-end driver lives in Dom0– Sets up “Virtual Interface” per guest VM

• Installs filters “on-demand” as traffic flows in/out

• Enforces quotas on front-end drivers


Migration

On migration, back-end sends ‘h/w unavailable’message to front-end– Everything then goes over slow-path

Possibly followed by ‘new h/w available’ message atthe other end

If no accelerated hardware available, or if no driver forthat hardware available: slow path!


SVA and Storage (Consider 3 Options)

iSCSI initiator runs in Dom0Dom0 DomU DomU

HardwareHypervisor

block i/fiSCSIinitiator

Dom0 DomU DomU

HardwareHypervisor

block i/f

iSCSIinitiator

iSCSI initiator runs inDomU

iSCSI initiator runs inDomU+ direct HW access


SVA Storage Performance

Back to back SFE4003 A1 Referencedesign (CX4) (v2.1.122) , Intel 5000Xchipset, quad core 2.6 GHz Xeon 53554GB RAM, 1500 byte MTU

Aggregate throughput of 6 iSCSIstreams (Disktest 64KB Seq Read), eachin a separate Xen virtualised Linux2.6.18 guest to Microsoft WUDSS iSCSItarget

Direct hardware access increasesbandwidth, reduces CPU overhead andeliminates the I/O bottleneck in theparent domain.

Responsiveness of machine alsosignificantly improved in the SolarflareAccelerated case.


iSCSI in the guest?

Why you want iSCSI in the guest– Potential for better performance

– Connections migrate with guest

– Less scheduling cross-talk

– Use existing iSCSI tools

– Security: authorization per guest initiator


iSCSI in the guest?

… and why you don’t:– Older guests may lack iSCSI support– Lots of different iSCSI stacks (quirks, tools, etc)– Boot device over iSCSI much simpler from Dom0

Conclusion:– No single best answer– Likely hot-spot between OS and virtualized OS

vendors


What's next for SVA and Xen?

Submission of SFC4000 accelerated drivers to xen-devel– Framework already merged

New features to integrate– MAC filtering transmit

– MSI-X (interrupt line per guest)

– DMA to other guests on the same host

– PCI-IOV


Conclusions

Smart I/O hardware can boost performance– Also less “QoS cross talk”

– Negates the need for driver domains

– iSCSI doesn’t necessarily belong in Dom0

Converged storage and virtualization architectures– High-performance, low-cost, easy to use

– Utilizing commodity hardware and driving consolidation

Technology

The Convergence of Storage and Server Virtualization