High Performance Pooled Block Storage with Cinder...accelerated storage backends Future plans. SPDK, PMDK & Vtune™ Amplifier Summit 18 Backup. SPDK, PMDK & Vtune™ Amplifier Summit

High Performance Pooled Block Storage with Cinder

Maciej Szwed

Tushar Gohad

SPDK, PMDK & Vtune™ Amplifier Summit

Agenda

• SPDK orchestration?

• OpenStack

• OpenStack NVMe-oF initiative

• Cinder and SPDK

• Usage

• Future plans

2

SPDK, PMDK & Vtune™ Amplifier Summit 3

Configuration file, JSON RPC calls, SPDK CLI

SPDK orchestration?Focused on building from blocks rather than complete solutions

Orchestration, Cloud Infrastructure?


• OpenStack and K8S the top contenders

• Start with OpenStack -

• API driven access to compute, network and storage resources

• Support for largest number of storage and networking systems

• Flexibility to deploy bare metal, VMs and containers

• Continually evolving

open Cloud infra and spdk


openstack

Open source software for creating private and public clouds

OpenStackClient HorizonCommand-line client web-based client

CinderSoftware Defined

Block Storage Service

NovaCompute Instances

Provisioner

NeutronNetwork

Connectivity Provisioner


• 2016: SPDK Team demo’ed Cinder/Nova NVMe-oF Volume Driver POC

• 2017: Joint project started with Mirantis, Mellanox, OpenStack community to add NVMe-oF base infrastructure to Cinder

• 2017-18: SPDK Logical Volume support (Cinder baseline requirement)

• 2018: OpenStack Rocky – Kernel NVMe-oF Target available

• 2019: Openstack Stein – SPDK volume and target drivers available

OpenStack NVMe-oF initiative

2016 2017 2018 2019

Barcelona Summit

NVMeT driverimplementation

started

Vancouver Summit

Cinder NVMeT driver merged

SPDK Cinder drivers merged

SPDK Logical Volumes


Cinder and SPDKHorizon

CinderSPDKSPDK bdev SPDK Volume

Driver

SPDK NVMe-oF Driver

Logical Volumes

NVMe-oF Target

Nova

VM

Cinder/Nova Control PathSPDK JSON RPC PathData Path


• Create/extend/copy/delete volume

• Create/delete snapshot

• Create volume from snapshot

• Attach/detach volume

• Create image from volume

• Create volume from image

Cinder and SPDKSupport for all common operations:

SPDK Cinder volume driver

SPDK Logical Volumes

SPDK NVMe-oF Target

SPDK Cinder target driver


openstack and SPDK NVMe-of

• OpenStack with fast storage

• Easily deployable infrastructure with SPDK

• Automatic resource tracking

• SPDK as a complete cloud solution


Usage – Cinder configuration file

[DEFAULT]

...

enabled_backends = spdk

...

[spdk]

spdk_rpc_ip = 192.168.0.3

spdk_rpc_port = 8000

spdk_rpc_username = user

spdk_rpc_password = password

target_ip_address = 192.168.0.2

target_port = 4260

target_protocol = nvmet_rdma

target_helper = spdk-nvmeof

target_prefix = nqn.2014-08.org.spdk

volume_driver = cinder.volume.drivers.spdk.SPDKDriver

volume_backend_name = SPDK


Usage – Openstack cli

DEMO


Usage – openstack horizon








• Cinder support for SPDK Logical Volumes on local NVMe and NVMe-over-TCP

• Kubernetes CSI plugin for NVMe-oF with SPDK support

• Nova support for SPDK vhost accelerated storage backends

Future plans


Backup


Usage – JSON RPC

# construct_lvol_bdev -l lvs lvol0 1024

# nvmf_create_transport -t rdma

nvmf_create_transport -t rdma# nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a

# nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 lvs/lvol0

# nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t rdma -a 192.168.0.2 -s 4260


Usage – Openstack cli

# openstack volume create --size 1 volume1

# openstack server create --flavor m1.small --image cirros-x86_64-disk

# nova volume-attach <server-ID> <volume-ID>

Documents

High Performance Pooled Block Storage with Cinder...accelerated storage backends Future plans. SPDK, PMDK & Vtune™ Amplifier Summit 18 Backup. SPDK, PMDK & Vtune™ Amplifier Summit