Object-based SSD (OSSD): Our Practice and Experience · OSD and Object-based SSD Introduction Our Practice and Experience Part 2. Application-Aware Storage Introduction 2 Our Practice

SAMSUNGFSWD

SAMSUNG

Object-based SSD (OSSD):

Our Practice and Experience

Jaesoo Lee

[email protected]

Flash Solution Team, Memory Division

Samsung Electronics Co.

SAMSUNGOutline

� Part 1. OSD and Object-based SSD

� Introduction

� Our Practice and Experience

� Part 2. Application-Aware Storage

� Introduction

2

� Introduction


� Future Directions

� Summary and Conclusion

SAMSUNGOutline


� Introduction



� Introduction

3

� Introduction




SAMSUNGOutdated Storage Stack

� HDD as main storage for decades

� Structure is relatively simple

� Physical location can be easily derived

� Storage stack has remained static

( ) ( ) 1secsec −+⋅+⋅⋅= sNhNNcLBA torstorsheads

4

Storage stack has remained static

� Narrow block interfaces (ATA, SCSI)

� No information flow except block reads/writes

� File subsystems make HDD-specific assumptions

� Sequential read is much faster than random read

� No write amplification, wear-out, background activity, …

� What if the underlying device changes ?

[Excerpted from Block Management in SSD, Usenix 2009 ]

SAMSUNG� Host-side optimization is no more feasible

� Confidential, subject to changes, different among SSDs, …

Challenges in SSD Storage Stack

□ Striping method

□ Buffer cache management policy

□ Logical-to-physical mapping

□ Garbage collection policy

□ Wear-leveling policy

□ Bad block management

□ # of channels

□ # of ways

□ # of planes per chip

□ # of blocks per plane

□ # of pages per block

5

Processor

Core

□ # of pages per block

□ Block size

□ Page size

[figure by Sang Lyul Min @ SNU]

SAMSUNGSSD-Aware File System

� Preliminary results with in-house SSD-aware file system

� Up to 1700% improvement for random writes

� 13% degradation in sequential write performance

12

14

16

18

No

rma

lize

d T

hro

ug

hp

ut

SSD-Aware FS

ext3

6

0

2

4

6

8

10

12

4 8 16 32 64 128 256 512 1024 2048 4096 8192

No

rma

lize

d T

hro

ug

hp

ut

I/O Size (KB)

SAMSUNGOSD: The Basic Concept

� Optional command set defined for SCSI device

� Provide object-based interface instead of traditional block-

based interface

� In OSD, an object is a flexible-sized data container

� Unique object ID

� A set of attributes

7

� A set of attributes

SAMSUNGSSD as OSD

� OSD manages space for objects

� Informed cleaning (utilize delete info)

� Stripe aligned accesses

� Logical to physical mapping

� OSD supports object attributes

8

� OSD supports object attributes

� Wear-leveling using cold data

information

� Priority assigned to objects (i.e., QoS)

� OSD handles low-level operations

� Block management in SSD

[Source: Block Management in SSD, Usenix 2009 ]

SAMSUNGLinux Support for OSD

� Linux provide a ready-to-use OSD prototype

� open-osd: open source initiator and exofs filesystem

� OSC-OSD: iSCSI OSD target

� Open-iSCSI: iSCSI transportvfs

exofs ext3 udf

9

scsi sata

iscsiiscsi

initiatorscsi initiator

SCSI/SCSI/

SATA

SCSI core (scsi_mod.ko)

bsg sd sr stosd_uld (osd.ko)

osd_lld (libosd.ko)

SAMSUNGTarget Platform

� OSD is the most promising for mobile storages

� Storage and system vendors are decoupled inherently

� QoS provisioning is essential

� e.g., managed flash memory applications including

� eMMC, UFD, miniSD, SD, …

� Our primary target is a Linux-based mobile platform

10

� Our primary target is a Linux-based mobile platform

SAMSUNGOutline


� Introduction



� Introduction

11

� Introduction




SAMSUNG� Prototyped in the Linux-based host

� MLD: object mapping, space management, …

� New type of SSD developed (called rawSSD)

OSSD Prototype

exofs

VFS□ Get rawSSD information

□ Erase all blocksfrom open-osd

12

Host (Linux)

OSSD ULD

OSSD MLD

OSSD LLD

raw SSD

Flash Cmd I/F

SCSI/SATA Flash Mng.

□ Erase all blocks

□ Erase a block

□ Read a flash page

□ Program a flash page

SATA Link

from open-osd

and Linux kernel

SAMSUNGMLD: Object Management Layer

� Overall architecture

(RB-Tree)

Index

(RB-Tree)

{PID, OID} [127:0]Allocator

GC

Wear-leveling

OSD Commands

13

Object descriptorWear-leveling

Meta data User data

Flash

media

OSSD LLD

READ / PROGRAM / ERASE

SAMSUNGMLD: Object Descriptor

� Containing metadata including extents and attributes

� Cached in memory (LRU)

Descriptor Header□ Object ID

□ PPN

□ Object information

14

Block Table

Attributes

1page

(16KB)

□ PPNs for object data

□ Extent: support 9MB

□ Indirect table: support 4GB

□ Attr. data of object

□ Usually contain i-node and length

□ Managed using original code

SAMSUNGMLD: Object Mapping

� Support page-wise mapping

� Based on page extents

� Association policies of an update block

� Fully associative

� Separation of index and user data

15

� No separation (called unified)

� Object associative (called per object)

� Garbage collection

� Victim selection

� Highest invalid pages

� Bitmap for invalid information

� Background GC supported

SAMSUNGLow Level Driver

� Provide block + Raw SSD specific

interfaces to upper layers

� Sanity check

� Page overwrite

� Out-of-order page write

OSSD LLD

Sanity Checker

Block interface

- submit_bio()

Raw SSD interface

- obtain_config()

- erase_block()

- format()

16

SCSI

subsystem

� Out-of-order page write

� Provide backend flexibility

� RawSSD

� Via SCSI-ATA Translation Layer (SATL)

� Loop

� RAM

Backend dispatcher

RAM LoopRaw

SSD

SAMSUNGExperimental Setup

� Host system

� Quad Q9650 3GHz

� 4GB RAM

� Linux kernel 2.6.33

� 4GB partition for OSSD

Write ThreadsRead Thread

…

17

Host

exofs

OSSD Driver

raw SSD

Flash Cmd I/F

SCSI/SATAFlash Mng.

VFS

SAMSUNGExperiments

� 1. Better performance for fragmented/multi-stream

writes

� Effect of incorporating space management in OSSD

� 2. QoS support (read prioritized service)

� Effect of having knowledge on the contexts of data blocks

18

� Effect of having knowledge on the contexts of data blocks

and a request

SAMSUNGExp1. Multi-thread File Write

� Scenario

� Write 800 MB 4 thread � delete 2 file � write 800 MB 2

thread

� Show the effect of fragmentation

Unified update

block

Index / Data Index / Per object

19

block

Write 800MB

4 thread

3m48.470s 3m52.263s 3m49.769s

After delete,

write 800MB

2 thread

6m34.535s 2m59.239s 1m59.375s

Erase 689 599 414

Valid copy 34887 23605 0

SAMSUNGExp2. Read Priority

� Scenario

� 200MB read over total 2.4G write (e.g., 300MB x 8)

20

SAMSUNGOutline


� Introduction



� Introduction

21

� Introduction




SAMSUNGCapacity Outran Bandwidth

� Capacity of HDD grows exponentially

� Ratio of the capacity to the interface bandwidth also

grows exponentially

100000

1000000

100000

1000000

Ca

pa

city

/ I

nte

rfa

ce B

an

dw

idth

(se

c)

Ca

pa

city

(M

B,

log

sca

le)

Capacity

C/B Ratio

22

1

10

100

1000

10000

1

10

100

1000

10000

1990 1995 2000 2005 2010

Ca

pa

city

/ I

nte

rfa

ce B

an

dw

idth

(se

c)

Ca

pa

city

(M

B,

log

sca

le)

Year

Maxtor 7000

(IDE)

Quantum Fireball ST

(UltraATA/33)

IBM Deskstar 16GP

(UltraATA/33)

Seagate Barracuda IV

(UltraATA/100)

Seagate Barracuda

(SATA/300)

SAMSUNGApplication-Aware Storage

� With a small database proxy, the amount of data

transferred can be decreased significantly

� scan, aggregation, join, sorting, …

23

� Other promising areas

� Data mining, search indexing, image processing, Anti-Virus, …

SAMSUNGApplication-Aware Storage

� Application-awareness can be easily achieved in OSSD

� Integrated object (i.e., file) management

� Fluent attribute mechanism

� Furthermore, SSD is no more dumb

� 4 Cortex-R4 CPUs, SATA 6Gbps, 512MB RAM, AXI bus matrix,

24

� 4 Cortex-R4 CPUs, SATA 6Gbps, 512MB RAM, AXI bus matrix,

16 flash memory controllers, …

� What about moving some of application’s work to

OSSD?

� Issues are models for programming, execution, and

deployment

SAMSUNGOutline


� Introduction



� Introduction

25

� Introduction




SAMSUNGPostgreSQL

DB-Aware OSSD

� PostgreSQL

� Free and open source

� Object-relational database

� Developed a plug-in for

accelerating:

Query PlannerQuery Planner

Query ParserQuery Parser

OSSD

Plug-in

Query

Optimizer

Query

Optimizer

ExecutorExecutor

26

accelerating:

� Aggregation

� Selection

� Query is processed in a similar

way to the active diskKernel

VFS

EXOFS

OSD Driver

DB-Aware OSSD

DB Proxy

ioctl

SAMSUNGDB Operators

� Aggregation

� Count, Sum, AverageSELECT count(*) FROM emp where age < 30;

Count(*)

--------

17

Result Result

27

Name Age Salary

Mc 29 3000

Kim 26 1200

Na 34 4000

Kang 28 1400

Lim 25 400

… … …

Seq. Read

Aggregate

Seq. Read

And aggregate

SAMSUNGDB Operators

� Selection, Projection

� Filtering (not yet)SELECT * FROM emp where age > 30;

Name Age Salary

----- ----- --------

Na 34 4000

Result Result

28

Seq. Read

Selection

Seq. Read

And selection

Name Age Salary

Mc 29 3000

Kim 26 1200

Na 34 4000

Kang 28 1400

Lim 25 400

… … …

SAMSUNGImplementation Mechanisms

SELECT AGGREGATE(target_col)

FROM t1 OSD

WHERE cond_col COND value;

Linux Host

PostgreSQL

� Currently,

� Running on existing exofs file system

29

WHERE cond_col COND value;

Translate inode to Object ID

Sequential Read

and

Processing Aggregate Function

exofs

OSSD ULD

OSSD LLD

SCSI/SATA

OSSD MLD

SAMSUNGExperiments

� Experimental setup

� Hardware

� AMD Atholon64x2 7750 2.7 Ghz

� 3GB RAM (PC6400)

� HDD WD3200AAKS (SATA2/7200/16M)

� OS and drivers

Linux kernel 2.6.33 (Fedora Core 13)

30

� Linux kernel 2.6.33 (Fedora Core 13)

� Target : OSC-OSD

� File system and Driver : Open-OSD exofs and drivers

� Database

� PostgreSQL 8.4.4

SAMSUNGExperiments

� Results will be available soon!!

31

SAMSUNGOutline


� Introduction



� Introduction

32

� Introduction




SAMSUNGFuture Directions of Work

� 1. OSD SSD

� Native support of OSD commands in SSD

� OSD command tunneling

� 2. Bi-directional communication extension in SATA

� Virtually any communication protocol over SATA

33

� Virtually any communication protocol over SATA

� 3. i2SSD: open storage software platform

� JVM and deployment middleware

� Storage programming framework

SAMSUNG

Linux Host OSSD

1. OSD SSD Architecture (Revised)

exofs

OSSD ULD OSSD Cmd Handler

34

OSSD LLD

SCSI/SATA

OSD Cmd I/F (via Tunneling Tech.)

SATA Link

Object Layer

FTL

SATA

Buffer Manager

SAMSUNG1. OSD SSD: OSD Cmd Tunneling

Host SSD

� 2-phased protocol

� OSD command is transferred as a payload of the 1st

command

35

1st phase

(cmd transfer)

2nd phase (optional)

(data transfer)

SATA vendor

command

(register FIS)

D2H DataH2D Data

SAMSUNG2. Bi-Directional Comm Extension

� ATA/SCSI protocol is too limited for application-specific

extension of OSSD

� We extended SATA with a simple bi-directional

communication protocol

� Based on reserved LBAs and notification mechanism

� Serve as a link layer of other advanced protocol, e.g., TCP/IP

36

� Serve as a link layer of other advanced protocol, e.g., TCP/IP

Application Main

ATA/SCSI Trasport

Bi-directiona l

Messaging Cmd.

Application Agent

ATA/SCSI Trasport

Bi-directional

Messaging Cmd.

FTL

SATA/SAS/… Link

Bi-directional

communication

HOST HDD/SSD

SAMSUNG3. i2SSD: Intelligent and Innovative SSD

� What is i2SSD ?

� Object-based SSD capable of extending its functionalities via

dynamic application deployment

� i.e., “Open Storage Architecture for Enabling Storage-Aware

Application Deployment”

� Benefit is significantly saved time and energy

� Computation at the near of the data

37

� Computation at the near of the data

dynamically deployedapplication

service library componentsi2SSD

i2SSD Run-time Framework

SAMSUNG3. Preliminary i2SSD Architecture

i2SSD Framework

Service Component Layer

Application Layer

38

i2SSD Framework

JVM

JNI OS(Linux?)

Secu

rity

Bi-directionalcomm.

Stream EditorRegEx Match

Compression/Decompression

Encoding/Decoding

Object Layer

FTL

HIL

..

SAMSUNGOutline


� Introduction



� Introduction

39

� Introduction




SAMSUNGConclusion

� OSD fits well for SSD

� Free from variations in SSD internals

� Easy application-specific extension

� Advanced features (e.g., QoS) made possible

� We have developed a proof-of-concept OSSD and

40

� We have developed a proof-of-concept OSSD and

demonstrated the benefits

� The results look promising

SAMSUNGCall for Participation

� Standardization

� SCSI/SATA extensions for OSD command tunneling

� SCSI/SATA extensions for bi-directional communication

command set

� i2SSD framework and programming interfaces

41

� Call for participation of Linux community

� OSSD bring-up issues in Linux

� End-to-end storage QoS support in Linux

SAMSUNGFSWD

SAMSUNG

Thanks

Documents

Object-based SSD (OSSD): Our Practice and Experience · OSD and Object-based SSD Introduction Our Practice and Experience Part 2. Application-Aware Storage Introduction 2 Our Practice