PoINT Software & Systemskonferenz-nz.dlr.de/pages/samfs2019/present/1... · Policy-based File Tiering, Archiving and Migration Multi-Cloud Support PSM supports up to 20 cloud providers

www.point.de Confidential

PoINT Software & Systems

© 2019 PoINT Software & Systems GmbH All rights reserved.

Managing storage classes for infrequently accessed data

DLR - HSM User Conference 2019

PoINT Software & Systems GmbH June 2019


2



Focused Data management software is our core competence

Sustainable Because archiving means keeping data for a long period of time

Flexible Understanding customers’ changing demands

Focused Data management software is our core competence




Who we are

Foundation of PoINT Software & Systems GmbH in 1994

Experienced ISV for data and storage management software

Software “Made in Germany”

Global customers from small businesses to large enterprises

World-wide reseller network and close cooperation with major storage system vendors

4 © 2019 PoINT Software & Systems GmbH All rights reserved.


4

PoINT Storage Manager Policy-based File Tiering, Archiving and Migration

Primary Storage NAS Systems

File Tiering & Archiving Policy-based archiving of inactive files from primary to secondary/archive storage

CIFS / NFS

Storage Systems ”off premises“

LAN

Primary Storage

Client and application transparency Archived files are visible through primary

storage Various options to access archived data

PoINT Storage Manager Software Solution Windows Server based

(VMware supported) HA by Secondary Access Server

or Windows Cluster

Secondary/Archive Storage HDD Appliances (CIFS, NFS) Object / Cloud (S3, Swift) Tape (LTO, LTFS, ACSLS) Optical Storage Systems

”on premises“

Conceptual Overview



5


PoINT EMC FileMover Protocol

PoINT NTFS/ReFS Service

PoINT NetApp FAS API

PoINT Web Link Service

Storage Manager Services

Object Store Connectors

HDD / NAS Connectors

Integration

Primary Storage

PSM Services

PSM Storage Connectors

Secondary / Archive Storage

Tape Library Connectors

Optical Library

Connectors Po

INT

Sto

rag

e M

anag

er

Applicance Connectors

Dell EMC Unity Windows NAS Systems NetApp FAS Generic

NAS Share Virtual „WORM“ File

System

PoINT Archive File System

Software Architecture


6



Focused Data management is our core competence





7


Stubbing File System

Web Links File System

Application Client System

Web GUI Browser

Transparent file system access

“Pass through“

Previous versions in snapshots

Primary storage support

Any NAS system

Indirect file system access

HTTP/Internet

Functionality

Versions

Search

Requires installation

Independent of operating system

HTTP/Internet

Search

Archive Access



8


User Controlled Archiving

Use cases Accounting department archives invoices

Development department moves closed projects

Speciality department recalls entire directories for big data analytics

…

Windows Explorer shell extension

Menu is fully customizable by administrator

Shell commands rely on PSM extensive policy framework


9



Focused Data management is our core competence





10


Multi-Cloud Support PSM supports up to 20 cloud providers per instance

AWS, Azure, Cloudian HyperStore, Dell EMC ECS, NetApp StorageGRID, Hitachi HCP, etc.

Up to 4 cloud providers in same Storage Vault

Reduce dependence on single cloud provider

Geographical distribution

Minimize cloud data egress cost

Different cloud providers for different data sets

Comply with legal requirements

Fast background migration of Archive Volumes (containers)

No interruption of operation

Private and/or public cloud



11


UDF Archive Volumes Universal Disc Format is well-known (e.g. used by optical discs)

Standardised format and platform-independent

Windows, Linux and MacOS can read UDF volumes

PSM uses UDF version 2.01 (sequential)

PSM gathers files in temporary image directory and creates Archive Volumes based on UDF

Volume size is appropriately adjustable to archive device/media

Files to be archived Creation of UDF Archive Volumes in image directory

Write finalized Archive Volumes to archive device

Vendor-Lock-In



12


Archive Migration

Archive Devices

Write to multiple targets in parallel Technology

separation Multi-Cloud …

New Archive Device

Long-Term Retention

Archive Volumes

In a few steps

Configure new archive device Choose source and target Start migration process

(accelerated by UDF Archive Volumes)

Logging of migration Remove legacy device




Object Storage

?

What about HSM for object storage?

Workloads

Archive Backup

Sync & Share Cloud native apps

IoT Analytics

…




? Storage classes at AWS, Google etc.

AWS Google

S3 Standard Multi-Regional / Regional

S3 Standard-IA Nearline

S3 Glacier Coldline

S3 Glacier Deep Archive

"Ice cold archive storage"

Acc

ess

fre

qu

en

cy

Pri

ce p

er

GB

Disk-based? Tape-based? Or optical media?




Is it really tape vs. cloud?

https://www.itwire.com/business-it-news/storage/82332-tape-is-the-new-tape,-says-azure-cto-for-now.html

“After evaluating various technologies including Blu-ray and magnetic disk, Microsoft came to the conclusion that tape was still the way to go for Azure's archival storage tier”




Don’t tier data from object storage to file system Limited metadata, limited character set, limited file name length, limited pathname

length, limited practical number of files, limited versioning FS overhead for maintaining file and directory structure (e.g. LTFS)

Don’t put S3 service on top of file system

Requires large amount of disk space for staging Intervening FS layer (see replacement of Ceph FileStore by BlueStore) Complexity and management overhead

Tiering object data DON’T



17

PoINT Archival Gateway High Performance S3 Gateway for Tape Storage

Object Storage (S3 REST API)

Up to 8 LTO Library Devices and up to 256 LTO Drives

Software Solution



18


Software Solution - Key Features

High performance in terms of data and object rates

High availability and reliability

High scalability (incl. load-balancing, redundancy and failover)

Data protection (erasure coding, authentication and encryption)

Self-monitoring, reporting and alerting

User management based on domain services (AD, LDAP)



19


System Components

Client Network

Tape Library

Administration Network

System Network (internal and

private)

PAG-IFN1

PAG-IFN2

PAG-IFN3

PAG-IFN4

PAG-DBN1

PAG-DBN2

PAG-DBN3

Client

Client

Client

Tape Library

Interface Nodes (IFN) S3 REST API Library Drives Control Up to 8 LTO drives Multiple active nodes

Database Nodes (DBN) Databases and Logs HTML admin interface Library Robotics Control Up to 4 nodes possible Up to 100 billion objects

per S3 bucket




PAG OEM Based on optical storage

technology Object interface Up to 181 PB capacity Up to 18 GB/sec

throughput

Cooperation with Sony OAI



21


DBN and IFN services on one node

Easy deployment

Single Node Solution

Tape Library Client Network

S3 API

DBN & IFN

Drives Control

Robotics Control



22


Read/Write Performance (examples)

1) May be limited by insufficient FC or Ethernet bandwidth or by maximum performance of clients

# Libraries # LTO-8 Drives

per Library Read/Write Rates

1 8 up to 7,200 MB/s1)

4 4 up to 14,400 MB/s1)

8 32 up to 230,400 MB/s1)

(theoretically)



23


Native Support for connected Drives, Loaders, Libraries No additional software or appliance required HPE, IBM, Overland-Tandberg, Qualstar, Quantum, Oracle StorageTek, … Rewritable and WORM media of LTO generations 5 and higher

Enhanced Support for Drive Features Logical Block Protection Data Compression and Encryption Drive Alarms, Error Logs and Cleaning

Management Functions for Offline Media Scheduled Media Quality Check Functions

Includes automatic copying and recovery if applicable

Barcode Labels required

LTO Device and Media Support



24


Architecture Overview

Archival Storage Device Group of up to 8 LTO tape

loaders or library units

Archival Storage Volume - LTO medium

Protected Volume Array (PVA) - 1-4 volumes - Protected by Erasure Codes

Archival Storage Partition Space provided by PVAs Expands automatically

Object Repository (Bucket)

S3

PVA PVA

Partition



25


Erasure Coding (Mirroring) Code rate 1/2 (or 1+1)

Ap

pli

cati

on

Ob

ject

Str

eam

S1 S1

S1

64K Segment Size {

Write Path

LTO Drives

S1

Ob

ject

Str

eam

Ap

pli

cati

on

S1

S1

Read Path

S3 PUT

Write Segmentation Encoding Write

Tape

Read Reassambling

S3 GET

Read



26


Erasure Coding Code rate 3/4 (or 3+1)

Ap

pli

cati

on

Ob

ject

Str

eam

S3 PUT

EC

S1

S2

S3

S1

S2

S3

Write Segmentation Encoding Write

Tape

EC

S1

S2

S3

64K Segment Size {

Write Path

LTO Drives

Read Path

Read

S1

S2

S3

Reassambling

Ob

ject

Str

eam

Ap

pli

cati

on

S3 GET

Read



27


PoINT Archival Gateway

Use Cases

Hyp

erS

tore

Au

to-T

ierin

g

Sto

rage

GR

ID

ILM

HD

FS

S3

a

ba

cke

nd

S3

AP

I

Any S3

compatible App

Ge

ne

ric S

3

Co

nn

ecto

r



28


Demo


29



www.point.de

Documents

PoINT Software & Systemskonferenz-nz.dlr.de/pages/samfs2019/present/1... · Policy-based File Tiering, Archiving and Migration Multi-Cloud Support PSM supports up to 20 cloud providers