VISION Cloud: Data-intensive Storage Services on Clouds

Dimosthenis Kyriazis

National Technical University of Athens

OGF 35 June 17-19, 2012

Delft, Netherlands

Data Deluge: The Emerging Zettabyte Age

2

CAGR Entities (67x)

(# files/containers)

Information (44x)

Storage (30x)

IT Professionals (1.4x)

2020 Source: IDC 2010 Digital Universe Study

Source: http://gigaom.files.wordpress.com/2010/05/2010-digital-universe-iview_5-4-10.pdf

Growth of Unstructured Data is a Key Driver for Storage Clouds

3

Some statistics

– Unstructured data accounts for more than 90% of the digital universe*

– Medical images may be up to 30% of the world’s storage**

Clouds hold a large portion of this storage

– Amazon S3 holds more than 900 billion objects (1Q12)***

– YouTube streams 4B videos daily****

*The Exploding Digital Universe, 2008, and Digital Universe Study, April 2010, June 2011

**http://www.healthcaretechnologyonline.com/doc.mvc/10-Healthcare-IT-Trends-To-Watch-In-2010-0001

***http://aws.typepad.com/aws/2012/04/amazon-s3-905-billion-objects-and-650000-requestssecond.html

**** http://www.youtube.com/t/press_statistics

According to the IDC, by 2020, 15% of

all digital info will be in a cloud and

another 20% will be touched by the

cloud

Source: IDC's Digital Universe Study, sponsored by EMC, June 2011

Adopting cloud storage solutions?

4

86%

[Above the Clouds: A Berkley View of Cloud Computing]

[L. Willcocks, W. Venters, E. Whitley - Accenture]

[Randy Bias – VP Technology Strategy of GoGrid, ServePath]

[CeBIT 2011]

[M. Malek - Google]

Goal

Architect and implement an infrastructure for the

reliable and effective delivery of data-intensive

storage services, facilitating the convergence of

ICT, media and telecommunications

Facts

A 3-year project, started Oct 2010

€15.709 M (total budget all partners)

www.visioncloud.eu

torage Srtualized Vi VISION Cloud:

etnn for the Future Interoces FoundatiiServ

5

Enabling data-intensive storage services on clouds (1/2)

Data Mobility and Federation

A layer enabling unified access to data across storage clouds and

federation of data objects maintained by users across different

administrative domains

Raise the Abstraction Level of Storage

A new data model enabling the encapsulation of storage into objects with

user-defined and system-defined attributes / metadata used for handling

both an object and its content

Content-Centric Storage Access

Access to data objects through information about their content and its

relationships (build content networks) rather than details of underlying

storage containers

6

Enabling data-intensive storage services on clouds (2/2)

Computational Storage

Computational agents (storlets) released into the cloud and activated by

events on data. The storlets define not only the computation but also

triggering conditions whereby storlets are activated

Advanced Capabilities for Cloud-based Storage

Support multi-tenancy, guarantee secure and authorized access to the data

and services, check compliance with standards and regulations, optimize

service provision with regard to cost, monitor, analyze and manage SLA

considering content-related terms

7

Emphasizing on Data Mobility

Goal: Single view of storage across multiple providers

Barriers are both technological and business-related

VISION Cloud is focusing on

techniques for data deduplication

data transfer technologies

Proposed architecture

A layer that enables unified access to data across storage clouds, and

federates sets of data objects maintained by users across different

administrative domains

Built in network optimizations to overcome limitations by employing new

mechanisms, extending lazy copying techniques, network deduplication

and network overlays

Allows leveraging of the information captured by metadata for executing

data transfers 8

Operating Layer

Access and

Interface Layer

Data Access Layer

(DAL)

Management Interface

Layer (MIL)

Data Operating Layer

(DOL)

Management Operating

Layer (MOL)

Data Access Management / Control

DATA SERVICE

Content networks/objects,

Computation on storage,

Mobility, availability, reliability, security

MANAGEMENT SERVICE

Monitoring, Metering, Billing,

Security management, Tenant/User management, SLA

The VISION Cloud Architecture

Unified Management Model

Use Cases

Multi-tenant cloud-native business applications Healthcare data access based on content and its relationships

Create, repurpose & distribute content through multiple channels Mobile users accessing content

Data: storing, retrieving, associating, analyzing, distributing…

12

5 Million TB of data on

the internet, Google has

indexed only the 200TB

(0.004%)

Session Overview - Agenda

VISION Cloud - Towards Interoperable Storage Clouds

A media use case for storage clouds (Mirko Lorenz - Deutsche

Welle)

Use of CDMI & challenges (Alex McDonald, SNIA Cloud

Storage Initiative)

Highlighting challenges on Federation & Interoperability for

data storage cloud (Massimo Villari - University of Messina)

Logging management for storage cloud compliance (Andrea

Manieri - Engineering)

Discussion / roundtable on VISION Cloud interoperability

issues and how OGF can address them

13

Thank You

Visit our Website: http://www.visioncloud.eu

Dimosthenis Kyriazis

(dimos@mail.ntua.gr)

National Technical University of Athens

Backup

16 VISION Cloud

Content Centric Interface

Content Centric Interface

Basic

Object

Interface

Object

Relations

Interface

Operational

Statistics

Interface

Metrics

Object

Interface

Metadata

Upload

Interface e.g. put object e.g. define relations e.g. how many

downloads e.g. search for

coordinates

e.g. import of large

metdata sets

Storlet Life Cycle and States

Physical Model

Data Center

Data Center

Data Center

100s of Data Centers (DC)

Data Center

Each DC 10s of storage clusters

Each storage cluster 100s of servers with direct attached disks

Storage Cluster 3

Storage Cluster 2 Storage

Cluster 1

Global View

Global

Catalog

Storlet

Execution Replicator

Catalog

Request Handler

Storage Service

Object

Service

Communicate with peer in other clusters

Communicate with peer in other clusters

Application requests

Secure Access

Service

Access Enforcer

Content Centric

Service

Logically global components

Components that have an independent instance on each cluster node Logically global components

User Services

Identity/Policy

Services

Data Access Layer

Data Operating Layer

Resource

Map

Data Access and Operating Layers

Computational

Storage Service

Secure Access Control Component

Global View

Management Interface and Operating Layers

Global

Catalog

Resource

Map

User Services

Identity/Policy

Services

API

Management Interface Layer

Models development Tenants, subtenants and user accounts management

SLA management

Containers management

Accounting and Billing

Secu

re A

cces

s

Management Operating Layer

Customer Developer Service

KB

Analysis

Monitor

SLA Management

DM Placement and Allocation

Compliance

Logical View of Storlet Execution Environment

We deduplicate content within a tenant while

ensuring there are replicas at each desired location

Tenant j’s

logical view