Object storage in Cloud Computing and Embedded Processing. Jan Jitze Krol. Systems Engineer. DDN | We Accelerate Information Insight. Established: 1998 Revenue: $226M (2011) – Profitable, Fast Growth Main Office: Sunnyvale, California, USA Employees: 600+ Worldwide - PowerPoint PPT Presentation
Object storage in Cloud Computing and Embedded ProcessingJan Jitze KrolSystems Engineerddn.com2012 DataDirect Networks. All Rights Reserved.
DDN | We Accelerate Information Insight2DDN is a Leader in Massively Scalable Platforms and Solutions for Big Data and Cloud ApplicationsEstablished: 1998Revenue: $226M (2011) Profitable, Fast GrowthMain Office: Sunnyvale, California, USAEmployees: 600+ WorldwideWorldwide Presence: 16 CountriesInstalled Base: 1,000+ End Customers; 50+ CountriesGo To Market: Global Partners, Resellers, DirectWorld-Renowned & Award-Winning
6/8/12ddn.com2012 DataDirect Networks. All Rights Reserved.Fact: amount of data is growing, fast3ddn.com2012 DataDirect Networks. All Rights Reserved.If all the words spoke since the dawn of mankind would be stored in ASCII format then it would take up about 5 to 6 Exa BytesIn 2007 94% of the data produced and stored was digital.
3Disk drives grow bigger, not faster or betterDisk drives havent changed that much over the last decadeThey just store more, ~40GB in 2002, 4000 GB in 2012Access times are about the same, ~ 5 10 msWrite/read speeds are about the same ~ 50 -100MB/sRead error rate is about the same 1 error per 10^14 bits read, or one guaranteed read error per 12 TB read.
4ddn.com2012 DataDirect Networks. All Rights Reserved.AgendaHighlight two of DDNs initiatives to deal with large repositories of data:
The Web Object Scaler, WOS
Embedded data processing, aka in-storage processing5ddn.com2012 DataDirect Networks. All Rights Reserved.Challenges of ExaByte scale storageExponential growth of dataThe expectation that all data will be available everywhere on the planet. Management of this tidal wave of data becomes increasingly difficult with regular NAS :Introduced in the 90s (when ExaByte was a tape drive vendor)With 16TB file system sizes that many still have Management intensiveLUNs, Volumes, Aggregates,Heroically management intensive at scaleAntiquated resiliency techniques that dont scaleRAID ( disk is a unit in RAID, whereas drive vendors consider a sector a unit)Cluster failover, standby replication, backupFile Allocation Tables, Extent ListsFocused on structured data transactions (IOPS)File locking overhead adds cost and complexity
6ddn.com2012 DataDirect Networks. All Rights Reserved.Hyperscale Storage | Web Object Scaler
NoFS Hyperscale Distributed Collaboration
ddn.com2012 DataDirect Networks. All Rights Reserved.What is WOSA data store for immutable dataThat means we dont need to worry about lockingNo two systems will write to the same object
Data is stored in objectsWritten with policiesPolicies drive replication
Objects live in zones
Data protection is achieved by replication or erasure codingReplicate within a zone or between zonesData is available from every WOS node in the cluster
Only three operations possible, PUT, GET and DELETE
ddn.com2012 DataDirect Networks. All Rights Reserved.Can handle ExaByte scale of data and beyondEasy to install, upgrade and manageReplication is part of the designA cluster of WOS nodes can be split in zonesReplication within a zones or between zonesMove data to where it will be usedNo RAIDProtection through replication orErasure coding orBothLoss of a sector not a disasterSelf healing
Simple, simple, keep it simple8
Universal Access to Support a Variety of Applications
WOS Hyperscale Geo-Distributed Storage Cloud
WOS Native Object Interface
NFS, CIFSLDAP,ADHA S3 & WebDAV APIs Secure File Sharing Multi-tenancyC++, Python, Java, PHP, HTTP, REST interfacesPUT, GET, DELETE
Rules EngineRich MetadataAutomationddn.com2012 DataDirect Networks. All Rights Reserved.What can you build with this? 10CIFS 1
Illumina Hi-SeqIonTorrent PGM
CIFS 1Remote site 1
Remote site 2Remote site 3Note: multiple name spacesMultiple storage protocolsBut ONE shared storage systemNFS 1iRODSiRODS
CIFS 1CIFS 2NFS 1CIFS 2iRODSCIFS 2iRODSddn.com2012 DataDirect Networks. All Rights Reserved.Storing ExaBytes, ZettaBytes or JottaBytes of data is only part of the story. The data needs to be processed too, which means as fast as possible access. ddn.com2012 DataDirect Networks. All Rights Reserved.The time it takes to read the data is lost time from the applications perspective.11What is Embedded Processing?And why ?Do data intensive processing as close to the storage as possible.Bring computing to the data instead of bring data to computingHADOOP is an example of this approach.
Why Embedded Processing?Moving data is a lot of workA lot of infrastructure needed
So how do we do that?
Client sends a request to storage (red ball)Storage responds with data (blue ball)ClientStorageBut what we really want is :ddn.com2012 DataDirect Networks. All Rights Reserved.Storage Fusion Architecture (SFA)RAID ProcessorsSystem memoryHigh-Speed CacheInterface processorInterface processorInterface processorInterface processor2345678PRAID 5,6QRAID 62341mQRAID 6PRAID 5,6111Interface VirtualizationInternal SAS SwitchingRAID ProcessorsSystem memoryHigh-Speed CacheInterface processorInterface processorInterface processorInterface processorInterface VirtualizationInternal SAS SwitchingCache Link8 x IB QDR or 16 x FC8 Host PortsUp to 1,200 disks in an SFA 10KOr 1,680 disks in an SFA 12Kddn.com2012 DataDirect Networks. All Rights Reserved.13Repurposing Interface ProcessorsIn the block based SFA10K platform, the IF processors are responsible for mapping Virtual Disks to LUNs on FC or IB
In the SFA10KE platform the IF processors are running Virtual Machines
RAID processors place data (or use data) directly in (or from) the VMs memoryOne hop from disk to VMs memory
Now the storage is no longer a block deviceIt is a storage appliance with processing capabilities
ddn.com2012 DataDirect Networks. All Rights Reserved.One SFA-10KE controllerRAID ProcessorsSystem memoryHigh-Speed CacheVirtualMachine
Interface VirtualizationInternal SAS SwitchingCache Link8 x IB QDR/10GbE Host Ports (No Fibre Channel)VirtualMachine
ddn.com2012 DataDirect Networks. All Rights Reserved.15Example configurationNow we can put iRODS inside the RAID controllersThis give iRODS the fastest access to the storage because it doesnt have to go onto the network to access a fileserver. It lives inside the fileserver.
We can put the iRODS catalogue, iCAT, on a separate VM with lots of memory and SSDs for DataBase storage
The following example is a mix of iRODS with GPFSThe same filesystem is also visible from an external compute cluster via GPFS running on the remaining VMs
This is only one controller, there are 4 more VMs in the other controller need some work too They see the same storage and can access it at the same speed.
On the SFA-12K we will have 16 VMs available running on Intel Sandy Bridge processors. (available Q3 2012)
ddn.com2012 DataDirect Networks. All Rights Reserved.Example configurationRAID ProcessorsSystem memoryVirtualMachine
Interface VirtualizationInternal SAS Switching8x 10GbE Host PortsVirtualMachine
Linux16 GB memory allocated SFA DriverLinux8 GB memory allocated
RAID sets with 2TB SSDLinuxLinuxHigh-Speed CacheCache LinkGPFSSFA DriverGPFSSFA DriverGPFSiCATSFA Driver8GB memory allocated 8GB memory allocated RAID sets with 300TB SATARAID sets with 30TB SASddn.com2012 DataDirect Networks. All Rights Reserved.17Running Micro Services inside the controllerSince iRODS runs inside the controller we now can run iRODS MicroServices right on top of the storage.
The storage has become an iRODS appliance speaking iRODS natively.
We could create hot directories that kick off processing depending on the type of incoming data.
ddn.com2012 DataDirect Networks. All Rights Reserved.Massive throughput to backend storage
Virtual Machine 5Virtual Machine 7Virtual Machine 6Virtual Machine 8
DDN | SFA10KE With iRODS and GridScaler parallel filesystemiRODSclients
Virtual Machine 1Virtual Machine 3Virtual Machine 2Virtual Machine 4DDNGRIDScalerClient DDNGRIDScalerNSDDDNGRIDScalerNSDDDNGRIDScalerNSDCompute clusterStart processingClients are sending data to the built-iniRODS server. iRODS was configured with a rule toconvert the orange ball into a purple one.Note that this is done on the storage itselfAfter the conversion another MicroServicesubmits a processing job on the cluster toprocess the uploaded and pre-processed dataMetadataComputingReading data via parallel paths withDDN | GRIDScaler embedded in the system.Faster access->faster processing->faster resultsWriting the result of the processingRegistering the data with iRODSAn iRODS rule requires thata copy of the data is sent to a remote iRODS serverAll of this happened because of a few rules in iRODS that triggered on the incoming data. In other words, the incoming data drives the processing.Here is a configuration with iRODS for datamanagementand GridScaler for fast scratch space. Data can come infrom clients or devices such as sequencers.
(click to continue)ddn.com2012 DataDirect Networks. All Rights Reserved.Thank Youddn.com2012 DataDirect Networks. All Rights Reserved.Backup slides21
ddn.com2012 DataDirect Networks. All Rights Reserved.Object Storage ExplainedObject storage stores data into containers, called objectsEach object has both data and user defined and system defined metadata (a set of attributes describing the object)
Objects are sto