ACG_Rackspace.pdf

Elastic “Big Data” Infrastructure Rackspace® Private Cloud powered by OpenStack® Use Case

by Natasha Gajic

Analytical Compute Grid (ACG)

October 17, 2012

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Rackspace’s EBI Environment

Current Environment

Windows and Linux operating systems

Oracle and Microsoft databases solutions

Microsoft and Oracle replication technology

SSIS

Informatica

Dedicated servers

Rapid data set growth

“Big Data” Problem

Cost of purchasing additional licenses

Time required to set up new hardware

Increased demand for DBA resources

System performance

System scalability

Capacity

2


Analytical Compute Grid (ACG) Features

•Host ever growing set of data

•Quick data collection and retrieval

•Rapid scalability

•Ease of maintenance

•Provide standard data access API

3


Analytical Compute Grid (ACG) Features

•Ability to provide variety of storage types:

• Columnar

• Relational

• HDFS

•Enable users to select optimal storage

type for information collected

•Leverage Rackspace® Private Cloud

powered by OpenStack® and open

source technology

4


Analytical Compute Grid (ACG) Quality Attributes

5


ACG on Rackspace® Private Cloud powered by OpenStack®

High Level Architecture

6



7

8 Hypervisor Servers each:

Dual Socket Six Core 2.4GHz Processors

96GB RAM

Terabytes of Storage

*The environment will grow significantly next year

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 8


Image



Database Engine Selection

Columnar Cassandra

Relational PostgreSQL

HDFS Hadoop



Node



Node



Node



Node



Controller



Controller



Controller



API



Indexing Structure

18



Indexing Structure

• ACG Indexing Structure:

• Resides on a set of Rackspace® Private Cloud powered by OpenStack® instances



Indexing Structure



• It is a set of pointers ultimately addressing database entities



Indexing Structure




• ACG Controller manages Indexing Structure



Indexing Structure




• ACG Controller manages Indexing Structure

• Dynamically expands vertically and horizontally to address a growing data set



Indexing Structure

• ACG Indexing Structure Enables:

• Distribution of data bases across many instances



Indexing Structure



• Splitting large data sets across many instances



Indexing Structure




• Parallelization of large data set queries



Indexing Structure





• Deploying data stores with optimal configuration, minimizing maintenance



Indexing Structure





• Deploying data stores with optimal configuration, minimizing maintenance

• Accessing data residing in

variety of storage types via

uniform interface



Sorter & Aggregator

• ACG Sorter & Aggregator Enables:

• Joining the results from multiple ACG nodes



Sorter & Aggregator



• Result sorting and aggregation



Sorter & Aggregator



• Result sorting and aggregation

• Together with temporary segment it will support

joining heterogeneous data sets



Quality Attributes

31



Quality Attributes - Performance

32

Rackspace® Private Cloud

powered by OpenStack®

ACG




33



Creates ACG node in 30 seconds

ACG




34




Creates ACG nodes concurrently

ACG




35





ACG

Controlled data set size resulting

in:




36





ACG


in: Quick data distribution




37





ACG



Query parallelization




38





ACG



Query parallelization

Fast data retrieval



Quality Attributes – Scalability

39



ACG




40



Quick and concurrent ACG node

creation

ACG




41




creation

Ability to re-size existing nodes

ACG




42




creation


Ability to remove nodes

ACG




43




creation


Ability to remove nodes

ACG

Indexing structure and controlled

data set size allow ACG to

stabilize quickly as it expands or

contracts



Quality Attributes – Availability

44



ACG




45



Rapidly replace failed ACG nodes

ACG




46



Rapidly replace failed ACG nodes

ACG

Deploys data store native

availability mechanisms

(replication, data distribution…)



Quality Attributes – Maintainability

47



ACG




48



Adding ACG nodes expands:

ACG




49




Storage capacity

ACG




50




Storage capacity

CPU power

ACG




51




Storage capacity

CPU power

RAM

ACG




52




Storage capacity

CPU power

RAM

No DBA or system administrators

activity required

ACG




53




Storage capacity

CPU power

RAM


activity required

ACG

Controlled data set size enables:




54




Storage capacity

CPU power

RAM


activity required

ACG

Controlled data set size enables: Optimal and stable data store

configuration




55




Storage capacity

CPU power

RAM


activity required

ACG


configuration

Reducing demand for managing

data store objects




56




Storage capacity

CPU power

RAM


activity required

ACG


configuration

Reducing demand for managing

data store objects

Stable query execution plans



Quality Attributes – Flexibility

ACG




ACG ACG

Variety of storage types:




ACG ACG

Variety of storage types: Columnar – Cassandra : time series data




ACG ACG


Relational – PostgreSQL : relational data




ACG ACG



HDFS – Hadoop : un-structured data




ACG ACG



HDFS – Hadoop : un-structured data

Ability to select optimal storage

type for individual use case



Quality Attributes – Usability

ACG ACG




ACG ACG

Standard interfaces:




ACG ACG


SQL language




ACG ACG


SQL language

JDBC API




ACG ACG


SQL language

JDBC API

Data store native calls




ACG ACG


SQL language

JDBC API


Native bulk loader utility




ACG ACG


SQL language

JDBC API


Native bulk loader utility

ACG will support joining

heterogeneous data sets



Rackspace Use Case

70


• Subject:

• Complex availability calculation sourcing 3 months of monitoring data and creating 1 billion records in initial calculation

71


Rackspace Use Case


• Environment 1

• Data Warehouse Microsoft SQL server database

• SSIS data loading

• SQL server with 24 CPUs and 250GB RAM was dedicated to the initial calculation

• SQL server stored procedure performed the calculation

• Source and result are stored in traditional data warehouse structure

72


Rackspace Use Case


• Environment 2

• In 30 seconds, ACG Node Manager instantiated new columnar data store consisting of 4 Cassandra nodes, and registered it in ACG Indexing Structure

• Each ACG node has 2CPUs and 8GB RAM

• Informatica data loading

• Calculation developed in Java

• Source and result are stored in columnar structure suitable for time series data

73


Rackspace Use Case


• Calculation Duration

•Microsoft SQL Server lasted 5 days

•ACG calculation completed in 3.5 hours

• Storage Size

• Microsoft SQL server 500GB

•ACG 20 GB

• Complexity of the calculation

•Columnar data store is optimal for time series data. Sourcing from columnar data store resulted in relatively simple Java calculation process comparing to SQL server stored procedure

74


Rackspace Use Case - Result


• Selecting optimal data store for use case resulted in:

• Substantial performance improvement

75


Rackspace Use Case - Conclusion




• Reduced storage demand

76







•Simplified processes

77








•Ability to process terabytes of data per day close to real-time and on-demand

78








•Ability to process terabytes of data per day close to real-time and on-demand

•Improved trending and reporting:

• enhances support capabilities

• improved Rackspace customer experience

79







• Simplified processes

• Ability to process terabytes of data per day close to real-time and on-demand

• Improved trending and reporting:

• enhances support capabilities

• improved Rackspace customer experience

• Significant cost reduction

80



81

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

Documents

ACG_Rackspace.pdf