Upload
qlw5
View
697
Download
0
Embed Size (px)
Citation preview
HugeTable:Application-Oriented Structure Data Storage System
China Mobile Research Institute
HugeTable Project Team
Qian Ling
Agenda
Motivations
Hadoop, Hive & HBase
HT Design & Development
HT Applications
Further Plans
Motivations
� Huge Data Volumes
� Total data volumes: Several PB per system
� Daily data volumes: Several TB per system
� Longer retention period: several months
� Big potential: 200% increase in some area
� Multiple Applications Areas
� BOSS BI NMS Internet ...
� Data Integration
� Traditional Application Model
� SQL support
� Fast Index Query
� Multiple Application support
� Sensitive data
� CRUD support
� Statistic & Reporting
Data Warehouse
•Scalable
•High Available
•Reliable
+ App Solution
… Affordable
Hadoop: Raw Techniques
HDFS: distributed file system with fault tolerance
MapReduce: parallel programming
environments over HDFS
�Similar to the situation of POSIX API + Local FS
High Level Toolkits are initiated
Yahoo: PIG/Latin
Business.com: Cloudbase/Hadoop+JDBC
China Mobile: BC-PDM
Facebook: Hive/SQL
Hive: A Petabytes Scale Data Warehouse
Source: ICDE 2010/Facebook
• Schema support
• Pluggable Storage Engine I/F
• SQL � MR translation
• xDBC Driver
• Tools: HQL Console
• Admin: HWI
Features:
• Reporting
• Ad hoc Analysis
• Machine Learning
• Others
•Log analysis
•Trend detection
�Facebook has huge clusters
>1000 nodes
Usage Scenarios
HBase: structured storage of sparse data for Hadoop
Source: ApacheCon2009/ HBase
• ColumnFamilies
• ACID
• Optimized R/W
• BigTable I/F + BU
• Tools: HBase Shell
• Admin: Jetty Based
Features
• Social Service
• MapReduce Analysis
• Content Repository
• Wiki, RSS
• Near Realtime Reporting
& analytics
• Store web pages
Usage Scenarios
… Replacing SQL Systems
HugeTable: Application-Oriented Structure Data Storage System
Address the missing blocksIndex store & Query Optimizer
Access Control List
Insert, Update and Delete
Web-based Administration
Build Solutions for Telco ApplicationsNetwork Management System – NMS
Value-added System – VAS
Business Intelligence – BI
Other areas
HugeTable
ToolsClient
s
HFile w/
CF
Index
Store
AdminData,
config,
FM, Log,
Perf
I/F
A Brief History of HugeTable
2008 2009 2010
HT-p1 HT-p2 HT-p3
1.HBase-based
2.Partial xDBC/SQL
support
3. Integration HBase
with ZK before
official release4.Secondary Index5.Support Schema6.ACL support7.SQL console
1.Connect Hive with
HBase
3.Support HFile, CF in
Hive
2.Global Indexing4.Secondary Index5.Multiple DB support6.ACL support7.MR & Scan I/F8.Loader Tools, HT-Client9.Admin Portal10.JDBC remote console
1. Move to higher version
of Hive, Hadoop and
HBase
2. New Storage Engine
3. Fruitful external I/F
4. Many other
improvements4. Application Solution
HugeTable Building Blocks
HugeTableHugeTableHugeTable
HadoopHadoop
CoreCore
HadoopHadoopHBaseHBase
HadoopHadoop
HiveHive
HadoopHadoop
ZookeeperZookeeper
Storage
Computing
KVStore SQL-MR Lock
Cloud Cloud
MasterMaster
NMS
……
………
ApplicationsApplications
HBase as HugeTable Index Store
Index Meta Data
Index Data
HBase
Create Index
Drop Index
Query Engine
Select … using index xxx
Select … where idxcol
Find Index
Load Service
Find Index
HT Loader
Write Index
Read Index
Check Index
Index Store Implementation
<10 sec<10 secN/AIndex Query
Load Speed
Memory
Consumption
20 Nodes,1TB/Node
>5MB/s·Node(Primary Index)
2.5MB/s·Node(Primar
y Index)
20MB/s·Node(No
Index)
2GB/Node*TB8GB/Node*TBNo Additional cost
HT-p2HTHTHTHT----p1p1p1p1Hive
Primary Index: index into data file
Secondary Index: index into primary index
Exact match and Range scan
Integrated with Hive ql and other modules
HugeTable IUD Support
Goal: Support Insert, Update and Delete on application data.
Meta Data
IUD Table
HBase
IUD Statement
Query EngineFind IUD table
Write IUD Data
Read IUD Data
Select
HT Data
HDFS
Offline Merger
HugeTable Access Control
Goal: Support Multiple Users from Multiple Applications , w/o mutual trust
Database privileges:
1. Meta Data: Index, Create,
Drop
2. User Data: IUD
User Access Level:
1. System Administrator
2. User Manager
3. User
Meta Data
DDL/DML
ACL ModuleCheck Privileges
Loader/PortalGrant/Revoke
Administration Portal
Goal: Unified HugeTable management point, decrease management effort
Data Management
DB/TBL/IDX
User Management
Add/Delete/Modify
Monitor & FM
Log/Alert/Service
Configuration
Deploy/Setup
HugeTable Application API
JDBC/SQL API MapReduce API
• Migration of traditional database
applications
• For SQL developer
• Batch processing & interactive
• Compatible with Hadoop MR API
• For data analysis, e.g. data mining
• Work with HT records format
• Access control
BigTable API
• BigTable/HBase style API
• For NoSQL application, on HFile2
• Range scan, Key-value access
• Access Control
Various kinds of ApplicationsVarious kinds of Applications
public void map(LongWritable key, HugeRecord value,
OutputCollector<HugeRecordRowKey, HugeRecord> output,
Reporter reporter);
public void reduce(HugeRecordRowKey key,
Iterator<HugeRecord> values,
OutputCollector<HugeRecordRowKey, HugeRecord> output,
Reporter reporter);
Table table = new Table("gdr", "admin", "admin");
String[] families = new String[] {"default"};
String[] partitions = new String[] {"dt=20100317"};
int limit = 10;
TableScannerInterface tsi = table.getScanner(
new byte[0],new byte[] {Byte.MAX_VALUE},families, partitions);
for (int i=0; i<limit; ++i) {
GroupValue gv = tsi.next();
for (String family : families) {
System.out.println(family + " = " + Bytes.toString(gv.getByteValue(family)));
}
}
HugeTable based Telco Application Solutions
Heavy Requirements, e.g.
Batch processing
Complex data analysis
Interactive query on CDR
Statistic and reporting
HugeTable
Cluster
+
DataMing
Tool kits
Data Source
Data Source
Data
Aggregator
Database
Data
warehouse
Telco App
Interactive
Simple Query
Interactive
Complex Query
Complex
Analyze
Reporting
Mass Data Store
Batch processing
Statistic
Future works
Column Sorage Engine
File Format
Compression
Local Index
Global Index
Query Optimization
Join Optimization: index
Load Optimization
Parallel Load
Application Solution
Thanks for your time!Thanks for your time!Thanks for your time!Thanks for your time!China Mobile Research Institute