Embed Size (px)
Citation preview
Hadoop Management Console
-User Account
-Share System
-Management Console
2015 中国 Hadoop 技术峰会
2015.07
About me
Gus, NieHadoop Engineer
Email: [email protected]
Oracle DBA -Teradata DBA - Hadoop Engineer –
Agenda
• Problems! Problems! Problems!
• We need Change
• HMC Solution
• Example
• Current & Next
• Q&A
Problems
dn2 dn3dn1
dn4 dn5
dn8 rank1dn7
dn2 dn3dn1
dn4 dn5
dn8 rank2dn7 dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn7
dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn7
Over 2000 nodes.
Account Increase
Problems
• All accounts share the cluster’s resource without limit.
Storage Usage
Problems
• All accounts share the cluster’s resource without limit.
Problems
• Small files always there. Small Files
>1GB
<1MB
<1GB
<128MB
<10MB
<1MB
Problems
• Small files always there.
0
10000000
20000000
30000000
40000000
50000000
60000000
Small Files Accounts
>1GB
<1MB
<1GB
<128MB
<10MB
Problems
• Compute resource - Queues Usage.
Problems
• Compute resource - Queues Usage.
Account - Queue
We need Change
• All accounts share the cluster’s resource without limit.
• Small files always there.
• Queues Usage in a mess.
• All accounts share the cluster’s resource without limit.
• Small files always there.
• Queues Access in a mess.
Resource Allocation
Self-Service
Dynamic
We need Change
Resource Allocation
Self-Service
Dynamic
Hierarchy Queue
Storage Quota
Apply Request
Auto Approve
Real time release
Pre-define capacity
We need Change
HMC Solution
Ruby on Rails
MySQL
KDC
Command Console
Resource Manager
LDAP
HDFS
HMC Solution
HMC
Fsimage
Delimiter File
Upload to HDFS
Aggregation to
Dir ViewFilter Quota Data
Hadoop Nodes HDFS
Update Real Time Data
Parse Schedul
er
Dump Schedu
ler
Command
Console
HMC Solution
HMC
Hadoop Nodes HDFS
Remote
Scripts
Set Name Quota
Set Space Quota
Dump to Schedule
r
Scheduler
XMLRefresh
Scheduler
Command
Console
HMC Solution
LDAP
Manager • Account• Kerberos• Directory
HMC Solution
3/6TBSpace quota /
Stand File Size
Name Quota
QuotaAdmin
Default
more
HMC Solution
Default QueueLDAP
HMC Solution
Default QueueLDAP
SLA Admin
Example
Default Resource
Example
Current & Next
• All accounts share the cluster’s resource without limit.
Storage Usage
Current & Next
• All accounts share the cluster’s resource without limit.
Storage Usage
• Small files always there.Small Files
>1GB
<1MB
<1GB
<128MB
<10MB <1MB
Current & Next
• Small files always there.
Current & Next
Small Files
>1GB
<1MB
<1GB
<128MB
<10MB<1MB
• Small files always there.
Current & Next
0
2000000
4000000
6000000
8000000
10000000
12000000
14000000
16000000
18000000
Small Files Accounts
>1GB
<1MB
<1GB
<128MB
<10MB
• Small files always there.
Current & Next
0
0.5
1
1.5
2
2.5
Account - Queue
• Time Window Scheduler.
• Dynamic mapping overwrite.
• Account full function self-service.
• ACL for HDFS
Current & Next
Q&A
