DeDu: Building a Deduplication Storage system over Cloud computing
This paper appears in : Computer Supported Cooperative work in Design(CSCWD) ,2011 15th InternationalData of Conference: 8-10 June 2011Author(s): Zhe Sun, Jun Shen, Fac. of inf., Univ. of Wollongong, Wollongong, NSW, Australia Jianming Yong, Fac. of bus., Univ. of Southern Queensland, Toowoomab, QLD ,Australia
Speaker: Yen-Yi Chen MA190104Date:2013/05/28
Outline
• Introduction• Two issues to be addressed• Deduplication• Theories and approaches • System design• Simulations and Experiments• Conclusions
Introduction
• 雲端運算興起、分散式系統架構• 資訊爆炸、資料海量• 儲存設備成本上升• 增加資料傳輸與減緩佔用網路頻寬
Introduction
• System name: DeDu• Front-end: deduplication application• Back-end: Hadoop Distributed File System
• HDFS• HBase
Two issues to be addressed
• How does the system identify the duplication? *hash function-MD5 and SHA-1
• How does the system manage the data? *HDFS and HBase
Deduplication
A C
B A
C
B
C
CA
AB
A
A
A
B
B
C
C
B
Data Store Data StoreData Store
a a
a
c
b
b
1. Data chunks are evaluated to determine a unique signature for
each
2. Signature values are compared to identify all
duplicates
3.Duplicate data chunks are replaced
with pointes to a single stored chunk. Saving
storage space
類別 File-level Block-level
重複資料比對層級 檔案 區塊
重複資料比對範圍 整個指定磁碟區 整個指定磁碟區
優點對單一檔案的容量刪減效果最好
可跨檔案比對,也能比對不同檔案底層的重複部份
缺點對已編碼檔案無效,對完全相同的兩份檔案仍會重複儲存
較消耗處理資源
重複資料刪檢比例 1:2~1:5 1:200甚至更高
Theories and approaches
A. The architecture of source data and link filesB. Architecture of deduplication cloud storage system
Source data and link files
Deduplication Cloud storage system
System design
A. Data organisationB. Storage of the filesC. Access to the filesD. Deletion of files
Data organisation
Storage of the files
Access to the files
Deletion of files
Simulations and Experiments
Performance evaluations
Conclusions
• 1. The fewer the data nodes, the writing efficiency is high; but the reading efficiency is low;• 2. The more data nodes, the writing efficiency is low, but reading efficiency is hight;• 3. single file is big, the time to calculate hash values becomes higher ; but transmission cost is low;• 4.single file is small, the time to calculate hash
values becomes lower ; but transmission cost is high.
Thanks for your listening