Nos 3 He thong file phan tan Google

5/27/2018 Nos 3 He thong file phan tan Google

1/20

H iu hnh mngnng cao

Ging vin: Hong Xun Du

Email: [email protected]

Khoa Cng ngh thng tin 1Hc vin Cng ngh BC-VT


2/20HH mng nng cao Google File System 2

Gii thiu h thng file

phn tn ca GoogleGFS - Google File Systemhttp://research.google.com/archive/gfs.html



Ni dung

Mc ch pht trin GFS

Kin trc h thng

Hat ng Hiu nng h thng



Mc ch pht trin GFS

Phc v nhu cu lu tr v x l d liu vi khilng rt ln ca Google.

Cho php lu tr v x l cc file c dung lngln, c nhiu GB.

Cung cp khng gian tn c lp vi v tr lutr.

H thng GFS c kh nng kt hp hng ngnnt:

Dung lng lu tr ln n hng trm TB. Cho chp hng trm client truy nhp ng thi vi tn

sut cao.



Cc gi thit thit k

H thng c xy dng t nhng phncng tng i r tin, nn kh nng gps c l tng i cao.

H thng phi thng xuyn kim tra phthin s c v phc hi sau s c.

Cc files lu trong h thng thng ckch thc ln, t 100MB n hng chcGB/file.



Cc gi thit thit k (tip)

Ti h thng cu thnh t hai thao tcchnh:

Cc thao tc c theo dng (streamingreads): thng h thng c mt lng lnd liu, t hng trm KB n 1MB, hoc lnhn.

Cc thao tc c ngu nhin vi khi lngd liu nh.


7/20


8/20

HH mng nng cao Google File System 8

Kin trc h thng


9/20


Kin trc h thng (tip)

Gm mt master v nhiu chunkservers. Master: qun l ton b cc thng tin v h

thng file:

Khng gian tn (namespace) Thng tin kim sot truy nhp nh x t cc files n cc chunks (phn on) V tr hin ti ca cc chunks

Ton b cc thng tin trn c master lutrong b nh -> gim thi gian tm kim. GFS cn khong 64 bytes qun l mt chunk.


10/20



Chui cc chunkservers: lu tr d liu ccfile.

File c chia thnh cc chunks c kch thc c

nh. Mi chunk c nhn dng bng mt tn nhn

dng ton cc (chunk handle), di 64bits.

Kch thc chunk c nh l 64MB.

Mi chunk thng c 2 bn sao th cp (replicas) Master giao tip vi cc chunkservers thng

qua cc thng ip HeartBeat.


11/20



Master c 2 shadow masters lu bnsao d liu h thng. Khi master chnhgp s c, cc shadow master s thay

th tm thi.

Ton b cc thao tc ca master clog vo a cc b v bn sao th cp

phc v cho khi phc khi gp s c.


12/20


Cc thao tc trn file

GFS h tr cc thao tc file c s, baogm: to, xo, m, ng v ghi.

Ngoi ra, GFS cn h tr cc thao tc: Snapshot: to copy ca file hoc cy th mc

vi chi ph thp.

Record append (ni ui file): cho php nhiu

clients cng ng thi ghi ni ui file, mvn m bo tnh nguyn t ca mi thao tcghi ca tng client.


13/20


iu khin ghi v dng d liu


14/20


iu khin ghi v dng d liu

1. Client yu cu master cung cp thng tin v chunkscn truy nhp (v tr cc replicas).

2. Master tm kim v thng bo kt qu cho client.3. Client y cc thng tin cn ghi n tt c cc replicas

ca chunk.4. Khi cc replicas xc nhn nhn c d liu, clientgi yu cu ghi n primary replica.

5. primary replica chn trt t ghi chuyn tip yu cughi n cc replicas cn li.

6. Cc replicas thng bo hon tt thao tc ghi.7. primary replica thng bo li cho client. Nu c liclient s lp li cc bc t 3-7. Nu vn c li clients lp li ton b qu trnh ghi.


15/20


Th nghim o hiu nng

Cu hnh GFS th nghim: 1 master 2 master replicas

16 chunkservers 16 clients

Tt c cc my s dng cu hnh PIII

1.4GHz, 2GB RAM, 2x80GB 5400vng/pht HDD.

Switch 100Mb/s full-duplex.


16/20


Th nghim o hiu nng (tip)


17/20




18/20




19/20


H thng GFS-2 clusters - Cu hnh


20/20


H thng GFS-2 clusters - Hiu nng

Documents

Nos 3 He thong file phan tan Google