View
224
Download
1
Category
Tags:
Preview:
Citation preview
Almaden
Rice University
Nache: Design and Implementation of a Caching Proxy for NFSv4
Ajay Gulati, Rice University Manoj Naik, IBM Almaden Renu Tewari, IBM Almaden
Almaden
© 2003 IBM Corporat2006n
Rice
Talk Outline
Federated File System
NFSv4: Overview and New Features
Nache Architecture
Nache Implementation
Experimental Results
Conclusions and Discussion
Almaden
© 2003 IBM Corporat2006n
Rice
Federated File System
File system federation with unified namespace
Flexible data management
– Replication, migration and read/write caching across distributed file servers
What are the options?
– AFS, DCE/DFS provide distributed FS over WAN, but…
• difficult to deploy, not widely used
– GPFS, Lustre are cluster filesystems, but…
• good for controlled environments, no federation
– NFS is a standard, but…
• v2/v3 have been designed mainly for LAN, are chatty and not suited for WAN, firewall issues
– NFSv4?
Almaden
© 2003 IBM Corporat2006n
Rice
Why NFSv4?
Open industry standard
Optimizations for WAN
– Compound operations• A single RPC request contains multiple ops, reduces round-trips
– Read-Write delegations • Efficient cache consistency, no need to contact server once a
delegation is obtained• Delegations can be recalled on conflicting access
Federation support
– Client redirection• Server returns NFS4ERR_MOVED to redirect clients• Client requests fs_locations attribute• Client connects to new location
Almaden
© 2003 IBM Corporat2006n
Rice
AlmadenAlmaden HoustonHouston New YorkNew York
Admin
ClientsClients
Filesystem Namespace Unification
Separate namespace across servers
Unified namespace for clients
Almaden
© 2003 IBM Corporat2006n
Rice
ClientsClients(San Jose)(San Jose)
Server (San Server (San Jose)Jose)
Server Server
(New York)(New York)
/project
1. OPEN /project/p12. NFS4ERR_MOVED3. GETATTR4. fs_locations
Data Access using NFSv4 Redirection
Limitations High network latency when clients access remote servers Delegation are unsuited for data sharing, fewer awarded and frequently broken No unified caching, every client has to access remote server
6. /project/p1
7. DELEGRETURN
5.Client connects to New York server
8. Server responds
Almaden
© 2003 IBM Corporat2006n
Rice
What is a Proxy Cache (Nache)?
An NFSv4 server acts as a proxy for another remote
NFS server
– caches data for reads and writes using delegations
– becomes the temporary “owner” of the data servicing
opens, lock requests and reads and writes
Multiple clients can share a proxy cache using a single
server-to-proxy delegation
Extends beyond web-proxy caching
Almaden
© 2003 IBM Corporat2006n
Rice
Data Access using a Proxy Cache
ClientsClients(San Jose)(San Jose)
Proxy Cache Proxy Cache
(San Jose)(San Jose)
ServerServer
(New York)(New York)
/project
3./project/p1
1.OPEN /project/p1
Benefits with NFSv4 Proxy Reduced network traffic and delay
– Clients access files through local proxy cache– Clients can share delegations – fewer conflicts/recalls
Improved performance for cached files – Clients can collectively take advantage of aggressive readahead, write back and local locking at the proxy– Both data and delegations are shared
2. Delegate /project/p1 to proxy
Almaden
© 2003 IBM Corporat2006n
Rice
Nache Architecture
Nache Server Nache Client
VFS Layer
Buffer Cache cacheFS
Local Client(export to)
Remote Server
(mount from)
(Linux nfsd module) (Linux nfs module)
Nache
Almaden
© 2003 IBM Corporat2006n
Rice
Nache Implementation
Cascaded mounts
NFS operation forwarding
Sub-operation RPC call handling
Almaden
© 2003 IBM Corporat2006n
Rice
Cascaded Mounts
Proxy exports a NFS mounted file system
Added export functionality for NFS
– export_ops added, not available in vanilla NFS
Nache Server
/watson/almaden
mount server:/ /almaden
ls /net at client shows contents of /watson at server
Client
mount nache:/ /net
/net /export
/export
Almaden
© 2003 IBM Corporat2006n
Rice
NFS Operation Forwarding
NFS ops are redirected from nfsd to nfs using vfs layer
Issues: certain operations are stateful
– OPEN, CREATE, LOCK, ULOCK, CLOSE
ClientClient NacheNache ServerServer
VFS
Nache Client NFSD
VFSVFS
NFS Client Nache Server
Application
Almaden
© 2003 IBM Corporat2006n
Rice
Sub-Operation RPC calls
Each FS operation at Nache server translates to a separate RPC call from Nache client
Solution:– Remove such redundant checks from code path
Nache Server Nache Client
VFS Layer
Local Client Remote Server
(Linux nfsd module) (Linux nfs module)1.OPEN
2.LOOKUP3.ACCESS4.OPEN
Almaden
© 2003 IBM Corporat2006n
Rice
Experimental Methodology
Setup
– 2-6 local NFS clients (San Jose)
– One local proxy with modified kernel modules (San Jose)
– One local NFS server (San Jose)
– One remote NFS server (New York)
Benchmarks
– Filebench (developed by Sun)
– Compilation of various packages – Linux kernel, emacs, gdb
– Our micro-benchmarks
Almaden
© 2003 IBM Corporat2006n
Rice
Experiment Categories
Delegation Benefits
Nache Benefits
Nache Overhead
Almaden
© 2003 IBM Corporat2006n
Rice
Read Delegation Benefits
Client repeats open-read-close on 100 files
Server ops reduced by 50% due to delegations
0
2
4
6
8
10
12
14
16
1 2 4 8 16 32
Number of Iterations
NF
S O
ps a
t ser
ver
(in'0
00) No Delegations
With Delegations
Almaden
© 2003 IBM Corporat2006n
Rice
Write Delegation Benefits
A client repeats open-read/write-close on 100 files
Server ops reduced by 4 times
0
10
20
30
40
50
60
1 2 4 8 16 32
Number of Iterations
NFS
Ops
at s
erve
r (in
'000
)
No Delegations
With Delegations
Almaden
© 2003 IBM Corporat2006n
Rice
Delegation Benefits (Server load)
Server ops reduced due to delegations
kernel emacs gdb webserver varmail
NF
S O
ps a
t ser
ver
(in'0
00)
With Delegations
Without Delegations
49
848
1.62.8
5.4
20.8
1529
2225
Almaden
© 2003 IBM Corporat2006n
Rice
Delegation Benefits (Client performance)
Time taken is lower and ops/sec is higher due to delegations
kernel emacs gdb webserver varmail
Thr
ough
put (
Ops
/sec
) or
Tim
e T
aken
(m
ins)
With Delegation
Without Delegation
14
29
2 6.5
565
390 9592
Time (minutes)
Ops/second
Almaden
© 2003 IBM Corporat2006n
Rice
Experiment Categories
Delegation Benefits
Nache Benefits
Nache Overhead
Almaden
© 2003 IBM Corporat2006n
Rice
Nache Benefits - Filebench (webserver)
Ops at server reduces by 38% with 4 clients
0
8
16
24
32
40
1 Client 2 Clients 3 Clients 4 Clients
NF
S O
ps a
t ser
ver
per
100
benc
hmar
k op
s
Without Nache
With Nache
Almaden
© 2003 IBM Corporat2006n
Rice
Nache Benefits – Kernel compile
Ops at server unaffected by increase in number of clients
0
30
60
90
120
150
180
210
240
1 Client 2 Clients 3 Clients 4 Clients
NF
S O
ps a
t ser
ver
(in '0
00)
Without Nache
With Nache
Almaden
© 2003 IBM Corporat2006n
Rice
Nache Benefits – Compile Time
Compile time unaffected by increase in number of clients
1
10
100
1000
Kernel Gdb Emacs
Tim
e ta
ken
over
WA
N (
min
utes
)
No Nache
With Nache (1 Client)
With Nache (2 Clients)
With Nache (4 Clients)
Almaden
© 2003 IBM Corporat2006n
Rice
Experiment Categories
Delegation Benefits
Nache Benefits
Nache Overhead
Almaden
© 2003 IBM Corporat2006n
Rice
Nache Overhead over WAN (Throughput)
Micro benchmark
Ops/s (Nache)
Ops/s (no Nache)
Overhead(%)
Create-Files 9 16 43
Random-Reads 77 48 -60
Random-Writes 8.6 10 14
Random-Append 2.2 2.4 8
Nache induces an overhead of 8-43% Code can be further optimized to reduce overheads
Almaden
© 2003 IBM Corporat2006n
Rice
Nache Overhead over WAN (Latency)
Micro benchmarkLatency (Nache)
Latency (no Nache)
Overhead (%)
Create-Files 5217 3009 42
Random-Reads 12.9 20.7 -37
Random-Writes 116.3 98 18
Random-Append 883.4 821 8
Proxy induces an overhead of 8-42% Code can be further optimized to reduce overheads
Almaden
© 2003 IBM Corporat2006n
Rice
Result Summary
Delegations provide better throughput, lower latency and reduced load at server
Nache improves performance in presence of sharing among clients
Overhead of Nache in absence of sharing is not too high except for few cases and can be further reduced
Almaden
© 2003 IBM Corporat2006n
Rice
Conclusions
NFSv4 is a good alternative for building federated file
systems
Delegations help in reducing the number of server
operations and provide better caching
Nache can be integrated with a federated file system
architecture and improve performance in presence of
sharing
Recommended