TFS: A Transparent File System for Contributory Storage · 2019. 2. 25. · TFS: A Transparent File System for Contributory Storage James Cipar Mark D. Corner Emery D. Berger Department

TFS: A Transparent File System for Contributory Storage

James Cipar Mark D. Corner Emery D. Berger

Department of Computer ScienceUniversity of Massachusetts Amherst

Amherst, MA 01003{jcipar, mcorner, emery}@cs.umass.edu

Abstract

Contributory applications allow users to donate unusedresources on their personal computers to a shared pool.Applications such as SETI@home, Folding@home, andFreenet are now in wide use and provide a variety of ser-vices, including data processing and content distribution.However, while several research projects have proposedcontributory applications that support peer-to-peer stor-age systems, their adoption has been comparatively lim-ited. We believe that a key barrier to the adoption ofcontributory storage systems is that contributing a largequantity of local storage interferes with the principal userof the machine.

To overcome this barrier, we introduce the TransparentFile System (TFS). TFS provides background tasks withlarge amounts of unreliable storage—all of the currentlyavailable space—without impacting the performance ofordinary file access operations. We show that TFS al-lows a peer-to-peer contributory storage system to pro-vide 40% more storage at twice the performance whencompared to a user-space storage mechanism. We an-alyze the impact of TFS on replication in peer-to-peerstorage systems and show that TFS does not appreciablyincrease the resources needed for file replication.

1 Introduction

Contributory applications allow users to donate unusedresources from their personal computers to a shared pool.These applications harvest idle resources such as CPUcycles, memory, network bandwidth, and local storage toserve a common distributed system. These applicationsare distinct from other peer-to-peer systems because theresources being contributed are not directly consumedby the contributor. For instance, in Freenet [8], allusers contribute storage, and any user may make useof the storage, but there is no relationship betweenuser data and contributed storage. Contributory appli-

cations in wide use include computing efforts like Fold-ing@home [17] and anonymous publishing and contentdistribution such as Freenet [8]. The research commu-nity has also developed a number of contributory appli-cations, including distributed backup and archival stor-age [30], server-less network file systems [1], and dis-tributed web caching [11]. However, the adoption ofstorage-based contributory applications has been limitedcompared to those that are CPU-based.

Two major barriers impede broader participation incontributory storage systems. First, existing contribu-tory storage systems degrade normal application perfor-mance. While transparency—the effect that system per-formance is as if no contributory application is running—has been the goal of other OS mechanisms for networkbandwidth [34], main memory [7], and disk schedul-ing [19], previous work on contributory storage systemshas ignored its local performance impact. In particu-lar, as more storage is allocated, the performance of theuser’s file system operations quickly degrades [20].

Second, despite the fact that end-user hard drives areoften half empty [10, 16], users are generally reluctantto relinquish their free space. Though disk capacity hasbeen steadily increasing for many years, users view stor-age space as a limited resource. For example, three of theFreenet FAQs express the implicit desire to donate lessdisk space [12]. Even when users are given the choiceto limit the amount of storage contribution, this optionrequires the user to decide a priori what is a reasonablecontribution. Users may also try to donate as little as pos-sible while still taking advantage of the services providedby the contributory application, thus limiting its overalleffectiveness.

Contributions: This paper presents the TransparentFile System (TFS), a file system that can contribute100% of the idle space on a disk while imposing a neg-ligible performance penalty on the local user. TFS oper-ates by storing files in the free space of the file systemso that they are invisible to ordinary files. In essence,

FAST ’07: 5th USENIX Conference on File and Storage TechnologiesUSENIX Association 215

normal file allocation proceeds as if the system were notcontributing any space at all. We show in Section 5 thatTFS imposes nearly no overhead on the local user. TFSachieves this both by minimizing interference with thefile system’s block allocation policy and by sacrificingpersistence for contributed space: normal files may over-write contributed space at any time. TFS takes severalsteps that limit this unreliability, but because contribu-tory applications are already designed to work with un-reliable machines, they behave appropriately in the faceof unreliable files. Furthermore, we show that TFS doesnot appreciably impact the bandwidth needed for repli-cation. Users typically create little data in the course ofa day [4], thus the erasure of contributed storage is neg-ligible when compared to the rate of machine failures.

TFS is especially useful for replicated storage systemsexecuting across relatively stable machines with plentifulbandwidth, as in a university or corporate network. Thisenvironment is the same one targeted by distributed stor-age systems such as FARSITE [1]. As others have shownpreviously, for high-failure modes, such as wide-areaInternet-based systems, the key limitation is the band-width between nodes, not the total storage. The band-width needed to replicate data after failures essentiallylimits the amount of storage the network can use [3].In a stable network, TFS offers substantially more stor-age than dynamic, user-space techniques for contributingstorage.

Organization: In Section 2, we first provide a de-tailed explanation of the interference caused by contrib-utory applications, and discuss current alternatives forcontributing storage. Second, we present the design ofTFS in Section 3, focusing on providing transparency tonormal file access. We describe a fully operating imple-mentation of TFS. We then explain in Section 4 the in-teraction between machine reliability, contributed space,and the amount of storage used by a contributory stor-age system. Finally, we demonstrate in Section 5 that theperformance of our TFS prototype is on par with the filesystem it was derived from, and up to twice as fast asuser-space techniques for contributing storage.

2 Interference from Contributing Storage

All contributory applications we are aware of are config-ured to contribute a small, fixed amount of storage—thecontribution is small so as not to interfere with normalmachine use. This low level of contribution has little im-pact on file system performance and files will generallyonly be deleted by the contributory system, not becausethe user needs storage space. However, such small, fixed-size contributions limit contribution to small-scale stor-age systems.

Instead of using static limits, one could use a dynamicsystem that monitors the amount of storage used by lo-cal applications. The contributory storage system couldthen use a significantly greater portion of the disk, whileyielding space to the local user as needed. Possibleapproaches include the watermarking schemes found inElastic Quotas [18] and FS2 [16]. A contributory storagesystem could use these approaches as follows: wheneverthe current allocation exceeds the maximum watermarkset by the dynamic contribution system, it could deletecontributory files until the contribution level falls belowa lower watermark.

However, if the watermarks are set to comprise allfree space on the disk, the file system is forced to deletefiles synchronously from contributed storage when writ-ing new files to disk. In this case, the performance of thedisk would be severely degraded, similar to the synchro-nous cleaning problem in LFS [31]. For this reason, Elas-tic Quotas and FS2 use more conservative watermarks(e.g., at most 85%), allowing the system to delete fileslazily as needed.

Choosing a proper watermark leaves the system de-signer with a trade-off between the amount of storagecontributed and local performance. At one end of thespectrum, the system can contribute little space, limitingits usefulness. At the other end of the spectrum, localperformance suffers.

To see why local performance suffers, consider the fol-lowing: as a disk fills, the file system’s block allocationalgorithm becomes unable to make ideal allocation de-cisions, causing fragmentation of the free space and al-located files. This fragmentation increases the seek timewhen reading and writing files, and has a noticeable ef-fect on the performance of disk-bound processes. In anFFS file system, throughput can drop by as much as 77%in a file system that is only 75% full versus an emptyfile system [32]—the more storage one contributes, theworse the problem becomes. The only way to avoid thisis to maintain enough free space on the disk to allow theallocation algorithm to work properly, but this limits con-tribution to only a small portion of the disk.

Though some file systems provide utilities to defrag-ment their disk layout, these utilities are ineffective whenthere is insufficient free space on the file system. For in-stance, the defragmentation utility provided with olderversions of Microsoft Windows will not even attempt todefragment a disk if more than 85% is in use. On mod-ern Windows systems, the defragmentation utility willrun when the disk is more than 85% full, but will givea warning that there is not enough free space to defrag-ment properly [22]. When one wants to contribute all ofthe free space on the disk, they will be unable to meetthese requirements of the defragmentation utility.

FAST ’07: 5th USENIX Conference on File and Storage Technologies USENIX Association216

0

10

20

30

40

50

60

0 10 20 30 40 50

Contribution (%)

Tim

e(s

)

Ext2 Copy

TFS Copy

Figure 1: The time required to perform a series of filesystem operations while contributing different amountsof storage. As the amount of contributed space increases,the time it takes for Ext2 to complete the experiment alsoincreases. However, the performance of TFS stays nearlyconstant. Error bars represent standard deviation.

Any user-space scheme to manage contributed storagewill be plagued by the performance problems introducedby the contributed storage—filling the disk with data in-evitably slows access. Given a standard file system in-terface, it is simply not possible to order the allocationof data on disk to preserve performance for normal files.As Section 3 shows, by incorporating the mechanismsfor contribution into the file system itself, TFS maintainsfile system performance even at high levels of contribu-tion.

Figure 1 depicts this effect. This figure shows the timetaken to run the copy phase of the Andrew Benchmarkon file systems with different amounts of space beingconsumed. The full details of the benchmark are pre-sented in Section 5. As the amount of consumed spaceincreases, the time it takes to complete the benchmarkincreases. We assume that the user is using 50% of thedisk space for non-contributory applications, which cor-responds to results from a survey of desktop file sys-tem contents [10]. The figure shows that contributingmore than 20% of the disk space will noticably affect thefile system’s performance, even if the contributed stor-age would have been completely idle. As a preview ofTFS performance, note that when contributing 35% ofthe disk, TFS is twice as fast as Ext2 for copying files.

3 Design and Implementation of TFS

The Transparent File System (TFS) allows users to do-nate storage to distributed storage systems with minimalperformance impact. Because the block allocation policyis one of the primary determinants of file system perfor-mance, designers have devoted considerable attention to

tuning it. Accordingly, deviating from that policy can re-sult in a loss of performance. The presence of data on thefile system can be viewed as an obstruction which causesa deviation from the default allocation policy. The goalof TFS is to ensure the transparency of contributed data:the presence of contributed storage should have no mea-surable effect on the file system, either in performance,or in capacity. We use the term transparent files for fileswhich have this property, and transparent data or trans-parent blocks for the data belonging to such files. Atransparent application is an application which strivesto minimize its impact on other applications running onthe same machine, possibly at the expense of its own per-formance.

Section 3.1 shows how we achieve transparency withrespect to block allocation in the context of a popularfile system, Ext2 [5]. Ext2 organizes data on disk usingseveral rules of thumb that group data on disk accord-ing to logical relationships. As we show in Section 5,TFS minimally perturbs the allocation policy for ordi-nary files, yielding a near-constant ordinary file perfor-mance regardless of the amount of contributed storage.

In exchange for this performance, TFS sacrifices filepersistence. When TFS allocates a block for an ordinaryfile, it treats free blocks and transparent blocks the same,and thus may overwrite transparent data. Files markedtransparent may be overwritten and deleted at any time.This approach may seem draconian, but because repli-cated systems already deal with the failure of hosts in thenetwork, they can easily deal with the loss of individualfiles. For instance, if one deletes a file from a BitTorrentpeer, other peers automatically search for hosts that havethe file.

The design of TFS is centered around tracking whichblocks are allocated to which kind of file, preserving per-sistence for normal user files, and detecting the overwrit-ing of files in the contributed space. As the file system isnow allowed to overwrite certain other files, it is imper-ative that it not provide corrupt data to the contributionsystem, or worse yet, to the user. While our design canpreserve transparency, we have also made several smallperformance concessions which have minimal effect onnormal file use, but yield a better performing contribu-tion system. Additionally, file systems inevitably havehot spots, possibly leading to continuous allocation anddeallocation of space to and from contribution. These hotspots could lead to increased replication traffic elsewherein the network. TFS incorporates a mechanism that de-tects these hot-spots and avoids using them for contribu-tion.


3.1 Block AllocationTFS ensures good file system performance by minimiz-ing the amount of work that the file system performswhen writing ordinary files. TFS simply treats transpar-ent blocks as if they were free, overwriting whatever datamight be currently stored in them. This policy allowsblock allocation for ordinary files to proceed exactly asit would if there were no transparent files present. Thisapproach preserves performance for ordinary files, butcorrupts data stored in transparent files. If an applicationwere to read the transparent file after a block was over-written, it would receive the data from the ordinary file inplace of the data that had been overwritten. This presentstwo issues: applications using transparent files must en-sure the correctness of all file data, and sensitive informa-tion stored in ordinary files must be protected from appli-cations trying to read transparent files. To prevent botheffects, TFS records which blocks have been overwrittenso that it can avoid treating the data in those blocks asvalid transparent file data. When TFS overwrites a trans-parent file, it marks it as overwritten and allocates theblock to the ordinary file.

This requires some modifications to the allocation pol-icy in Ext2. Blocks in a typical file system can only be inone of two states: free and allocated. In contrast, in TFSa storage block can be in one of five states: free, allo-cated, transparent, free-and-overwritten, and allocated-and-overwritten.

Figure 2 shows a state transition diagram for TFSblocks. Ordinary data can be written over free or trans-parent blocks. If the block was previously used bytransparent data, the file system marks these blocks asallocated-and-overwritten. When a block is denoted asoverwritten, it means that the transparent data has beenoverwritten, and thus “corrupted” at some point. Trans-parent data can only be written to free blocks. It can-not overwrite allocated blocks, other transparent blocks,or overwritten blocks of any sort. Figure 3 shows thisfrom the perspective of the block map. Without TFS,appending to a file leads to fragmentation, leading to aperformance loss in the write, and in subsequent reads.In TFS, the file remains contiguous, but the transparentfile data is lost and the blocks are marked as allocated-and-overwritten.

When a process opens a transparent file, it must verifythat none of the blocks have been overwritten since thelast time it was opened. If any part of the file is over-written, the file system returns an error to open. Thissignals that the file has been deleted. TFS then deletesthe inode and directory entry for the file, and marks allof the blocks of the file as free, or allocated. As ordi-nary files cannot ever be overwritten, scanning the al-location bitmaps is not necessary when opening them.This lazy-delete scheme means that if TFS writes trans-

!"##$%%&'()#*

+"(,-.("#,)!"##/(,*/

01#"2"3))#,

$%%&'()#*/(,*/

01#"2"3))#,

4#%#)#

+"(,-.("#,)

5"3)#6

+"(,-.("#,)

5"3)#6

7%#(,7%#(,

+"(,-.("#,)

4#%#)#

4#%#)#

5"3)#

5"3)#

Figure 2: A state diagram for block allocation in TFS.The Free and Allocated states are the two allocationstates present in the original file system. TFS adds threemore states.

parent files and never uses them again, the disk will even-tually fill with overwritten blocks that could otherwise beused by the transparent storage application. To solve this,TFS employs a simple, user-space cleaner that opens andcloses transparent files on disk. Any corrupted files willbe detected and automatically deleted by the open oper-ation.

Though scanning the blocks is a linear operation inthe size of the file, very little data must be read from thedisk to scan even very large files. On a typical Ext2 filesystem, if we assume that file blocks are allocated con-tiguously, then scanning the blocks for a 4GB file willonly require 384kB to be read from the disk. In theworst case—where the file is fragmented across everyblock group, and we must read every block bitmap—approximately 9.3MB will be read during the scan, as-suming a 100GB disk.

Many file systems, including Ext2 and NTFS, denote ablock’s status using a bitmap. TFS augments this bitmapwith two additional bitmaps and provides a total of threebits denoting one of the five states. In a 100GB file sys-tem with 4kB blocks, these bitmaps use only 6.25MBof additional disk space. These additional bitmaps mustalso be read into memory when manipulating files. How-ever, very little of the disk will be actively manipulatedat any one time, so the additional memory requirementsare negligible.

3.2 Performance ConcessionsThis design leads to two issues: how TFS deals with opentransparent files and how TFS stores transparent meta-data. In each case, we make a small concession to trans-parent storage at the expense of ordinary file system per-


formance. While both concessions are strictly unneces-sary, their negative impact on performance is negligibleand their positive impact on transparent performance issubstantial.

First, as TFS verifies that all blocks are clean only atopen time, it prevents the file system from overwritingthe data of open transparent files. One alternative wouldbe to close the transparent file and kill the process withthe open file descriptor if an overwritten block is de-tected. However, not only would it be difficult to traceblocks to file descriptors, but it could also lead to datacorruption in the transparent process. In our opinion,yielding to open files is the best option. We discuss otherstrategies in Section 7.

It would also be possible to preserve the transparentdata by implementing a copy-on-write scheme [26]. Inthis case, the ordinary file block would still allocate itstarget, and the transparent data block would be moved toanother location. This is to ensure transparency with re-gards to the ordinary file allocation policy. However, touse this strategy, there must be a way to efficiently de-termine which inode the transparent data block belongsto, so that it can be relinked to point to the new block.In Ext2, and most other file systems, the file system doesnot keep a mapping from data blocks to inodes. Accord-ingly, using copy-on-write to preserve transparent datawould require a scan of all inodes to determine the ownerof the block, which would be prohibitively expensive.It is imaginable that a future file system would providean efficient mapping from data blocks to inodes, whichwould allow TFS to make use of copy-on-write to pre-serve transparent data, but this conflicts with our goal ofrequiring minimal modifications to an existing operatingsystem.

Second, TFS stores transparent meta-data such as in-odes and indirect blocks as ordinary data, rather thantransparent blocks. This will impact the usable space forordinary files and cause some variation in ordinary blockallocation decisions. However, consider what would hap-pen if the transparent meta-data were overwritten. If thedata included the root inode of a large amount of trans-parent data, all of that data would be lost and leave aneven larger number of garbage blocks in the file sys-tem. Determining liveness typically requires a full trac-ing from the root as data blocks do not have reverse map-pings to inodes and indirect blocks. Storing transparentstorage metadata as ordinary blocks avoids both issues.

3.3 Transparent Data AllocationAs donated storage is constantly being overwritten by or-dinary data, one concern is that constant deletion willhave ill effects on any distributed storage system. Everytime a file is deleted, the distributed system must detectand replicate that file, meanwhile returning errors to any

!""#$%&'(

)*''

+*%,-.%*',&/

0#,&*123&'(45.%$'!""#$%&'(6%,(6

78'*9*1&&',

:1&;#3& +

)5 :1&; +)5

Figure 3: The block map in a system with and withoutTFS. When a file is appended in a normal file system itcauses fragmentation, while in TFS, it yields two over-written blocks.

0.00001

0.0001

0.001

0.01

0.1

1

10

50 60 70 80 90 100

Percent of Disk Available to TFS

Blo

ckA

lloca

tion

Rat

e(k

B/s

)Machine 1

Machine 2

Figure 4: Cumulative histogram of two user machine’sblock allocations. 70% of the blocks on machine 2’s diskwere never allocated during the test period, and 90% ofthe blocks were allocated at a rate of 0.1kB/s or less. Thiscorresponds to the rate at which TFS would overwritetransparent data if it were to use a given amount of thedisk.

peers that request it. To mitigate these effects, TFS iden-tifies and avoids using the hot spots in the file system thatcould otherwise be used for donation. The total amountof space that is not used for donation depends on thebandwidth limits of the distributed system and is con-figurable, as shown in this section.

By design, the allocation policy for Ext2 and otherlogically organized file systems exhibits a high degreeof spatial locality. Blocks tend to be allocated to onlya small number of places on the disk, and are allocatedrepeatedly. To measure this effect, we modified a Linuxkernel to record block allocations on two user worksta-tions machines in our lab. A cumulative histogram of thetwo traces is shown in Figure 4. Machine 1 includes 27trace days, and machine 2 includes 13 trace days. We can


observe two behaviors from this graph. First, while oneuser is a lot more active than the other, both show a greatdeal of locality in their access—machine 2 never allo-cated any blocks in 70% of the disk. Second, an averageof 1kB/s of block allocations is approximately 84MB ofallocations per day. Note that this is not the same as cre-ating 84MB/s of data per day—the trace includes manyshort-lived allocations such as temporary lock files.

Using this observation as a starting point, TFS can bal-ance the rate of block deletion with the usable storageon the disk. Using the same mechanism that we usedto record the block allocation traces shown in Figure 4,TFS generates a trace of the block addresses of all or-dinary file allocations. It maintains a histogram of thenumber of allocations that occurred in each block and pe-riodically sorts the blocks by the number of allocations.Using this sorted list, it finds the smallest set of blocksresponsible for a given fraction of the allocations.

The fraction of the allocations to avoid, f , affects therate at which transparent data is overwritten. Increasingthe value of f means fewer ordinary data allocations willoverwrite transparent data. On the other hand, by de-creasing the value of f , more storage becomes availableto transparent data. Because the effects of changing f aredependent on a particular file system’s usage pattern, wehave found it convenient to set a target loss rate and allowTFS to determine automatically an appropriate value forf . Suppose ordinary data blocks are allocated at a rate ofα blocks per second. If f is set to 0 – meaning that TFSdetermines that the entire disk is usable by transparentdata – then transparent data will be overwritten at a rateapproximately equal to α. The rate at which transparentdata is overwritten t is approximately β = (1 − f)α.Solving for f gives f = 1 − β

α . Using this, TFS candetermine the amount of storage available to transparentfiles, given a target rate. Using this map of hot blocks inthe file system, the allocator for transparent blocks treatsthem as if they were already allocated to transparent data.

However, rather than tracking allocations block-by-block, we divide the disk into groups of disk blocks,called chunks, and track allocations to chunks. Eachchunk is defined to be one eighth of a block group.This number was chosen so that each block group couldkeep a single byte as a bitmap representing which blocksshould be avoided by transparent data. For the defaultblock size of 4096 bytes, and the maximum block groupsize, each chunk would be 16MB.

Dividing the disk into multi-block chunks ratherthan considering blocks individually greatly reduces thememory and CPU requirements of maintaining the his-togram, and due to spatial locality, a write to one blockis a good predictor of writes to other blocks in the samechunk. This gives the histogram predictive power inavoiding hot blocks.

It should be noted that the rate at which transparentdata is overwritten is not exactly α because, when atransparent data block is overwritten, an entire file is lost.However, because of the large chunk size and the highlocality of chunk allocations, subsequent allocations forordinary data tend to overwrite other blocks of the sametransparent file, making the rate at which transparent datalost approximately equal to the rate at which blocks areoverwritten.

0.00001

0.0001

0.001

0.01

0.1

1

10

50 55 60 65 70 75 80 85 90 95 100

Percent of Disk Available to TFS

Rat

eof

Dat

aL

oss

(kB

/s) Machine 1

Machine 2

Figure 5: This shows the simulated rate of TFS data losswhen using block avoidance to avoid hot spots on thedisk. For example, when TFS allows 3% of the disk togo unused, the file system will allocate data in the usedportion of the disk at a rate of 0.1kB/s. By using blockavoidance, TFS provides more storage to contributoryapplications without increasing the rate at which data islost.

The figure of merit is the rate of data erased for a rea-sonably high allocation of donated storage. To examineTFS under various allocations, we constructed a simula-tor using the block allocation traces used earlier in thesection. The simulator processes the trace sequentially,and periodically picks a set of chunks to avoid. When-ever the simulator sees an allocation to a chunk which isnot being avoided, it counts this as an overwrite. We ranthis simulator with various fixed values of f , the fractionof blocks to avoid, and recorded the average amount ofspace contributed, and the rate of data being overwrit-ten. Figure 5 graphs these results. Note that the graphstarts at 80% utilization. This is dependent on the high-est value for f we used in our test. In our simulation, thiswas 0.99999. Also notice that, for contributions less thanapproximately 85%, the simulated number of overwritesis greater than the number given in Figure 4. This is be-cause, for very high values of f , the simulator’s adap-tive algorithm must choose between many chunks, noneof which have received very many allocations. In this


case, it is prone to make mistakes, and may place trans-parent data in places that will see allocations in the nearfuture. These results demonstrate that for machine 2’susage pattern, TFS can donate all but 3% of the disk,while only erasing contributed storage files at 0.08kB/s.As we demonstrate in the section 4, when compared tothe replication traffic due to machine failures, this is neg-ligible.

3.4 TFS ImplementationWe have implemented a working prototype of TFS in theLinux kernel (2.6.13.4). Various versions of TFS havebeen used by one of the developers for over six monthsto store his home directory as ordinary data and Freenetdata as transparent data.

TFS comprises an in-kernel file system and a user-space tool for designating files and directories as eithertransparent or opaque, called setpri. We implementedTFS using Ext2 as a starting point, adding or modifyingabout 600 lines of kernel code, in addition to approxi-mately 300 lines of user-space code. The primary modifi-cations we made to Ext2 were to augment the file systemwith additional bitmaps, and to change the block alloca-tion to account for the states described in Section 3. Ad-ditionally, the openVFS call implements the lazy-deletesystem described in the design. In user-space, we modi-fied several of the standard tools (including mke2fs andfsck) to use the additional bitmaps that TFS requires.We implemented the hot block avoidance histograms inuser-space using a special interface to the kernel driver.This made implementation and experimentation some-what easier; however, future versions will incorporatethose functions into the kernel. An additional benefit isthat the file system reports the amount of space availableto ordinary files as the free space of this disk. This causesutilities such as df, which are used to determine disk uti-lization, to ignore transparent data. This addresses theconcerns of users who may be worried that contributoryapplications are consuming their entire disk.

In our current implementation, the additional blockbitmaps are stored next to the original bitmaps as filesystem metadata. This means that our implementationis not backwards-compatible with Ext2. However, if wemoved the block bitmaps to Ext2 data blocks, we couldcreate a completely backwards-compatible version, eas-ing adoption. We believe that TFS could be incorporatedinto almost any file system, including Ext3 and NTFS.

4 Storage Capacity and Bandwidth

The usefulness of TFS depends on the characteristics ofthe distributed system it contributes to, including the dy-namics of machine availability, the available bandwidth,and the quantity of available storage at each host. In this

section, we show the relationship between these factors,and how they affect the amount of storage available tocontributory systems. We define the storage contributedas a function of the available bandwidth, the uptime ofhosts, and the rate at which hosts join and leave the net-work. Throughout the section we will be deriving equa-tions which will be used in our evaluation.

4.1 Replication DegreeMany contributory storage systems use replication to en-sure availability. However, replication limits the capacityof the storage system in two ways. First, by storing re-dundant copies of data on the network, there is less over-all space [2]. Second, whenever data is lost, the systemmust create a new replica.

First we calculate the degree replication needed as afunction of the average node uptime. We assume thatnodes join and leave the network independently. Thoughthis assumption is not true in the general case, it greatlysimplifies the calculations here, and holds true in net-works where the only downtime is a result of node fail-ures, such as a corporate LAN. In a WAN where thisassumption does not hold, these results still provide anapproximation which can be used as insight towards thesystem’s behavior. Mickens and Noble provide a morein-depth evaluation of availability in peer-to-peer net-works [21].

To determine the number of replicas needed, we use aresult from Blake and Rodrigues [3]. If the fraction oftime each host was online is u, and each file is replicatedr times, then the probability that no replicas of a file willbe available at a particular time is

(1 − u)r. (1)

To maintain an availability of a, the number of replicasmust satisfy the equation

a = 1 − (1 − u)r. (2)

Solving for r gives the number of replicas needed.

r =ln(1 − a)ln(1 − u)

(3)

We consider the desired availability a to be a fixedconstant. A common rule of thumb is that “five nines” ofavailability, or a = 0.99999 is acceptable, and the valueu is a characteristic of host uptime and downtime in thenetwork. Replication could be accomplished by keepingcomplete copies of each file, in which case r would haveto be an integer. Replication could also be implementedusing a coding scheme that would allow non-integer val-ues for r [28], and a different calculation for availability.In our analysis, we simply assume that r can take anyvalue greater than 1.


4.2 Calculating the Replication BandwidthThe second limiting effect in storage is the demand forreplication bandwidth. As many contributory systemsexhibit a high degree of churn, the effect of hosts fre-quently joining and leaving the network [13, 33], repair-ing failures can prevent a system from using all of itsavailable storage [3]. When a host leaves the network forany reason, it is unknown when or if the host will return.Accordingly, all files which the host was storing mustbe replicated to another machine. For hosts that werestoring a large volume of data, failure imposes a largebandwidth demand on the remaining machines. For in-stance, a failure of one host storing 100GB of data every100 seconds imposes an aggregate bandwidth demand of1GB/s across the remaining hosts. In this section, weconsider the average bandwidth consumed by each node.When a node leaves the network, all of its data must bereplicated. However, this data does not have to be repli-cated to a single node. By distributing the replication,the maximum bandwidth demanded by the network canbe very close to the average.

The critical metric in determining the bandwidth re-quired for a particular storage size is the session timeof hosts in the network: the period starting when thehost joins the network, and ending when its data must bereplicated. This is not necessarily the same as the time ahost is online—hosts frequently leave the network for ashort interval before returning.

Suppose the average storage contribution of each hostis c, and the average session time is T . During a host’ssession, it must download all of the data that it will storefrom other machines on the network. With a session timeof T , and a storage contribution of c, the average down-stream bandwidth used by the host is B = c

T . Becauseall data transfers occur within the contributory storagenetwork, the average downstream bandwidth equals theaverage upstream bandwidth.

In addition to replication due to machine failures, bothTFS and dynamic watermarking cause an additional bur-den due to the local erasure of files. If each host loses filedata at a rate of F , then the total bandwidth needed forreplication is

B =c

T+ F. (4)

Solving for the storage capacity as a function of band-width gives

c = T · (B − F ). (5)

The file failure rate F in TFS is measurable using themethods of the previous section. The rate at which filesare lost when contributing storage by the dynamic wa-termarking scheme is less than the rate of file loss withTFS. When using watermarking, this rate is directly tiedto the rate at which the user creates new data.

If the value c is the total amount of storage contributedby each host in the network, then for a replication factorof r, the amount of unique storage contributed by eachhost is

C =c

r=

T · (B − F )r

. (6)

The session time, T , is the time between when a hostcomes online and when its data must be replicated to an-other host, because it is going offline, or has been offlinefor a certain amount of time. By employing lazy repli-cation—waiting for some threshold of time, t, beforereplicating its data—we can extend the average sessiontime of the hosts [2]. However, lazy replication reducesthe number of replicas of a file that are actually online atany given time, and thus increases the number of replicasneeded to maintain availability. Thus, both T , the sessiontime, and r, the degree of replication, are functions of t,this threshold time.

C =T (t)(B − F )

r(t)(7)

The functions T (t) and r(t) are sensitive to the fail-ure model of the network in question. For instance, in acorporate network, machine failures are rare, and sessiontimes are long. However, in an Internet-based contribu-tory system, users frequently sign on for only a few hoursat time.

5 TFS Evaluation

Our goal in evaluating TFS is to assess its utility forcontributing storage to a peer-to-peer file system. Wecompare each method of storage contribution that we de-scribe in Section 2 to determine how much storage canbe contributed, and the effects on the user’s applicationperformance. We compare these based on several met-rics: the amount of storage contributed, the effect on theblock allocation policy, and the overall effect on localperformance.

In scenarios that are highly dynamic and bandwidth-limited, static contribution yields as much storage capac-ity as any of the other three. If the network is more stable,and has more bandwidth, the dynamic scheme providesmany times the storage of the static scheme; however,it does so at the detriment of local performance. Whenbandwidth is sufficient and the network is relatively sta-ble, as in a corporate network, TFS provides 40% morestorage than dynamic watermarking, with no impact onlocal performance. TFS always provides at least as muchstorage as the other schemes without impacting local per-formance.


5.1 Contributed Storage CapacityTo determine the amount of storage available to a con-tributory system, we conduct trace-based analyses usingthe block avoidance results from Section 3, the analysisof the availability trace outlined in Section 3.3, and therelationship between bandwidth and storage described inSection 4.2. From these, we use Equation 7 to determinethe amount of storage that can be donated by each hostin a network using each of the three methods of contri-bution.

We assume a network of identical hosts, each with100GB disks that are 50% full, and we use the blocktraces for Machine 2 in Section 3.3 as this machine rep-resents the worst-case of the two machines. Given a fixedrate of data loss caused by TFS, we determine the maxi-mum amount of data that can be stored by TFS based onthe data from Figure 5. We assume that the fixed con-tribution and the dynamic contribution methods causeno data loss. Though the dynamic contribution methoddoes cause data loss as the user creates more data on thedisk, the rate of data creation by users in the long term islow [4]. We assume that the amount of non-contributedstorage being used on the disk is fixed at 50%. For fixedcontribution, each host contributes 5% of the disk (5GB), the dynamic system contributes 35% of the disk,and TFS contributes about 47%, leaving 3% free to ac-count for block avoidance.

To determine the functions T (t) and r(t), we analyzeavailability traces gathered from two different types ofnetworks which exhibit different failure behavior. Thesenetworks are the Microsoft corporate network [4] andSkype super-peers [14]. The traces contain a list of timeintervals for which each host was online contiguously.To determine the session time for a given threshold t, wefirst combine intervals that were separated by less than t.We then use the average length of the remaining intervalsand add the value t to it. The additional period t repre-sents the time after a node leaves the network, but beforethe system decides to replicate its data.

We use these assumptions to calculate the amountof storage given to contributory systems with differentamounts of bandwidth. For each amount of bandwidth,we find the value for the threshold time (Section 4.2) thatmaximizes the contributed storage for each combinationof file system, availability trace, and bandwidth. We usethis value to compute the amount of storage available us-ing Equation 7. Figure 6 shows the bandwidth vs. stor-age curve using the reliability model based on availabil-ity traces of corporate workstations at Microsoft [4]. Fig-ure 7 shows similar curves using the reliability model de-rived from traces of the Skype peer-to-peer Internet tele-phone network [14].

Each curve has two regions. In the first region, thetotal amount of storage is limited by the available band-

0

5

10

15

20

25

0 200 400 600 800 1000 1200

Bandwidth (kB/s)

Con

trib

utio

n(G

B)

TFSWatermarkingFixed Contribution

Figure 6: The amount of storage that can be donatedfor contributions of network bandwidth, assuming acorporate-like failure model. With reliable machines,TFS is able to contribute more storage than other sys-tems, even when highly bandwidth-limited.

0

5

10

15

20

25

0 200 400 600 800 1000 1200

Bandwidth (kB/s)

Con

trib

utio

n(G

B)

TFSWatermarkingFixed Contribution

Figure 7: The amount of storage that can be donatedfor contributions of network bandwidth, assuming anInternet-like failure model. Because peer-to-peer nodeson the Internet are less reliable, the amount of contribu-tion by TFS does not surpass the other techniques untillarge amounts of bandwidth become available.

width, and increases linearly as the bandwidth is in-creased. The slope of the first part of the curve is deter-mined by the frequency of machine failures and file fail-ures. This line is steeper in networks where hosts tendto be more reliable, because less bandwidth is neededto replicate a large amount of data. In this case, theamount of available storage does not affect the amountof usable storage. This means that, for the first part ofthe curve when the systems are bandwidth-limited, TFScontributes an amount of storage similar to the other twosystems. Though TFS contributes slightly less becauseof file failures, the additional bandwidth needed to han-dle failures is small.


The second part of the curve starts when the amountof storage reaches the maximum allowed by that contri-bution technique. For instance, when the small contribu-tion reaches 5% of the disk, it flattens out. This part ofthe curve represents systems that have sufficient replica-tion bandwidth to use all of the storage they are given,and are only limited by the amount of available storage.In this case, TFS is capable of contributing significantlymore storage than other methods.

In the Microsoft trace, the corporate systems have arelatively high degree of reliability, so the bandwidth-limited portion of the curves is short. This high relia-bility means that, even for small bandwidth allocations,TFS is able to contribute the most storage. The Skypesystem shows a less reliable network of hosts. Muchmore network bandwidth is required before TFS is ableto contribute more storage than the other storage tech-niques can—in fact, much more bandwidth than is typ-ically available in Internet connected hosts. However,even when operating in a bandwidth-limited setting, TFSis able to contribute as much as the other techniques. Onemethod to mitigate these bandwidth demands is to em-ploy background file transfer techniques such as TCP-Nice [34].

From these results, we can conclude that TFS donatesnearly as much storage as other methods in the worstcase. However, TFS is most effective for networks ofreliable machines, where it contributes 40% more stor-age than a dynamic watermarking system. It is importantto note that these systems do not exhibit the same impacton local performance, which we demonstrate next.

5.2 Local File System PerformanceTo show the effects of each system on the user’s file sys-tem performance, we conduct two similar experiments.In the first experiment, a disk is filled to 50% with or-dinary file data. To achieve a realistic mix of file sizes,these files were taken from the /usr directory on a desk-top workstation. These files represent the user’s data anddo not change during the course of the experiment. Afterthis, files are added to the system to represent the con-tributed storage.

We considered four cases: no contribution, small con-tribution, large contribution, and TFS. The case wherethere is no simulated contribution is the baseline. Anydecrease in performance from this baseline is interfer-ence caused by the contributed storage. The small con-tribution is 5% of the file system. This represents a fixedcontribution where the amount of storage contributedmust be set very small. The large contribution is 35% ofthe file system. This represents the case of dynamicallymanaged contribution, where a large amount of storagecan be donated. With TFS, the disk is filled completelywith transparent data. Once the contributed storage is

added, we run a version of the Modified Andrew Bench-mark [25] to determine the contribution’s effect on per-formance.

We perform all of the experiments using two identicalDell Optiplex SX280 systems with an Intel Pentium 43.4GHz CPU, 800MHz front side bus, 512MB of RAM,and a 160GB SATA 7200RPM disk with 8MB of cache.The trials were striped across the machines to account forany subtle differences in the hardware. We conduct tentrials of each experiment, rebooting between each trial,and present the average of the results. The error bars inall figures in this section represent the standard deviationof our measurements. In all cases, the standard deviationwas less than 14% of the mean.

5.2.1 The Andrew BenchmarkThe Andrew Benchmark [15] is designed to simulate theworkload of a development workstation. Though mostusers do not compile programs frequently, the AndrewBenchmark can be viewed as a test of general small-fileperformance, which is relevant to all workstation file sys-tems. The benchmark starts with a source tree located onthe file system being tested. It proceeds in six phases:mkdir, copy, stat, read, compile, and delete.

Mkdir During the mkdir phase, the directory structureof the source tree is scanned, and recreated in an-other location on the file system being tested.

Copy The copy phase then copies all of the non-directory files from the original source tree to thenewly created directory structure. This tests smallfile performance of the target file system, both inreading and writing.

Stat The stat phase then scans the newly created sourcetree and calls stat on every file.

Read The read phase simply reads all data created dur-ing the copy phase.

Compile The compile phase compiles the target pro-gram from the newly created source tree.

Delete The delete phase deletes the new source tree.

The Andrew Benchmark has been criticized for being anold benchmark, with results that are not meaningful tomodern systems. It is argued that the workload beingtested is not realistic for most users. Furthermore, theoriginal Andrew Benchmark used a source tree whichis too small to produce meaningful results on modernsystems [15]. However, as we stated above, the Bench-mark’s emphasis on small file performance is still rel-evant to modern systems. We modified the AndrewBenchmark to use a Linux 2.6.14 source tree, which con-sists of 249MB of data in 19577 files. Unfortunately,even with this larger source tree, most of the data usedby the benchmark can be kept in the operating system’spage cache. The only phase where file system perfor-mance has a significant impact is the copy phase.


Despite these shortcomings, we have found that theresults of the Andrew Benchmark clearly demonstratethe negative effects of contributing storage. Though theonly significant difference between the contributing caseand the non-contributing case is in the copy phase of thebenchmark, we include all results for completeness.

The results of this first experiment are shown in Fig-ure 8. The only system in which contribution causes anyappreciable effect on the user’s performance is the caseof a large contribution with Ext2. Both the small con-tribution and TFS are nearly equal in performance to thecase of no contribution.

It is interesting to note that the performance of TFSwith 50% contribution is slightly better than the perfor-mance of Ext2 with 0% contribution. However, this doesnot show that TFS is generally faster than Ext2, but thatfor this particular benchmark TFS displays better perfor-mance. We suspect that this is an artifact of the way theblock allocation strategy was be modified to accommo-date TFS. As we show in Section 5.3, when running ourAndrew Benchmark experiment, TFS tends to allocatethe files used by the benchmark to a different part of thedisk than Ext2, which gives TFS slightly better perfor-mance compared to Ext2. This is not indicative of a ma-jor change in the allocation strategy; Ext2 tries to allocatedata blocks to the same block group as the inode theybelong to [5]. In our Andrew Benchmark experiments,there are already many inodes owned by the transparentfiles, so the benchmark files are allocated to different in-odes, and therefore different block groups.

The second experiment is designed to show the ef-fects of file system aging [32] using these three sys-tems. Smith and Seltzer have noted that aging effectscan change the results of file system benchmarks, andthat aged file systems provide a more realistic testbed forfile system performance. Though our aging techniquesare purely artificial, they capture the long term effects ofcontinuously creating and deleting files. As contributoryfiles are created and deleted, they are replaced by fileswhich are often allocated to different places on the disk.The long term effect is that the free space of the disk be-comes fragmented, and this fragmentation interferes withthe block allocation algorithm. To simulate this effect,we ran an experiment very similar to the first. However,rather than simply adding files to represent contributedstorage, we created and deleted contributory files at ran-dom, always staying within 5% of the target disk utiliza-tion. After repeating this 200 times, we proceeded tobenchmark the file system.

Figure 9 shows the results of this experiment. As withthe previous experiment, the only system that causes anyinterference with the user’s applications is the large con-tribution with Ext2. In this case, Ext2 with 35% contri-bution takes almost 180 seconds to complete the bench-

0

20

40

60

80

100

120

140

160

180

200

Ext2 (nocontribution)

Ext2 (5%contribution)

Ext2(35%contribution)

TFS

Tim

e(s)

DeleteCompileReadStatCopyMkdir

Figure 8: Andrew benchmark results for 4 different un-aged file systems. The first is an Ext2 system with nocontributory application. The second is Ext2 with a min-imal amount of contribution (5%). The third has a signif-icant contribution (35%). The fourth is TFS with com-plete contribution. TFS performance is on par with Ext2with no contribution. Error bars represent standard devi-ation in total time.

0

20

40

60

80

100

120

140

160

180

200

Ext2 (nocontribution)

Ext2 (5%contribution)

Ext2(35%contribution)

TFS

Tim

e(s)

DeleteCompileReadStatCopyMkdir

Figure 9: Andrew benchmark results for 4 different agedfile systems. The first is an Ext2 system with no contrib-utory application. The second is Ext2 with a minimalamount of contribution (5%). The third has a signifi-cant contribution (35%). The fourth is TFS with com-plete contribution. TFS performance is still comparableto Ext2 with no contribution. Error bars represent stan-dard deviation in total time.

mark. This is about 20 seconds longer than the same sys-tem without aging. This shows that file activity causedby contributory applications can have a noticeable im-pact on performance, even after considering the impactof the presence of those files. On the other hand, the time


!"#

$%&'()*

$%&'(+)*

Figure 10: Block allocation pattern for several contribu-tion levels in Ext2 and TFS. Both TFS and Ext2 with 0%contribution show high locality. Ext2 with 40% contribu-tion does not because the contributed files interfere withthe block allocation policy.

that TFS takes to complete the benchmark is unchangedby aging in the contributory files. There are no long-termeffects of file activity by contributory systems in TFS.

5.3 Block Allocation LayoutA closer look at the block allocation layout reveals thecause of the performance difference between Ext2 andTFS. We analyzed the block layout of files in four occu-pied file systems. Two of these systems were using Ext2,the third was TFS. Each file system was filled to 50%capacity with ordinary data. Data was then added to sim-ulate different amounts of data contribution. For Ext2,we consider two levels of contribution: 0% and 40%.0% and 40% were chosen as examples of good and badExt2 performance from Figure 1. The TFS system wasfilled to 50% capacity with ordinary data, and the rest ofthe disk was filled with transparent files. We then copiedthe files that would be used in the Andrew Benchmarkinto these disks, and recorded which data blocks wereallocated for the benchmark files.

Figure 10 shows the results of this experiment. Thehorizontal axis represents the location of the block on thedisk. Every block that contains data to be used in the An-drew Benchmark is marked black. The remaining blocksare white, demonstrating the amount of fragmentation inthe Andrew Benchmark files. Note that both Ext2 with0% contribution, and TFS show very little fragmentation.However, the 40% case shows a high degree of fragmen-tation. This fragmentation is the primary cause of theperformance difference between Ext2 with and withoutcontribution in the other benchmarks.

6 Related Work

Our work brings together two areas of research: tech-niques to make use of the free space in file systems, andthe study of churn in peer-to-peer networks.

Using Free Disk Space: Recognizing that the file sys-tem on a typical desktop is nearly half-empty, researchershave investigated ways to make use of the extra storage.

FS2 [16] is a file system that uses the extra space on thedisk for block-level replication to reduce average seektime. FS2 dynamically analyzes disk traffic to determinewhich blocks are frequently used together. It then cre-ates replicas of blocks so that the spatial locality on diskmatches the observed temporal locality. FS2 uses a pol-icy that deletes replicas on-demand as space is needed.We believe that it could benefit from a TFS-like allo-cation policy, where all replicas except the primary onewould be stored as transparent blocks. In this way, theentire disk could be used for block replication.

A number of peer-to-peer storage systems have beenproposed that make use of replication and free disk spaceto provide reliability. These include distributed hash ta-bles such as Chord [24] and Pastry [29], as well as com-plete file systems like the Chord File System [9], andPast [30].Churn in Peer-to-Peer Networks: The research com-munity has also been active in studying the dynamicbehavior of deployed peer-to-peer networks. Measure-ments of churn in live systems have been gatheredand studied as well. Chu et al. studied the availabil-ity of nodes in the Napster and Gnutella networks [6].The Bamboo DHT was designed as an architecture thatcan withstand high levels of churn [27]. The BambooDHT [23] is particularly concerned with using a min-imum amount of “maintenance bandwidth” even underhigh levels of churn. We believe that these studies givea somewhat pessimistic estimate of the stability of fu-ture peer-to-peer networks. As machines become morestable and better connected, remain on continuously, andare pre-installed with stable peer-to-peer applications ormiddleware, the level of churn will greatly diminish, in-creasing the value of TFS.

7 Future Work

TFS was designed to require minimal support from con-tributory applications. The file semantics provided byTFS are no different than any ordinary file system. How-ever, modifying the file semantics would provide oppor-tunities for contributory applications to make better useof free space. For instance, if only a small number ofblocks from a large file are overwritten, TFS will deletethe entire file. One can imagine an implementation ofTFS where the application would be able to recover thenon-overwritten blocks. When a file with overwrittenblocks is opened, the open system call could return avalue indicating that the file was successfully opened,but some blocks may have been overwritten. The ap-plication can then use ioctl calls to determine whichblocks have been overwritten. An attempt to read datafrom an overwritten block will fill the read buffer withzeros.


In the current implementation of TFS, once a transpar-ent file is opened, its blocks cannot be overwritten untilthe file is closed. This allows applications to assume, asthey normally do, that once a file is opened, reads andwrites to the file will succeed. However, this means thattransparent files may interfere with ordinary file activ-ity. To prevent this, it would be possible to allow datafrom opened files to be overwritten. If an application at-tempts to read a block which has been overwritten, theread call could return an error indicating why the blockis not available.

Both of these features could improve the service pro-vided by TFS. By allowing applications to recover filesthat have been partially overwritten, the replication band-width needed by systems using TFS is reduced to the rateat which the user creates new data. By allowing openfiles to be overwritten, transparent applications may keeplarge files open for extended periods without impactingthe performance of other applications. Despite these ben-efits, one of our design goals for TFS is that it should beusable by unmodified applications. Both of these fea-tures would require extensive support from contributoryapplications, violating this principle.

8 Conclusions

We have presented three methods for contributing diskspace in peer-to-peer storage systems. We have includedtwo user-space techniques, and a novel file system, TFS,specifically designed for contributory applications. Wehave demonstrated that the key benefit of TFS is that itleaves the allocation for local files intact, avoiding is-sues of fragmentation—TFS stores files such that theyare completely transparent to local access. The designof TFS includes modifications to the free bitmaps and amethod to avoid hot-spots on the disk.

We evaluated each of the file systems based theamount of contribution and its cost to the local user’s per-formance. We quantified the unreliability of files in TFSand the amount of replication bandwidth that is needed tohandle deleted files. We conclude that out of three tech-niques, TFS consistently provides at least as much stor-age with no detriment to local performance. When thenetwork is relatively stable and adequate bandwidth isavailable, TFS provides 40% more storage over the bestuser-space technique. Further, TFS is completely trans-parent to the local user, while the user-space techniquecreates up to a 100% overhead on local performance. Webelieve that the key to encouraging contribution to peer-to-peer systems is removing the barriers to contribution,which is precisely the aim of TFS.

The source code for TFS is available at http://prisms.cs.umass.edu/tcsm/.

Acknowledgments

This material is based upon work supported by the Na-tional Science Foundation under CAREER Awards CNS-0447877 and CNS-0347339, and DUE 0416863. Anyopinions, findings, and conclusions or recommendationsexpressed in this material are those of the author(s) anddo not necessarily reflect the views of the NSF.

We would also like to thank our anonymous reviewersfor their helpful comments. We would like to give specialthanks to Erez Zadok for his careful shepherding.

References

[1] Atul Adya, William J. Bolosky, Miguel Castro,Gerald Cermak, Ronnie Chaiken, John R. Douceur,Jon Howell, Jacob R. Lorch, Marvin Theimer,and Roger Wattenhofer. FARSITE: Federated,available, and reliable storage for an incompletelytrusted environment. In 5th Symposium on Op-erating System Design and Implementation (OSDI2002). USENIX Association, December 2002.

[2] Ranjita Bhagwan, Kiran Tati, Yu-Chung Cheng,Stefan Savage, and Geoffrey Voelker. Total Recall:System support for automated availability manage-ment. In Proceedings of the First ACM/UsenixSymposium on Networked Systems Design and Im-plementation (NSDI), pages 73–86, San Jose, CA,May 2004.

[3] Charles Blake and Rodrigo Rodrigues. High avail-ability, scalable storage, dynamic peer networks:Pick two. In Ninth Workshop on Hot Topics inOperating Systems (HotOS-IX), pages 1–6, Lihue,Hawaii, May 2003.

[4] William J. Bolosky, John R. Douceur, David Ely,and Marvin Theimer. Feasibility of a serverless dis-tributed file system deployed on an existing set ofdesktop PCs. In Proceedings of the ACM SIGMET-RICS International Conference on Measurementand Modeling of Computer Systems (SIGMETRICS2000), pages 34–43, New York, NY, USA, 2000.ACM Press.

[5] Remy Card, Thodore Ts’o, and Stephen Tweedie.Design and implementation of the second extendedfilesystem. In Proceedings of the First Dutch Inter-national Symposium on Linux. Laboratoire MASI— Institut Blaise Pascal and Massachussets Insti-tute of Technology and University of Edinburgh,December 1994.

[6] Jacky Chu, Kevin Labonte, and Brian Neil Levine.Availability and locality measurements of peer-to-peer file systems. In Proceedings of ITCom: Scala-


bility and Traffic Control in IP Networks II Confer-ence, July 2002.

[7] James Cipar, Mark D. Corner, and Emery D.Berger. Transparent contribution of memory. InUSENIX Annual Technical Conference (USENIX2006), pages 109–114. USENIX, June 2006.

[8] Ian Clarke, Oskar Sandberg, Brandon Wiley, andTheodore W. Hong. Freenet: A distributed anony-mous information storage and retrieval system. InProceedings of Designing Privacy Enhancing Tech-nologies: Workshop on Design Issues in Anonymityand Unobservability, pages 46–66, July 2000.

[9] Frank Dabek, M. Frans Kaashoek, David Karger,Robert Morris, and Ion Stoica. Wide-area cooper-ative storage with CFS. In Proceedings of the 18thACM Symposium on Operating Systems Principles(SOSP ’01), Chateau Lake Louise, Banff, Canada,October 2001.

[10] John R. Douceur and William J. Bolosky. A large-scale study of file-system contents. In Proceedingsof the ACM SIGMETRICS International Confer-ence on Measurement and Modeling of ComputerSystems (SIGMETRICS 1999), pages 59–70, NewYork, NY, USA, 1999. ACM Press.

[11] Michael J. Freedman, Eric Freudenthal, and DavidMazieres. Democratizing content publication withCoral. In Proceedings of the 1st USENIX Sym-posium on Networked Systems Design and Imple-mentation (NSDI ’04), San Francisco, California,March 2004.

[12] http://freenetproject.org/faq.html.

[13] P. Brighten Goedfrey, Scott Shenker, and Ion Sto-ica. Minimizing churn in distributed systems. InProc. of ACM SIGCOMM, 2006.

[14] Saikat Guha, Neil Daswani, and Ravi Jain. An Ex-perimental Study of the Skype Peer-to-Peer VoIPSystem. In Proceedings of The 5th InternationalWorkshop on Peer-to-Peer Systems (IPTPS ’06),Santa Barbara, CA, February 2006.

[15] John H. Howard, Michael L. Kazar, Sherri G.Menees, David A. Nichols, M. Satyanarayanana,Robert N. Sidebotham, and Michael J. West. Scaleand performance in a distributed file system. ACMTransactions on Computer Systems, 6(1):51–81,February 1988.

[16] Hai Huang, Wanda Hung, and Kang G. Shin. FS2:Dynamic data replication in free disk space for im-proving disk performance and energy consumption.In Proceedings of 20th ACM Symposium on Oper-ating System Principles, October 2005.

[17] Stefan M. Larson, Christopher D. Snow, MichaelShirts, and Vijay S. Pande. Computational Ge-nomics. Horizon, 2002. Folding@Home andGenome@Home: Using distributed computing totackle previously intractable problems in computa-tional biology.

[18] Ozgur Can Leonard, Jason Neigh, Erez Zadok, Jef-ferey Osborn, Ariye Shater, and Charles Wright.The design and implementation of Elastic Quotas.Technical Report CUCS-014-02, Columbia Univer-sity, June 2002.

[19] Christopher R. Lumb, Jiri Schindler, and Gre-gory R. Ganger. Freeblock scheduling outside ofdisk firmware. In Proceedings of the Conference onFile and Storage Technologies (FAST), pages 275–288, 2002.

[20] Marshall K. McKusick, William N. Joy, Samuel J.Leffler, and Robert S. Fabry. A fast file system forUNIX. Computer Systems, 2(3):181–197, 1984.

[21] James Mickens and Brian D. Noble. Exploit-ing availability prediction in distributed systems.In Proceedings of the ACM/Usenix Symposium onNetworked Systems Design and Implementation(NSDI), 2006.

[22] Microsoft Corporation. http://www.microsoft.com/technet/prodtechnol/winxppro/reskit/c28621675.mspx.

[23] Ratul Mahajan Miguel. Controlling the cost of re-liability in peer-to-peer overlays. In Proceedingsof the 2nd International Workshop on Peer-to-PeerSystems (IPTPS ’03), 2003.

[24] Robert Morris, David Karger, Frans Kaashoek, andHari Balakrishnan. Chord: A Scalable Peer-to-PeerLookup Service for Internet Applications. In ACMSIGCOMM 2001, San Diego, CA, September 2001.

[25] John K. Ousterhout. Why aren’t operating systemsgetting faster as fast as hardware? In Proceedingsof USENIX Summer, pages 247–256, 1990.

[26] Zachary Peterson and Randal Burns. Ext3cow: atime-shifting file system for regulatory compliance.Trans. Storage, 1(2):190–212, May 2005.

[27] Sean Rhea, Dennis Geels, Timothy Roscoe, andJohn Kubiatowicz. Handling churn in a DHT.Technical Report UCB/CSD-03-1299, EECS De-partment, University of California, Berkeley, 2003.

[28] Rodrigo Rodrigues and Barbara Liskov. High avail-ability in DHTs: Erasure coding vs. replication. InProceedings of the 4th International Workshop onPeer-to-Peer Systems, February 2005.

[29] Antony Rowstron and Peter Druschel. Pastry: Scal-able, decentralized object location, and routing for


large-scale peer-to-peer systems. In Proceedings ofMiddleware, November 2001.

[30] Antony Rowstron and Peter Druschel. Storagemanagement and caching in PAST, a large-scale,persistent peer-to-peer storage utility. In Proceed-ings of the 18th SOSP (SOSP ’01), Chateau LakeLouise, Banff, Canada, October 2001.

[31] Margo Seltzer, Keith Bostic, Marshall Kirt McKu-sick, and Carl Staelin. An implementation of a log-structured file system for UNIX. In Winter USENIXTechnical Conference, January 1993.

[32] Keith Smith and Margo Seltzer. File system aging.In Proceedings of the 1997 Sigmetrics Conference,Seattle, WA, June 1997.

[33] Daniel Stutzbach and Reza Rejaie. Towards a betterunderstanding of churn in peer-to-peer networks.Technical Report UO-CIS-TR-04-06, Departmentof Computer Science, University of Oregon, No-vember 2004.

[34] Arun Venkataramani, Ravi Kokku, and MikeDahlin. TCP Nice: A mechanism for backgroundtransfers. SIGOPS Oper. Syst. Rev., 36(SI):329–343, 2002.


Documents

TFS: A Transparent File System for Contributory Storage · 2019. 2. 25. · TFS: A Transparent File System for Contributory Storage James Cipar Mark D. Corner Emery D. Berger Department