Next Generation File SystemsNext Generation File Systems

Insight into Journaled file systems and the development on ReiserFS 4.0

By Jason Moiron

ContentsContents

Introduction & A Quick Recap

Types of (on disk) File Systems

Hierarchical, Journalled, Other

Traditional, Hierarchical File System Ext2, Fat32

Data versus Meta Data

Problems with Hierarchical File Systems

Fsck'ing File Systems

Contents Contd.Contents Contd.

Journalling File Systems

How Journalling Solves Problems In HFS'

XFS, JFS, ReiserFS 3.6, Ext3

ReiserFS 3.6

Balanced B+ Trees

ReiserFS 4.0

“Dancing” Trees

Pseudo File Meta Data Access

Some Notes ...Some Notes ...

FS will usually abbreviate “Filesystem”

HFS will, unless explicit, abbreviate the generic term “Hierarchical File System”, and NOT Apple Computer's “Hierarchical Filing System” (HFS)

HDD stands for “Hard Disk Drive”

RDB stands for “Relational Data Base”

An Introduction (on design)An Introduction (on design)

File Systems is an Old Metaphor

Grudgingly: Punch Cards

“Flat” File Systems

Many File Systems Tailored to Hardware

ISO9660

Any Modern HDD FS

File System design's shining hour has passed: memory resident RDB's now optimal.

A Quick Recap (1980 - 1995)A Quick Recap (1980 - 1995)

“Flat” File Systems to Hierarchy

Main Frames (IBM)

The need to deal with less files (HFS, UFS)

Meta Data

The simplest meta data : file names

Other examples: timestamps, permissions, etc.

Hardware Driving FS Growth

CPU's get faster, Barriers Broken (14, 8.3, 31)

Types Of FilesystemsTypes Of Filesystems

Flat

Just a bunch of files; not in modern use

Hierarchical

Filesystem's providing organization (directories)

Journalled

Usually hierarchical, with data about the filesystem itself

Other Interesting FS' include SFS (MIT), Plan9's KFS (Bell/Lucent), Log-structured FS

Traditional Hierarchical FS'Traditional Hierarchical FS'

Hierarchical FS'

Ext, Ext2, FAT(X), HFS (Apple), UFS (old)

Need for less CPU intensive FS'

Mainframes could handle “flat” FS searching

Directories increase/create namespaces

Provide a system to follow a name to data (usually inodes)

Data and Meta DataData and Meta Data

Data is not enough: Information about files is wanted, not just their contents (the data itself)

Enter meta data, and Apple's HFS: first (or early) use of rich meta data.

Meta Data kept with inodes in *NIX FS' This means meta data has to be kept in sync with the files they represent.

If this fails...

Fsck'ing Partitions!Fsck'ing Partitions!

Systems Crash

When data and meta data are out of sync, fsck (filesystem check) attempts to fix inodes

Time and Space:

Disks have grown in size tremendously, but not (much) in speed.

1GB 10GB 500GB0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

fsck run time (in sec)

fsck run time

Problem #1!Problem #1!

Original FS were too simple, created bottlenecks:

Original UNIX FS (UFS) only got 5 – 10% of maximum possible disk throughput

Math to the rescue

Much early FS research spent on optimizing FS' for HDD's : Minimize seek time, maximize effective buffering, caching, etc.

Problem #2Problem #2

HDD's space progresses faster than FS' capacities

FS' have to evolve to fit more files more efficiently

Directory with many files has flat FS problems.

# of files avg file size0

2500

5000

7500

10000

12500

15000

17500

20000

22500

Linux Kernel Stats

1.0

2.6.5

Solving Consistency : JournalsSolving Consistency : Journals

The Problem: Manage consistency

The Solution: Log FS activity as transactions and keep the log in a separate part of the FS called a Journal

Transaction : borrowed from RDB's : unit of work

Journaling FilesystemsJournaling Filesystems

Ext3

Built to add a journal to popular Ext2 FS (Linux)

JFS

Built by IBM for AIX; FS in OS2/Warp

XFS

Built by SGI with efficient large file support in mind

ReiserFS v3.x

Built by Hans Reiser; first Journaling FS in Linux, built to optimize for space and small files / large directories

Extended 3 FSExtended 3 FS

Designed for backwards compatibility with Ext2

Several Journaling Modes (mount options):

Writeback – typical meta data journaling

Ordered – data written first, then meta data

“Journal” - full meta and data journaling

Quirks:

can be mounted Ext2, “.journal” files, abnormally high performance during simultaneous reading/writing

XFSXFS

Barely more interesting than JFS

Replaced EFS in early 90's (IRIX 5.3)

Optimized for large files (render farms)

Tremendous scalability due to wide usage of B+Trees (Inode management, Free space management, etc)

Quirks

X 1.0 had odd bugs, known for slow file deletion, delayed allocation

ReiserFS 3 : HistoryReiserFS 3 : History

ReiserFS 3.x first Journaling FS included in Linus Kernel

Developed by Hans Reiser ->

ReiserFS 3.x : Boring Technical SpecsReiserFS 3.x : Boring Technical Specs

Max files : 232

Max subdirs in a dir : 216 -1k

Max filesize : 260, 232 in ReiserFS 3.5

Max links to a file : 232, 215 in ReiserFS 3.5

Max FS size : 232 4k blocks (17.6 TB)

Dynamic “inode” creation

ReiserFS 3.x : The DesignReiserFS 3.x : The Design

Philosophy : extra layers (RDBs) come from bad design

Traditional inode link structure flawed

Replacing inode bitmap with B tree (XFS) not enough

Solution : Use 1 B tree for the whole FS

How exactly do you use the B Tree?

B - TreesB - Trees

B-Tree stores data with nodes

Arbitrary fanout

Data, Keys, and Pointers

B+TreesB+Trees

Store only keys and pointers on internal nodes

Superior to B-trees; increases fanout, and increases caching of internal nodes

Problems with B+TreesProblems with B+Trees

B+Trees flawed for FS design

Files change; ensuring structure among leaves is costly

ReiserFS 3's solution same as RDB's :

B+Trees with BLOBS

A ReiserFS 3.x TreeA ReiserFS 3.x Tree

Using Trees :Using Trees :

Using trees ensures better performance because :

Searching trees gets faster as fanout increases

Temporal Locality of files to meta data

Temporal Locality of directories to meta data

Temporal Locality due to the structure of a tree and methods of insertion

Tail PackingTail Packing

Normal FS use blocks and chunks:

blocks are sized 1, 2, 4k (usually)

chunks take up some fraction of a block

ReiserFS3 internally uses blocks for keeping track of extents, but...

Being a tree with exact locational meta data, ReiserFS can pack tails

Performance?Performance?

Tail Packing makes FS' utilize space more efficiently

Regularly reported as 8 to 15 times faster on files < 1k than Ext2

Regularly reported as slower on average for writing large files than Ext3 & XFS

Overall : journaling without penalty

ReiserFS 4ReiserFS 4

Still experimental (3/26/04 2.6.5-pre2 has 2 bugs before its let into Linus Kernel)

Even more unique than Reiser3

Aims to fix design deficiencies in Reiser3 and FS in general

Is really cool

ReiserFS4 : DesignReiserFS4 : Design

Dancing Trees

Spatial Locality

Atomicity

Wandering Logs

Repacking & V4.1

Plugins

Meta Data Pseudo's

Problems With B+TreesProblems With B+Trees

ReiserFS3 trees kept pointers with data when dealing with blobs

Decreases the ability to cache pointers, hurts performance

Impacts searching performance because now we must search leaves

Dancing TreesDancing Trees

Bad name, Brilliant Concept

Keep BLOB pointers at twigs

Keep fanout, cache all pointers, increase search speeds

Problems With Temporal LocalityProblems With Temporal Locality

Temporal locality fine when user is creating most files

Often, files are created as batch jobs in many places (installing a package);

hurts overall user performance

Dancing Trees save enough time to allow for spatial locality trade off

AtomicityAtomicity

✟ Crucial in avoiding inconsistencies and bad data

✟ ReiserFS4 guarantees atomicity of file operations (transactions) without writing data twice

✟ How is this possible?

CommittingCommitting

✟ Transactions preserve until commit

✟ “Dirty blocks” separated into 3 groups

� relocatable✟ blocks that have 'dirty' parents

� relocate✟ blocks in relocatable that will go somewhere 'new'

� overwrite✟ dirty blocks that re-write their parents

More SetsMore Sets

✟ 'Wandered' set is where overwrite blocks go before committal

✟ On committal, write mapping of wander list to overwrite list & update pointer to point to wander list

� This update is atomic; before the update, the FS is not changed

� After the update, the log is played; on crash, its still played

RepackingRepacking

✟ Goes through the Dancing Tree and shoves all leaf nodes to left.

✟ Later, shoves all to right.

� This enables tighter packing

� Can be tuned overtime to allow for “air holes” for faster insertion (planned for 4.1)

PluginsPlugins

✟ Ensures Reiser4 can adapt; new features without reformat

✟ All file operations done with plugins

✟ Examples

� files, directories

� security, hash keys, node searching

✟ Quick, safe method of modifying FS without touching core