File Systems and Mass Storage Sorin Manolache [email protected]

File Systems and Mass Storage

Sorin Manolache

[email protected]

2S. Manolache, Process programming and operating systems, File systems and mass storage

Last on TTIT61

BindingCompile time, load time, execution time

SwappingContiguous memory allocation

External fragmentationPaging

Internal fragmentation, sharing, protectionSegmentation

External fragmentation, sharing, protectionVirtual memory

Page replacementThrashing


Lecture Plan

1. What is an operating system? What are its functions? Basics of computer architectures. (Part I of the textbook)

2. Processes, threads, schedulers (Part II , chap. IV-VI)

3. Synchronisation (Part II, chap. VII)

4. Primary memory management. (Part III, chap. IX, X)

5. File systems and secondary memory management (Part III, chap. XI, XII, Part IV)

6. Security (Part VI)


Outline

The concept of fileOperations on filesAccess methods

DirectoriesOperations on directoriesDirectory hierarchies

File sharingProtectionDisk scheduling


Files

Named collection of related information that is stored on secondary storage

Smallest allotment of logical secondary storage (when we want to store something on the secondary storage, we store it in files)

Format of files is typically defined by the creator


File Attributes

Name (identifier for human use) Identifier (typically a numeric identifier for the internal use of

the OS)Type (for OS that support file types)SizeTime, date, user identification (last access, last modification,

creation)Location (on the device)Protection (permissions to read/write/execute/etc.)


Operations on Files

CreationNameProtection information

DeletionName

WritingReadingTruncatingRepositioning within a file


Opening a File If the file to be read from (written to) was specified by its

name to the read (write) system callsThe OS would have to lookup the disk blocks

corresponding to the named file for each system call invocation performance penalty

Most OS require the user to perform an open system call thatMaps the file name to an identifier Initialises a memory structure with the disk location of the

opened file and other data (see slides)

Implicit opening: automatically open at first access, close at process exit


Disks

Cylinder

Heads


Disk Organisation

Tracks

Gap

Sector


Disk Organisation

The geometry of disks is given in C/H/S (Cylinders/Heads/Sectors)

Initially used to corresponded to the true physical geometry

The access granularity is the physical block (sector)The operating system maps logical records on physical

blocks

Disk space is always allocated in blocksFiles may not have a size equal to an integer multiple of

the block size last block not fully used Internal fragmentation


Access Methods

Direct accessSequential access Indexed access


Direct Access

Direct accessRead (write) system call do specify the relative block

number from where to read (where to write to)E.g.

write(fd, buf, sizeof(buf), 30); -- writes sizeof(buf) bytes from the buffer buf to the 30th block of the file identified by fd.


Sequential Access

Sequential accessThe block from where to read (where to write) is not

specified. The OS keeps a file pointer that it modifies accordingly.

A read (write) operation reads (writes) data from the current file offset, stored in the file pointer

After the read or write, the value of the file pointer is incremented with the amount of transferred records.

E.g.: write(fd, buf, sizeof(buf)) – write sizeof(buf) bytes

from the buffer buf to the file identified by fd at the current file offset. Increment the file pointer with sizeof(buf)


Indexed Access

143520, 1311607896, 1312853661, 1400

143520, $10245679, $30509877, $5510978, $15

Block 1311

607896, $20610942, $10790134, $15829842, $8

Block 1312

853661, $10898541, $20934625, $6973147, $7

Block 1400

Index file

File


Writing to Files in Unix

WritingWhich file to write toWhat to write

The write system call does not specify a file offset (a write pointer pointing at the position in the file where the writing should begin)

Sequential access


Reading from Files in Unix

ReadingWhich file to read fromHow much to readWhere to put what we read

The read system call does not specify a file offset (a read pointer pointing at the position in the file where the reading should begin)


Read/Write Pointers

Typically, a file is used either for reading or for writing by a process

The OS keeps just one single file pointer for both reading and writing


File Operations in Unix

int creat(const char *name, int permissions) int open(const char *name, int flags) int read(int fd, char *buffer, int requested_size); int write(int fd, const char *buffer, int size); int close(int fd); long lseek(int fd, long offset, int whence); int stat(const char *name, struct stat *status);


Usage Example

int fd, n;

char wbuf[] = “Hello!\n”, rbuf[MAX_BUF];

fd = creat(“source.txt”, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);

write(fd, wbuf, sizeof(wbuf));

n = read(fd, rbuf, MAX_BUF);

close(fd);

From which file offset will read read? What's the value of n? What will rbuf contain?


Usage Example

int fd, n;

char wbuf[] = “Hello!\n”, rbuf[MAX_BUF];

fd = creat(“source.txt”, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);

write(fd, wbuf, sizeof(wbuf));

close(fd);

fd = open(“source.txt”, O_RDONLY);

n = read(fd, rbuf, MAX_BUF);

close(fd);


Directories

Special files that contain directory entriesA directory entry is a data structure containing the file

attributes

/

bin/

ls

lib/

libc.solibgcc.so

vmlinux-2.6.11

bin, dir, root:root, rwxr-xr-x, 10050lib, dir, root:root, rwxr-xr-x, 52175vmlinux-2.6.11, reg, root:root, r-x------, 120311

ls, reg, root:root, r-xr-xr-x, 10052

libc.so, reg, root:root, r-xr-xr-x, 52177libgcc.so, reg, root:root, r-xr-xr-x, 60621

root dir

bin dir

lib dir

directory entry


Operations on Directories

List the contents of the directoryDelete (unlink) a fileRename a fileOpen and close the directory

Search for a fileTraverse the file system


Directory Hierarchies

Tree-structured directoriesLeaf nodes are files, all other nodes are directories

Acyclic-graph directoriesSame with the exception that the structure is an acyclic

graphGeneral graph directories

May contain cycles


Acyclic Graph Directories

We need reference counters. A file is removed and its blocks marked as free when the reference counter reaches 0.

Removing = unlinking

/

bin/

ls

lib/

libc.solibgcc.sokernel

vmlinux-2.6.11



libc.so, reg, root:root, r-xr-xr-x, 52177libgcc.so, reg, root:root, r-xr-xr-x, 60621kernel, reg, sys:root, r-xr-x---, 120311

root dir

bin dir

lib dir


Symbolic Links

A hard link is a directory entry pointing to a different file. It contains no blocks of its own

A symbolic link is a special file, very short one, that contains the name of the file that it points to

/

bin/

ls

lib/

libc.solibgcc.sokernel

vmlinux-2.6.11



libc.so, reg, root:root, r-xr-xr-x, 52177libgcc.so, reg, root:root, r-xr-xr-x, 60621kernel, link, guest:guest, rwxrwxrwx, 71220

root dir

bin dir

lib dir


File Sharing

When files can be sharedShould all writes be allowed to occur or should the OS

protect the user actions from each other?Should a write be immediately visible to all the other

users who share the file?


yyyyyyyyyyxxxxxxxxxx

File Sharing

fd = open(“file.txt”, O_RDWR);n = read(fd, buf, 10);

n = read(fd, buf, 5);printf(“%s\n”, buf);

fd = open(“file.txt”, O_RDWR);n = read(fd, buf, 10);

write(fd, “xxxxxxxxxx”, 10);printf(“%s\n”, buf);

n = read(fd, buf, 5);printf(“%s\n”, buf);

printf(“%s\n”, buf);write(fd, “yyyyyyyyyy”, 10);

printf(“%s\n”, buf);n = read(fd, buf, 10);

000000000088888888885555555555

0

0file.txt, ftp:users, rw-rw-rw-, ,10228

01

18

…

183

186

10

10

15

20

2030

30

close(fd);

121

close(fd);

0

OS-wide open file table

Proc A open file table Proc B open file table


Protection

Keep safe from improper accessWe introduce the notion of file owner

A user ID kept on the disk in the directory entryUsers have IDsProcesses, besides their process IDs (pid), have user

IDs, typically the user ID of the user that executes them (user ID (uid) or effective user ID (euid))

Files typically are owned by the user who creates themThe effective user ID of the file creator is written in the

directory entryBesides owner, a file is characterised by its group


Protection

Controlled access is introduced by specifying which users (or user groups) are allowed to perform operations on the file

Examples of controlled operations:ReadWriteExecuteAppendDeleteList


Unix File Protection

Read:Read for files, list for directories

Write:Write/modify for files, create/delete new entries for

directoriesExecute:

Execute for files, change directory rights for directories


Access Control Lists (ACL)

Each file (directory) has an access control list attachedThe access control list specifies for each controlled

operation the users that are allowed to perform this operation

Advantage:Very general and flexible

Disadvantages:Difficult to construct if we do not know all the users

beforehandDirectory entry of variable size, more difficult to manage


Condensed ACL Use of condensed ACLs instead Use per-owner, per-group, per-others permissions

E.g.: Sara writes a book, Jim, Dawn, and Jill help her. Sara has all the rights, Jim, Dawn, and Jill may read or write but not delete, all the others may only read

Sara is the owner, has rw- permissions A group book is created, the file is owned by user Sara and group book,

Jim, Dawn, and Jill are added to group book, the group has rw- permissions

Others have r-- permissions The directory in which the book resides has rwxr-xr-x permissions If Sara wants Joe to have read/write access to chapter 1, she cannot add

him to group book Instead, user Joe is added to the ACL


Condensed ACL

What if:kim:staff rw-r-xr-- script.shUser kim belongs to group staffShould kim be allowed to execute script.sh? If we consider that the permissions of the owner apply,

then no If we consider that the permissions of the group apply,

then yesPrecedence given to most specific


Disk Access Scheduling

OS has to ensure that the resources are used efficientlyBandwidth = transferred bytes / length of interval between

first request and completion of last request

Seek timeRotational latency


FCFS Disk Scheduling

Following requests: cylinders 98, 183, 37, 122, 14, 124, 65, 67

Head initially at cylinder 53

Total head movement of 640 cylinders

0 98 1831223714 6553


Shortest Seek Time First

Movement of only 236 cylindersMay cause starvationNot optimal!! If we moved from 53 to 37 and then 14,

before 65, 67, etc. 208 cylinders

0 98 1831223714 6553


SCAN Scheduling

When we reach one end, it is more likely that requests are closer to the other end than close to the read/write head

Those also waited the longest C-SCAN algorithm

0 98 1831223714 6553


Circular-SCAN Algorithm

0 98 1831223714 6553


LOOK and C-LOOK Algorithms

0 98 1831223714 6553

0 98 1831223714 6553


Which One?

Depends on the loadDepends on file allocationSSTF and LOOK seem reasonable alternatives

Real disk geometry is hidden to the OSDisk manufacturers include a scheduling in the hard disk

controllerThen why not let the hard disk do all the scheduling?Because some requests have different semantics and have

to be treated differently (accesses of a higher priority process, or paging, for example)


Summary

FilesOperationsSharingProtection

Directory hierarchiesDisk scheduling algorithms

Documents

File Systems and Mass Storage Sorin Manolache [email protected]