1GWU CS 259 Brad Taylor Spring 2004 Systems Programming Meeting 3: Handling Input/Output Files, Directories & Special Files

1GWU CS 259 Brad Taylor Spring 2004

Systems Programming

Meeting 3:Meeting 3:

Handling Input/OutputHandling Input/OutputFiles, Directories & Special FilesFiles, Directories & Special Files


Objectives

•Complete OS Basis MaterialComplete OS Basis Material

•OS Input/Output Handling Means OS Input/Output Handling Means

•User & Program Interaction (Files, User & Program Interaction (Files, Directories)Directories)

•Process Communication Primitives Process Communication Primitives (Pipes, Named Pipes, …)(Pipes, Named Pipes, …)


Input / Output, Files & Directories

• What are they? Why use them?What are they? Why use them? • How do they work?How do they work? • How are they integrated?How are they integrated? • How deal with resource & time How deal with resource & time

contention concurrently?contention concurrently?


Brainstorm onto board …Brainstorm onto board …

How many How many differentdifferent means means can we list / discuss?can we list / discuss?

Inputting Data

HelloHal.

Hello Dave

!

Its not behaving … what do Its not behaving … what do we do???we do???


Inputting Data

KeyboardKeyboard

Mouse / TrackballMouse / Trackball

Drawing Tablet Drawing Tablet

Handwritten charactersHandwritten characters

Magnetic-ink & Optical Magnetic-ink & Optical character recognitioncharacter recognition

ScannerScanner

Optical mark recognitionOptical mark recognition

Digital CameraDigital Camera

Bar codesBar codes

Voice inputVoice input

Touch screensTouch screens

Eye Motion (Look)Eye Motion (Look)

Midi DevicesMidi Devices

Sensor DevicesSensor Devices

Clock, GPS Clock, GPS

Network Network

SERIAL, vs. PARALLEL, SERIAL, vs. PARALLEL, GPIB GPIB


Sensor Devices

HomeHome-- Smoke Detectors Smoke Detectors-- Room Temperature Room Temperature-- Motion (Presence) Motion (Presence) DetectorsDetectors- - Continuity Continuity DetectionDetection-- Security Code or Security Code or Personal Personal IdentificationIdentification

IndustrialIndustrial- Valve Position- Valve Position- Electrical Contacts- Electrical Contacts- Gas & Fluid Flow - Gas & Fluid Flow - Photocell- Photocell- Pressure Detection- Pressure Detection- Level Measurement- Level Measurement- Safety Circuits- Safety Circuits- Electrical Current- Electrical Current- Finger /Voice - Finger /Voice /Retinal ID/Retinal ID


Digital Cameras

Digital cameras take photographs and store Digital cameras take photographs and store them internallythem internally

Photos are then sent to the computer for Photos are then sent to the computer for output, or stored on disk.output, or stored on disk.

Photographic Data Compression StandardsPhotographic Data Compression Standards


Output of Information

Ideas?Ideas?

Another list …Another list …


Output of Information

Useful user information takes a variety of forms:Useful user information takes a variety of forms:Computer screens Computer screens (or virtual/holographic)(or virtual/holographic)Printers Printers (Impact vs. non)(Impact vs. non)Voice output Voice output

- by Analysis (words) or Rules (sounds)- by Analysis (words) or Rules (sounds)Music and Sound Music and Sound (Napster or peers …)(Napster or peers …)Actuating DevicesActuating DevicesNetworkNetworkStorage Devices (FDD/HDD/tapes/USB …)Storage Devices (FDD/HDD/tapes/USB …)HapticsHapticsGPIBGPIB


Actuating Devices

HomeHome-- Fire Alarms Fire Alarms-- Heating & Heating & CoolingCooling-- Lighting Lighting (Inside & Outside)(Inside & Outside)- Refrigerator- Refrigerator

IndustrialIndustrial- Valve Positioners- Valve Positioners- Electrical - Electrical SwitchesSwitches- Laser Projection- Laser Projection- Safety Cutoffs- Safety Cutoffs- Mixers- Mixers- Lights- Lights- Sirens- Sirens- Ovens, Heaters, - Ovens, Heaters, Chillers …Chillers …


UNIX Device Files

Peripheral DevicesPeripheral Devices accessed through filenames accessed through filenames in file systemin file system

readreads and s and writewrites s to theseto these device files transfers device files transfers data directly between system and devicesdata directly between system and devices

Block device files Block device files (Transfer unit is block; ex: disk)(Transfer unit is block; ex: disk)Character device files Character device files (Transfer unit is character; (Transfer unit is character;

terminal, modem, printer)terminal, modem, printer)Action Sequence Action Sequence

- Access device file’s inode- Access device file’s inode- Determine transfer unit- Determine transfer unit- Call specific device driver- Call specific device driver


Files: named bytes on disk

File abstraction:File abstraction:– user’s view: named sequence of bytes user’s view: named sequence of bytes

– FS’s view: collection of disk blocksFS’s view: collection of disk blocks– file system’s job: translate name & offset to disk blocks file system’s job: translate name & offset to disk blocks

File operations:File operations:– create a file, delete a filecreate a file, delete a file– read from file, write to fileread from file, write to file

Want: operations to have as few disk accesses as possible & have minimal Want: operations to have as few disk accesses as possible & have minimal space overheadspace overhead

offset:int disk addr:int

int main() { … foo.c


Low-level I/O: open, read, write, close

Example 1: Example 1: append.cappend.c main(int argc, char **argv)main(int argc, char **argv){{int n, in, out;int n, in, out;

char buf[1024];char buf[1024];

/* Open the first file for reading *//* Open the first file for reading */in = in = openopen (argv[1], O_RDONLY); (argv[1], O_RDONLY);

/* Open the second file for writing *//* Open the second file for writing */out = out = openopen (argv[2], O_WRONLY | O_APPEND); (argv[2], O_WRONLY | O_APPEND);

/* Copy data from the first file to the second *//* Copy data from the first file to the second */while ((n = while ((n = readread (in, buf, sizeof(buf))) > 0) (in, buf, sizeof(buf))) > 0)writewrite out, buf, n); out, buf, n);

}}


select & poll functions

Means to monitor & access various I/O devices Means to monitor & access various I/O devices when unsure which is readywhen unsure which is ready

Considerations:Considerations:Blocking vs. non-blockingBlocking vs. non-blockingConditional vs. time-slicesConditional vs. time-slices


In some sense, the problems we will look at are no different than those in In some sense, the problems we will look at are no different than those in virtual memoryvirtual memory– like page tables, file system meta data are simply data structures like page tables, file system meta data are simply data structures

used to construct mappings.used to construct mappings.– Page table: map virtual page # to physical page #Page table: map virtual page # to physical page #

– file meta data: map byte offset to disk block addressfile meta data: map byte offset to disk block address

– directory: map name to disk block addressdirectory: map name to disk block address

What hard Grouping Blocks?

Page table28 33

Unix inode 418 8003121

directoryfoo.c 3330103

Like usual, we’re going to call the same thing by different names. We’ll be using lists and trees of arrays to track integers, but instead of calling them that or page tables, now meta data. Purpose the same: construct a mapping.

Like usual, we’re going to call the same thing by different names. We’ll be using lists and trees of arrays to track integers, but instead of calling them that or page tables, now meta data. Purpose the same: construct a mapping.


In some ways problem similar: In some ways problem similar: – Desire location transparency: oblivious to size & protectionDesire location transparency: oblivious to size & protection

In some ways the problem is easier: In some ways the problem is easier: – CPU time to do FS mappings not a big deal (no TLB)CPU time to do FS mappings not a big deal (no TLB)– Page tables deal with sparse address spaces and random access, Page tables deal with sparse address spaces and random access,

files are dense (0 .. filesize-1) & ~sequentialfiles are dense (0 .. filesize-1) & ~sequential

In some ways problem is harder:In some ways problem is harder:– Each layer of translation = potential disk accessEach layer of translation = potential disk access– Space a huge premium! (But disk is huge?!?!) Reason? Never Space a huge premium! (But disk is huge?!?!) Reason? Never

enough cache space: amount of data you can enough cache space: amount of data you can getget in one fetch in one fetch– Range very extreme: Many files <10k, some more than GB.Range very extreme: Many files <10k, some more than GB.– Implications?Implications?

File System vs. Virtual Memory

Recall: can fetch a track at a time, or about 64KRecall: can fetch a track at a time, or about 64K


Some Working IntuitionsFS performance dominated by # of disk accessesFS performance dominated by # of disk accesses

– Each access costs 10s of millisecondsEach access costs 10s of milliseconds– Touch the disk 50-100 extra times = 1 *second*Touch the disk 50-100 extra times = 1 *second*– Can easily do 100s of millions of ALU ops in same timeCan easily do 100s of millions of ALU ops in same time

Access cost dominated by movement, not transferAccess cost dominated by movement, not transfer– Can get 20x the data for only ~5% more overheadCan get 20x the data for only ~5% more overhead– 1 sector = 10ms + 8ms + 50us (512/10MB/s) = 18ms1 sector = 10ms + 8ms + 50us (512/10MB/s) = 18ms– 20 sectors = 10ms + 8ms + 1ms = 19ms20 sectors = 10ms + 8ms + 1ms = 19ms

Observations:Observations:– All blocks in file tend to be used together sequentiallyAll blocks in file tend to be used together sequentially– Most files in a directory tend to be used togetherMost files in a directory tend to be used together– Most names in a directory tend to be used togetherMost names in a directory tend to be used together– How to exploit?How to exploit?


Common addressing patterns

Sequential:Sequential:– File data processed in sequential orderFile data processed in sequential order– By far, the most common modeBy far, the most common mode– Ex: editor writes out new file, compiler reads in file, etc.Ex: editor writes out new file, compiler reads in file, etc.

Random access: Random access: – Address any block in file directly without passing through Address any block in file directly without passing through

predecessorspredecessors– Ex: data set for demand paging, databasesEx: data set for demand paging, databases

Keyed access: Keyed access: – Search for block with particular valuesSearch for block with particular values– Ex: associative data base, indexEx: associative data base, index– Usually not provided by OS Usually not provided by OS


Problem: how to track file’s data?

Disk management: Disk management: – Need to keep track of where file contents are on diskNeed to keep track of where file contents are on disk– Must be able to use this to map byte offset to disk blockMust be able to use this to map byte offset to disk block– The data structure used to track a file’s sectors is called a The data structure used to track a file’s sectors is called a file file

descriptordescriptor– File descriptors often stored on disk along with fileFile descriptors often stored on disk along with file

Things to keep in mind while designing file structure:Things to keep in mind while designing file structure:– Most files are small Most files are small – Much of the disk is allocated to large filesMuch of the disk is allocated to large files– Many of the I/O operations are made to large filesMany of the I/O operations are made to large files– Want good sequential and good random access (what do these Want good sequential and good random access (what do these

require?)require?)

Fixed cost must be low, must be able to nicely represent large files, and accessing them must not take too much time

Fixed cost must be low, must be able to nicely represent large files, and accessing them must not take too much time


File Representation:Handles & Descriptors

File HandlesFile HandlesTwo methods of Two methods of doing I/O in UNIX:doing I/O in UNIX:File descriptors: File descriptors: this is part of the this is part of the UNIX UNIX specification, specification, POSIX, etc.POSIX, etc.File pointers: this File pointers: this is from the C is from the C standard standard

File DescriptorsFile DescriptorsUse the POSIX Use the POSIX names names STDIN_FILENO, STDIN_FILENO, STDOUT_FILENO STDOUT_FILENO and and STDERR_FILENOSTDERR_FILENO


Parent & Child Share Same File Pointer

Example 2: Example 2: fileshare.cfileshare.c main ()main ()

{{int in;int in;pid_t pid;pid_t pid;char buf[1024];char buf[1024];

system ("echo abc123 > junk");system ("echo abc123 > junk");in = open("junk", O_RDONLY);in = open("junk", O_RDONLY);

pid = fork();pid = fork();if (pid == 0) {if (pid == 0) {read(in, buf, 3);read(in, buf, 3);write(1, buf, 3); write(1, buf, 3); }}else {else {read(in, buf, 3);read(in, buf, 3);write(1, buf, 3);write(1, buf, 3);}}write(1,"\n", 1);write(1,"\n", 1);

}}


Used since CTSS (1960s)Used since CTSS (1960s)– Unix picked up and used really nicely.Unix picked up and used really nicely.

Directories stored on disk just like regular filesDirectories stored on disk just like regular files– inode contains special flag bit setinode contains special flag bit set– Users can read just like any other fileUsers can read just like any other file– Only special programs can write (why?)Only special programs can write (why?)– File pointed to by the index may be File pointed to by the index may be

another directoryanother directory– Makes FS into hierarchical treeMakes FS into hierarchical tree

Simple. Simple. ++ Speeding up file ops = Speeding up dir Speeding up file ops = Speeding up dir ops! ops!

Hierarchical Unix

/

afs bin cdrom dev sbin tmp

awk chmod chown

<name, inode#><afs, 1021><tmp, 1020><bin, 1022><cdrom, 4123><dev, 1001><sbin, 1011> ...


inodes

Directory entries Directory entries contain only a contain only a name and index name and index into file into file information information table. The table table. The table and index are and index are both referred both referred to as an inode.to as an inode.


ptr 1ptr 2 …ptr 128

Multi-level indexed files: ~4.3 BSD

File descriptor (File descriptor (inodeinode) = 14 block pointers + “stuff”) = 14 block pointers + “stuff”

Ptr 1ptr 2ptr 3ptr 4...

ptr 13ptr 14

stuffdata blocks

ptr 1ptr 2 …ptr 128

Indirect block

Double indirect block

Indirect blks


Indexed files (DOS FAT)Each file has an array holding all of it’s block pointersEach file has an array holding all of it’s block pointers

– (purpose and issues = those of a page table)(purpose and issues = those of a page table)– Max file size fixed by array’s size (static or dynamic?)Max file size fixed by array’s size (static or dynamic?)– Create: allocate array to hold all file’s blocks, but allocate on Create: allocate array to hold all file’s blocks, but allocate on

demand using free listdemand using free list

– Pro: both sequential and random access easyPro: both sequential and random access easy– Con? (When did you last defragment your HDD?)Con? (When did you last defragment your HDD?)

file a file b


Pro:Pro:– Simple, easy to build, fast access to small filesSimple, easy to build, fast access to small files– Maximum file length fixed, but large. (With 4k blks?)Maximum file length fixed, but large. (With 4k blks?)

Cons:Cons:– What’s the worst case # of accesses?What’s the worst case # of accesses?– What’s some bad space overheads?What’s some bad space overheads?

An empirical problem:An empirical problem:– Because you allocate blocks by taking them off unordered Because you allocate blocks by taking them off unordered

freelist, meta data and data get strewn across diskfreelist, meta data and data get strewn across disk

Unix discussion


Latency tricksLatency tricks– Caching (data centric)Caching (data centric)– Code migration (code centric)Code migration (code centric)– Prefetching (dual: asynchronous writes)Prefetching (dual: asynchronous writes)– Clever data layoutClever data layout

Throughput tricks:Throughput tricks:– Increase bandwidth? Duplicate device N timesIncrease bandwidth? Duplicate device N times– Hide latency? Run another thread while waitingHide latency? Run another thread while waiting– ““Money can buy bandwidth. Latency requires bribing God.”Money can buy bandwidth. Latency requires bribing God.”

These tricks are eternal themesThese tricks are eternal themes– Use whenever need fast access to data living in slow placeUse whenever need fast access to data living in slow place– Common straws: I/O bus, memory bus, network, space (bi-Common straws: I/O bus, memory bus, network, space (bi-

directional), time (uni-directional)directional), time (uni-directional)

Sucking & Blowing Data through Slow Straws

Once there, can usually get a chunk rather than one thing. So clusterOnce there, can usually get a chunk rather than one thing. So cluster


Importance of writes?Importance of writes?

Write through (simple, but slow):Write through (simple, but slow):– Whenever modify cached block, write block to disk.Whenever modify cached block, write block to disk.– Con: makes writes slow.Con: makes writes slow.– Pro: keeps FS in consistent state. PC OSes do this. Pro: keeps FS in consistent state. PC OSes do this.

Write back (faster, but complex (and dangerous)):Write back (faster, but complex (and dangerous)):– Whenever modify block, mark as dirty. Flushed laterWhenever modify block, mark as dirty. Flushed later– Pro: fast writes, absorbs writes, enables batchingPro: fast writes, absorbs writes, enables batching– Con: More you defer write back, the worse a crash is.Con: More you defer write back, the worse a crash is.

Cache Write back Policies

size

time

Memory size

File size Implications?

Which do DOS and MacOS do as user can pull floppy out at any time?Which do DOS and MacOS do as user can pull floppy out at any time?


Fundamental tension:Fundamental tension:– On crash, all modified data in cache is lost.On crash, all modified data in cache is lost.– Longer you postpone write back, the worst the damage is, but Longer you postpone write back, the worst the damage is, but

the faster you are.the faster you are.

Four times to flush:Four times to flush:– When block evicted (this is ~ what VM does)When block evicted (this is ~ what VM does)– When a file is closed (distributed file systems do this)When a file is closed (distributed file systems do this)– On an explicit flush (“man sync”)On an explicit flush (“man sync”)– When a time interval elapses (30 secs in Unix)When a time interval elapses (30 secs in Unix)

Write back doesn’t respect orderingWrite back doesn’t respect ordering– Crash = file in weird jumbled stateCrash = file in weird jumbled state

Finally: OS may not have any choice!Finally: OS may not have any choice!– Disk could be doing write buffering.Disk could be doing write buffering.

Write back complications


Losing some blocks worse than losing others. Usually:Losing some blocks worse than losing others. Usually:

File system effects:File system effects:– Flush modified meta data Flush modified meta data

back quickly back quickly – Note: was an implicit Note: was an implicit

side-effect of synchronous writes for meta data modsside-effect of synchronous writes for meta data mods

Application effects:Application effects:– Have a really important file? Issue a manual flush (sync) to Have a really important file? Issue a manual flush (sync) to

make sure its saved to disk.make sure its saved to disk.– Databases are forced to do this. Frequently, they will just by-Databases are forced to do this. Frequently, they will just by-

pass file system. pass file system.

Flushing nuance: All blocks are not equal

Data block

inode

directory

importance

superblock


File Sharing between Processes (Inheritance)

Active


Filters and Redirection

Filters, pipes (“|”) Filters, pipes (“|”) provide a means provide a means for UNIX utilities for UNIX utilities to transform to transform input (read from input (read from standard input) standard input) to an output (on to an output (on standard output)standard output)

Examples: Examples: moremore, , grepgrep

Redirection: take Redirection: take input (“<“) or input (“<“) or provide output provide output (“>“) elsewhere(“>“) elsewhere


Pipelines: Inter-Process Communication (IPC)

pipepipe calls, similar calls, similar to use at the to use at the command command prompt (“|”) prompt (“|”) provide a provide a communication communication buffer means to buffer means to process dataprocess data

A FIFO (or named A FIFO (or named pipe) processes pipe) processes data on first in, data on first in, first out basis first out basis and persists and persists beyond process beyond process execution.execution.


Child Writes to Parent

Example 3: Example 3: pipedate.cpipedate.cmain()main()

{{pid_t pid;pid_t pid;int pfd[2];int pfd[2];int i, status;int i, status;char line[64];char line[64];

pipe(pfd);pipe(pfd);

pid = fork();pid = fork();

if (pid == 0) {if (pid == 0) {dup2(pfd[1], 1);dup2(pfd[1], 1);close(pfd[1]);close(pfd[1]);close(pfd[0]);close(pfd[0]);

execl("/bin/date", "date", 0);execl("/bin/date", "date", 0);}}

close(pfd[1]);close(pfd[1]);

read (pfd[0], line, read (pfd[0], line, sizeof(line));sizeof(line));printf("date from child is: %s\printf("date from child is: %s\n", line);n", line);

close(pfd[0]);close(pfd[0]);waitpid(pid, &status, 0);waitpid(pid, &status, 0);exit(0);exit(0);}}


Parent Writes to Child

Example 4: Example 4: pipemail.cpipemail.cmain()main()

{{pid_t pid;pid_t pid;int pfd[2];int pfd[2];int i, status;int i, status;char *username;char *username;

username = cuserid(NULL);username = cuserid(NULL);

pipe(pfd);pipe(pfd);

pid = fork();pid = fork();

if (pid == 0) {if (pid == 0) {

dup2(pfd[0], 0);dup2(pfd[0], 0);close(pfd[1]);close(pfd[1]);

execl("/bin/mail", "mail", execl("/bin/mail", "mail", username, 0);username, 0);}}

close(pfd[0]);close(pfd[0]);

write(pfd[1], "Greetings and write(pfd[1], "Greetings and salutations,\n\n", 28);salutations,\n\n", 28);write(pfd[1], "This is your write(pfd[1], "This is your program saying hello.\n", 35);program saying hello.\n", 35);write(pfd[1], "Have a nice write(pfd[1], "Have a nice day.\n\n", 18);day.\n\n", 18);write(pfd[1], "Bye.\n", 5);write(pfd[1], "Bye.\n", 5);

close(pfd[1]);close(pfd[1]);waitpid(pid, &status, 0);waitpid(pid, &status, 0);exit(0);exit(0);}}


I/O, Files & Directories Summary

. • Completes discussion of basic issuesCompletes discussion of basic issues• I/O handling in terms of filesI/O handling in terms of files• Manipulation of files, directoriesManipulation of files, directories


Project Discussions

Code & concept issues …Code & concept issues …

Documents

1GWU CS 259 Brad Taylor Spring 2004 Systems Programming Meeting 3: Handling Input/Output Files, Directories & Special Files