N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 3 February, 1999 Effective Use of NERSC File Systems 1 Effective Use of NERSC File Systems Thomas

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

3 February, 1999 Effective Use of NERSC File Systems 1

Effective Use of NERSC File Systems

Thomas M. DeBoni

NERSC/USG



Effective Use of NERSC File Systems

Contents

•Home Directories

•Scratch Space

•Mass Storage

•Networked File Systems

•Resource Conservation

•Examples

•DetailsSee alsohttp://home.nersc.gov/training/tutorials/file.management.html



Home Directories• Your private portion of the file systems on a computer

• Default “current working directory” when you log in

• Knows as system environment variable $HOME”

• Usage limited in bytes– Max size is 2 GB on T3E, 5 GB on J-90s

– Warnings issued at 75-90% usage

– There’s not enough space for everybody to use this much at the same time, so “migration” sometimes happens

• Usage limited in “inodes”– An “inode” is a file or directory

– Max number is 6000 on T3E, unsettled on J-90s

– Warnings issued at 75-90% usage



Home Directories, cont.

• $HOME– is routinely backed up

– is shared among all J-90’s

– is NOT the fastest file system available

– should be used for development, debugging, pre- and post-processing, and other administrative tasks

– should NOT be routinely used by large jobs requiring high performance

• Startup files (.cshrc, .login, etc.) may change your working directory on login– Remove all references to $WRK from these files




• Files can be “migrated” to backing store– Largest and oldest files first

– De-migrate with “dmget” command before using, or

– Automatically de-migrated when referenced, but with unknown delay

– Example listing: killeen 257: ls -al

total 64

drwx------ 2 u10101 zzz 4096 Sep 21 11:11 .

drwxr-xr-x 5 u10101 zzz 4096 Sep 21 11:11 ..

mrw------- 1 u10101 zzz 2414 Sep 21 11:11 decomp.job.log

mrw------- 1 u10101 zzz 2712 Sep 21 11:11 decomp.job.out

-rw------- 1 u10101 zzz 2381 Sep 21 11:11 decomp.job2.log

-rw------- 1 u10101 zzz 11490 Sep 21 11:11 decomp.job2.out

“m” in first column means file has been migrated




• A word about “quotas”– Use the “quota” command to view them:mcurie 154: quota

File system: /u5

User: deboni, Id: 9950

Aggregate blocks (512 bytes) Inodes

User Quota: 3906240* ( 19.2%) 3500* ( 17.2%)

Warning: 3515616* ( 21.4%) 2975* ( 20.2%)

Usage: 751416 602

Current usageis this % of max usageis this % of warning level

•Maximum usage allowed•Level at which warning will be issued•Current usage



Scratch Space

• Also known as temporary storage or working storage

• A pool of fast RAID drives

• The fastest file system available

• Unique to each batch system

• Not backed up

• Usage limits are larger than for $HOME– 75 GB and 5000-6000 inodes on T3E, unsettled on J-90s

• Should be used for large files and high performance jobs

• This is transient space, and persistence will vary with usage and demand



Scratch Space, Cont.

• System environment variable “$TMPDIR”– Created for each session or batch job

– Randomly named, so always use $TMPDIR to refer to it

– Deleted at the end of session or job

• Use “$TMPDIR” if you want the OS to manage your scratch space usage for you.– E.g., on the J-90s, you can’t log on to batch machines, and each has

its own scratch space, so you can’t get at it directly, as you can on the T3E.



Scratch Space, cont.

• /tmp or /usr/tmp– Create directories there for yourself

– Watch out for name collisions with other users’ directories

– Delete files and directories as you finish with them

– This space will be scavenged depending on demand: largest and oldest files are usually deleted first

– “It should be safe for 7 to 14 days…”

• You must manage this scratch space for yourself



Scratch Space, cont.• Pre-staging files to scratch space is a good idea, but...

– You don’t know when your batch job will run, so it may not work when batch queues are heavily loaded

• Staging files in a batch script is a good idea, but...– It idles your processor ensemble and uses up serial time

• So, do it both ways:#!/bin/csh -f

# Change to scratch directory

cd /tmp/mydir

# Check for presence of pre-staged files

if (-e foo.input) then

echo "input file prestaged"

else

echo "fetching input file"

hsi "get foo.input"

endif



Scratch Space, cont.

• What about intermediate I/O? Example to follow…

• What about I/O from parallel programs?– This is a deep topic

– Beware of interleaving of multiple outputs to a single file

– J-90 codes typically use a small number of files at a time

– T3E codes may use hundreds of files at a time

– Special mechanisms exist to manage parallel files and I/O

– General rule: the more you do, the faster it should be

– See these sources for further info:• http://home.nersc.gov/software/prgenv/opt/binary.html • http://home.nersc.gov/training/tutorials/T3E/IO/• http://www.cray.com/swpubs/

(See, especially , " CRAY T3E Fortran Optimization Guide, SG-2518 3.0”)



Mass Storage

• NERSC provides the High Performance Storage System (HPSS)

• A modern, flexible, hierarchical system

• Optimized for large files and fast transfers

• Built on multiple disk farms and tape libraries

• Used for system backups, file migration, and user archives

• Has multiple user interface utilities– HSI - powerful, flexible, convenient utility, from SDSC and NERSC– pftp - parallel ftp, locally customized, fastest for large files– ftp - traditional version, available everywhere

• The proper place to save large, important, long-lived files (e.g. raw output and restart data)

• Requires a separate storage account (“DCE account”), but can automatically authenticate after initial setup



Networked File Systems

• “Networked” or “distributed” file systems are intended to decouple a file’s physical location from it’s logical location

• Can be very convenient, but also dangerous

• There are three of interest– NFS - Developed at Sun and has become a standard in workstation

environments• Used as little as possible at NERSC, due to security and performance concerns

– AFS - More modern, global in scope, with pretty good security• Used at NERSC via the “gateway system” dano.nersc.gov

• Use AFS with care - it can ruin performance

– DFS - A “coming standard” that NERSC is evolving toward; also has good security and will be global in scope



Resource Conservation

• Critical resources are expensive and rare

• They are shared among (competed for by) all users

• Four critical resources related to file systems use:– Storage space - the actual files and bytes of data

– File system entries - “inodes”; one per file or directory

– Bandwidth - bits per second, in transfers between devices

– Time - servers, I/O devices, and CPU cycles

• NERSC meters (charges for) all these types

• Resource conservation must be engineered in (don’t depend on luck)



Bandwidth Conservation

• Design parallel I/O carefully– Human readable I/O probably should be done by a single (master)

process(or) to/from a single file

– Binary I/O may be done by one process(or) or by many, as required

– Binary data may occupy many files which match problem decomposition for parallel execution

– Limits exist on the number of files that can be open at any one time

– Flushing larger buffers is usually a better idea than flushing smaller ones more often.

• For further info, see– Cray publication “Application Programmer's I/O Guide”– NERSC web doc http://home.nersc.gov/training/tutorials/T3E/IO/

– Man page for the “assign” command



Bandwidth Conservation, cont.

• Transfer files carefully– Session setup is not free - move files to/from mass storage in as few

sessions or commands as possible; don’t run pftp in a loop

– Use multiple-file transfer commands, such as “mget”, when possible

• Use the appropriate utility for the job– “Meta-data operations do not involve actual file access

• Renaming files or directories or moving files around within HPSS

• changing file or directory permissions in HPSS

– Use hsi for these sorts of operations, for efficiency

• For further info see– NERSC web doc http://home.nersc.gov/hardware/storage/hpss.html– NERSC web doc http://home.nersc.gov/hardware/storage/hsi.html



Bandwidth and Time Conservation• Use the fastest utility available

– Use pftp when moving within the NERSC domain

– Use multiple-file transfer commands, such as “mget”, when possible

– Use ftp when moving files into or out of the NERSC domain

• Use the fastest devices and networks available– This is a deep area … and oversimplified here ...

– Don’t make a fast machine wait on a slow one

– Pre-stage and de-migrate files needed by batch jobs into fast storage space

– Sometimes a multiple-step process is better• First, move files from outside NERSC onto a NERSC computer (a workstation)

• Then, move files from the NERSC computer to the destination device

– Avoid networked file systems

• For further info see– NERSC web doc http://home.nersc.gov/hardware/storage/hpss.html– Man pages for ftp, pftp, and hsi



Bandwidth and Storage Conservation

• Shrink files, if appropriate– If the file contains redundant or unimportant data

• Such as white space in formatted output

– Use Unix commands “compress” and “gzip”

• Combine files into archives– If the files are small, transferring them individually may involve

more setup time than transfer time

– Use Unix commands “tar”, “ar”, and “cpio”

• For more info, see– Man pages for all the above commands



Example 1 - batch pftp multiple file access with a here-doc

#!/bin/csh

...

# First, copy the source from the submitting directory and compile it.

pftp -i -v archive <<+

cd my_HPSS_directory

mget data* myprog

quit

+

ja

./myprog <data >outfile

ja -cst

# Save the output file in HPSS.

pftp -i -v archive <<+

cd my_HPSS_directory

mput outfile* restart*

quit

+

exit

Here-document



Example 2 - batch hsi multiple file access

#!/bin/csh

...

# First, copy the source from the submitting directory and compile it.

hsi archive “cd my_HPSS_directory; mget data* myprog”

ja

./myprog <data >outfile

ja -cst

# Save the output file in HPSS.

hsi archive “cd my_HPSS_directory; mput outfile* restart*”

exit



Example 3 - Minimizing parallel cpu idling during pftp I/O#!/bin/csh -f

# preliminary job steps, including fetching executables and input files

. . .

# parallel code execution with mpprun

set i = 1

mpprun -n 128 a.out < bigjob.input > bigjob.output

# Intermediate file movement, to save output file to mass storage:

mv bigjob.output bigjob.output$i

mv bigjob.restart bigjob.restart$I

# Generate a separate serial job to do the actual I/O

echo "pftp -i -v archive <<EOF\

mkdir bigjobs/job.06.15.99\

cd bigjobs/job.06.15.99 \

mput bigjob.output$i bigjob.restart$i \

ls\

quit\

EOF" | qsub -q serial

. . .

# further parallel code execution, perhaps through shell script looping

@ i = $i + 1

. . .



Example 4 - Minimizing parallel cpu idling during HSI I/O#!/bin/csh -f

# preliminary job steps, including fetching executables and input files

. . .

# parallel code execution with mpprun

set i = 1

mpprun -n 128 a.out < bigjob.input > bigjob.output

# Intermediate file movement, to save output file to mass storage:

mv bigjob.output bigjob.output$i

mv bigjob.restart bigjob.restart$i

# Generate a separate serial job do to the actual I/O

echo ”hsi archive ‘mkdir bigjobs/job.06.15.99; cd bigjobs/job.06.15.99; mput bigjob.output$i bigjob.restart$i; ls’“ | qsub -q serial

. . .

# further parallel code execution, perhaps through shell script looping

@i = $i + 1

. . .



Example 5 - Minimizing usage with tar and compressmcurie 181: ls -al STD* bigjob*-rw-r--r-- 1 deboni mpccc 0 Dec 23 12:26 STDIN.e48938-rw-r--r-- 1 deboni mpccc 0 Dec 23 12:27 STDIN.e48939-rw-r--r-- 1 deboni mpccc 1126 Dec 23 12:26 STDIN.l48938-rw-r--r-- 1 deboni mpccc 1126 Dec 23 12:27 STDIN.l48939-rw-r--r-- 1 deboni mpccc 7181 Dec 23 12:26 STDIN.o48938-rw-r--r-- 1 deboni mpccc 7181 Dec 23 12:27 STDIN.o48939-rw------- 1 deboni mpccc 486 Feb 4 11:24 bigjob.output-rw------- 1 deboni mpccc 486 Feb 4 12:10 bigjob.output1-rw------- 1 deboni mpccc 486 Feb 4 12:10 bigjob.output2-rw------- 1 deboni mpccc 972 Feb 4 11:24 bigjob.restart-rw------- 1 deboni mpccc 972 Feb 4 12:10 bigjob.restart1-rw------- 1 deboni mpccc 972 Feb 4 12:10 bigjob.restart2#--------------------total space = 20988 bytes and 12 inodesmcurie 182: tar cf batch.tar STD* bigjob*mcurie 183: ls -al batch.tar-rw------- 1 deboni mpccc 65536 Feb 5 09:15 batch.tarmcurie 184: compress batch.tarmcurie 185: ls -al batch.tar*-rw------- 1 deboni mpccc 9313 Feb 5 09:15 batch.tar.Zmcurie 186: rm STD* bigjob*#--------------------total space = 9313 bytes and 1 inode



Example 6 - Minimizing usage with cpio

cd $HOME

/bin/find . -type f -size -15000c -atime +90 ! \

( -type m -o -type M \) \ -print > hitlist

vi hitlist

cat hitlist | cpio -co > myfiles.cpio

cat hitlist | xargs rm -f

Here's what the above commands (NOT a shell script!) do:

1) First, cd to home directory and generate a list of eligible files;

2) the find command will find regular files smaller than 15000 chars, that have not been accessed in 90 days, and are not migrated.

3) Use "vi" to examine the list and delete items from it that you do not want removed.

4) The fourth line will create the cpio file archive.

5) The fifth line will remove all the files now stored in the archive.



Details - some useful ftp and pftp commands

FTP Commands Meanings or actions PFTP Variants get <rf> [<lf>] retrieve a file pgetput <lf> [<rf>] store a file pputmget <f> [<f>…] retrieve multiple files mpgetmput <f> [<f>…] store multiple files mpputdel <f> delete a filemdel <f> [<f>…] delete multiple filesmkdir <d> create a remote directoryrmdir <d> delete a remote directorycd <d> change to remote dirctorylcd <d> change to local directoryls, dir list files in directoryldir list files in local directory!<cmd> perform <cmd> locally outside ftp/pftp<f> = file name, <lf> = local file name, <rf> = remote file name

Caveats:Be aware of where your actions will take placeWatch out for name collisions



Details - HSI commands

HPSS File and Directory Commandsget, mget, recv - Copy file(s) from HPSS to a local

directory

cget - Copy file from HPSS to a local

directory if not already there

put, mput, replace,

save, store, send - Copy local file(s) to HPSS

cput - Copy local file to HPSS if it

doesn’t already exist there

cp, copy - Copy file within HPSS

mv, move, rename - Rename/relocate an HPSS file

delete, mdelete, erase, rm - Remove a file from HPSS

ls, list - List directory



Details - HSI commands, cont.

HPSS File and Directory Commands, cont.find - Traverse a directory tree looking

for a file

mkdir, md, add - Create an HPSS directory

rmdir, rd, remove - Delete an HPSS directory

pwd - Print current directory

cd, cdls - Change current directory

Local File and Directory Commandslcd, lcdls - Change local directory

lls - List local directory

lpwd - Print current local directory

! - Issue shell command



Details - HSI commands, cont.File Administrative Informationchmod - Change permissions of file or directory

umask - Set file creation permission mask

Miscellaneous HSI commandshelp - Display help file

quit, exit, end - Terminate HSI

in - Read commands from a local file

out - Write HSI output to a local file

log - Write commands and responses to a log file

prompt - Toggles prompting for mget, mput, mdelete



Details - HSI commands, cont. HSI can accept input several different ways:• From a command session, consisting of multiple lines and ending with an

explicit termination command • From a single line command, with semicolons (“;”) separating commands

hsi “mkdir foo; cd foo; put data_file”

• From a command file hsi “in command_file”

• HSI can read from standard input and write to standard output tar cvf - . | hsi put - : datadir.tar

hsi get - : datadir.tar | tar xvf -

Wildcards are supported, but quoting must be used in one-line commands to prevent shell interpretation.

hsi “cd foo; mget data*”



Details - HSI commands, cont.WARNING: For 'get' and 'put' operations, HSI uses a

different syntax than ftp; a colon (“:”) is used to separate local and remote file names.

put local_file : hpss_file

get local_file : hpss_file

Recursive operations are allowed for the following commands:cget, chgrp, chmod, chown, cput, delete, get, ls,

mdelete, mget, mput, put, rm

Special commands exist for setting up variables whose values are directories, commands, and command-sets.

The complete HSI manual is online at http://home.nersc.gov/hardware/storage/hsi.html



Details - Tasks that HSI Simplifies• Accessing segmented CFS files:

– CFS handled files larger then 400 MB by spliting them into smaller subfiles and storing the subfiles. HSI is the only utility that can read and rejoin segmented CFS files to reproduce their original state. The procedure for handling such files is quite simple: simply read the first of the segmented subfiles from the archive storage system.

• Renaming/moving or copying an entire subdirectory :– <mv/cp> path1 path2 renames/copies path1 to path2

• Changing the permissions of several files at once:– chmod perms files changes the permissions of all files

to perms; the file specifications may include wildcards; the permissions may be given as octal numbers or via symbolic designators.



Details - Getting Access to AFS Directories

Don’t do this in batch jobs!

killeen 210: telnet dano.nersc.gov

Trying 128.55.200.40...

Connected to dano.nersc.gov.

Escape character is '^]'.

Hello killeen.nersc.gov.

=========================================================================

> WARNING: Unauthorized access to this computer system is <

> prohibited, and is subject to criminal and civil penalties. <

> Use constitutes consent to security testing and monitoring. <

=========================================================================

UNIX(r) System V Release 4.0 (dano)

login: u10101

Password:

AFS (R) 3.4 Login

Last login: Mon Sep 21 10:00:14 from killeen.nersc.gov



Details - Getting Access to AFS Directories, cont. *********************************************************

* AFS gateway user interface *

* *

* used to enable AFS access on J90's and T3E *

* (select enable attached hosts (1) before exiting) *

*********************************************************

* *

* 1) enable attached hosts (knfs) *

* 2) disable attached hosts (unlog) *

* 3) list tokens (tokens) *

* 4) authenticate to another cell (klog) *

* *

* 5) help *

* 0) exit (logoff) *

*********************************************************

enter command(0-5): 3

Tokens held by the Cache Manager:

User's (AFS ID 9950) tokens for [email protected] [Expires Oct 17 15:55]

--End of list--



Details - Getting Access to AFS Directories, cont.

enter command(0-5):0

Connection closed by foreign host.

killeen 213: pwd

/U3/u10101

killeen 214: cd /afs/nersc.gov

killeen 215: pwd

/afs/nersc.gov

Option 4 is used to attach other cells, regardless of location, but you must have login and password to use in the “klog” process.



Details - Dealing With Your DCE Account

• DCE is a modern authentication methodology that will likely evolve into general use at NERSC

• Right now, it merely control access to HPSS

• DCE accounts and login/password info must be gotten from NERSC Support staff

• Initial login is necessary to change from initial password, and to set up future automatic authentication– It has been occasionally necessary for a few users to re-initialize

their accounts

– Both procedures are easy

– DCE is currently most reliable on killeen.nersc.gov



Details - Dealing With Your DCE Account, cont.Initial Setup: Once you have your initial DCE login and password, change it

with the following procedure, on any NERSC mainframe: % dce_login Enter Principal Name: <HPSS_user_name>

Enter Password: <current_or_temporary_HPSS/DCE_password>

% chpass -p

Changing registry password for HPSS_user_name

New password: <new_HPSS/DCE_passwd>

Re-enter new password: <new_HPSS/DCE_passwd>

Enter your previous password for verification: <current_or_temporary_HPSS/DCE_password>

% kdestroy

% exit

%

You will need to log in to HPSS only on your next use, and thereafter you will be automatically authenticated.



Details - Dealing With Your DCE Account, cont.

If you should get the following message from HPSS...mcurie 224: hsi hpss

credential user mismatch

use -l option to generate a new cred file

DCE Principal:

…it means automatic authentication has failed.

You must authenticate manually, until you re-initialize authentication:mcurie 232: hsi -l hpss

DCE Principal: u10101

Password:

*-----------------------------------------------------------*

* NERSC HPSS USER SYSTEM(hpss) *

*-----------------------------------------------------------*

V1.5 Username: u10101 UID: 0123

? quit

Subsequent usage should not require full login.

In rare and unusual situations, do “rm .hsipw” and then repeat the above.

Documents

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 3 February, 1999 Effective Use of NERSC File Systems 1 Effective Use of NERSC File Systems Thomas