Filesystems and Quotas

Allocation Areas & Quotas

There are three main directories or filesystems of importance that exist on the Hydra. You are allocated a finite amount of diskspace on each.

Directory Quota Backed Up? Description
/home/userid 300MB Yes Upon login, you will be situated in /home/userid, where userid is your logon name. This area is automatically replicated for safety. So in effect the maximum disk space you can occupy here is 600MB. The disk system supplying this space is directly attached to four (external) Power 520 systems and made available to all the cluster nodes via ethernet. For that reason it is the lowest performing of the three disk areas that users have access. Please be mindful of this fact when you compile and run codes in your home directory. The response will not be nearly as good, especially when large files are involved, compared to working in the /scratch or /work areas. It is also the only area that will be backed up nightly. This directory can be referenced with the $HOME environment variable.
/scratch/userid 10GB No This is a high I/O performance area, intended to temporarily hold rather large files that you will need for current processing needs. The disk supplying this storage is hosted on a fiber-channel DataDirect Networks (DDN) raid array. The DDN array is directly attached to four p575 nodes, the so called I/O nodes of the cluster by fiber channel links. Because this is a high performance area, its intended role is to hold files that undergo frequent processing. It is not intended as a long-term store for needed but largely inactive files. Both /scratch/username and /scratch/username/tmp are automatically available upon login. Upon login to the cluster, the environment variable TMPDIR has the value of /scratch/username/tmp. This is only true for interactive processing. We will adjust your quota for justifiable processing needs. You are automatically granted an allocation of diskspace in the scratch directory. Please notify the helpdesk when one is not already in place.
/work/batch_jobid n/a No No user quota is in effect here. This directory is available only to executing batch jobs and only for the duration of the job. A LoadLeveler job gets access to this area by using the command, cd $TMPDIR. The value of the environment variable TMPDIR is /work/batch_jobid during job execution. Like /scratch above, /work also resides on the DDN. It has the highest I/O performance on the cluster. This is especially the case with sequential I/O involving large transfer sizes.

All three areas, /home, /scratch, and /work are configured as parallel (GPFS) file systems and are globally accessible from each node. All data transfers from any node that are associated with /scratch or /work employ the HPS switch and the DDN raid array. Data movement that involves only /home uses Gigabit Ethernet and Ultra-320 SCSI drives that are attached to four Power 520 IBM workstations.

REMINDER: The /scratch area is NOT for long-term storage of valuable but largely inactive files. Valuable files which undergo infrequent use should be moved else- where.


Of the three filesystems mentioned above only your /home directory is backed up. Files can be restored upon request.

Extensions to your /home space will be strongly resisted beyond the standard allocation. The staff will consider increases to your /scratch space only for justifiably legitimate needs.

Needed input files that are unavailable on hydra should first be copied locally, say, to /scratch. These files can then be copied to a job's $TMPDIR using the standard 'cp' command. Large output files resulting from batch job execution should be handled similarly.

The saving of data files on hydra that are unrelated to processing on it is NOT allowed. Users should also delete all files that are no longer needed. Please cooperate with the staff in this important task.

Common File Commands

Display disk space allotments: myquota

hydra# myquota aggie
     Filesystem   blocks      quota    limit    grace    files      quota    limit    grace
           /var      320      10240    10240                 3       1000     1000         
           /tmp        0      10240    10240                 0       1000     1000
                         Block Limits                                    |     File Limits
Filesystem type             KB      quota      limit   in_doubt    grace |    files   quota
    limit in_doubt    grace  Remarks
gpfs_home  USR             240     614400     614400          0     none |       15   20000
    30000        0     none gpfs520.gpfs1-priv
gpfs_scratch USR              32   10485760   10485760          0     none |        1   30000
    30000        0     none 

The four leftmost columns indicate the filesystem, blocks used, quota, and upper limit. The system will prevent you from writing files beyond your upper limit. Numerical amounts of disk space are given in kilobytes.

Display files and directories: ls

% ls -l      
total 464
drwx------    2 aggie  user           18 May 26 11:03 dira
drwx------    2 aggie  user           30 May 26 11:03 dirb
drwx------    2 aggie  user           18 May 26 11:20 dirc
drwx------    2 aggie  user           18 May 26 11:26 dird
-rw-------    1 aggie  user        43008 May 26 10:56 filea
-rw-------    1 aggie  user        57344 May 26 10:56 fileb
-rw-------    1 aggie  user       353280 May 26 10:56 filec

Sizes are given in bytes. The -a option lists files preceded by a dot. Consult the ls man page for more information about the ls command..

Display (recursively) disk usage: du

% du -H
32k     ./.ssh
48k     ./dira
416k    ./dirb
23M     ./dirc
45M     ./dird
69M     .
69M     total

This indicates the amount of disk space used for each directory including the current working directory. Consult the du man page for more information about the du command.

Locate files in a directory tree: find

Files can be searched according to a variety of conditions. Options that are most commonly used are -name and -size. The full list is given in the find man page.

% find . -size +5000k

In this example, we are searching for files that are greater than 5 megabytes. Consult the find man page for more information about the find command.

View Protection & File Security: ls -al

File security in unix involves three different catagories of users. Files and directories may be owned by user, group and other. Each of class of user can possess read, write, and execute permissions. Permissions for a particular file are listed with the 'ls -al' command.

% ls -al
drwx------    7 aggie  user        16384 May 26 13:34 .
drwxr-xr-x   29 root   root        16384 May 21 12:22 ..
drwx------    2 aggie  user           29 May 26 12:21 dira
-rw-rw----    1 aggie  user        43008 May 26 10:56 filea
-rw----rw-    1 aggie  user        57344 May 26 10:56 fileb
-rw-------    1 aggie  user       353280 May 26 10:56 filec
-rwx------    1 aggie  user        23552 May 26 13:34 script
-rw-------    1 aggie  user         3072 May 26 14:24 scriptb

The left column indicates what class of user and what kind of permission belongs to a file. The first field indicates whether or not it is a directory. The next three indicate permissions for the user class. Three more indicate permissions for the group class and the last three highlight permissions for all others.

As an illustrated example, we see that 'dira' is a directory that has read, write, and execute permissions available to user 'aggie'. Execute permissions for directories permit traversal for that class of user.

'filea' has read and write permissions for the user 'aggie' and the group 'user'.

'fileb' has read and write permissions for the user 'aggie' and all other users not in the group user. This is undesirable if 'fileb' is intended to be a private file.

Notice that the current directory, as indicated by the single dot, has full permissions for aggie and no other. If this is aggie's home directory then no other user would be allowed to view the directory contents. Under normal circumstances, modifications to the default home directory permissions are not needed.

Change permissions to files and directories: chmod

The chmod command adds or removes specified permissions for any of the three classes of users.

% ls -l fileb
-rw----rw-    1 aggie  user        57344 May 26 10:56 fileb
% chmod o-rw fileb
% ls -l fileb
-rw-------    1 aggie  user        57344 May 26 10:56 fileb

Here we have revoked read and write permissions for all other users not in the group 'user'.

% ls -l scriptb
-rw-------    1 aggie  user         3072 May 26 14:24 scriptb
% chmod u+x scriptb
% ls -l scriptb
-rwx------    1 aggie  user         3072 May 26 14:24 scriptb

'aggie' has now applied user execute permissions to the file 'scriptb'. Consult the chmod man page for more information about the chmod command.

Additional Information

All of the commands outlined above are described in detail in on-line manual pages. To call up the manual page for the 'find' command simply type at the command prompt:

% man find

An introduction to UNIX commands can be found here.