Basic UNIX Concepts and Commands

This chapter is intended to be a quick primer on unix for those who have little or no previous experience with it. We will guide the new user through some very basic and essential concepts of the unix operating system. Simple examples will be used to illustrate these concepts and the user will learn a small set of essential unix commands in the process.

We will introduce the following concepts (not necessarily in this order):

1. The unix shell
2. The hierarchical unix filesystem
3. Processes in unix
4. I/O redirection
5. Environment variables and the login control files

Unix is a multi-user, multi-tasking operating system with roots going back to the 1960's. It was the first complete programming environment designed by programmers to make software development easier. Although it has a reputation for being user-unfriendly and archaic, people in research and academic circles have continued to thrive in its elegance and simplicity. A brief tour of unix should be sufficient to allay the apprehensions of the novice user.

Logging in to a UNIX System

Let's start at the very beginning, by connecting and gaining access to a unix system (also known as "logging in"). You can connect to one of our machines over the network by using one of several methods. You could use a telnet or ssh client, or an X-windows client, for instance. For more information on these different methods, see How to Access Our Machines Remotely. Regardless of which method you use, at some point during the process you will be presented with a screen similar in essence to the following:

AIX Version 5
(C) Copyrights by IBM and by others 1982, 2000.
login: faisal
faisal's Password: 

Access to a unix system requires a login name and a password. UNIX login names (sometimes called "usernames") and passwords are case sensitive. When attempting to log in to a system, it will first prompt you for your login name and then your password; if both are correct, the system will grant access.

   _______________________________________________________________________
  |       Welcome to agave, a 32-cpu IBM p690 Running AIX 5.1             |
  |   A Component  of the Texas A&M University Supercomputing Facility    |
  | Website: http://sc.tamu.edu  Consulting: help@sc.tamu.edu or 845-0219 |
  |_______________________________________________________________________|

   Jun 12 ------ Abaqus 6.3

   Abaqus 6.3 is now installed on k2. Please consult the man page (man
   abaqus) to find out how to use the new version. Note that Abaqus 6.2
   remains the default version.

   Jun 27 ------ PBS queue reconfiguration

   We have eliminated the PBS extra small and extra large memory queues
   (*_xsm and *_xlm). The large memory queues` memory limit has been
   set to 8GB.

   Jul 25 ------ Scheduled downtime

   Agave will be brought down for hardware repairs on the morning of
   July 28 (Monday).  The machine will be unavailable for use most of the
   day starting at 10 am.   The longer cpu-time LSF queues are being
   drained starting today.

               Use the "msg" command to review the notices above
................................................................................

Last login: Wed Jul 23 16:17:23 CDT 2003 on ssh from jerusalem.tamu.edu

>

Once you successfully log on, the system will print some information to the screen. This information includes important messages from the TAMU-SC staff regarding issues such as software upgrades, system problems, and scheduled downtimes; it is referred to as the "motd" or the "message of the day" and it may change from day to day or whenever there is a need for the staff to notify users about anything. The system also prints the date and time of your last login.

After all of this, the system will present you with a "prompt". By default, the prompt will be a symbol such as '>', or '%', or '$'. The appearance of the prompt can be customized by the user, however, as we will show later. The the prompt indicates that the system is now ready to accept commands from the user. Every time a command completes execution, the system returns with a new prompt, awaiting further commands.

The program that writes the prompt on the screen and waits for user input is called the unix shell. The shell acts mostly as a medium through which other programs are invoked. While it has a set of built-in functions which it performs directly, most commands cause it to execute programs that are, in fact, external to the shell. So the shell is essentially an intermediary between the user and the unix operating system and it provides certain services to make the life of the user easier and more productive. Different types of shells are available to the user. While for the most part the specific shell used will not make a big difference to most users, certain shells are recommended for certain types of work. We'll discuss this subject in greater detail later.

Changing Passwords

If you log in to any unix system for the very first time, chances are the system will force you to change your password for security purposes; so let's talk briefly about passwords.

It is a good practice to change your login password from time to time, even if the system does not enforce regular changes as a matter of policy. Your password can be changed manually by issuing the "passwd" command at the prompt once you are logged in to a system. This command will ask you for your old password, and then will ask you to enter your proposed new password twice (for verification). Note that some systems are configured to require that users change their passwords after a certain period of time has elapsed since their last password was established.

Passwords should be easy enough for you to remember that you do not feel the need to write them down on paper (that defeats the purpose of passwords). Yet, they should not be simple dictionary words that can be guessed by programs designed to crack passwords. Remember, passwords are case sensitive, so "PaSSworD" and "pAssWORd" are considered two different animals! Good passwords will contain a mixture of upper and lower case letters, as well as digits and special characters ( _ , $ @ ! # etc).

Never share passwords with anyone and never send them in email messages or give them to anyone claiming to be a system administrator!

Anatomy of a UNIX Command

Going back to our session on the unix machine which we just logged into, we notice that the shell is presenting us with a prompt and therefore awaiting our command.

UNIX commands are executed by typing them at the prompt and pressing enter. The figure below illustrates the anatomy of a simple unix command.

A command is a sequence of "words" separated by blanks. The first word is the name of the command to be executed. The remaining words are passed as arguments to the command that is invoked. Unix commands can be thought of as being somewhat similar to function calls in programming languages, except that their arguments are not enclosed in parentheses nor separated by commas. In the example above we specify the ls or list command, which produces a listing of files in the directory called "/scratch". An important caveat to bear in mind when using unix is that it is case sensitive, meaning that the ls command cannot be typed in as LS.

While many arguments to commands specify file names or user names, some arguments rather specify an optional capability of the command which the user wishes to invoke. By convention, such arguments begin with the character '-' (hyphen) and are referred to as flag arguments, or simply "flags". In our example with the ls command, the flag speficied by the letter "a" forces the command to print even the names of files that are normally hidden in regular ls output. The flag specified by the letter "l" forces a longer (more detailed) version of the output. Flags are usually optional but can occasionally be required as arguments. If optional, their absence simply results in certain pre-defined default behaviors by the command.

While many unix commands and utilities perform only one or two functions rather than having a large number of hard-to-remember options, there are many others that do offer many flag arguments. It is hard to remember options of commands which are not used very frequently, so users need to refer to the relevant man pages whenever detailed documentation needs to be referenced.

The UNIX Filesystem

Let's see the output of an ls command on our IBM machine:

> ls -al
total 320
drwxr-xr-x   4 faisal   staff          1024 Aug 04 16:08 .
drwxr-xr-x 149 bin      bin            2560 Aug 02 13:27 ..
-rw-r--r--   1 faisal   staff            18 Nov 06 2002  .login
drwx------   2 faisal   staff           512 May 07 10:58 .lsbatch
-rw-r--r--   1 faisal   staff            74 Sep 26 2002  .ncbirc
-rwxr-----   1 faisal   staff           315 Jul 21 09:46 .profile
drwxr-xr-x   2 faisal   staff           512 Jun 10 10:48 .qt
-rw-------   1 faisal   staff           828 Nov 06 2002  .sh_history
-rw-r--r--   1 faisal   staff           463 Nov 06 2002  .tcshrc
-rw-r--r--   1 faisal   staff          5040 Jun 17 16:38 WebSM.pref
-rw-r--r--   1 faisal   staff          5719 May 07 10:58 errfile
drwxr-xr-x   2 faisal   staff           512 Aug 05 15:42 mysubdirectory
-rw-r--r--   1 faisal   staff          3172 May 07 10:58 outfile.2835
-rwxr--r--   1 faisal   staff         37799 Mar 27 10:22 regatta_intro.txt
-rw-r--r--   1 faisal   staff           184 May 07 09:29 sample.job
-rw-r--r--   1 faisal   staff           789 Jan 14 2003  smit.log
-rw-r--r--   1 faisal   staff             0 Jan 14 2003  smit.script
-rw-r--r--   1 faisal   staff         20404 Sep 26 2002  test.out
-rw-r--r--   1 faisal   staff           574 Sep 26 2002  test.txt
-rw-r--r--   1 faisal   staff         26965 Jun 17 16:15 websm.script
-rw-r--r--   1 faisal   staff            84 Jun 17 14:28 wsmmonitoring.data

Here, we did not explicitly specify the name of the directory whose contents we wanted to view. By default, ls printed the contents of the current working directory. The first column of the output shows the file permissions for each file or directory listed. We'll discuss the meaning of this portion of the output in more detail later. The second column displays the number of "links" to the file. The UNIX OS allows the creation of alternate filenames that are in reality references or pointers to the same file. These alternate names are called links, through which the original file can be accessed or modified. When a link is deleted, the contents of the referenced file remain intact and only the link to the file is removed and the link count is decremented by one. By default, every file has a link count of 1 unless the file is being accessed under alternate names as well. The third column of output above lists the username of the user who owns the file, and the fourth column lists the user group to which this file belongs. We'll talk about the concepts of user and group ownership later. The fifth column lists the size of the file in bytes, and this is followed by the date and time of the last modification made to the file. For older files, only the date is listed while for recent files the time of day is also displayed. Finally, the name of the file is in the last column of output.

On UNIX, all files are alike and the system does not impose any constraints on the way a file can be used. Each file is simply a sequence of bytes, whether it contains text, program source code, or executable object code. This is different from some operating systems that force the user to specify the kind of file the user intends to work with before providing access to the file, and files of a certain type can only be accessed in pre-defined ways. Such an operating system may store random-access files differently than it stores sequential files or database files.

Another thing to note is that as an operating system, UNIX does not impose any file naming conventions. A file with the suffix .txt need not contain ASCII text, it could be a binary executable file; UNIX does not care. Users and application developers, however, do voluntarily adhere to sensible naming conventions to maintain sanity, even though UNIX itself does not mandate any such measures.

Let us go back to our unix prompt and once again enter the ls command but this time without any options:

> ls
WebSM.pref          outfile.2835        smit.log            test.txt
errfile             regatta_intro.txt   smit.script         websm.script
mysubdirectory      sample.job          test.out            wsmmonitoring.data
>

Now, the output of the command is missing much of the detail of the previous listing because the default behavior of the ls command simply prints the names of the files and directories in the current working directory.

Let's now move about in the file system and explore what's out there. Using the cd command (the change directory command) we'll descend down into the subdirectory called mysubdirectory:

> cd mysubdirectory
> ls
file1  file2  file3  file4  file5
> pwd
/home/faisal/mysubdirectory
>

Having changed our current working directory to mysubdirectory, we now have a different vantage point of the filesystem. So now when we issue the ls command, we see a different listing than one we had seen previously. This new listing reflects the contents of our current working directory, not the directory we were in before. To determine the current working directory at any time, you can simply issue the pwd (the print working directory command) as shown above.

UNIX has what is called a hierarchical file system. All information on disk is organized into files and files can be grouped together into directories. Directories are like folders on a windows operating system. The top level directory is called the root directory and is represented by the slash (/) character. Within the root directory, there are many other directories, some of which contain executable programs and utilities, some others contain configuration files and settings, and still others contain user data. Since directories can be created within other directories, we can have multiple levels of directories. An illustration of such a hierarchy of files and directories looks like an upside down tree structure.

When a user logs into a UNIX system, she is placed in her home directory. When the user wants to create a new grouping of files that belong together, she simply issues the command to create a new directory and moves those files into it. The new directory looks almost like any other file except that the user can use the files placed in that directory any time. Or the user can "move down" to the new directory: move, because the new directory becomes the new vantage point for examining files, and down, because the new directory can be thought of as being below the home directory. This process can be repeated any reasonable number of times by creating new subdirectories one level below the home directory or at deeper and deeper levels. Each directory is logically distinct from all others, so two files with identical names can reside in separate directories without causing confusion.

Each file in a UNIX filesystem has a unique location that can be specified unambiguously by its pathname. If a file called unix resides in the top-level (root) directory, then its pathname is /unix. The slash (/) character in front of the filename signifies that it will be found under the root directory. Each level of descent into the filesystem is denoted by an additional slash, so the home directory of the user faisal would be /home/faisal, two levels down from the root, and one of his files would be called /home/faisal/.login. Putting all the slashes in the filename identifies the unique path followed down from the root and thus uniquely identifies the file itself. This is known as specifying the full path name.

Another type of path is known as the relative path. For instance, if the user's current working directory is /home/faisal and he wants to print the contents of the file named "file3" one level below in the subdirectory called mysubdirectory, he can do so by issuing the following command from his current location:

> more mysubdirectory/file3

This is a sample file.  It consists of simple ASCII text typed
using the vi editor.  This message was created for use in
examples used to demonstrate unix commands and concepts.

>

Here, the name of the file to be viewed on screen (using the more command) was specified by a path name, but this was not a full path name. The full path name of this file would have been "/home/faisal/mysubdirectory/file3". In the example above, the path name used was relative to the user's current location in the filesystem, not relative to the root directory (/), as a full path name would be.

Filename Expansion

Most filenames consist of a number of alphanumeric characters and '.'s (periods). In fact, all printing characters except '/' (slash) may appear in filenames. It is inconvenient to have most non-alphabetic characters in filenames because many of these have special meaning to the shell. The character '.' (period) is not a shell-metacharacter and is often used to separate the extension of a file name from the base of the name. Thus

   prog.c prog.o prog.errs prog.output

If we wished to refer to all four of these files in a command, we could use the notation

   prog.*

   echo prog.*

   prog.c prog.errs prog.o prog.output

Other notations for filename expansion are also available. The character '?' matches any single character in a filename. Thus

   echo ? ?? ???

Another mechanism consists of a sequence of characters between '[' and ']'. This metasequence matches any single character from the enclosed set. Thus

   prog.[co]

   prog.c prog.o

   chap.[1-5]

   chap.1 chap.2 chap.3 chap.4 chap.5

   chap.[12345]

An important point to note is that files with the character '.' at the beginning are treated specially. Neither '*' or '?' or the '[' ']' mechanism will match it. This prevents accidental matching of the filenames '.' and '..' in the working directory which have special meaning to the system, as well as other files such as .cshrc which are not normally visible.

Another filename expansion mechanism gives access to the pathname of the home directory of any user. This notation consists of the character '~' (tilde) followed by a user's login name. For instance the word '~keith' would map to the pathname '/home/keith' if the home directory for 'keith' was '/home/keith'. The '~' on its own (without any user name) expands to the home directory pathname of the user executing the command.

Quotation

We have already seen a number of metacharacters used by the shell. These metacharacters pose a problem in that we cannot use them directly as parts of words. Thus the command

   echo *

The recommended mechanism for placing a character which is neither a number, digit, '/', '.' or '-' in an argument word to a command is to enclose it with single quotation characters ('), i.e.

   echo '*'

   echo \'

   '

   echo \''*'

'*

File Ownership and Permissions

Since UNIX is a multi-user operating system, it implements mechanisms to regulate access to files and directories to prevent unauthorized access. We would not want a situation where other users could simply move into our home directory and view our files or execute our programs. Let us take a few moments to understand unix filesystem security.

Let's go back and re-examine the detailed output of the ls command we discussed earlier:

> ls -al
total 320
drwxr-xr-x   4 faisal   staff          1024 Aug 04 16:08 .
drwxr-xr-x 149 bin      bin            2560 Aug 02 13:27 ..
-rw-r--r--   1 faisal   staff            18 Nov 06 2002  .login
drwx------   2 faisal   staff           512 May 07 10:58 .lsbatch
-rw-r--r--   1 faisal   staff            74 Sep 26 2002  .ncbirc
-rwxr-----   1 faisal   staff           315 Jul 21 09:46 .profile
drwxr-xr-x   2 faisal   staff           512 Jun 10 10:48 .qt
-rw-------   1 faisal   staff           828 Nov 06 2002  .sh_history
-rw-r--r--   1 faisal   staff           463 Nov 06 2002  .tcshrc
-rw-r--r--   1 faisal   staff          5040 Jun 17 16:38 WebSM.pref
-rw-r--r--   1 faisal   staff          5719 May 07 10:58 errfile
drwxr-xr-x   2 faisal   staff           512 Aug 05 15:42 mysubdirectory
-rw-r--r--   1 faisal   staff          3172 May 07 10:58 outfile.2835
-rwxr--r--   1 faisal   staff         37799 Mar 27 10:22 regatta_intro.txt
-rw-r--r--   1 faisal   staff           184 May 07 09:29 sample.job
-rw-r--r--   1 faisal   staff           789 Jan 14 2003  smit.log
-rw-r--r--   1 faisal   staff             0 Jan 14 2003  smit.script
-rw-r--r--   1 faisal   staff         20404 Sep 26 2002  test.out
-rw-r--r--   1 faisal   staff           574 Sep 26 2002  test.txt
-rw-r--r--   1 faisal   staff         26965 Jun 17 16:15 websm.script
-rw-r--r--   1 faisal   staff            84 Jun 17 14:28 wsmmonitoring.data

The first column in the output above displays file permissions. Each file (or directory) has an associated set of protection bits (also known as mode bits) which the owner of the file can control individually. If a particular bit is enabled, its value is visible in the output, otherwise only a hyphen (-) is displayed for that bit. For each file, the first bit in this ten bit field is called the directory bit, the next three are "user" bits, the following three are "group" bits, and the final three are "other" bits. An enabled directory bit means that the file in question is a directory: the bit will have the value of "d" (a valuse of "l" in this bit is also used to mark files that are links to other files). The following three sets of protection bits determine how the user (the owner of the file), his or her working group (a collection of other people wishing to share file access for a project), and all other system users can access the file. These three sets can be thought to represent three different classes of users whose access permissions can be defined separately.

By default, a user is the owner of his or her own home diectory as well as of every file and subdirectory that he or she creates. On our machines, every user automatically belongs to a group called "user". The staff members of our facility belong to a group called "staff". Our system administrators can define any number of groups and give these groups new names. For instance, if Dr. Jones is collaborating with a number of users on one of the machines and would like to share files with his co-researchers over the lifetime of his gene sequencing project, we could define a new group called "genes" and make these users part of this group. The group members could then set the group permissions on their files to give other members the ability to read, modify, or execute their files.

When the read or write bit is enabled for one of the three classes of users mentioned above, a user belonging to that class is permitted read or write access. When the read bit, but not the write bit, is set, a user cannot add to, change, or destroy the file. If the write bit, but not the read bit, is set, a user has write only permission. When the execute bit is set, it means that the file may be executed as a program because it is either object code or a shell program.

The significance of the permission bits changes slighty when applied to directories. With directories, when the write bit is enabled, it means that a user may create or delete files within that directory. When the execute bit is turned on, the user may search through the listing of files in that directory and read, write, or execute them if allowed to do so by the permission bits on the files themselves. If only the read bit is enabled on a directory, a user may only produce a simple listing of the files contained therein.

Keep in mind that the system administrator (otherwise known as the superuser) can bypass all file protections and has access to everything.

There are two different methods for setting file permissions. They can be set "relative" to their current settings or directly, by using a numeric code. Let's examine how to assign file permissions using a numeric code (the user may refer to the man page for chmod to learn about the other method). The ten bit field that shows the set of permissions for each file can be represented with an equivalent numeric code. The UNIX command chmod is used to change file permissions. chmod stands for "change mode". The syntax for using the chmod command is:

chmod  nnn  file_name

So, to assign file permissions that would allow the owner to read, write, and execute, the group to read and execute, and the other users to only execute a file, we could use the following command:

 > chmod  751  file_name
 

Input and Output Re-direction

Most programs need input to complete their tasks. This input could be in the form of mouse clicks on a graphical user interface that tell a program how to behave, or it could be what a user types in at the system prompt in response to questions posed by the program, and it could also be information contained in a data file. Similarly, the output of a program can be sent to the screen, or to another data file, or a number of other places.

On unix, commands that normally read input or write output on the terminal can also be executed with this input and/or output done to a file. This is called input (or output) re-direction. Let's look at how unix handles I/O re-direction.

Using the date command, we can re-direct its output to a file instead of to screen:

 > date > now
 

The "greater than" (>) character sends the output of the date command to a file called "now" instead of printing it to the screen. The leading ">" character is the unix prompt as we mentioned earlier, so it is not to be confused with the output re-direction character. In fact, let's change the way our prompt looks and come up with a less confusing prompt.

> PS1="agave% "
agave%

agave% cat now
Tue Aug 19 16:03:48 CDT 2003
agave%
 

One other thing to note here is that the file 'now' need not have existed before the date command was executed; the shell would have created the file if it did not exist. If it had existed previously its previous contents would have been discarded! A shell option called noclobber can prevent this type of over-writing from happening accidentally (the user may issue the command set noclobber to enable this safeguard).

It is also possible to redirect the standard input of a command from a file. For instance, we could say

agave% sort < data

A most useful capability is the ability to combine the standard output of one command with the standard input of another, i.e. to run the commands in a sequence known as a pipeline. For instance the command

agave% ls -s

agave% ls -s | sort -n

agave% ls -s | sort -n -r | head -5

The notation introduced above is called the pipe mechanism. Commands separated by '|' characters are connected together by the shell and the standard output of each is run into the standard input of the next. The leftmost command in a pipeline will normally take its standard input from the terminal and the rightmost will place its standard output on the terminal.

Combining Standard Output and Error

By default, any process (which can be thought of as a program in execution) has three 'channels' of communication with the outside world. Through one of these channels, it receives information; we have been referring to this channel as the "standard input". Another channel is used to send out information; we have called this the "standard output". There is a third channel which is also used to send out information from the process, but by convention it is used to send out error messages or diagnostic messages as distinguished from ordinary output; this channel is called the "standard error". The technical unix term for these channels is "streams". Each of these I/O streams is represented internally (within the operating system) by a "file descriptor" which is referenced by an integer: standard input (often written as "stdin") is assigned descriptor 0, standard output (stdout) is descriptor 1, and standard error (stderr) is descriptor 2. Initially, all three of these descriptors are connected to the terminal by default. When a file descriptor is assigned to something other than the terminal, redirection is said to have occurred.

Until now, we have not talked about the standard error stream. From the user's perspective, it is often useful to separate output that consists of error, warning, or diagnostic messages from that which constitutes the normal output of a program. This kind of separation can allow a user to redirect the regular output of a command to a file (or to another command) and yet still see error or warning messages appear on the terminal while the command is in operation. Alternatively, it can allow the user to see standard output on the terminal but redirect the diagnostic or warning messages to a file so they do not clutter and confuse the regular output on the screen.

Occasionally, there are also circumstances in which it is more convenient to combine the standard output and standard error streams into one. Such a capability would allow a user to send both relgular output as well as error messages to one and the same file simeltaneously.

Let's examine how we can manipulate the standard output and error streams in the following examples:

agave% cd /tmp
agave% find . -name faisal -print
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
./faisal
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
find: 0652-081 cannot change directory to :
  : The file access permissions do not allow the specified action.
agave%

Here we invoked the find command to search the /tmp filesystem for a file called "faisal". The command returned numerous warnings alerting us to the fact that the file permissions on many subdirectories under /tmp did not allow us to conduct such a search within those specific directories. The command also returned the path to the file named faisal, which in fact was found immediately under the /tmp directory. The output we see on the terminal looks very cluttered and it is not easily apparent that the find command did in fact locate the file we were looking for (in addition to warning us about many other things). It would be much better to redirect the standard error of this command to a file so we can focus on the useful ouput on the screen:

agave% find . -name faisal -print 2> find-errors.txt
./faisal
agave%

In this example, all the warning messages of the previous run have been saved in the file we called "find-errors.txt". The symbol 2> redirected the I/O stream referenced by descriptor number 2 into a file. Descriptor 2 denotes the standard error stream as we have mentioned earlier.

Had we used the > symbol without the digit 2, we would have redirected the standard output of the command to the file, while all the warning messages would have been sent to the terminal.

If we wanted to save both the standard output and the standard error in a single file for later examination, we could do the following:

agave% find . -name faisal -print > find-errors.txt 2>&1

The 2>&1 symbols instruct the shell to send the output of descriptor 2 to the same place where the output of descriptor 1 is being sent (so standard error is in effect routed to the same destination as standard output). In this example, we have managed to merge both streams into one.

The examples above show how to achieve re-direction under the bourne shell family. Some diffenences in syntax will apply when attempting the same things using one of the shells in the C shell family. See the section More on I/O Re-Direction for more information. The concept of shells is discussed next.

The Shell and its Environment

We have mentioned that the shell is a special program used as an interface between the user and the heart of the unix operating system, a program called the kernel. The kernel is loaded into memory at boot-up and manages the system until shutdown. It creates and controls processes, manages memory, file systems, communications, etc. All other programs, including the shell program, reside out on the disk. The kernel loads those programs into memory, executes them, and cleans up the system when they terminate. The shell is a utility program that starts up when you log in and it interprets commands that are typed either at the command line (the prompt on the screen) or in a script file.

New users typically spend most of their time using the shell interactively. If a user types the same set of commands on a regular basis though, he would find it useful to automate those tasks by putting the commands in a script file and allowing the shell to execute that script. A script is analagous to a DOS batch file. More sophisticated scripts contain programming constructs for making decisions, looping, file testing, etc. When executing commands from within a script, the user has in fact begun to use the shell as a programming language.

The three popular and supported shells on most unix systems are the Bourne shell (written by S. Bourne at AT&T), the C shell (written by William Joy at Berkeley), and the Korn shell (written by David Korn at AT&T). The Bourne shell, otherwise known simply as sh, is the standard unix shell. Most system scripts are written using the bourne shell and system administrators commonly use it as their default administrator shell. The Bourne shell is known for being concise in its syntax, compact in size, and fast in execution. The C shell (csh) added a number of user friendly features to the bare-bones approach of the bourne shell. It provides features such as command line history, aliasing, built-in arithmetic, and filename completion. These features are all designed to make interactive use of the shell less painful, making C shell and its derivatives (like tcsh) popular for interactive use. For scripting, though, the Bourne shell is still preferred since Bourne scripts are simpler and faster than the same scripts written in the C shell. (The C shell family is not recommended for scripts due to inconsistent syntax at different levels of nesting.) The Korn shell (ksh) is a superset of the Bourne shell. The Korn shell enchances the features of sh by providing an editable history, aliases, functions, regular expression wildcards, built-in arithmetic, and other features. The Bourne shell is almost completely upward compatible with the Korn shell, so older Bourne shell programs should run fine in the Korn shell.

To the shell, commands are generally one of four types: aliases, functions, built-in, or external programs. Aliases are user-defined abbreviations or nicknames for existing commands and apply only to the C shell, tcsh, and Korn shell. Functions apply only to the Bourne and Korn shells; they are user-defined groups of commands organized as separate routines. Aliases and functions are defined within the shell's memory. Built-in commands are internal routines within the shell program itself (they do not require separate processes to be executed), and executable programs reside on disk.

While built-in commands and external executables are invoked directly by the user, aliases and functions need to be defined before they can be used. Read the section The Alias Command for more information on how to define aliases (a brief note about functions is also included).

Apart from interpreting commands and executing them, a shell also serves to customize a user's working environment. This is normally done by setting what are called variables. The statements which set these variables can be saved in files that are read by the shell. Such files are called shell initialization files and they typically contain definitions for setting terminal characteristics and window charactertistics, variables that define the search path, the appearance of the prompts, the terminal type, variables required by the shell to locate specific applications and programming libraries within the filesystem, as well as other things.

Variables are of 2 types with regards to their scope: shell and environment. Shell variables are known only to the shell in which they are defined while environment variables are passed down and inherited by processes created by the shell. This distinction is analagous to the one between local and global variables in common programming languages. Some shell variables are special in that they are created automatically when the shell starts up. Apart from such shell-defined variables, there are also user-defined shell variables. A user may create a variable of any name and assign it any value. Users may also define the scope of the variables they create (in other words, they can create both shell as well as environment variables).

For more information on how to set and read variables, read the sections entitled:

• Shell Variables and Environment Variables
• Some Important Variables

Processes

While a program can be defined as an executable file, a process is an instance of a program that is being executed by the operating system. Some operating systems use the term "task" instead of process. Operating systems that are capable of executing more than one task (process) at a time are called multi-tasking systems.

The unix kernel (which is what the operating system is called) provides many "access points" through which an active process can obtain services from the kernel. These are called system calls. The standard unix C library provides a C interface to each system call; as a result, the actual system calls appear as normal C functions to the programmer.

A process consists of several components: the executable program code itself (referred to as the "code" or sometimes the "text" portion of the process), the data on which the program will execute, the resources required for the execution (such as memory workspace and access to various files), and information about the state of the process. The data portion contains items such as program variables and their values. Among the resources required for execution is memory space, which can be divided into two types: heap and stack.

The heap is a portion of memory allocated dynamically (as needed, at runtime) for the use of the process. Whenever the malloc or calloc functions are used in C for instance, they reserve space in heap memory. The stack portion of memory, on the other hand, is used by the process to aid the invocation of functions. Every time a function is called, the process reserves a portion of stack memory to store the values of parameters passed to the functions as well as for results returned by the functions and the local variables used within the functions. The stack is also where space for all declared data types and structures is reserved at compile time.

Unix is a multi-tasking system because it allows the concurrent execution of multiple processes. In such a system, the CPU switches automatically from process to process running each for tens or hundreds of milliseconds, giving the impression that more than one process is running simeltaneously. Technically, though, the CPU can actually work on only one program at any given instance in time. An operating system like unix allows the successful sharing of the CPU by ensuring that each process gets a chance to run and that no process is able to modify the state of another process.

In order to manage and control processes, the operating system must know certain specific information about each process (stored in what is called the Process Control Block). This information defines the state of the process execution and allows the operating system to temporarily de-activate the process, giving some other process an opportunity to work on the CPU, and then at some point in the future to restore the state of the de-activated process, allowing it to resume where it stopped work previously. This swapping out of one process and replacing it with another is called "context-switching" since the context of one process is saved for later restoration and the context of another process is loaded and activated. In this manner, the operating system keeps cycling through its list of "concurrently" running processes, giving everyone a chance at the CPU.

All processes in unix exist in a hierarchy of parent-child relationships. Any process that creates or spawns another process becomes the parent of the created process. The created process itself is called the child of the creating process. A process can have multiple child processes, but a child process can have only one parent process.

Every process has a unique process ID (or PID). The PID is an integer that is assigned by the kernel when the process is created. The process with PID 0 is a special kernel process called the "swapper" (or sometimes called the "scheduler") which implements the concurrent execution of multiple processes on a single CPU as mentioned above. PID 1 is also a special process called "init" which initializes the system and makes it ready for use by users. init is considered the parent of all other processes since it creates them.

The fork and exec system calls

We mentioned earlier that the unix kernel provides the mechanism of "system calls" for processes to request and obtain services from the operating system. Two of the most fundamental services provided through system calls are 1) the creation of new processes and 2) the execution of processes. These two services are provided by the fork and exec system calls.

The fork system call creates a clone of the calling process and this clone becomes its child. The child process has its own unique PID. Whenever a process needs to execute another program, it must do so by first making a copy of itself through fork. After this, the fork'ed copy (the child process) issues an exec system call to execute the program. This is typical for programs such as shells, which invoke other programs at the behest of the user.

The exec system call causes the calling process to be replaced by another program. So exec allows unix to use the context of an existing process to load a new program into memory and to start executing it. Unlike fork, exec does not assign a new PID to the process (because no new process is actually created).

Let's look at (a simplified model of) what typically happens when a system comes up and a user logs in and runs various commands.

When the init process brings up the system and a user attempts to connect to it, init forks a copy of itself and that copy execs the getty program, which prints the login banner to the user's terminal and prompts the user for his or her username. When the user enters the username, the getty process then execs the login program, passing it the username as an argument. The login process prompts the user for the account password, which it then compares to the one stored in the /etc/passwd file. If these match, the login process execs the appropriate shell program (in our example /bin/sh). The shell now presents the user with a command prompt and waits. In our example the user first enters the ls command to view the contents of the current working directory. Then she uses the more command to view the contents of the file file.c. Next, she compiles this file (which apparently contains C source code) using the C compiler. Finally, she executes the resulting executable program called a.out. Every time the user executes a command, the shell forks a copy of itself and that forked child process then execs the command that was invoked. When the command completes execution, it issues the exit system call and returns control to the waiting parent process (the shell). The shell then prints another command prompt to the terminal, awaiting the next user command.

Signals

A signal is a notification to a process that some event has occurred. Signals are sometimes also referred to as "software interrupts." Signals usually occur asynchronously, meaning that a process doesn't know ahead of time exactly when a signal will occur. Signals can be sent 1) by one process to another process, or 2) by the kernel to a process.

Different types of signals are intended to notify processes of different types of events and occurences. Each type of signal is given a different name and all these names are listed in a system header file called signal.h, typically found in the /usr/include directory.

There are several conditions that generate signals; let's examine some of them:

1. The kill system call.
2. The kill command.
3. Certain terminal characters.
4. Certain hardware conditions.

1. The process can provide a special function that is called whenever a specific type of signal arrives.
2. The process can choose to ignore the signal.
3. The process can allow the system defined default behavior to take place