Job Script Hints

Last modified: Friday October 04, 2013 1:03 PM

Below are various hints to help you write a job script. For examples, see example job scripts. For more information about batch processing, see the Introduction to Batch Processing section of this user guide.

Record the job script for future reference

Sometimes, coming up with the proper job script can be a bit confusing. Often times it takes many iterations to get right. When one does get it right, it is often good to have a record of what was run so that: a) if you make changes and it quits working, you have a reference to what did work, b) you have a record to reproduce the run in the future as well as a record of what actually was run to collect the data you have.

To that end, it is often a good idea to include a copy of the job in the output. To do this, add this to your script (after all your #PBS variables):

echo "############################# JOB FILE ###############################"
cat $0
echo "######################################################################"
If you add that to your job file, then you will always have a record of what job was used to collect what output.

Change directory to where job was launched from

Often times, users will have the job scripts in proximity to the the data they want to use. By default, PBS/Torque launches your job in your home directory which often leads to confusion due to "file not found" errors when the file in question is quite clearly in the directory where the job was launched from. To avoid this problem it is recommended you add something like this early in your job script.
cd $PBS_O_WORKDIR # change directory to where job was launched from
echo "The current directory is `pwd`" # output the current directory so it is clear where it ran from

Set your default "billto" account

On the Introduction to Batch Processing Page under Submission options is a description of the "billto" option to specify the account to use. Many of our users have at least two accounts: 1) their personal, "Basic", allocation (usually 800 BUs) and 2) the allocation granted to them by their PI (often thousands of BUs). By default, the queue and billing system will charge the individual's account, which is often then quickly depleted. By using the "billto" option, one can choose to pick which account to use (usually the larger one). But, sometimes, that gets overlooked. So, it generally recommended that you set the default account with:
myams -p <account_number>
BEFORE running your job (NOT in the job script itself). Depending upon the number of accounts you have and what projects they are allocated to, this may or may not be useful and you might prefer to use the "billto" option. But for most of our users who are primarily working on a single research project, this helps to minimize problems due to forgetting to specify an account and having their job abruptly terminated due to depleting their "Basic" account (which often leads to a waste of time and effort).

For details, see our AMS Functionality page.

Create a .forward file for mail

As noted in the Submission options section of the Introduction to Batch Processing Page, users have the option of receiving mail when their job: a) is aborted, b) begins running and c) quits running (via the "-m abe" option). However, unless one includes the "-M" option to provide an external email address, the mail will be silently stored away on the system.

If you want to skip specifying the "-M" option each time and guarantee that any email from gets routed to an email address you use regularly, it is recommended you create a "~/.forward" file (where "~" expands to the name of your home directory). To create such a file, do this (in an interactive login session and NOT in your batch file).

cd # change to home directory
echo > .forward
cd # change to home directory
echo > .forward
That will guarantee that all email from eos, either from the batch system or other programs you use that send email will be routed to someplace where you can read it. And it will save you having to add the "-M" option to all of your batch files.

Be aware of the difference between Unix and DOS/Windows text files

A majority of our users are logging in from Microsoft Windows based systems. A great number of our users haven't yet effectively learned to use native Unix editors like vi and emacs and often resort to editting their files on their local (Windows) system and then using PuTTY's SCP or SFTP to transfer their files to our Unix systems. When using SFTP, there is an option to transfer in either text or binary mode. Provided the user remembers to switch to text mode before doing their transfer, the translation from DOS to Unix text file formats is done automatically. (The difference being that on DOS, the end of line is indicated by a carriage return and a line feed whereas on Unix the end of line is marked by only a line feed.) However, users don't always remember to switch to text mode and if they use SCP, then no such option to exists. If you do decide to edit on a Windows machine and then upload to a Unix machine, then you should double check your file on the Unix side to make sure it was translated correctly. To do that, simply run:
file <filename>
and if it indicates "ASCII text, with CRLF line terminators", then you likely have a DOS formatted file which can cause a number of problems. To remedy the problem you can either: a) transfer the file again using SFTP in text mode or b) (much easier) run:
dos2unix <filename>
to have the file converted to the proper file format (the file command will just say "ASCII text" at that point). Note that this applies to all text files that you may have edited on Windows and not just batch job scripts.

It is recommended all users learn to use a native Unix editor like vi or emacs to avoid problems when transferring files back and forth between Unix and Windows (Note: when transferring Unix files back to Windows you will likely want to run unix2dos on the file before doing the transfer). However, it is understood that many command line based editors can be difficult to learn.

Given that, we also have a GUI-based editor available on our systems by the name of gedit. To use this from Windows, you will need to enable X11 forwarding with Xming before logging in with PuTTY. For details, see the Remote Display of Programs with Graphical Interfaces section of our Remote Access Using Windows page. Follow the instructions there on using xterm to test your X11 connection. Once it is working, you can simply run:

gedit <filename>
to use a nice GUI-based editor to edit your files locally (without having to transfer back and forth between Windows).

Make sure non-Unix editors haven't inserted "strange" formatting characters

Again, this is sometimes a problem people have when editing their files on other systems. Depending upon your editor, it may, unbeknownst to you, insert special characters or use a special character set. The easiest way to check this is using the file command mentioned above. In most cases, you will want your text files to indicate "ASCII text". If you run the file command and get something like "ISO-8859 text" (or worse yet, "ISO-8859 text, with CRLF line terminators") then there is the distinct possiblity that you will have problems. In addition to the file command you might also find it useful to concatenate the file to the standard output (your terminal) with:
cat <filename>
If you see any funny looking characters or oddly constructed sequences, then you are likely looking at a non-ASCII file and might run into problems.

Be sure to load the desired modules

So you finally got your custom code built and running in an interactive session. However, whenenever you go to run it in the queue you run into "file not found" or "unresolved symbol" errors. Remember that you need to load whatever modules that you used when you built your project. By default, the batch environment is started with no modules loaded. So, if you do a module list in your batch file before loading any modules, you will see "No Modulefiles Currently Loaded." You need to explicitly module load <module_name> any modules you need. [Note: since there are no modules loaded by default there is no need to do a module purge before loading your modules.]

Don't specify a queue unless you want a specific one

Unless you have very specific needs or are involved in a research project supported by one of our reserved queues, you are usually better off not including a queue specification (the PBS "-q" option). By default, if no queue is selected, the system will try to find a queue that best matches the other options provided (i.e. memory, # of nodes, # CPUS/node). By specifying a particular queue, you confine the batch system to just that queue which often times will mean your job will have to wait for that queue when other queues are available. So, in general, unless you are using a reserved queue, it is usually best not to specify one at all.