NOTE:For information about Matlab on hydra visit the Matlab on hydra page

Interactive mode

The interactive limit (CPU time) per login session on all systems is 1 hour. This means your Matlab code can not run for more than 1 hour or it will get killed by the system automatically. Interactive mode is intended only for debugging your Matlab script, running testing small jobs, or using the matlab GUI

Batch jobs

When your sequential matlab needs more than 1 hour to execute (or requires more memory than allowed in an interactive session) you need to run your matlab job using the batch system. You can either write you own batch script and submit it manually to the batch scheduler or you can use the matlabsubmit script to run your matlab scripts automatically.

Using matlabsubmit

For your convenvenience we created a simple program named matlabsubmit to run matlab scripts (extension .m) without the need to create your own batch script. matlabsubmit can be used to run both sequential and parallel programs and will automatically set the required resources (e.g. nodes and ppn).

To submit your Matlab script just type:

   matlabsubmit myscript.m
This will execute matlab script myscript.m in batch mode. The matlab script myscript.m has to be in the current directory. After printing information about the job to run (including jobid) control is returned immediately.Screen output will be redirected to file

After your job finishes matlab will exit and all your results will be lost. If you want to save the results make sure you add the save command to your matlab code. You can use the load command later to load the results into matlab.

Options with matlabsubmit

Below is the output of running "matlabsubmit -h"
   Usage: matlabsubmit [options] SCRIPTNAME

   This script submits matlab script SCRIPTNAME to the job scheduler

   OPTIONS:
      -h Shows this message
      -m set the amount of requested memory (e.g. -m 20gb)
      -t sets the walltime; form hh:mm:ss (e.g. -t 03:27:00)
      -w sets the number of additional workers
      -n sets the number of nodes to assign the workers to
      -g indicates script needs GPU  (no value needed)
      -b sets the billing account to use
      -q sets the queue to be used 

   DEFAULT VALUES:
    nodes    : 1
    memory   : 12gb
    time     : 02:00:00
    workers  : 0
    gpu      : no gpu

For example the command matlabsubmit -t "03:27:00" -m 17gb myscript.m will request 17gb of memory and 3 hours and 27 minutes of computing time and executes Matlab script myscript.m.

matlabsubmit will scan your script for matlabpool open/close commands and adjust them to use the correct cluster profile and requested number of workers so there is no need to change your script manually.

Output files

matlabsubmit will create a directory named MatlabSubmitLOGN (where N is the jobid printed by the matlabsubmit command) where it stores output from the matlab window (matlab.log) and error messages from matlab and PBS (matlab.err). This directory will also contain the adjusted matlab script and the actual submission script.

NOTE: if your job needs more than 1 node, it will also create a number of other directories and files named JobNxxx (where N is matlab batch job id). Do not delete these directories when the computation is not finished yet. Matlab needs these directories and files for execution. After computation is finished you can safely remove them.

Checking run status of matlabsubmit

To check the status of the matlabsubmit command you can use the standard "qstat -u user" command (where user is your login id). The name of the job is the same as the name of the script you are executing (NOTE: when you need more than 1 node the name will be JobN).

Writing your own batch file

You can also write your own batch file and submit it manually to the batch system. Below is a simple example of a batch script
#PBS -l nodes=1:ppn=1,walltime=02:00:00,mem=12gb
#PBS -N mymatlab
#PBS -S /bin/bash
#PBS -j oe

# load the matlab module
module load matlab

#$PBS_O_WORKDIR is the directory from which the job was submitted
cd $PBS_O_WORKDIR

## run the matlab command
matlab -nosplash -nodesktop << EOF
a = zeros(10,1);
for i=1:10
  a(i) = a(i) + i;
end
a
exit
EOF

The code between the EOF markers is the actual sample matlab snippet. You will replace this with your own matlab code. Always add the exit command at the end of your matlab code, it tells matlab to exit cleanly.

Instead of writing the actual matlab code in your batch script you can also create a separate Matlab script file (needs to have extension .m) where you write your Matlab code. As mentioned before, make sure the last matlab command in your matlab script will be the exit command. Suppose you name your matlab script myscript.m you can replace the matlab command above with:

  
matlab -nosplash -nodesktop -r myscript

Screen output will be redirected to file as in any EOS batch run (in the sample batch sript it will be redirected to file mymatlab.oXXX, where XXX is the jobid).

Parallel Matlab

Matlab has excellent capabilities to run your matlab code in parallel; either using multiple cpus or using available gpus. Currently, the Supercomputing Facility has a license for 32 tokens (which means you can run your matlab code in parallel using at most 32 cpus). For parallel computing capabilities Matlab uses Cluster profiles. A cluster profile lets you define certain properties of your cluster (e.g. how to submit jobs, submission parameters, job requirements, etc). Matlab will use the cluster profile to offload parallel (or sequential) matlab code to one or more worker and run it there.

Cluster Profiles

NOTE: Instructions below are specific for Matlab version r2012b. Click here for instructions to import a parllel profile for version r2010b.

Importing profile

The Supercomputing facility already created a template Cluster Profile for your convenience. Before you can use this profile you need to import it first (you only need to do this once).

1. In the Matlab window, click 'Parallel/Manage Cluster Profiles/Import'
2. Choose '/g/software/matlab/CP2012b.settings' file to import
You can also just type on the matlab commandline:
cpname = parallel.importProfile('/g/software/matlab/CP2012b');
cpname is the name of the cluster profile

Setting default profile

To set the default profile do the following:
1. In the Matlab window, click 'Parallel/Set Default'
2. select CP2012b.
You can also type on the command line:
parallel.defaultClusterProfile(cpname)

Changing Profile properties

Currently the default job requirements specified for the template cluster profile (imported above) are:

'-l nodes=^N^,mem=12gb,walltime=4:00:00'

(the ^N^ will be replaced automatically by Matlab with the actual number of workers you specify).

If your job requirements are different (e.g. walltime, amount of memory) you need to modify them
1. In the Matlab window, click 'Parallel/Manage Profile/'
2. select the profile you want to adjust
3. click "Edit"
4. make the changes.
5. click "Done"
You can also change it on the command line but this requires a few more steps.

STEP 1: get a cluster object from the default cluster:
dcluster=parcluster
STEP 2: set the job requirements using the ResourceTemplate property:
dcluster.ResourceTemplate='-l nodes=^N^,walltime=02:00:00,mem=20gb'
In this case your matlab job would request 2 hours, 20GB.

Note: although not as flexible, you can set the number of nodes explicetely by replacing ^N^ with actual values (e.g. 2::ppn=8).

STEP 3: if needed, add aditional parameters (e.g. billing account) using the SubmitArguments property:
dcluster.SubmitArguments='-l billto=XXX'
STEP 4: Save the changes to the cluster:
dcluster.saveProfile
Now the default profile has been updated with the above changes.

In case you want to save it as a different profile:
dcluster.saveAsProfile('NEWNAME')
Where NEWNAME is the name you choose for the profile.

Parallel constructs

We will discuss briefly some of the most common parallel matlab concepts. For more detailed information about these constructs, as well as additional parallel constructs consult the Parallel Computing Toolbox User Guide

matlabpool

The matlabpool functions enables the full functionality of the parallel language features (parfor and spmd, will be discussed below). matlabpool creates a special job on a pool of workers, and connects the pool to the MATLAB client. For example:
matlabpool open 4
    :
    :
matlabpool close
This code starts a worker pool using the default cluster profile, with 4 additional workers.

NOTE: only instructions within parfor and spmd blocks are executed on the workers. All other instructions are executed on the client.

NOTE: all variables declared inside the matlabpool block will be destroyed once the block is finished.

For more detailed information please visit the Matlab matlabpool page.

parfor

The concept of a parfor-loop is similar to the standard Matlab for-loop. The difference is that parfor partitions the iterations among the available workers to run in parallel. For example:
matlabpool open  2
parfor i=1:1024
   A(i)=sin((i/1024)*2*pi);
end
matlabpool close
This code will open a matlab pool with 2 workers using the default cluster profile and execute the loop in parallel.

For more information please visit the Matlab parfor page.

spmd

spmd runs the same program on all workers concurrently. A typical use of spmd is when you need to run the same program on multiple sets of input. For example, Suppose you have 4 inputs named data1,data2,data3,data4 and you want run funcion myfun on all of them:
matlabpool open  4
spmd (4)
    data = load(['data' num2str(labindex)])
    myresult = myfun(data)
end
matlabpool close
NOTE: labindex is a Matlab variable and is set to the worker id, values range from 1 to number of workers.

Every worker will have its own version of variable myresult. To access these variables outside the spmd block you append {i} to the variable name, e.g. myresult{3} represents variable myresult from worker 3.

For more information please visit the Matlab spmd page.

batch

The parallel constructs we discussed so far are all interactive, meaning that the client starts the workers and then waits for completion of the job before accepting any other input. The batch command will submit a job and return control back to the client immediately. For example, suppose we want to run the parfor loop from above without waiting for the result. First create a matlab function myloop.m
parfor i=1:1024
  A(i)=sin((i/1024)*2*pi);
end
To run using the batch command:
myjob = batch('myloop','matlabpool',4)
This will start the parallel job on the workers and control is returned to the client immediately. To see all your running jobs click on Parallel/Monitor Jobs. Use the wait command, e.g. wait(myjob), to wait for the job to finish, use the load command, e.g. load(myjob), to load all variables from the job into the client workspace.

For more information please visit the Matlab batch page.

Using GPU

Normally all variables reside in the client workspace and matlab operations are executed on the client machine (e.g. your desktop, or an eos login node). However, Matlab also provides options to utilize available GPUs to run code faster.

Running code on the gpu is actually very straightforward. Matlab provides GPU versions for many build-in operations. These operations are executed on the GPU automatically when the variables involved reside on the GPU. The results of these operations will also reside on the GPU. To see what functions can be run on the GPU type:

methods('gpuArray')
This will show a list of all available functions that can be run on the GPU, as well as a list of available static functions to create data on the GPU directly (will be discussed later).

NOTE: There is significant overhead of executing code on the gpu because of memory transfers.

Another useful function is:
gpuDevice
This functions shows all the properties of the GPU. When this function is called from the client (or a node without a GPU) it will just print an error message.

Adjusting Cluster Profile

to use the gpus on EOS we need to adjust the job requirements to make sure the job is scheduled on a node with a gpu, the same way you would do it with a regular eos job.
dcluster = parcluster
dcluster.ResourceTemplate='-l nodes=1:ppn=1:gpus=1,walltime=02:00:00,mem=20gb'
The above job requirements are just an example. You can adjust the various properties to suit your needs.
More detailed information about changing Profile Properties can be found here

Copying between client and GPU

To copy variables from the client workspace to the GPU, you can use the gpuArray command. For example:
carr = ones(1000);
garr = gpuArray(carr);
will copy variable carr to the GPU wit name garr. If variable carr is not used in the client workspace you can write it as:
garr = gpuArray(ones(1000));
The two versions have the same problem. They both need to copy the 1000x1000 matrix from client workspace to the GPU. We mentioned above that Matlab provides methods to create data directly on the GPU to avoid the overhead of copying data to the GPU. For example:
garr=gpuArray.ones(1000)
This will create a 1000x1000 matrix directly on the GPU consisting of all ones.

You can find a list of all methods to create data directly on the GPU here.


To copy data back to the client workspace Matlab provides the gather operation.
carr2 = gather(garr)
This will copy the array garr on the GPU back to variable carr2 in the client workspace.

Overhead

As mentioned before there is considerable overhead involved when using the GPU. Actually, there are two types of overhead.

Warming up

When the GPU is just starting up computation, there are many things that need to be done, both on the Matlab part and the GPU device itself (e.g. loading libraries, initializing the GPU state, etc). For example:
matlabpool open 1
spmd 1
tic
gpuArray.ones(10,1);
toc
end
This code only creates a 10x1 array of ones on the GPU device. The first run takes an astounding 21.5 seconds to execute while every successive run only needs about 0.00017 seconds. This shows the huge cost of warming up the GPU.

NOTE:These are running times on EOS. Other systems might have very different timing results.

Data transfer

GPU operations in Matlab can only be done when the data is physically located on the GPU device. Therefore data might need to be transferred to the GPU device (and vice versa). This is a significant overhead. For example:
spmd 1
tic;ag=gpuArray(ones(10000));toc;
end
The above code only copies a 10000x10000 matrix from client workspace to GPU device. The time it takes is almost 0.6 seconds. This is a significant overhead.

Example

Here is a little example that performs a matrix multiplication on the client, a matrix multiplication on the GPU, and prints out elapsed times for both. The actual cpu-gpu matrix multiplication code can be written as:
a =  rand(1000);
tic; b = a*a; toc;
tic; ag = gpuArray(a); bg = ag*ag; toc;
c = gather(cg) 
Almost no additional steps are required to use the gpu. Actually, copying the results to the client workspace is not even needed. Variables that reside on the gpu can be printed or plotted just like variables in the client workspace.

The above code will run without problems if Matlab is installed on a computer with a gpu attached. Since EOS does not have gpus attached to the login nodes (where the client is running) we need to ensure the above code is run on a gpu node. We will show how to do it in interactive mode (using matlabpool), and by using the Matlab batch command.

For convenience the code above is saved as mymatrixmult.m

Interactive using matlabpool

A matlabpool needs to be opened since a gpu node is needed and the client is running on one of the login nodes (no gpu available) and mymatrixmult needs to be inside a spmd block to ensure code will actually run on the worker instead of the client (see matlabpool section). The code will be as follows:
matlabpool open 1
spmd 1
mymatrixmult
end
matlabpool close

Using Matlab batch command

This example is a basic sequential code (i.e. uses only one cpu core), so in this case a matlabpool is not even needed. The Matlab batch command will start the job on one of the workers (which has a gpu). The code will look as follows:
batch('mymatrixmult')

Warming up the GPU

there is considerable overhead involved when using the GPU. Besides the data transfer overhead mentioned before, there is another kind of overhead; warming up time. When the GPU is just starting up computation, there are many things that need to be done, both on the Matlab part and the GPU device itself (e.g. loading libraries, initializing the GPU state, etc). To get an indication how much time is needed look at the following example:
                                                                                     
matlabpool open 1
spmd 1
tic
gpuArray.ones(10,1);
toc
end
This code only creates a 10x1 array of ones on the GPU device. The first run takes 0.026 seconds to execute while every sucessive run only needs about 0.00017 seconds (of course different runs will produce slightly different results). This shows the huge cost of warming up the GPU .

NOTE:These are running times on EOS. Other systems might have very different timing results.