Athena Cluster

From IPRE Wiki
Revision as of 15:19, 25 October 2013 by Doug Blank (Talk | contribs)

Jump to: navigation, search

Hello MPI in Python

This tutorial will demonstrate how to use Bryn Mawr College's Athena Cluster with Python and MPI, the Message Passing Interface.

Here are the steps, briefly, for running a MPI Python Script on Athena

  1. log into
  2. find out which /data path you have a shared directory (either /data1, or /data2)
  3. create a python program in your /dataN/userid/ directory
  4. create a PBS script
  5. submit the PBS script
  6. check on status
  7. see the results

We will now go through each step in detail.

Step 1: Log into Athena

If you do not have an Athena account, please email ... to request one. It can take up to one business day.

After you have your account, you can log into the account from anywhere. You can use ssh on Linux and Mac, or putty on Windows.

You can use the X-forwarding flag, -X, if you have X running:

ssh -X

Using X-forwarding will allow you to use graphical programs, such as gedit.

Step 2: Where is my /data?

Next, you need to find out where your /data directory (either /data1 or /data2) has been located. The easiest way is to list the files in the directories:

[dblank@powerwulf ~]$ ls /data1
dblank     gdavis  pbrodfue
dmtpc      josh
emaccorma  mrice  xwang
[dblank@powerwulf ~]$ ls /data2
dblank     gdavis  pbrodfue
emaccorma  josh
galaxy     mrice  xwang
[dblank@powerwulf ~]$ 

Step 3: Create a Python script

Next, you will want to create a Python script in your /data directory.

First, change to your directory:

cd /data1/dblank

Next, use a text editor (emacs, vi, gedit, or nano) to create a file in your /data directory. We use the /data directory because it is available to all of the nodes.

Here is a sample program:

# More tutorials available at:

from mpi4py import MPI
import platform

rank = comm.Get_rank()
size = comm.Get_size()
machine = platform.node()

print("Hello MPI from %s %d of %d" % (machine, rank, size))

Save the script to a file, such as: /data1/dblank/

Step 4: Submit PBS script

To submit a Python script, you will need a shell program to give to qsub. Here is a sample:


    #PBS -q default
    #PBS -l nodes=6:ppn=12

mpiexec -machinefile /opt/machinelist -n 10 python /data1/dblank/

echo "end"

Save this in a file, such as /home/dblank/pbs/

PBS is a language that allows you to set many aspects of your running program. For more details see ...

In this example, we tell mpiexec to choose from the machines listed in /opt/machinelist (all of the nodes) and to create 10 instances of our running code across those machines.

NOTE: the program will start in your home directory, so by default you will be in /home/dblank/. Use full pathnames to /data1/ for shared data.

Finally you will submit this script to qsum, like so:


NOTE: the results of qsub will appear here. Therefore, you may want to move your shell script to a folder before beginning.

Step 5: Check status

When you start your program a number of things will happen:

  1. your program will be scheduled to run
  2. when qsub runs your script, the script be submitted to each of the machines, perhaps a number of times

You can run "qstat -a" to see the status of your program.

Step 6: Get Results

Finally, any output from your program will be saved in a file with the same name as your shell script, with a dot, the letter "o", and job number appended.