Version: Amber 12 (Aug 2012), Amber 11 (Aug 2010), Amber 10 (May 2008)

The following builds of Amber are available on Biowulf:

In your batch script, you need to set the appropriate path for the Amber version and nodes, as in the table above.

LEaP is a graphical builder of input files for AMBER modules. LEaP can be used via the Xwindows graphical interface xleap, or the terminal version tleap. To run xleap,

1. Open an Xwindows session to Biowulf. (More information about Xwindows on Macs, Windows, and Unix desktop machines.)
2. On Biowulf, type

   biowulf% export AMBERHOME=/usr/local/amber (bash)
   biowulf% setenv AMBERHOME /usr/local/amber (C shell)
   biowulf% /usr/local/amber/exe/xleap
   

   You should see the xleap window appear, in which you can type any LEaP commands.
3. The AMBER tutorials

For basic information about setting up an Amber job, see the Amber manual and the Amber tutorials . Also see Batch Queuing System in the Biowulf user guide.

The Amber executables can run in parallel on all Biowulf computational nodes. However, benchmark runs showed that sander does not scale beyond a single IB node. Amber 12 has the best performance/scaling on the ipath nodes. (see benchmarks). Therefore we recommend that users run on the ipath or gige nodes.

Sample script for a Sander run on ipath nodes:

#!/bin/bash
# This file is amber.run
#
#PBS -N sander
#PBS -m be
#PBS -k oe

export AMBERHOME=/usr/local/amber12

cd /data/user/amber/myproject
date
/usr/local/openmpi_intel_ipath/bin/mpirun -machinefile $PBS_NODEFILE -np $np \
   /usr/local/amber12/bin/sander.MPI -O -inf mdinfo -r md.rst -x md.crd \
   -e md.en -p md.top -o md.out -i md.in

qsub -v np=16 -l nodes=8:ipath jobscript

Sample script for a Sander or pmemd run on gige nodes

#!/bin/bash
# This file is amber.run
#
#PBS -N sander
#PBS -m be
#PBS -k oe

export AMBERHOME=/usr/local/amber12

cd /data/user/amber/myproject
date
/usr/local/openmpi_intel_eth/bin/mpirun -machinefile $PBS_NODEFILE -np $np \
   /usr/local/amber12/bin/sander.MPI -O -inf mdinfo -r md.rst -x md.crd \
   -e md.en -p md.top -o md.out -i md.in

For a pmemd run, the executable sander.MPI in the last line would be replaced by pmemd.MPI with appropriate parameter changes.

This script is submitted with a command like:

qsub -v np=16 -l nodes=1:e2666 /data/user/amber/amber.run

Replica Exchange jobs can reasonably be run on the gige nodes since the inter-node communication are less of an issue, and the jobs will scale well.

NOTE: this refers to Amber 10 only. See the Amber 12 documentation for replica exchange with Amber 12.

First read the tutorial on Replica Exchange simulations with Amber 10. Once the topology and coordinate files are generated, the input structure minimized, and the number of replicas and temperatures determined, the equilibration and the REMD simulations can be run as batch jobs. A detailed explanation of these steps including sample input files is in the tutorial. It is possible to run each replica on multiple processors as in the example below.

Set up a batch script along the following lines:

#!/bin/bash
#  this file is called amber_re.bat
#

export PATH=/usr/local/mpich/bin:$PATH
export AMBERHOME=/usr/local/amber11

cd /data/myname/amber/re
mpirun -machinefile $PBS_NODEFILE -np $np $AMBERHOME/exe/sander.MPI -ng 8 \
                                       -groupfile equilibrate.groupfile

Submit this job with

qsub -v np=8 -l nodes=4:o2800 amber_re.bat       (to use 8 processors on 4 o2800 nodes).

or

qsub -v np=16 -l nodes=8:o2800 amber_re.bat      (to use 16 processors on 8 o2800 nodes.)
                                               (Each replica will run on 2 processors)

or

qsub -v np=32 -l nodes=8:dc amber_re.bat       (to use 32 processors on 8 dual-core nodes)
                                               (Each replica will run on 4 processors)

As with all batch jobs, you will need to monitor the job to ensure that the load is reasonable.

Note: This sample script is for Amber 11. See the Amber 12 documentation for details about MMPBSA on Amber 12.

Sample batch script for a parallel MM-PBSA run. Courtesy of Tae Jin Kim (NCI).

#!/bin/bash
# This file is mmpbsa.run
#
#PBS -N MMPBSA
#PBS -m be


export AMBERHOME=/usr/local/amber11

cd /data/user/MMPBSA/

/usr/local/openmpi/bin/mpirun -machinefile $PBS_NODEFILE -np $np $AMBERHOME/bin/MMPBSA.MPI -O -i mmpbsa.in -o mmpbsa.dat -sp 1err.solvated.prmtop -cp complex.prmtop -rp receptor.prmtop -lp ligand.prmtop -y 1err_prod.mdcrd > progress.log

qsub -v np=24 -l nodes=1:c24 mmpbsa.run

Memory usage: The required memory for normal mode calculation is { 9*N*(3*N-1)/2 } *8 byte, where N is total atom number. (from the Amber 9 manual, p 280) Each process in a parallel MM-PBSA job loads each snapshot. Therefore the total memory used by the job will be { 9*N*(3*N-1)/2 } *8 * number_of_processes Users should adjust the number of processes and submit to an appropriate node so that the entire job fits in memory. For example, if the system has 5000 atoms, the memory required for a single process is 2.7 GB. 8 of these processes (21.6 GB total) will fit on a 24GB node, so the job should be submitted with

qsub -v np=8 -l nodes=1:g24 mmpbsa.run

If you are new to Amber, you may want to work through some of the tutorials on the Amber tutorial page. Note that the entire Protein Data Bank is available on Biowulf in /pdb and is updated weekly. Some of these tutorials have been worked through by the Biowulf staff, and the input, output and batch scripts specific to Biowulf are in /usr/local/amber11/tutorials/. If you want to examine these files, you could do something like:

cd /scratch/user
tar xvzf /usr/local/amber11/tutorials/tutorial-a6.tar.gz
cd tutorial-a6
more section4.biowulf

The Biowulf staff benchmarks indicate that sander jobs should be run on a single gige node or up to 8 ipath nodes. If you submit a parallel Amber job to multiple gige nodes, more than 8 ipath nodes, or multiple IB nodes, you should run your own benchmarks to determine the appropriate number of nodes.

To determine the optimal number of nodes:

• Set up a small version of your job. e.g. if your project involves a simulation of 100 ps, set up a 1ps job.
• Submit this job to 1 node, 2 nodes, 3 nodes... and record the time taken for each job. (or the ns/day as reported in the mdinfo output).
• Examine the results. You want to pick the number of processors with at least 60% efficiency.

  
  100 * Time on 1 processor
  ---------------------------    = Efficiency  
  n * Time on n processors
  

You can use the Biowulf system monitors to watch your job. Click on 'List status of Running Jobs only', and then on your username in the resultant web page. This will pop up a small window showing the status of all nodes that are running your jobs, as in the picture on the right. In the ideal situation, all your nodes should be yellow (both processors used). Watch for green or red nodes. Clicking on a node will pop up a window with information about the processes running on that node.

The PBS batch system will write 2 files called JOBNAME.oJOBNUM and JOBNAME.eJOBNUM in your home directory (e.g. sander.o90763 and sander.e90763). They contain the standard output and standard error from your job. Most problems can be figured out by examining these files.

Factor IX benchmark from the Amber benchmark suite. [Details] (Aug 2012)

• Amber 12 manual
• AmberTools 12 manual
• AMBER tutorials
• AMBER 10 Manuals
• The AMBER homepage
• AMBER Mail Reflector Archive at Vanderbilt University.