FreeSurfer on the Biowulf Linux CLuster

FreeSurfer on Biowulf

FreeSurfer is a set of automated tools for reconstruction of the brain's cortical surface from structural MRI data, and overlay of functional MRI data onto the reconstructed surface. It was developed at the Martinos Center for Biological Imaging at Harvard. FreeSurfer website.

FreeSurfer is not a parallel program. The only advantage of running FreeSurfer on Biowulf would be to process different subjects simultaneously. Serial processing via FreeSurfer is best done on Helix.

FreeSurfer on Biowulf is a 64-bit program. This means that it will not run on the Biowulf head node or a 32-bit (p2800) node. The GUI-based FreeSurfer programs should be run by allocating an interactive node (as described below) or on Helix.

FreeSurfer environment:
The following commands are required to set up the FreeSurfer environment:

ash shell:
export FREESURFER_HOME=/usr/local/freesurfer   
source $FREESURFER_HOME//SetUpFreeSurfer.sh

csh/tcsh shell:
setenv FREESURFER_HOME /usr/local/freesurfer
source $FREESURFER_HOME//SetUpFreeSurfer.csh

Running a FreeSurfer job in batch

The following example demonstrates a recon-all job run via the swarm command on Biowulf. (Thanks to Nikhil Sharma at NINDS for this example.)

In the working directory for this project (e.g. /data/user/freesurfer), create a directory called subjects, and create a subdirectory for each subject. See the Freesurfer documentation for a detailed explanation of the directories and files required and created during the recon-all job.

In the working directory, create a swarm command file, e.g. freesurfer.swarm, containing the commands you wish to run on each subject.

cd /data/user/freesurfer/; recon-all -subject hv1 -all
cd /data/user/freesurfer/; recon-all -subject hv2 -all
cd /data/user/freesurfer/; recon-all -subject hv3 -all
[...]

with one line for each subject (hv1, hv2, hv3 etc.).

Submit this swarm set with the command:

swarm -f freesurfer.swarm

Running a FreeSurfer GUI program

FreeSurfer GUI programs can easily be run on Helix. Open an Xwindows session to Helix, set up the FreeSurfer environment, and enter a FreeSurfer GUI command.

Alternatively, allocate an interactive node on Biowulf, making sure to export your graphics environment with the '-V' flag, and enter a Freesurfer GUI command. Sample session (user input in bold):

[user@biowulf ~]$ qsub -I -V -l nodes=1 (this commandallocates a 64-bit node by default) qsub: waiting for job 2113609.biobos to start qsub: job 2113609.biobos ready [user@p4 ~]$ export FREESURFER_HOME=/usr/local/freesurfer [user@p4 ~]$ source /usr/local/freesurfer/SetUpFreeSurfer.sh -------- freesurfer-Linux-centos4_x86_64-stable-pub-v4.2.0 -------- Setting up environment for FreeSurfer/FS-FAST (and FSL) FREESURFER_HOME /usr/local/freesurfer FSFAST_HOME /usr/local/freesurfer/fsfast FSF_OUTPUT_FORMAT nii SUBJECTS_DIR /usr/local/freesurfer/subjects MNI_DIR /usr/local/freesurfer/mni FSL_DIR /usr/local/fsl [user@p4 ~]$ tkmedit bert orig.mgz Setting subject to bert Reading 0 control points... Reading 0 control points... Reading /usr/local/freesurfer/lib/tcl/tkm_common.tcl Reading /usr/local/freesurfer/lib/tcl/tkm_wrappers.tcl Reading /usr/local/freesurfer/lib/tcl/fsgdfPlot.tcl Reading /usr/local/freesurfer/lib/tcl/tkUtils.tcl

[user@p4 ~] exit qsub: job 2113609.biobos completed

You should see the GUI windows appear on your desktop as above. Be sure to type 'exit' when you have finished, so that the node is de-allocated.