Programming Tools and Libraries
Biowulf is intended to run code programmed by our users as well as commercial and open-source codes that may need to be built for our platform(s) if they do not come in a useable binary format. Accordingly, we host a number of compilers and build environments to suit the needs of developers and individuals that need to build projects from source. This page provides information specific to the Biowulf development environment as well as a rough overview of the various compilers, libraries and programs used on our system. The linked documentation on specific packages and programs will usually need to be consulted for any useful understanding of them. |
The Biowulf head-node is a 32-bit system that can only be used to compile 32-bit applications. In most cases this is the preferred mode since 32-bit applications will run on any node in the cluster and applications with sufficiently undemanding memory and integer/floating-point requirements will often perform better in a 32-bit mode. However some users may require or benefit from the larger per-process available memory possible in a 64-bit architecture (>4GB) or benefit from the additional registers and instruction extensions present on 64-bit Intel/AMD processors − only the developer, familiar with his/her project can ascertain this.
A large majority of the cluster nodes are 64-bit systems, however some use 32-bit Intel Xeon processors. In the event that a user builds a 64-bit application s/he will need to specify (at least) "x86-64" in their qsub options to keep the job from dieing on a 32-bit node.
If a 64-bit application is required or beneficial, the user will need to gain a session on one of the 64-bit interactive nodes to compile and test. This is done by logging into Biowulf and requesting an interactive job from the scheduler:
% qsub -I -l nodes=1The above example will give the user an interactive session on a system with a 64-bit development toolchain.
In addition to the standard cluster nodes, we maintain a 32-processor Itanium2 system (Firebolt) with 96GB of RAM intended for large memory or floating-point intensive tasks. This system has the usual gcc compiler suite available as well as the Intel compiler suite. Complete information on Firebolt's hardware and software configuration is available here.
|
All Biowulf cluster nodes include the GCC compiler suite which includes C, C++, FORTRAN77 and FORTRAN90/95 compilers (gcc, g++, g77 and gfortran respectively) along with the GNU debugger (gdb). In addition to these default compilers, there are three other popular suites available to Helix/Biowulf users that may improve the performance of your project or better accommodate certain code bases - these are the Intel, Pathscale and Portland Group International (PGI) compiler suites. |
Each compiler suite has a listing which includes a chart that shows the location of a set-up script that will enable the compiler in your environment, lists common front-ends for each compiler and shows the locations of various MPI installations depending on target architecture and desired interconnect (see MPI section below for details on MPI installations). For instance, if you wanted to use the Intel compilers to build your project, and your current shell is bash, the following command would set up your environment:
% source /usr/local/intel/intelvars.sh Arch is i386. setting up for Intel C compiler version 10.1.018. setting up for Intel Fortran compiler version 10.1.018. setting up for Intel debugger verion 10.1.018.
The venerable GNU compiler suite is available in the user's PATH by default. Though not considered "high-performance" or "optimized," they are usually the best choice for pre-existing source codes since build-systems are often created with this compiler in mind. Consequently, sensible compiler flags are generated and building will be comparatively trouble-free with these build-systems. However, if performance is an issue, you should consult the documentation distributed with the source distribution that you're trying to build to see if other compilers are supported. If you're developing a high-performance application "in-house," you may want to explore the other compilers available on Biowulf.
GCC quick-chart
Current Version: 4.1.2 Documentation: Try "man gcc" or "man g++" or "man g77" or "man gfortran". Primary front-ends:
C gcc C++ g++ Fortran77 g77 Fortran90/95 gfortran MPI Installations:
Ethernet (32-bit): /usr/local/mpich2 Ethernet (64-bit): /usr/local/mpich2-gnu64 Myrinet (32-bit): /usr/local/mpich-gm2k Infiniband: /usr/local/mvapich2
Some users need the latest features offered by the latest release of the GNU compiler suite, so we also manually maintain recent 4.x releases. Using these newer GNU compilers requires the user to source a set-up script and also specify an LD_LIBRARY_PATH before executing programs built with it. Or, alternatively, the user can specify "-static-libgcc", "-static-libgfortran" or "-static--libg++" (depending on the language) during the build phase to compile-in these runtimes as thus avoid needing to set a run-time library path. The MPI installations below can be used only after you have sourced the appropriate gccvars set-up script.
Latest GCC quick-chart
Current Version: 4.3.3 Setup Scripts
bash /usr/local/gcc/gccvars.sh csh/tcsh /usr/local/gcc/gccvars.csh Documentation: Try "man gcc" or "man g++" or "man gfortran". Primary front-ends:
C gcc C++ g++ Fortran77 gfortran Fortran90/95 gfortran MPI Installations:
Ethernet (32-bit): /usr/local/mpich2 Ethernet (64-bit): /usr/local/mpich2-gnu64 Myrinet (32-bit): /usr/local/mpich-gm2k Infiniband: /usr/local/mvapich2
The Portland Group suite includes the usual set of C, C++ FORTRAN77 and FORTRAN90/95 compilers. Also included is an OpenMP implementation, preliminary support for FORTRAN2000 and PGDBG, a graphical debugger (see debugging section below).
PGI quick-chart
Current Version: 8.0-1 Setup Scripts
bash /usr/local/pgi/pgivars.sh csh/tcsh /usr/local/pgi/pgivars.csh Documentation: PGI Compiler Documentation Primary front-ends:
C pgcc C++ pgCC Fortran77 pgf77 Fortran90 pgf90 Fortran95 pgf95 MPI Installations:
Ethernet (32-bit): /usr/local/mpich2-pgi Ethernet (64-bit): /usr/local/mpich2-pgi64 Myrinet (32-bit): /usr/local/mpich-gm2k-pg Infiniband: /usr/local/mvapich2-pgi
The Intel suite includes C, C+, FORTRAN77 and FORTRAN90/95 compilers along with OpenMP and the Intel debugger. Anecdotal evidence suggests that this compiler suite frequently provides the best performance for calculation-intensive applications. Included with these compilers are the Intel Math Kernel Library (MKL), LINPACK and Intel Performance Primitives (IPP) - all discussed in the scientific libraries section below.
Intel quick-chart
Current Version: 10.1.018 Setup Scripts
bash /usr/local/intel/intelvars.sh csh/tcsh /usr/local/intel/intelvars.csh Documentation: C/C++
Fortran
Debugger
LINPACK
Math Kernel Library
Intel Performance PrimitivesPrimary front-ends:
C icc C++ icpc Fortran77/90/95 ifort MPI Installations:
Ethernet (32-bit): /usr/local/mpich2-intel Ethernet (64-bit): /usr/local/mpich2-intel64 Myrinet (32-bit): /usr/local/mpich-gm2k-i Infiniband: /usr/local/mvapich2-intel
The Pathscale suite is often used to generate highly optimized binaries on Opteron systems. It includes a debugger and an option generator that can analyze source code and suggest compiler options. This is the only compiler that can be used to build MPI binaries for Biowulf's Infinipath network.
Pathscale quick-chart
Current Version: 3.1 Setup Scripts
bash /usr/local/pathscale/pathvars.sh csh/tcsh /usr/local/pathscale/pathvars.csh Primary front-ends:
C pathcc C++ pathCC Fortran77 pathf90 (not pathf77) Fortran90 pathf90 Fortran95 pathf95 MPI Installations:
Ethernet (32-bit): /usr/local/mpich2-pathscale Ethernet (64-bit): /usr/local/mpich2-pathscale64 Myrinet (32-bit): /usr/local/mpich-gm2k-ps Infiniband: /usr/local/mvapich2-pathscale Infinipath: MPI wrappers in /usr/bin on the Infinipath nodes will use the Pathscale compilers and link the correct (Infinipath) libraries by default. Documentation:
Several Java Development Kits are installed in /usr/local/java. Older versions are available for applications that require them. The latest is usually the best choice. For very specific situations, 64-bit JDKs are available in /usr/local/java64. These are not recommended when the 32-bit JDKs will do, but are available here for completeness and for users that know that they need a 64-bit Java.
While not usually appropriate for high-performance calculations or distributed memory tasks, scripting languages can be very useful when managing jobs or processes at a higher level, sorting data or doing an infinite number of simple tasks. Biowulf includes many scripting languages which are made available by the operating system and by the Biowulf staff.
Among these, Perl is arguably the most useful. The Perl installation available in the default PATH on all nodes includes an extensive set of modules and extensions built, created or made available by the Biowulf staff. Other scripting environments include Python, Ruby, PHP, tcl, a host of small, special-purpose languages and of course, there's your shell.
|
Parallel applications on Biowulf typically use MPI as the means inter-process communication across our various network interconnects. MPI is an application programming interface specification that currently exists in two major versions: MPI1 and MPI2. These APIs are implemented by many vendors and projects in many versions. The Biowulf staff maintains some popular MPI implementations for the convenience of our users, they are outlined here. |
NOTE:This section pertains mainly to new development as the Biowulf staff has moved from MPICH1 to MPICH2 as a currently maintained MPI implementation. MPICH1 is still supported - users with existing binaries/build systems can consult the MPICH1 page to reference Biowulf's site-specific information on MPICH1.MPI over Ethernet on the Biowulf cluster is provided by MPICH2, an implementation developed at Argonne National Laboratories and OpenMPI, a very actively developed MPI project implementing MPI 1 & 2. Choosing one should be based on the performance of your application under each.
Building Ethernet MPI apps using MPICH2To use this MPI, the user will first have to decide on a compiler and target architecture and then consult the chart below to find the correct MPICH installation.
MPICH installations for Ethernet
Compiler Architecture MPI installation GCC i686 (32-bit) /usr/local/mpich2x86_64 (64-bit) /usr/local/mpich2-gnu64PGI i686 /usr/local/mpich2-pgix86_64 /usr/local/mpich2-pgi64Intel i686 /usr/local/mpich2-intelx86_64 /usr/local/mpich2-intel64Pathscale i686 /usr/local/mpich2-pathscalex86_64 /usr/local/mpich2-pathscale64As an example, here were are preparing to build an MPI project that uses Ethernet as its MPI network and we want to use the Intel compilers to create 64-bit binaries. Note that when building a 64-bit application, the user will need to first gain an interactive session on a 64-bit node - our default interactive nodes are all 64-bit (for 32-bit builds, stay on the Biowulf head-node).
Here we get an interactive session, source the appropriate environment set-up script so that the compilers referenced in the MPI wrappers are available, and then add the MPI bin directory to the begining of our PATH:
% qsub -I -l nodes=1
qsub: waiting for job 1579075.biobos to start
qsub: job 1579075.biobos ready
[janeuser@p2 ~]$ source /usr/local/intel/intelvars.sh
Arch is x86_64.
setting up for Intel C compiler version 10.1.018.
setting up for Intel Fortran compiler version 10.1.018.
setting up for Intel debugger verion 10.1.018.
[janeuser@p2 ~]$ export PATH=/usr/local/mpich2-intel64/bin:$PATHNow we can use the MPI wrappers in our PATH to build MPI programs:
[janeuser@p2 ~]$ mpif90 -o mpitest hello_world.f90For complete documentation on using MPICH2, consult the latest version of the MPICH2 user's guide here (Argonne National Laboratory's MPICH2 site).
Running Ethernet MPI Applications under MPICH
Ethernet MPI using OpenMPIOpenMPI is currently installed for x86_64 only, 32-bit installations can be built upon request. To use OpenMPI to build your project first decide on a compiler and then consult the chart below to find the appropriate OpenMPI installation:
Compiler MPI installation GCC /usr/local/openmpi-gccPGI /usr/local/openmpi-pgiIntel /usr/local/openmpi-intelPathscale /usr/local/openmpi-pathscaleHere's an example that uses the OpenMPI compiler wrappers built for the Intel compilers:
% qsub -I -l nodes=1
qsub: waiting for job 1579075.biobos to start
qsub: job 1579075.biobos ready
[janeuser@p2 ~]$ source /usr/local/intel/intelvars.sh
Arch is x86_64.
setting up for Intel C compiler version 10.1.018.
setting up for Intel Fortran compiler version 10.1.018.
setting up for Intel debugger verion 10.1.018.
[janeuser@p2 ~]$ export PATH=/usr/local/openmpi-intel/bin:$PATH
Now we can use the MPI wrappers in our PATH to build MPI programs (see the OpenMPI FAQ page for more information on OpenMPI and how to use it):
[janeuser@p2 ~]$ mpicc -o mpitest hello_mpi.c
A portion of the Biowulf cluster has access to a Myrinet2000 network (also called GM2k), which has some performance advantages over Ethernet: lower latency and roughly twice the bandwidth of gigabit Ethernet. Currently, only 32-bit applications are supported on the GM2k network. This chart shows the MPI installations used on Biowulf for building MPI applications that will run on the Myrinet network.
MPICH installations for Myrinet 2000 (GM2k)
Compiler MPI installation GCC /usr/local/mpich-gm2kPGI /usr/local/mpich-gm2k-pgIntel /usr/local/mpich-gm2k-iPathscale /usr/local/mpich-gm2k-psThe process for building GM2k applications is similar to the Ethernet MPI build process. Fist, select and source a compiler set-up script, then set your PATH to include the bin directory from the appropriate GM2k installation, then use the wrappers in your PATH to build the project.
% source /usr/local/pgi/pgivars.sh
Arch is i386
PGI Server Suite version: 7.2-3
% export PATH=/usr/lcoal/mpich-gm2k-pg/binNow we can use the MPI wrappers in our PATH (mpicc, mpicxx, mpif90 etc.) to build MPI programs for Myrinet.
A portion of the Biowulf cluster has access to an Infinipath network (Infinipath being a subset of the Infiniband standard optimized for message passing) that is intended for high-performance parallel applications.
Building and running MPI applications for Infinipath requires the user to log into an Infinipath node, source the Pathscale compiler set-up script before using the MPI wrapper scripts that are in the default PATH on the Infinipath nodes.
Note: only the Pathscale compiler suite should be used to build binaries for the Infinipath cluster as the MPI libraries for that interconnect seem to have stability problems when combined with the run-times of other compilers (your milage may vary).
% qsub -I -l nodes=1:ipath
qsub: job 1580340.biobos ready
% source /usr/local/pathscale/pathvars.sh
Setting Pathscale compiler version 3.1
Architecture is x86_64
% mpicc -o mympiprog mympisourcefile.c
Infinipath MPI Wrappers
Language/Compiler MPI wrapper C (pathcc) /usr/bin/mpiccC++ (pathCC) /usr/bin/mpicxxFortran77 (pathf90) /usr/bin/mpif77Fortran90 (pathf90) /usr/bin/mpif90Documentation: The Infinipath MPI libraries and wrapper scripts were derived from the MPICH1 MPI implimentation. Consult the MPICH1 pages for specific documentation on using these wrapper scripts keeping in mind that the target interconnet is an Infinipath network and not Ethernet. The above listed wrapper scripts themselves contain much useful information in the comment areas.
A portion of the Biowulf cluster has access to an Infiniband network, which is currently the fastest network on this cluster in terms of bandwidth. By default, a user with an interactive session on an IB node has MVAPICH2 compiler wrappers and runtimes already available. However a user may want to use MVAPICH2 with a different compiler or may want to use OpenMPI for their application. Consult the charts below to locate the desired compiler/MPI implementation.
Infiniband using MVAPICH2Log into an Infiniband node and use the MPI wrappers found in the default PATH to build programs with a gcc back-end.
% qsub -I -l nodes=1:ib
qsub: job 1580340.biobos ready
% mpicc -o mympiprog mympisourcefile.c
Or alternatively, if the user wants to use one of the proprietary compilers available on the cluster, s/he may use one of the MVAPICH2 installations listed in the table below.
Infiniband MVAPICH2 Installations
Language/Compiler MPI Installation GNU Suite (gcc) /usr/local/mvapich2Intel /usr/local/mvapich2-intelPortland Group (PGI) /usr/local/mvapich2-pgiPathscale /usr/local/mvapich2-pathscaleExample using MVAPICH2 with the Pathscale compilers:
% qsub -I -l nodes=1:ib qsub: waiting for job 2078457.biobos to start qsub: job 2078457.biobos ready % source /usr/local/pathscale/pathvars.sh % export PATH=/usr/local/openmpi-pathscale/bin:$PATH % mpicxx -o dothings mpi_cpp_src.cppMVAPICH2 Documentation can be found here.
Running MVAPICH2 Infiniband Jobs
Infiniband using OpenMPIUsing OpenMPI requires the user to find the appropriate installation in set his/her PATH accordingly. Consult the chart below:
Infiniband-aware OpenMPI Installations
Language/Compiler MPI Installation GNU Suite (gcc) /usr/local/openmpi-gccIntel /usr/local/openmpi-intelPortland Group (PGI) /usr/local/openmpi-pgiPathscale /usr/local/openmpi-pathscaleExample using OpenMPI with the PGI compilers for Infiniband:
% qsub -I -l nodes=1:ib qsub: waiting for job 2078457.biobos to start qsub: job 2078457.biobos ready % source /usr/local/pgi/pgivars.sh % export PATH=/usr/local/openmpi-pgi/bin:$PATH % mpif90 -o dothings mpi_fortran_src.f90
Listed here are a few of the more notable libraries and suites available to developers of scientific and/or high-performance software. These are mostly various implementations of BLAS, LAPACK, etc., however the developer should review each to find the one that best fits his/her needs.
FFTW is an popular open-source fast Fourier transform library. 32- and 64-bit versions of the library can be found here:
/usr/local/fftw-2.1.5/
/usr/local/fftw-2.1.5_x86_64/
Then Intel Math Kernel Library (MKL) is a set of optimized and threadable math routines for scientific, engineering and financial applications. It includes BLAS, LAPACK, ScaLAPACK, FFTs, a vector math library and random number generators. It is installed on biowulf in:
/usr/local/intel/mkl/current
AMD's implementation of several common math routine libraries: Full 1, 2 and 3 BLAS, LAPACK, FFTs and a number of routine sets specific to the ACML. These should run exceptionally well on Biowulf's Opteron nodes. AMD was kind enough to build the ACML for the major Fortran compiler suites. Installations are available here according to compiler:
Compiler ACML installation GNU (gfortran) /usr/local/acml/gfortranIntel (ifort) /usr/local/acml/ifortPGI (pgf77/90/95) /usr/local/acml/pgiPathscale (pathf90/95) /usr/local/acml/pathscale
From Intel's website:
Integrated Performance Primitives (Intel® IPP) is an extensive library of multi-core-ready, highly optimized software functions for multimedia data processing, and communications applications.
It includes, among other things, routines for audio/video encoding and decoding, image processing, signal processing, Vector/Matrix math and data compression. IPP is installed here according to target architecture:
/usr/local/intel/ipp/current
Documentation:
This is the open-source scientific library provided by GNU for C and C++ developers. It provides a large array of math routines with an extensive test-suite. A complete list of routines and capabilities is available on the GSL website. It is installed by architecture:
/usr/local/gsl-x86_64
/usr/local/gsl-i686
Debuggers and memory/thread profilers are often associated with a specific compiler to be used only with the accompnying compiler, however some may work across compiler suites as well. Also included here are a couple generic debugger/profilers.
GDB is part of the GNU project and is available on all nodes by default. Documentation is available on the GDB website and by typing "man gdb".
Valgrind is a common tool-chain for memory-profiling and debugging. Documentation is avaible on the Valgrind website or by typing "man valgrind".
The Intel debugger comes with the Intel compiler suite in 32- and 64-bit flavors. Installations of IDB are available in the Intel compiler suite area:
/usr/local/intel/idb/current (32-bit)
/usr/local/intel/idbe/current (64-bit)
The PGI compilers come with a graphical debugger and memory profiler (pgdbg). Using the GUI requires X to be installed on your workstation, however it will drop to a console-only version when X is not available. The debugger is present in your PATH after sourcing the appropriate PGI set-up script (see PGI Compilers above).
The Pathscale compilers come with a command-line debugger (pathdb). The debugger is present in your PATH after sourcing the appropriate Pathscale set-up script (see Pathscale Compilers above).
Compiler | Ethernet MPI | Myrinet MPI | Infinipath MPI | Infiniband MPI |
GCC (gcc, g++, g77, gfortran) | Yes | Yes | -- | Yes |
PGI (pgcc, pgCC, pgf77/90/95) | Yes | Yes | -- | Yes |
Intel (icc, icpc, ifort) | Yes | Yes | -- | Yes |
Pathscale (pathcc, pathCC, pathf90/95 | Yes | Yes | Yes | Yes |
Compiler | Ethernet | Myrinet | Infiniband |
GCC 32-bit | /usr/local/mpich2/bin | /usr/local/mpich-gm2k/bin | -- |
GCC 64-bit | /usr/local/mpich2-gnu64/bin | -- | /usr/local/mvapich2/bin |
PGI 32-bit | /usr/local/mpich2-pgi/bin | /usr/local/mpich-gm2k-pg/bin | -- |
PGI 64-bit | /usr/local/mpich2-pgi64/bin | -- | /usr/local/mvapich2-pgi/bin |
Intel 32-bit | /usr/local/mpich2-intel/bin | /usr/local/mpich-gm2k-i/bin | -- |
Intel 64-bit | /usr/local/mpich2-intel64/bin | -- | /usr/local/mvapich2-intel/bin |
Pathscale 32-bit | /usr/local/mpich2-pathscale/bin | /usr/local/mpich-gm2k-ps/bin | -- |
Pathscale 64-bit | /usr/local/mpich2-pathscale64/bin | -- | /usr/local/mvapich2-pathscale/bin |