PETSc

where X.Y.Z is the PETSc software release version number and spec describes how the package was built as described below. All versions include a full PETSc implementation with internal linear and non-linear solvers.

spec = blank or (default)

A standard full version of PETSc with debugging flags (an alias for _g described below).

spec = -opkgs

These versions include external packages that interface with PETSc. Currently (Mar 12, 2012), the versions installed on Hopper are 3.2_p5 and these external packages are available: SuperLU, SuperLU_Dist, HYPRE, METIS, MUMPS, and PARMETIS. 

spec = _O (the capital letter "O")

A version built with compiler optimizations enabled.

spec = _g

A version built with the -g debugging flag and no compiler optimizations.

spec = _complex

Uses complex arithmetic.

spec = _c++

C++ bindings.

How to Access PETSc at NERSC

You must use the module command to access the PETSc libraries. To see all the available PETSc modules on a given machine use: module avail petsc.

When you run the module avail petsc command on Hopper you will see the Cray-supported versions listed under /opt/modulefiles and the ACTS-supported options listed under /usr/common/acts/Modules/modulefiles.

To select the default version use module load petsc.  To select a specific version use, for example, module load petsc/3.1
To get some specific usage information for a particular version use, for example, module help petsc/petsc/3.0.0_g_complex. 

Using PETSc on Hopper

Cray Versions

The Cray (i.e., not ACTS) installation is the default one on Hopper.  All Cray versions should work under all programming environments, except as noted in the documentation (i.e., Pathscale is not supported in some versions). In addition to different modulefiles for different PETSc release versions, there are also versions for real and complex arithmetic.

By loading the PETSc modulefile, all header and library locations are automatically set corresponding to your environment.

Note: If you use the Cray Fortran compiler to create a PETSc executable, you need to either add the directive

!dir$ PREPROCESS EXPAND_MACROS 

to the source code or add the -F option to the ftn command line.

Using PETSC on Carver

The following is an example but you should look at the tutorials directories (see Documentation, below) for more extensive examples.

mpif90 -I$PETSC_DIR/$PETSC_ARCH/include -I$PETSC_DIR/include -o ex2f ex2f.F

Documentation

On the Cray systems, after you have loaded a Cray-installed PETSc modulefile, you can use man intro_petsc to get some basic information.  More info is available in Cray's Programming Environment User's Guide.  The base document number for this is S–2396.  Cray documents are generally of the form S-xxxx-nn, where nn relates to the release.  The June, 2010 release of the Programming Environment User's Guide is S–2396–50.

There are no man pages available with the ACTS installed versions; however, there are doc and tutorials directories with many useful documents and examples.  Use the module show version command to see the location.

Full documentation about PETSc is available on the PETSc home page.

For questions about using PETsc at NERSC contact the consultants, consult@nersc.gov

Using Packages Within the Cray PETSc Release

MUMPS

You can view a sample MUMPS Fortran example code here.  This code happens to use double precision MUMPS routines.  To compile this code on Hopper using the Cray installation of PETSc and MUMPS you might use:

module load petsc
ftn -Wl,-ydmumps_parallel_analysis_ dsimpletest.F

where the optional -Wl,-y compile flag simply tells the loader to state where the routine "dmumps_parallel_analysis" is loaded from and if you try this you'll see that it comes from the Cray "Third Party Software Library" (tpsl) libdmumps library, which is loaded automatically - both header files and library object archives - with the 'module load petsc' command.  A C example program is here.  Compile with cc simpletest.c.

HYPRE

A C example code is here.  Compile it on Hopper with cc ex11.c after loading the Cray PETSc modulefile. 

Availability